Question: By("starting Nimbus") framework.RunKubectlOrDie("create", "-f", nimbusPodJson, nsFlag) framework.RunKubectlOrDie("create", "-f", nimbusServiceJson, nsFlag) <del> err = framework.WaitForPodNameRunningInNamespace(c, "nimbus", ns) </del> <ins> err = f.WaitForPodRunningSlow(nimbusPod) </ins> Expect(err).NotTo(HaveOccurred()) err = framework.WaitForEndpoint(c, ns, "nimbus")

https://k8s- Failed: [] [Feature:Example] [] Storm should create and stop Zookeeper, Nimbus and Storm worker servers {Kubernetes e2e suite}
https://k8s- Failed: [] [Feature:Example] [] Storm should create and stop Zookeeper, Nimbus and Storm worker servers {Kubernetes e2e suite}
https://k8s- Failed: [] [Feature:Example] [] Storm should create and stop Zookeeper, Nimbus and Storm worker servers {Kubernetes e2e suite}
https://k8s- Failed: [] [Feature:Example] [] Storm should create and stop Zookeeper, Nimbus and Storm worker servers {Kubernetes e2e suite}
https://k8s- Failed: [] [Feature:Example] [] Storm should create and stop Zookeeper, Nimbus and Storm worker servers {Kubernetes e2e suite}
[FLAKE-PING] This flaky-test issue would love to have more attention...
This has been marked with the "needs attention" label and needs to be addressed for the 1.4 release. Could you please close if it's fixed update on your progress to fix it make a case for it not to block the release or find a new owner who can do one of the above soon. Once you or the new owner update this issue and are actively working on it, you can remove the "needs attention" label.
the following message looks suspicious, looks a DNS issue?
Actually more like a network connectivity issue
do you happen to have the Dockerfile for these images? We better push them into
I don't think this is a blocking issue. It looks a transient issue. Since the example is largely dependent on whether the image can be fetched and how long it takes to pull the image, it is very environment sensitive. I would recommend removing label. Longer term, we should move the images used in examples to gcr.
https://k8s- Failed: [] [Feature:Example] [] Storm should create and stop Zookeeper, Nimbus and Storm worker servers {Kubernetes e2e suite}
[FLAKE-PING] This flaky-test issue would love to have more attention.

for the last flake, based on , image pull (mattf/zookeeper:latest) started at 42:40, finished at 48:31, about 5min51sec, well passed 5 min
storm-nimbus, storm-worker and zookeeper are from they're all automated builds from github (quick link: and )
[FLAKE-PING] This flaky-test issue would love to have more attention.
https://k8s- Failed: [] [Feature:Example] [] Storm should create and stop Zookeeper, Nimbus and Storm worker servers {Kubernetes e2e suite}


Question: }{ {64, 3, 1 << uint(math.Ceil(math.Log2(float64(ff(64, 3))))+spare)}, {128, 3, ff(128, 3)}, <del> {128, 3, 3 * ff(128, 3)}, {70, 4, ff(70, 4)}, </del> <ins> {50, 4, ff(50, 4)}, </ins> } for _, test := range tests { dealer, err := NewDealer(test.deckSize, test.handSize)

<!-- Please only use this template for submitting reports about flaky tests or jobs (pass or fail with no underlying change in code) in Kubernetes CI --Which test(s) are flaking: Testgrid link: While there aren't any visible failures in the above testgrid link, we've observed flaky timeouts downstream. Reason for failure: Test times out Anything else we need to know: We don't have full logs for failed unit tests, so it's hard to pinpoint exactly which test is timing out, but I believe the offending function is this: You will see that the test iterates times: Summing up the hashMax for all four subtests, we see that the function iterates ~34 million times. While the test takes just ~9 seconds on my machine to complete, the CI is probably struggling with it (other tests are running in parallel and iterating 34 million times over a non-trivial loop is very CPU hungry). A successful execution of the unit test on CI takes ~3-4 minutes on average. The test doesn't have any of the usual suspects that can cause a timeout (like <-chan receive, i/o reads, wait.Poll(), time.Sleep(), etc.), so I suspect the large number of iterations to be the culprit. I propose making the unit test smaller and quicker to complete. Hopefully someone who understands shufflesharding better than me can take over :) /sig api-machinery /cc
when run with , the runtime increases significantly: that time is dominated by TestUniformDistribution
The test does a lot of slice appends. Maybe is constraining the allocator? There isn't much else in the test that might be slowed down by : no mutexes, no RNGs, no atomic operations, no channels, etc.
It looks like the buffer is pre-allocated to be the correct size though, so none of those appends should be resulting in allocations?
/triage accepted
BTW, in another branch I tried taking the s out of the test, instead calling with a lambda that writes into at a position selected by an advancing index. The got a little slower!


Question: apiVersion: apps/v1 kind: DaemonSet metadata: <del> name: fluentd-es-v2.2.0 </del> <ins> name: fluentd-es-v2.2.1 </ins> namespace: kube-system labels: k8s-app: fluentd-es <del> version: v2.2.0 </del> <ins> version: v2.2.1 </ins> kubernetes.io/cluster-service: "true" addonmanager.kubernetes.io/mode: Reconcile spec: selector: matchLabels: k8s-app: fluentd-es <del> version: v2.2.0 </del> <ins> version: v2.2.1 </ins> template: metadata: labels: k8s-app: fluentd-es kubernetes.io/cluster-service: "true" <del> version: v2.2.0 </del> <ins> version: v2.2.1 </ins> # This annotation ensures that fluentd does not get evicted if the node # supports critical pod annotation based priority scheme. # Note that this does not guarantee admission on the nodes (#40573).

Follow-up of Docker 1.13 introduced a change into the logging mechanism (), that split the log lines longer than 16K into chunks of no more than that. Current default logging integrations (fluentd-gcp and fluentd-es addons) do not support that and this might result in broken ingestion, i.e. when JSON parsing is broken for entries longer than 16K. One possible solution is introducing a fluentd plugin to the configuration to concat such entries in the similar way does it (assuming there is one out there, cc for that) /cc
Presumably this should be done in the Docker fluentd logging driver... The discussion on seems to be leaning that way...
In fluentd logging driver side, journald driver approach is better: I'm not sure buffering partial message in logging driver is good or not... In fluentd side, implementing or is needed.
do we need to block 1.8 release? can we please move this out of 1.8 if it's not?
Moved to 1.9, since it's not specific for 1.8
Is there any way this can be fixed by running a custom fluentd? Or is there a fluentd docker driver github issue I can follow for updates on when they expect to have this fixed? Txh
[MILESTONENOTIFIER] Milestone Issue Needs Approval Action required: This issue must have the label applied by a SIG maintainer. <details<summaryIssue Labels</summary- : Issue will be escalated to these SIGs if needed. : Never automatically move out of a release milestone; continually escalate to contributor and SIG through all available channels. : Fixes a bug discovered during the current release. </details<details<summaryHelp</summary<ul<li<a href=""Additional instructions</a</li<li<a href=""Commands for setting labels</a</li</ul</details
Problem is that there's no fluentd plugin to fix the problem, so running a custom fluentd won't help, there's nothing to add to the configuration to make it work (unless you want to write your own solution) They don't think it's a docker problem and don't plan to fix it. However, it will be fixed in GKE
That seems fair from Docker's perspective.

I'd rather have stable containers as well :) So you're thinking of switching from the current log drive - at least that's what my Google Container Optimized image in GKE 1.8.1 uses - to the journald driver to circumvent the problem? And then use a plugin like to recombine the partial messages?
No, that's not the case. Solution is TBD, but if you export your logs to Stackdriver (that has a limit of 100KB per entry) you won't see the difference
I think there's still a difference in that the 100KB stackdriver logging limit is larger than the 16KB docker limit at which the partial messaging starts, so without proper recombination of those chunked messages you'll lose the ability to send log lines larger than 16KB into stackdriver. However we're also running our own fluentd that sends messages straight into pub/sub, which has a limit of 10MB per message. And although we're trying to keep the messages much smaller than 16KB it would be nice to be able to send bigger messages without instrumenting our applications to push the messages into pub/sub themselves. I hope you find a nice solution that will make this possible :)
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten /remove-lifecycle stale
/remove-lifecycle rotten Still relevant, hasn't been fixed
I've been reading up on this issue, it's really too bad docker would not allow the configurable log-max-size option, this would be an easy temporary workaround to the issue until fully resolved. Regardless, I also wanted to point out a 2nd issue that the log splitting causes which might help increase the priority of this issue.

If you are using json logging and have MCD or Structured Arguements attributes in your log messages, what ends up happening unfortunately is the attributes are lost on the split messages. This is a huge drawback right now. In my case I am using spring cloud sleuth which adds span and traceid's to all log messages allowing you to easily filter and correlate all related logging messages. When the messages are split, the traceid's are not part of the message in elasticsearch making filtering by traceid impossible. My application logging is setup as follows: is it still the plan to have fluentd put the logs back together into a single message? can anyone update if any progress is being made? thanks for your help.
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten
/remove-lifecycle rotten Still relevant, hasn't been fixed
Would using the "concat" fluentd plugin be relevant? It seems to have explicit support for our specific use case. Edit: For EFK, Adding fluentd-plugin-concat to fluentd-es image (, ) and adding to the Works well. Would a PR with those changes be acceptable?
trying to solve this as well. Having the same issue as where long lines are being truncated and MDC gone.
Please see The big problem here is concurrency. Just concatenating all partial log messages as they arrive may be wrong. Let's say your app logs two events, called and here. Then the Docker log messages may be written to the JSON log file (or whatever Docker logging driver is used, e.g. in plain Docker or Docker Swarm) in this order: The fluentd concat plugin cannot decide what to do. What records belong together?

The good message is: This can be solved (similar to ) as Docker logs contain all the needed information here: So you just have to extend these methods and the struct by the partial log meta data: Then you have to fix Mind the ID and ordinal of the partial log metadata so that you concatenate the right records in the correct order.
I just edited my above comment because I am not sure how the case of linebreaks (i.e. those written by the application) in log messages relate to this issue. However, often you can control the logging of your application and let it write JSON to STDOUT (e.g. for Java / Spring Boot applications) so you do not have linebreaks. Back to the original issue: I have serious doubts that will do it under load with a lot of concurrent partial messages. I observed this in a Docker Swarm setup with fluentd and the concat plugin configured analogously.
Could you please reopen this issue? The concurrency is the problem here: Let your container simultaneously produce two log messages of 16KB (let's say 20 KB). Then fluentd will concatenate all four messages to one large (16+16+4+4 KB = 40 KB) message. The fluentd (or whatever you use) needs to know the ID of the partial log metadata to concatenate only those messages with identical IDs. So this needs to be implemented in the (default) local JSON file log driver in Docker (moby).
Problem is that ID comes with Docker 18.06 if I'm not wrong, this is why I didn't use it: do we want to require a docker version which is 2 months old?
is no longer much involved with the project. if you would like to open this back up to alter the implementation, please file a new issue
If you do so, please ping me as well.
I have the concat plugin but still it is not concatenating, Any Idea?
No idea why it does not work. For up-to-date information, I have inserted: Just after the kubernetes_metadate injection.


Question: // There is no need to bind Node IP to dummy interface, so set parameter `bindAddr` to `false`. if err := proxier.syncService(svcNameString, serv, false); err == nil { activeIPVSServices[serv.String()] = true <del> if err := proxier.syncEndpoint(svcName, svcInfo.OnlyNodeLocalEndpoints, serv); err != nil { </del> <ins> if err := proxier.syncEndpoint(svcName, false, serv); err != nil { </ins> glog.Errorf("Failed to sync endpoint for service: %v, err: %v", serv, err) } } else {

Is this a BUG REPORT or FEATURE REQUEST?: /kind bug What happened: The test is flaky with the error something like: What you expected to happen: Test should pass /assign
/sig network /area ipvs
/close
Cases are green now. :tada:


Question: definitions MUST be published for custom resource definitions. release: v1.16 file: test/e2e/apimachinery/crd_publish_openapi.go <ins> - testname: Event resource lifecycle codename: '[sig-api-machinery] Events should ensure that an event can be fetched, patched, deleted, and listed [Conformance]' description: Create an event, the event MUST exist. The event is patched with a new message, the check MUST have the update message. The event is deleted and MUST NOT show up when listing all events. release: v1.19 file: test/e2e/framework/events/events.go </ins> - testname: Garbage Collector, delete deployment,  propagation policy background codename: '[sig-api-machinery] Garbage collector should delete RS created by deployment when not orphaning [Conformance]'

This issue proposes the writing of a new test for the API endponts , , , , , and with the intention of promoting it to Conformance. This issue contains a very basic mock test in order to start the conversation about what a good e2e test would look like for this endpoint. The code below is verified to be hitting the intended endpoint, show as per the queries in APIsnoop's live view of the cluster's audit logs. According to these query results from APIsnoop, we are lacking in tests for the following endpoints related to Events We chose the following 5 endpoints for this mock test: - - - Test outline: Event in namespace default with custom fields all events in all namespaces find event created in in response of the content of the event created in events by name in namespace default ensure that the event updated from event by name all events in namespace ensure event in namespace default was deleted This mock test uses client-go's standard functions for talking to the Event area of the API. Discover useragents Display endpoints hit by the test Show the change in coverage after the mock test If a test with these calls gets merged in, conformance will go up by 5 endpoints. /sig architecture /area conformance
lgtm according to the SIG-Arch conformance office hours 17th December 2019.
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/reopen can you put the checkboxes at the top for your apisnoop org pr, the test, and link to the promotion PR? I think we are ready.
/open
Reopened this issue. <detailsIn response to : /reopen can you put the checkboxes at the top for your apisnoop org pr, the test, and link to the promotion PR? I think we are ready. Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/retitle Promote CoreV1Event test - +5 coverage
/assign
This test was written before the tickets were stored in APISnoop branches. Unsure what to do with it in regards to the APISnoop PR


Question: if replicationController == nil { framework.Failf(rcIsNil) } <del> return int(replicationController.Status.Replicas) </del> <ins> return int(replicationController.Status.ReadyReplicas) </ins> case KindDeployment: deployment, err := rc.framework.ClientSet.Extensions().Deployments(rc.framework.Namespace.Name).Get(rc.name, metav1.GetOptions{}) framework.ExpectNoError(err) if deployment == nil { framework.Failf(deploymentIsNil) } <del> return int(deployment.Status.Replicas) </del> <ins> return int(deployment.Status.ReadyReplicas) </ins> case KindReplicaSet: rs, err := rc.framework.ClientSet.Extensions().ReplicaSets(rc.framework.Namespace.Name).Get(rc.name, metav1.GetOptions{}) framework.ExpectNoError(err) if rs == nil { framework.Failf(rsIsNil) } <del> return int(rs.Status.Replicas) </del> <ins> return int(rs.Status.ReadyReplicas) </ins> default: framework.Failf(invalidKind) } return 0 } <del> func (rc *ResourceConsumer) WaitForReplicas(desiredReplicas int) { duration := 15 * time.Minute </del> <ins> func (rc *ResourceConsumer) WaitForReplicas(desiredReplicas int, duration time.Duration) { </ins> interval := 20 * time.Second err := wait.PollImmediate(interval, duration, func() (bool, error) { replicas := rc.GetReplicas()

Currently the time it takes for HPA and CA to scale up after load increase is not measured at all, and this can lead to regressions that will be missed. This issue is about adding an e2e test that will prevent such regressions.
There are no sig labels on this issue. Please by:<br(1) mentioning a sig: <br(2) specifying the label manually: <br<brNote: method (1) will trigger a notification to the team. You can find the team list .
Filip, can you reopen? I cannot do this myself, and this issue is clearly not closed yet.
Thanks Wojtek!


Question: execAffinityTestForLBService(f, cs, svc) }) <del> // TODO: Get rid of [DisabledForLargeClusters] tag when issue #56138 is fixed. </del> // [LinuxOnly]: Windows does not support session affinity. <del> ginkgo.It("should be able to switch session affinity for LoadBalancer service with ESIPP off [Slow] [DisabledForLargeClusters] [LinuxOnly]", func() { </del> <ins> ginkgo.It("should be able to switch session affinity for LoadBalancer service with ESIPP off [Slow] [LinuxOnly]", func() { </ins> // L4 load balancer affinity `ClientIP` is not supported on AWS ELB. e2eskipper.SkipIfProviderIs("aws")

Is this a BUG REPORT or FEATURE REQUEST?: /kind bug This is a followup from filed separately now that we're not tracking this as part of 1.9 release. The tests have been disabled with [DisabledForLargeClusters] tag for now, and this fact will be mentioned in the scalability validation report for v1.9. This issue tracks fixing the test so the tag can be removed and the test ran again in the large cluster. /assign /kind e2e-test-failure
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale
Is this e2e test still broken?
Sorry for the confusion here. Currently this e2e test is disabled on extra large cluster due to some scalability issues with the load balancer related components. At the moment the issues haven't been completely resolved yet.
ok, adding priority so that I know it's not a 1.13 blocker. /priority important-longterm
cc - this is why the relevant tests for are not part of the e2e suite.
This issue is still valid --- some action items on top of my head: Figure out a plan to make service controller properly handle operations in parallel. Check on the status for cloud providers (especially GCP, which the test ran against) on handling large number of nodes as LB backends. /remove-triage unresolved
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or .

/lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten
/remove-lifecycle rotten /lifecycle frozen
/reopen Reopening as is being reverted in
Reopened this issue. <detailsIn response to : /reopen Reopening as is being reverted in Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
these tests are flaking in xref
cc since these were enabled by
Yeah - though these failures doesn't seem to be scale related, they seem to be test related to me. This basically means nodes without public ips: It's pretty much the same thing as (just for a different test). should be addressing these kinds of errors. [Not a release blocker for sure]
BTW - has been opened directly for this one.
... so since is closed, Can we close this and re-enable for large clusters ? (i.e. remove the Tag in that disables this test) ? I'm not up to speed on the history but 2 related notes I came across digging into related test today... 's a couple of TODOs in the code referencing this specific issue, but your above direct issue is now closed, so we can close this as well right?... And if so, we can remove the TODO on the test... -enabling the test ... not sure wether the timing is good right now (given that is open right now , which sparked my stumbling upon this issue :))
We tried to enable all of them and that caused us a bunch of issues and those had to be reverted: A couple tests were enabled later in: and those seem fine. The remaining (mostly session-affinity ones) had some issue, which so far we didn't have time to debug and I haven't heard a good explanation why those were failing. It's possible that they were improved with , but I doubt those would really cause the failures we've seen in the past. So until this is debugged (or at least understood) I'm reluctant to enabling them.


Question: Output:    true, }, { <ins> Inspected: dockertypes.ImageInspect{RepoTags: []string{"docker.io/kubernetes/pause:latest"}}, Image:     "kubernetes/pause:latest", Output:    true, }, { </ins> Inspected: dockertypes.ImageInspect{ ID: "sha256:2208f7a29005d226d1ee33a63e33af1f47af6156c740d7d23c7948e8d282d53d", },

I cannot create pod. shows the following error shows the following message Tracing back to the source, the newly image check function from prints the message
when you run into such issue, have you tried docker pull to see if docker hub is down? Also you are using image kubernetes/pause for testing?
when did you build your cluster? was merged yesterday.
yes, docker pull worked for that image and I compared container id, it matches the sha in the kubelet log. The new fix is not in my tree yet. I'll check it out. Thank you for helping.
still not working console commit Merge: Author: Kubernetes Submit Queue <k8s-merge- Wed Aug 17 00:54:33 2016 -0700 Merge pull request from hodovska/-non-k8scontainers Automatic merge from submit-queue Add note: kubelet manages only k8s containers. Kubelet wrote log when accesing container which was not created in k8s, what could confuse users. That's why we note about it in documentation and lowered log level of the message to 5. Here is example of the message: bug Bugzilla link commit Author: Dominika Hodovska <: Thu Aug 11 10:00:13 2016 +0200 Add note: kubelet manages only k8s containers.

bug Bugzilla link 3m 58s 5 {kubelet rootfs-dev} spec.containers{web} Normal Pulling pulling image "kubernetes/pause" 3m 56s 5 {kubelet rootfs-dev} spec.containers{web} Warning Failed Failed to pull image "kubernetes/pause": image pull failed for unknown error 3m 56s 5 {kubelet rootfs-dev} Warning FailedSync Error syncing pod, skipping: failed to "StartContainer" for "web" with ErrImagePull: "image pull failed for unknown error" 3m 10s 13 {kubelet rootfs-dev} spec.containers{web} Normal BackOff Back-off pulling image "kubernetes/pause" 3m 10s 13 {kubelet rootfs-dev} Warning FailedSync Error syncing pod, skipping: failed to "StartContainer" for "web" with ImagePullBackOff: "Back-off pulling image "kubernetes/pause"" I0817 15:26:14. ] Connecting to docker on I0817 15:26:14. ] Start docker client with request timeout=2m0s I0817 15:30:36. ] Pulling image kubernetes/pause:latest without credentials I0817 15:30:38. ] Inspected image ("c80eebea8921a5e97f82fcf60da84f4de6399b9fc0ba683ceff67cd0dd319644") does not match kubernetes/pause:latest I0817 15:30:52. ] Pulling image kubernetes/pause:latest without credentials I0817 15:30:55. ] Inspected image ("c80eebea8921a5e97f82fcf60da84f4de6399b9fc0ba683ceff67cd0dd319644") does not match kubernetes/pause:latest I0817 15:31:19. ] Pulling image kubernetes/pause:latest without credentials I0817 15:31:21. ]

Inspected image ("c80eebea8921a5e97f82fcf60da84f4de6399b9fc0ba683ceff67cd0dd319644") does not match kubernetes/pause:latest
could you post the output of ?
I use to start my cluster, maybe there is a missing option
Uh...I think this is the opposite case of what PR was fixing. Uploaded to fix this.
Thanks for taking care of this!


Question: return nil, err } <del> if s.tlsConfig == nil { s.tlsConfig = &tls.Config{} </del> <ins> tlsConfig := s.tlsConfig switch { case tlsConfig == nil: tlsConfig = &tls.Config{ServerName: host} case len(tlsConfig.ServerName) == 0: tlsConfig = tlsConfig.Clone() tlsConfig.ServerName = host </ins> } <del> if len(s.tlsConfig.ServerName) == 0 { s.tlsConfig.ServerName = host } tlsConn := tls.Client(rwc, s.tlsConfig) </del> <ins> tlsConn := tls.Client(rwc, tlsConfig) </ins> // need to manually call Handshake() so we can call VerifyHostname() below if err := tlsConn.Handshake(); err != nil {

an APIService that routes to an external API server (e.g. service catalog) via a service in namespace a caBundle in the APIService the external API server with a certificate signed for Result: Normal API requests (get/put/post) work as expected Normal watch requests work as expected Upgrade requests (e.g. websocket watch requests) return a 500 error with text like cc


Question: CaptureStderr bool // If false, whitespace in std{err,out} will be removed. PreserveWhitespace bool <ins> Quiet              bool </ins> } // ExecWithOptions executes a command in the specified container, // returning stdout, stderr and error. `options` allowed for // additional parameters to be passed. func (f *Framework) ExecWithOptions(options ExecOptions) (string, string, error) { <del> Logf("ExecWithOptions %+v", options) </del> <ins> if !options.Quiet { Logf("ExecWithOptions %+v", options) } </ins> config, err := LoadConfig() ExpectNoError(err, "failed to load restclient config")

What happened: ExecUtil in has mandatory logging side effect hardcoded to INFO What you expected to happen: Since INFO level logging may or may not be an implementation detail of a test that is worth showing all the time, it would be better if we can turn this logging off optionally with a Quiet (false by default) boolean field How to reproduce it (as minimally and precisely as possible): Run any test that execs across pods, i.e. exec tests
There are no sig labels on this issue. Please add an appropriate label by using one of the following commands: - Please see the for a listing of the SIGs, working groups, and committees available. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: fwk.EventRecorder().Eventf(p.Pod, nil, v1.EventTypeWarning, "FailedScheduling", "Scheduling", msg) } called := make(chan struct{}) <del> stopFunc := eventBroadcaster.StartEventWatcher(func(obj runtime.Object) { </del> <ins> stopFunc, err := eventBroadcaster.StartEventWatcher(func(obj runtime.Object) { </ins> e, _ := obj.(*eventsv1.Event) if e.Reason != item.eventReason { t.Errorf("got event %v, want %v", e.Reason, item.eventReason) } close(called) }) <ins> if err != nil { t.Fatal(err) } </ins> sched.scheduleOne(ctx) <-called if e, a := item.expectAssumedPod, gotAssumedPod; !reflect.DeepEqual(e, a) {

Follow-up cleanup work for: Most of the information about this issue can be found in , where the work merged with a to change the function signature of to return a func() as well as an error. This issue will help to track that. All TODOs introduced in are resolved. N/A N/A <detailsN/A </details<detailsN/A </details<details</details<details</details<details</details<details</details
/sig api-machinery
The label(s) cannot be applied, because the repository doesn't have them. <detailsIn response to : /priority low Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/cc /triage accepted
/priority important-longterm
/assign
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Mark this issue or PR as rotten with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
/remove-lifecycle stale


Question: dirty = true } <del> frontEndPorts := map[int64]bool{} </del> <ins> frontEndPorts := map[int64]map[string]bool{} </ins> for i := range mappings { <del> frontEndPorts[mappings[i].FrontendPort] = true </del> <ins> if frontEndPorts[mappings[i].FrontendPort] == nil { frontEndPorts[mappings[i].FrontendPort] = map[string]bool{} } frontEndPorts[mappings[i].FrontendPort][mappings[i].FrontendProtocol] = true </ins> } // handle deletions <del> for port, listener := range actual { if _, ok := frontEndPorts[port]; !ok { err := c.deleteListenerV2(listener) if err != nil { return nil, err </del> <ins> for port := range actual { for protocol := range actual[port] { if _, ok := frontEndPorts[port][protocol]; !ok { err := c.deleteListenerV2(actual[port][protocol]) if err != nil { return nil, err } dirty = true </ins> } <del> dirty = true </del> } } } if err := c.reconcileLBAttributes(aws.StringValue(loadBalancer.LoadBalancerArn), annotations); err != nil { return nil, err }

What happened: What you expected to happen: kubernetes creating an AWS NLB for UDP How to reproduce it (as minimally and precisely as possible): Create service with annotation and port with Anything else we need to know?: AWS recently announced that UDP is available within their NLB Environment: Kubernetes version (use ): 1.11.9 Cloud provider or hardware configuration: AWS
/sig aws
/kind feature
/remove-kind bug
/milestone v1.16
You must be a member of the GitHub team to set the milestone. If you believe you should be able to issue the /milestone command, please contact your and have them propose you as an additional delegate for this responsibility. <detailsIn response to : /milestone v1.16 Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/milestone v1.16
I am taking up this Feature Development. How can this be assigned to me?
/assign
+1
+1 I am experiencing the same issue. NLB also supports UDP_TCP type, any service using "nlb" should be able to mix TCP and UDP port configs.
Any progress? I can collaborate on this, if you need a hand.
+1
+1 On Thu, Jul 18, 2019 at 12:58 Adam Murphy <:
+1
I just ran into this. Any updates?
Attempting to fix it with
Thanks, The Changes looks good.
This should be closed I think.
This feature only for 1.16?
Could feature back ported into 1.15 1.14 1.13
Could someone review please?
/sig cloud-provider
Any update on this?
As we are in the code freeze and this issue is still not fixed, I'll move it to the 1.17 milestone. I'm also going to add the PR that fixes this issue to the milestone. If there are any concerns, please ping me or ping SIG-Release on Slack. /milestone v1.17
Do you have a timeline when this will be released?
This is a breaking issue in multiple versions but the is not prioritized. Does anyone have a workaround at the moment?
For AWS you can associate elastic IP with pod. Need make aws cli call.

If you need static private IP then you can create DaemonSet attach NIC with static IP as second NIC to EC2 instance and node some label (like ). Later you start pod with network, get node label, bound to required host IP. If you need static public and private IPs, then you can customize node-group init script and attach NIC with permanent private IP and Elastic IP at node startup. Later simple start pod with network
Bug triage for release 1.17 here. Since this issue has been tagged for the milestone, we would like to check it's status. Code freeze is on Nov. 14. Will this issue be resolved before then? If not, we should move it out of the milestone.
Hi bug triage team for 1.17, will this issue be resolved before code freeze on Nov. 14 (Next Thursday)?
/milestone v1.18 (v1.17 code freeze)
looking for a first issue to work, can I take this
gave some good feedback to the original PR. Does this need a reset with a fresh PR? I also have a need for this. I don't have a strong urge to write, or reuse, an operator for NLBs in AWS for UDP. This will eventually get done in k8s anyways, so I'd rather put the effort there.
BTW, you can always setup an NLB manually to forward UDP to a nodePort service on your k8s instances. The issue with that is that you have to pick a NodePort service for UDP before hand, and manually handle the NLB and the target groups. One cluster upgrade and your target group behind the NLB disappears since all of the instances get replaced. It's not ideal but in a pinch...
I don't mind updating a the PR, seems like a good first issue for me.
/assign
+1
+1
+1
+1
+1
Hello ! Bug Triage team here for the 1.18 release. This is a friendly reminder that code freeze is scheduled for 5 March, which is a few weeks from now. Is this issue still intended for milestone 1.18? Thanks in advance!

C/O Code freeze for release 1.18 is starting in 5 March (less than 1 week from now) and we want to ensure that each issue has a chance to be finished on time. Is this still on track for this deadline?
/remove-kind bug
Is there any work remaining to be done on this? Willing to contribute if so, would really love to see this make it into 1.18!
Thanks for the reply This has been a long running issue, and given that there has not been any recent activity, would it make more sense to once again move the milestone? We are 2 days away from code freeze (5 March).
I wonder what "activity" you are expecting.
We are now in code freeze for 1.18, so I am moving this to the next milestone. /milestone v1.19
Hello, as of now, I see is 65% completed. May we please have an update on estimated release date of UDP support for AWS NLB?
People have been trying to be constructive here. But the team doesn't even try to tell us what is pending. Sad.
Would also love to see this feature for EKS - and am willing to help out. Even if we could get it in 1.19 it would still take a long time to become available in EKS (which is currently at 1.15). So the sooner we can start work on this - the better :-)
+1 Also hope we could get in 1.19 and happy to help out!
any update on this pretty please ?
whats the update here?
From what I can see, the main issues that remain unresolved since the first PR last year: is still pending if you need a service that supports both TCP and UDP, e.g., DNS. DNS is also a more-complicated example because it supports TCP and UDP on the same port number, and that'll need more refactoring here, as AWS treats that as a single multi-protocol target. Edit: To be clear, not really a blocker for UDP support, but it's been raised in the discussion earlier. AWS NLB can only do TCP-based health checks (including HTTP and HTTPS), so your service needs to have a health-check TCP port listening. Current PRs silently assume it's on the same port number as the UDP service you're advertising.

If you're using , then it'll use the health check from  instead anyway. So this is probably only an issue for LoadBalancer Services with set. See "Things to know" on the .


Question: "k8s.io/apimachinery/pkg/util/validation/field" "k8s.io/apimachinery/pkg/util/wait" "k8s.io/apimachinery/pkg/watch" <ins> "k8s.io/apiserver/pkg/endpoints/handlers/fieldmanager" </ins> genericapirequest "k8s.io/apiserver/pkg/endpoints/request" "k8s.io/apiserver/pkg/registry/generic" "k8s.io/apiserver/pkg/registry/rest"

Sending a patch built from a typed client to SSA will force a write to etcd (and bump resourceVersion) since creationTimestamp is serialized to null. This is a follow-up of which doesn't solve the issue properly. Sending a no-op SSA request with creationTimestamp to null shouldn't do anything Kubectl apply an object with creationTimestamp null twice and see that it updates the object. Probably something like this would cause it:
/sig api-machinery /wg api-expression /triage accepted /assign /priority important-soon


Question: <ins> /* Copyright 2017 The Kubernetes Authors. Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. */ package autoscaling import ( "strings" "time" metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" "k8s.io/kubernetes/pkg/api/v1" "k8s.io/kubernetes/test/e2e/common" "k8s.io/kubernetes/test/e2e/framework" . "github.com/onsi/ginkgo" . "github.com/onsi/gomega" ) var _ = framework.KubeDescribe("[Feature:ClusterSizeAutoscalingScaleUp] [Slow] Autoscaling", func() { f := framework.NewDefaultFramework("autoscaling") framework.KubeDescribe("Autoscaling a service", func() { BeforeEach(func() { // Check if Cloud Autoscaler is enabled by trying to get its ConfigMap. _, err := f.ClientSet.CoreV1().ConfigMaps("kube-system").Get("cluster-autoscaler-status", metav1.GetOptions{}) if err != nil { framework.Skipf("test expects Cluster Autoscaler to be enabled") } }) Context("from 1 pod and 3 nodes to 8 pods and >=4 nodes", func() { const nodesNum = 3       // Expect there to be 3 nodes before and after the test. var nodeGroupName string // Set by BeforeEach, used by AfterEach to scale this node group down after the test. var nodes *v1.NodeList   // Set by BeforeEach, used by Measure to calculate CPU request based on node's sizes. BeforeEach(func() { // Make sure there is only 1 node group, otherwise this test becomes useless. nodeGroups := strings.Split(framework.TestContext.CloudConfig.NodeInstanceGroup, ",") if len(nodeGroups) != 1 { framework.Skipf("test expects 1 node group, found %d", len(nodeGroups)) } nodeGroupName = nodeGroups[0] // Make sure the node group has exactly 'nodesNum' nodes, otherwise this test becomes useless. nodeGroupSize, err := framework.GroupSize(nodeGroupName) framework.ExpectNoError(err) if nodeGroupSize != nodesNum { framework.Skipf("test expects %d nodes, found %d", nodesNum, nodeGroupSize) } // Make sure all nodes are schedulable, otherwise we are in some kind of a problem state. nodes = framework.GetReadySchedulableNodesOrDie(f.ClientSet) schedulableCount := len(nodes.Items) Expect(schedulableCount).To(Equal(nodeGroupSize), "not all nodes are schedulable") }) AfterEach(func() { // Scale down back to only 'nodesNum' nodes, as expected at the start of the test. framework.ExpectNoError(framework.ResizeGroup(nodeGroupName, nodesNum)) framework.ExpectNoError(framework.WaitForClusterSize(f.ClientSet, nodesNum, 15*time.Minute)) }) Measure("takes less than 15 minutes", func(b Benchmarker) { // Measured over multiple samples, scaling takes 10 +/- 2 minutes, so 15 minutes should be fully sufficient. const timeToWait = 15 * time.Minute // Calculate the CPU request of the service. // This test expects that 8 pods will not fit in 'nodesNum' nodes, but will fit in >='nodesNum'+1 nodes. // Make it so that 'nodesNum' pods fit perfectly per node (in practice other things take space, so less than that will fit). nodeCpus := nodes.Items[0].Status.Capacity[v1.ResourceCPU] nodeCpuMillis := (&nodeCpus).MilliValue() cpuRequestMillis := int64(nodeCpuMillis / nodesNum) // Start the service we want to scale and wait for it to be up and running. nodeMemoryBytes := nodes.Items[0].Status.Capacity[v1.ResourceMemory] nodeMemoryMB := (&nodeMemoryBytes).Value() / 1024 / 1024 memRequestMB := nodeMemoryMB / 10 // Ensure each pod takes not more than 10% of node's total memory. replicas := 1 resourceConsumer := common.NewDynamicResourceConsumer("resource-consumer", common.KindDeployment, replicas, 0, 0, 0, cpuRequestMillis, memRequestMB, f) defer resourceConsumer.CleanUp() resourceConsumer.WaitForReplicas(replicas, 1*time.Minute) // Should finish ~immediately, so 1 minute is more than enough. // Enable Horizontal Pod Autoscaler with 50% target utilization and // scale up the CPU usage to trigger autoscaling to 8 pods for target to be satisfied. targetCpuUtilizationPercent := int32(50) hpa := common.CreateCPUHorizontalPodAutoscaler(resourceConsumer, targetCpuUtilizationPercent, 1, 10) defer common.DeleteHorizontalPodAutoscaler(resourceConsumer, hpa.Name) cpuLoad := 8 * cpuRequestMillis * int64(targetCpuUtilizationPercent) / 100 // 8 pods utilized to the target level resourceConsumer.ConsumeCPU(int(cpuLoad)) // Measure the time it takes for the service to scale to 8 pods with 50% CPU utilization each. b.Time("total scale-up time", func() { resourceConsumer.WaitForReplicas(8, timeToWait) }) }, 1) // Increase to run the test more than once. }) }) }) </ins>

Currently the time it takes for HPA and CA to scale up after load increase is not measured at all, and this can lead to regressions that will be missed. This issue is about adding an e2e test that will prevent such regressions.
There are no sig labels on this issue. Please by:<br(1) mentioning a sig: <br(2) specifying the label manually: <br<brNote: method (1) will trigger a notification to the team. You can find the team list .
Filip, can you reopen? I cannot do this myself, and this issue is clearly not closed yet.
Thanks Wojtek!


Question: package v1_test import ( <ins> "encoding/json" </ins> "reflect" "testing"

<!-- Thanks for filing an issue! Before hitting the button, please answer these questions.--Is this a request for help? (If yes, you should use our troubleshooting guide and community support channels, see ): No What keywords did you search in Kubernetes issues before filing this one? (If you have found any duplicates, you should instead reply there.): initContainers imagePullPolicy initContainers required value Is this a BUG REPORT or FEATURE REQUEST? (choose one): BUG REPORT <!-- If this is a BUG REPORT, please: Fill in as much of the template below as you can. If you leave out information, we can't help you as well. If this is a FEATURE REQUEST, please: Describe in detail the feature/behavior/change you'd like to see. In both cases, be ready for followup questions, and please respond in a timely manner. If we can't reproduce a bug or think a feature already exists, we might close your issue. If we're wrong, PLEASE feel free to reopen it and explain why. --Kubernetes version (use ): Environment: Cloud provider or hardware configuration: GCE OS (e.g. from /etc/os-release): Kernel (e.g. ): Install tools: cluster/kube- Others: What happened: When trying to create either a ReplicaSet or Replication Controller whose Pod template contains an Init Container, the validation fails saying is a required value. What you expected to happen: The create should succeed and the Init Container should receive a default . How to reproduce it (as minimally and precisely as possible): Try to create either of the following: ReplicaSet Replication Controller Anything else do we need to know: If I add to the Init Container in the ReplicaSet or Replication Controller Pod template, the create succeeds. When I create an Init Container without in a StatefulSet Pod template, it succeeds. When I create an Init Container without in a bare Pod, it succeeds:
Looks like defaulting not being applied on server? That is probably due to removing defaulting from convert. Surprised we didn't catch this earlier
Some values still not being defaulted
, I'm going to create a PR if none works on it.
We decode from bytes, set defaults and then convert; but for init containers, we delay the decode from "decode phase" to "convert phase" at .

The defaults has been handled for initContainer at . /cc I'm working to find a solution for that; if any suggestion, please let me know.
Would this bug affect the deletion of jobs with an init-container? I ran into this bug and adjusted my manifest to add the pull policy to the init-container. However, I now cannot delete said job claiming a field is immutable. Example:
, not sure now. Let me check it,
Same error here with version 1.5.1, 1.5.2 The Job "grafana-import-dashboards" is invalid: [0].imagePullPolicy: Required value With this setup: kubectl apply -f monitoring Not working even if I specify a pullpolicy in the yaml. If I roll back to 1.4.7, it is working. Thank you. Greg.
any updates on a PR to address this issue? Is anyone from able to pick this issue up otherwise?
, here's the latest status: there's is a PR to generate for each Object (), it works for Object (Pod, Job) now but List; I'm working to make it generating for List.
Why would you need List to work? No client side logic should be decoding lists except as a generic submission.
The List tests are failed in . There's no for List, so no code to decode init-container annotation, e.g. of should call 's recursively.
Thanks for the update
+1
Any update on this? I still getting this issue with the last version of 1.5.3 on GKE?
ping for the comments? should we merge the PR for 1.5?
, there's a PR () that handled that issue; but we're discussing not to merge it because initContainers had been moved to fields, and fields is the recommended way to use since 1.6. But I'd like to get & 's comments on 1.5.
This is still an issue in 1.5.3 when using annotations: Without the deployment would not be accepted
ping & :).
i dont think this is fixed in 1.5 I just encountered this bug on 1.5.3
, yes, the PR is NOT merged to any branch, so the issue is still here. I'll create a PR for 1.5.
Hi Anyone help this issue.

#cat mongo- apiVersion: v1 kind: Pod metadata: labels: name: mongo-svc-a name: mongo-deployment spec: replicas: 1 template: metadata: labels: name: mongo spec: containers: image: mongo name: mongo ports: name: mongo containerPort: hostPort: volumeMounts: mountPath: "/data/db" name: mongopv volumes: name: mongopv persistentVolumeClaim: claimName: mongopvc When i want create mongo db container by using this file folowing error is occured. The Pod "mongo-deployment" is invalid:spec.containers: Required value
, it seems there're two here, please remove the first one and have a try.
Hi my kubectl binary is configured at ubuntu 16.04 machine.

NAME LABELS STATUS 10.1 Ready,SchedulingDisabled 10.2 10.1.10.2 Ready 10.3 Ready apiVersion: v1 kind: Node metadata: creationTimestamp: 2017-03-03T14:41:52Z labels: 10.1.10.1 name: 10.1.10.1 resourceVersion: "" selfLink: /api/v1/nodes/10.1.10.1 uid: -001f-11e7-8e83- spec: externalID: 10.1.10.1 status: addresses: address: 10.1.10.1 type: LegacyHostIP address: 10.1.10.1 type: InternalIP capacity: cpu: "2" memory: 4041352Ki pods: "110" conditions: lastHeartbeatTime: 2017-04-05T18:58:42Z lastTransitionTime: 2017-03-03T14:41:52Z message: kubelet has sufficient disk space available reason: KubeletHasSufficientDisk status: "False" type: OutOfDisk lastHeartbeatTime: 2017-04-05T18:58:42Z lastTransitionTime: 2017-04-05T15:27:52Z message: kubelet is posting ready status reason: KubeletReady status: "True" type: Ready nodeInfo: bootID: -6124-4ac7-ab96- containerRuntimeVersion: docker://1.12.6 kernelVersion: 4.9.16-coreos-r1 kubeProxyVersion: v1.2.4+coreos.1 kubeletVersion: v1.2.4+coreos.1 machineID: osImage: Container Linux by CoreOS 1298.7.0 (Ladybug) systemUUID: C3FD9852-2DFB-21F4-CE07-2597A746D99D When i create pv for creation pod is shows pv is pending. Please give any suggestions.....
Is there solution of this problem with version 2.7. Still facing the same. what exactly expectation of helm upgrade --install relname chartsdirectory is is expected to work on first time installation in case with CI/CD Pipelines.


Question: Name string // Username for registry (required) Username string <del> // Email for registry (required) </del> <ins> // Email for registry (optional) </ins> Email string // Password for registry (required) Password string

<!-- Thanks for filing an issue! Before hitting the button, please answer these questions.--Is this a request for help? (If yes, you should use our troubleshooting guide and community support channels, see ): What keywords did you search in Kubernetes issues before filing this one? (If you have found any duplicates, you should instead reply there.): Is this a BUG REPORT or FEATURE REQUEST? (choose one): <!-- If this is a BUG REPORT, please: Fill in as much of the template below as you can. If you leave out information, we can't help you as well. If this is a FEATURE REQUEST, please: Describe in detail the feature/behavior/change you'd like to see. In both cases, be ready for followup questions, and please respond in a timely manner. If we can't reproduce a bug or think a feature already exists, we might close your issue. If we're wrong, PLEASE feel free to reopen it and explain why. --Kubernetes version (use ): Environment: Cloud provider or hardware configuration: OS (e.g. from /etc/os-release): Kernel (e.g. ): Install tools: Others: What happened: When I create a docker registry secret, the is also a MUST, I think we should make this parameter as optional as there is no need always put when creating a docker registry secret. What you expected to happen: How to reproduce it (as minimally and precisely as possible): Anything else we need to know:
Agree it should not be required
Did this change get reverted? I do not see as optional on stable.
I feel that this code should be changed because "--docker-email flag" is no longer required.
error: flag docker-email is required. On kubenetes 1.10.3, the above error message still exists
The field is still required. This is very frustrating as does not require an email but Kubernetes does? Is there a good reason for this?
Why is this closed? As far as I can see it is still required?


Question: <del> /* Copyright 2015 The Kubernetes Authors. Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. */ package unstructured import ( "fmt" "reflect" "testing" "k8s.io/apimachinery/pkg/util/diff" "k8s.io/apimachinery/pkg/util/json" ) // Definte a number of test types. type A struct { A int    `json:"aa,omitempty"` B string `json:"ab,omitempty"` C bool   `json:"ac,omitempty"` } type B struct { A A                 `json:"ba"` B string            `json:"bb"` C map[string]string `json:"bc"` D []string          `json:"bd"` } type C struct { A []A `json:"ca"` B `json:",inline"` C string         `json:"cc"` D *int64         `json:"cd"` E map[string]int `json:"ce"` F []bool         `json:"cf"` G []int          `json:"cg"` H float32        `json:"ch"` I []interface{}  `json:"ci"` } type D struct { A []interface{} `json:"da"` } type E struct { A interface{} `json:"ea"` } type F struct { A string            `json:"fa"` B map[string]string `json:"fb"` C []A               `json:"fc"` D int               `json:"fd"` E float32           `json:"fe"` F []string          `json:"ff"` G []int             `json:"fg"` H []bool            `json:"fh"` I []float32         `json:"fi"` } type G struct { Custom Custom `json:"custom"` } type Custom struct { data []byte } func (c Custom) MarshalJSON() ([]byte, error) { return c.data, nil } func doRoundTrip(t *testing.T, item interface{}) { data, err := json.Marshal(item) if err != nil { t.Errorf("Error when marshaling object: %v", err) return } unstr := make(map[string]interface{}) err = json.Unmarshal(data, &unstr) if err != nil { t.Errorf("Error when unmarshaling to unstructured: %v", err) return } data, err = json.Marshal(unstr) if err != nil { t.Errorf("Error when marshaling unstructured: %v", err) return } unmarshalledObj := reflect.New(reflect.TypeOf(item).Elem()).Interface() err = json.Unmarshal(data, &unmarshalledObj) if err != nil { t.Errorf("Error when unmarshaling to object: %v", err) return } if !reflect.DeepEqual(item, unmarshalledObj) { t.Errorf("Object changed during JSON operations, diff: %v", diff.ObjectReflectDiff(item, unmarshalledObj)) return } newUnstr, err := DefaultConverter.ToUnstructured(item) if err != nil { t.Errorf("ToUnstructured failed: %v", err) return } newObj := reflect.New(reflect.TypeOf(item).Elem()).Interface() err = DefaultConverter.FromUnstructured(newUnstr, newObj) if err != nil { t.Errorf("FromUnstructured failed: %v", err) return } if !reflect.DeepEqual(item, newObj) { t.Errorf("Object changed, diff: %v", diff.ObjectReflectDiff(item, newObj)) } } func TestRoundTrip(t *testing.T) { intVal := int64(42) testCases := []struct { obj interface{} }{ { // This (among others) tests nil map, slice and pointer. obj: &C{ C: "ccc", }, }, { // This (among others) tests empty map and slice. obj: &C{ A: []A{}, C: "ccc", E: map[string]int{}, I: []interface{}{}, }, }, { obj: &C{ A: []A{ { A: 1, B: "11", C: true, }, { A: 2, B: "22", C: false, }, }, B: B{ A: A{ A: 3, B: "33", }, B: "bbb", C: map[string]string{ "k1": "v1", "k2": "v2", }, D: []string{"s1", "s2"}, }, C: "ccc", D: &intVal, E: map[string]int{ "k1": 1, "k2": 2, }, F: []bool{true, false, false}, G: []int{1, 2, 5}, H: 3.3, I: []interface{}{nil, nil, nil}, }, }, { // Test slice of interface{} with empty slices. obj: &D{ A: []interface{}{[]interface{}{}, []interface{}{}}, }, }, { // Test slice of interface{} with different values. obj: &D{ A: []interface{}{3.0, "3.0", nil}, }, }, } for i := range testCases { doRoundTrip(t, testCases[i].obj) if t.Failed() { break } } } // Verifies that: // 1) serialized json -> object // 2) serialized json -> map[string]interface{} -> object // produces the same object. func doUnrecognized(t *testing.T, jsonData string, item interface{}, expectedErr error) { unmarshalledObj := reflect.New(reflect.TypeOf(item).Elem()).Interface() err := json.Unmarshal([]byte(jsonData), &unmarshalledObj) if (err != nil) != (expectedErr != nil) { t.Errorf("Unexpected error when unmarshaling to object: %v, expected: %v", err, expectedErr) return } unstr := make(map[string]interface{}) err = json.Unmarshal([]byte(jsonData), &unstr) if err != nil { t.Errorf("Error when unmarshaling to unstructured: %v", err) return } newObj := reflect.New(reflect.TypeOf(item).Elem()).Interface() err = DefaultConverter.FromUnstructured(unstr, newObj) if (err != nil) != (expectedErr != nil) { t.Errorf("Unexpected error in FromUnstructured: %v, expected: %v", err, expectedErr) } if expectedErr == nil && !reflect.DeepEqual(unmarshalledObj, newObj) { t.Errorf("Object changed, diff: %v", diff.ObjectReflectDiff(unmarshalledObj, newObj)) } } func TestUnrecognized(t *testing.T) { testCases := []struct { data string obj  interface{} err  error }{ { data: "{"da":[3.0,"3.0",null]}", obj:  &D{}, }, { data: "{"ea":[3.0,"3.0",null]}", obj:  &E{}, }, { data: "{"ea":[null,null,null]}", obj:  &E{}, }, { data: "{"ea":[[],[null]]}", obj:  &E{}, }, { data: "{"ea":{"a":[],"b":null}}", obj:  &E{}, }, { data: "{"fa":"fa","fb":{"a":"a"}}", obj:  &F{}, }, { data: "{"fa":"fa","fb":{"a":null}}", obj:  &F{}, }, { data: "{"fc":[null]}", obj:  &F{}, }, { data: "{"fc":[{"aa":123,"ab":"bbb"}]}", obj:  &F{}, }, { // Only unknown fields data: "{"fx":[{"aa":123,"ab":"bbb"}],"fz":123}", obj:  &F{}, }, { data: "{"fc":[{"aa":"aaa","ab":"bbb"}]}", obj:  &F{}, err:  fmt.Errorf("json: cannot unmarshal string into Go value of type int"), }, { data: "{"fd":123,"fe":3.5}", obj:  &F{}, }, { data: "{"ff":["abc"],"fg":[123],"fh":[true,false]}", obj:  &F{}, }, { // Invalid string data data: "{"fa":123}", obj:  &F{}, err:  fmt.Errorf("json: cannot unmarshal number into Go value of type string"), }, { // Invalid string data data: "{"fa":13.5}", obj:  &F{}, err:  fmt.Errorf("json: cannot unmarshal number into Go value of type string"), }, { // Invalid string data data: "{"fa":true}", obj:  &F{}, err:  fmt.Errorf("json: cannot unmarshal bool into Go value of type string"), }, { // Invalid []string data data: "{"ff":123}", obj:  &F{}, err:  fmt.Errorf("json: cannot unmarshal number into Go value of type []string"), }, { // Invalid []string data data: "{"ff":3.5}", obj:  &F{}, err:  fmt.Errorf("json: cannot unmarshal number into Go value of type []string"), }, { // Invalid []string data data: "{"ff":[123,345]}", obj:  &F{}, err:  fmt.Errorf("json: cannot unmarshal number into Go value of type string"), }, { // Invalid []int data data: "{"fg":123}", obj:  &F{}, err:  fmt.Errorf("json: cannot unmarshal number into Go value of type []int"), }, { // Invalid []int data data: "{"fg":"abc"}", obj:  &F{}, err:  fmt.Errorf("json: cannot unmarshal string into Go value of type []int"), }, { // Invalid []int data data: "{"fg":["abc"]}", obj:  &F{}, err:  fmt.Errorf("json: cannot unmarshal string into Go value of type int"), }, { // Invalid []int data data: "{"fg":[3.5]}", obj:  &F{}, err:  fmt.Errorf("json: cannot unmarshal number 3.5 into Go value of type int"), }, { // Invalid []int data data: "{"fg":[true,false]}", obj:  &F{}, err:  fmt.Errorf("json: cannot unmarshal number 3.5 into Go value of type int"), }, { // Invalid []bool data data: "{"fh":123}", obj:  &F{}, err:  fmt.Errorf("json: cannot unmarshal number into Go value of type []bool"), }, { // Invalid []bool data data: "{"fh":"abc"}", obj:  &F{}, err:  fmt.Errorf("json: cannot unmarshal string into Go value of type []bool"), }, { // Invalid []bool data data: "{"fh":["abc"]}", obj:  &F{}, err:  fmt.Errorf("json: cannot unmarshal string into Go value of type bool"), }, { // Invalid []bool data data: "{"fh":[3.5]}", obj:  &F{}, err:  fmt.Errorf("json: cannot unmarshal number into Go value of type bool"), }, { // Invalid []bool data data: "{"fh":[123]}", obj:  &F{}, err:  fmt.Errorf("json: cannot unmarshal number into Go value of type bool"), }, { // Invalid []float data data: "{"fi":123}", obj:  &F{}, err:  fmt.Errorf("json: cannot unmarshal number into Go value of type []float32"), }, { // Invalid []float data data: "{"fi":"abc"}", obj:  &F{}, err:  fmt.Errorf("json: cannot unmarshal string into Go value of type []float32"), }, { // Invalid []float data data: "{"fi":["abc"]}", obj:  &F{}, err:  fmt.Errorf("json: cannot unmarshal string into Go value of type float32"), }, { // Invalid []float data data: "{"fi":[true]}", obj:  &F{}, err:  fmt.Errorf("json: cannot unmarshal bool into Go value of type float32"), }, } for i := range testCases { doUnrecognized(t, testCases[i].data, testCases[i].obj, testCases[i].err) if t.Failed() { break } } } func TestFloatIntConversion(t *testing.T) { unstr := map[string]interface{}{"fd": float64(3)} var obj F if err := DefaultConverter.FromUnstructured(unstr, &obj); err != nil { t.Errorf("Unexpected error in FromUnstructured: %v", err) } data, err := json.Marshal(unstr) if err != nil { t.Fatalf("Error when marshaling unstructured: %v", err) } var unmarshalled F if err := json.Unmarshal(data, &unmarshalled); err != nil { t.Fatalf("Error when unmarshaling to object: %v", err) } if !reflect.DeepEqual(obj, unmarshalled) { t.Errorf("Incorrect conversion, diff: %v", diff.ObjectReflectDiff(obj, unmarshalled)) } } func TestCustomToUnstructured(t *testing.T) { testcases := []struct { Data     string Expected interface{} }{ {Data: `null`, Expected: nil}, {Data: `true`, Expected: true}, {Data: `false`, Expected: false}, {Data: `[]`, Expected: []interface{}{}}, {Data: `[1]`, Expected: []interface{}{int64(1)}}, {Data: `{}`, Expected: map[string]interface{}{}}, {Data: `{"a":1}`, Expected: map[string]interface{}{"a": int64(1)}}, {Data: `0`, Expected: int64(0)}, {Data: `0.0`, Expected: float64(0)}, } for _, tc := range testcases { result, err := DefaultConverter.ToUnstructured(&G{Custom: Custom{data: []byte(tc.Data)}}) if err != nil { t.Errorf("%s: %v", tc.Data, err) continue } fieldResult := result["custom"] if !reflect.DeepEqual(fieldResult, tc.Expected) { t.Errorf("%s: expected %v, got %v", tc.Data, tc.Expected, fieldResult) // t.Log("expected", spew.Sdump(tc.Expected)) // t.Log("actual", spew.Sdump(fieldResult)) continue } } } </del>

Kubernetes version (use ): master. What happened: Unstructured converter from does not roundtrip . What you expected to happen: When I call on a which is backed by it should basically make a copy of the content (can be optimized btw using deep copy ) and not a json object with key. How to reproduce it (as minimally and precisely as possible): See the test in my branch Test output: Anything else we need to know: For some reason changed the argument types from to . I guess the intent is to make the converter a more generic tool. But based on the previous signature I assume I'm doing the right thing. Actually, my code works with and I don't know if the actual object behind the interface is an or not - I just pass it to the converter and it works for usual API types, but not for unstructured. Also it would be useful to have a more type-safe version of the converter (i.e. the previous signature) that just delegates to a lower-level converter that does not care about types.
There are no sig labels on this issue. Please by:<br(1) mentioning a sig: <br(2) specifying the label manually: <br<brNote: method (1) will trigger a notification to the team. You can find the team list and label list
/sig api-machinery
/kind bug
/assign
Looked into this issue just now. It is caused by the pointer vs non-pointer receiver of the method - similar to . In detail: is passed as to when it calls passing as . if passed value implements (has method ) does not implement this interface, does. Because on it . So perhaps the fix is to do the dereferencing inside of the ? Or maybe the same way as - check if the value is addressable, get a pointer to the value and then see if it matches the interface (I dunno if it will work that way)? In any case, the check whether a value implements the interface should be done on both the value itself () and the pointer to the value (; if the value is addressable) to cover both valid cases. This is what stdlib package does. Right now it is broken - it dereferences the pointer and then checks if the thing implements the interface. Does it make sense?

Reiterating the mentions to trigger a notification: <detailsIn response to : Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
does this helps? Look like we need to do MarshalJSON on non-pointers for it to work. I didn't get into the details but if that is the case then making Unstructured's MarshalJSON() receiver, a object instead of pointer should fix your problem, right?
Only if we don't fix the actual cause of the bug which I described above. From the stack overflow question that is mentioned in the link you provided: This code is buggy - to work it should be passing into and into . Yes, json package tries to take the address of the passed (so a pointer receiver should satisfy the interface check) but it cannot take an address of a function argument ( will return false). So, pointer receiver will be invoked if either is a field in a struct (so that an address can be taken via reflection) and a struct is passed (either by value or via a pointer) to / OR is passed. Please take a look at - both issues have the same cause.
I'm happy to take a stab at it and submit a PR but I don't want to spend my time if it is not going to get merged - if we could merge first to make sure people who can approve/lgtm are happy with the approach taken. I strongly believe it is the right thing to do - that is how standard library does it and, well, it is just logical and makes sense :)


Question: "docker_stats_unsupported.go", "docker_stats_windows.go", "docker_streaming.go", <ins> "docker_streaming_others.go", "docker_streaming_windows.go", </ins> "exec.go", "helpers.go", "helpers_linux.go",

<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . If this may be security issue, please disclose it privately via --Is this a BUG REPORT or FEATURE REQUEST?: /kind bug What happened: Ran on windows pod and received the error message: What you expected to happen: The port be available on my localhost How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: socat is not available on windows nodes Environment: Kubernetes version (use ): Client Version: version.Info{Major:"1", Minor:"8", GitVersion:"v1.8.1", GitCommit:"", GitTreeState:"clean", BuildDate:"2017-10-11T23:27:35Z", GoVersion:"go1.8.3", Compiler:"gc", Platform:"linux/amd64"} Server version: 1.10.2 Cloud provider or hardware configuration: Azure - acs-engine OS (e.g. from /etc/os-release): Windows Server 1803 Kernel (e.g. ): Install tools: Others:
/sig windows
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale
I can confirm this on 1803:
What would be the approach to fixing this? If connecting to a windows machine use a different tool other than socat? Are there any tools that are used in the windows world for port-forwarding? If I got some guidance on tools to use I could take a try at a PR for this.
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
The problem is that the portForward streaming operation invokes and on the node. Those aren't available (or required) on Windows.

On Linux, the command is something like
Would using be the command to get this to work? (v=ws.10)#add-v4tov4
/remove-lifecycle stale
I looked into this, I think that it could in theory work, the hard part about trying to use is that it needs the IP address, whereas in this method all you have is the PID and . I don't know how to do mapping to get the IP, but if you can figure that out then it's just a matter of allocating a random port to use as the listen port, assuming people are okay with that solution. It has the disadvantage of being less secure than the Linux side, which only exposes access to the stream to the Kubelet process instead of all processes, but I'm not sure that's a big security concern.
This function implemented by the CRI shim, which is capable of finding out the IP address by . /cc
I'm a bit ignorant of all things windows. Does "all processes" really mean everything, including unprivileged containers, or do you just mean all processes in the host network namespace (or whatever the windows equivalent is)?
It would have been just the host network namespace. This approach didn't really pan out though. is investigating another approach that more closely matches the way it works on Linux.


Question: "k8s.io/apimachinery/pkg/util/wait" "k8s.io/client-go/kubernetes/fake" core "k8s.io/client-go/testing" <ins> fakecloud "k8s.io/kubernetes/pkg/cloudprovider/providers/fake" </ins> kubeletapis "k8s.io/kubernetes/pkg/kubelet/apis" "k8s.io/kubernetes/pkg/kubelet/cm" kubecontainer "k8s.io/kubernetes/pkg/kubelet/container"

BUG REPORT: /kind bug What happened: After upgrading the kubernetes cluster from 1.6.4 with the VSphere Cloud Provider enabled to 1.6.5/1.6.6 or 1.6.7, the cluster nodes don't have anymore InternalIPs Several issues occur at this step. For example, it breaks heapster with such an error : What you expected to happen: The node should have an InternalIP. How to reproduce it (as minimally and precisely as possible): Deploy a 1 node kubernetes cluster in a VM, and enable the vsphere cloud provider. Anything else we need to know?: with kubernetes 1.6.4 we get : Downgrading to 1.6.4 fixes the issue. Environment: Kubernetes version (use ):Cloud provider or hardware configuration: vsphere OS (e.g. from /etc/os-release):Kernel (e.g. ):Install tools: Others: It may be related to some changes around PR kubernetes /area platform/vsphere cloudprovider
There are no sig labels on this issue. Please by:<br(1) mentioning a sig: <bre.g., for API Machinery<br(2) specifying the label manually: <bre.g., for sig/scalability<br<brNote: method (1) will trigger a notification to the team. You can find the team list and label list
/sig cluster-ops /sig network /area platform/vsphere /area cloudprovider
Performing some more tests on a mini instance outside of the production cluster, I think I've identified the issue. The configuration required to start kubelet with the option (otherwise the wrong IP/interface is used to communicate between nodes). The issue is triggered when this option is set. The cloud provider looks OK. But the issue is located here : When the is not , the loop iterates on the node addresses and returns only the first one found. Since PR kubernetes, the list always returns an ExternalIP first. I think the loop in should iterate on all the addresses to rebuild the list, so that it will include the ExternalIP and the InternalIP. Can someone confirm the analysis ?
/sig node
ping
Modifying the code like this solved the issue : which returns : I guess it can be done in a cleaner way, but it was to quickly test the fix.

CC:
Thanks for filing this issue related to vSphere cloud provider. I agree with your analysis. It seems the current function assumes that the IP address specified by user can only be InternalIP. Your fix to is a better solution than work-around in cloud provider.
Hi, what are the next steps to discuss about the PR ? Are you able to make some tests ?


Question: # To copy an image between projects: # `gcloud compute --project <to-project> disks create <image name> --image=https://www.googleapis.com/compute/v1/projects/<from-project>/global/images/<image-name>` # `gcloud compute --project <to-project> images create <image-name> --source-disk=<image-name>` <del> GCE_IMAGES=e2e-node-ubuntu-trusty-docker10-image,e2e-node-ubuntu-trusty-docker9-image,e2e-node-coreos-stable20160622-image,e2e-node-containervm-v20160321-image </del> <ins> GCE_IMAGES=e2e-node-ubuntu-trusty-docker9-v1-image,e2e-node-ubuntu-trusty-docker10-v1-image,e2e-node-coreos-stable20160622-image,e2e-node-containervm-v20160321-image </ins> GCE_ZONE=us-central1-f GCE_PROJECT=kubernetes-jenkins-pull GCE_IMAGE_PROJECT=kubernetes-node-e2e-images

Doesn't seem to haven been caused by my change:
/cc
The test only failed on coreos, and the is full of error messages. Plus the log is almost unreadable because of the extremely high verbosity. I filed for that. shows that the node has been rebooted a couple of times. And the kernel log shows This looks like . The hairpin mode is set to hairpin-veth. /cc
Yeah don't run with hairpin-mode I guess. is coreos using kubenet?
Node problem detector should detect this issue and report KernelDeadLock condition.
Yeah, but this is node e2e test, we don't have node problem detector running on node e2e cluster.
Not sure how to assign coreos issues... Assigning to people to reassign based on thread activity. Please reassign correctly!
ping!
/cc
This happens in node e2e test right? I believe node e2e test is NOT using kubenet.
Running node e2e with kubenet+hairpin-veth could prevent this.
Sorry, this is lost in github notifications.. Feel free to ping me on slack if I am not responding in a day next time :)
To run kubenet on a node, the framework needs to download and unpack the cni plugin in kubelet needs to run with and a fixed pod CIDR (since there is no node controller) I did those in , but it failed on the trusty images because is not installed. We need to either install this in the node e2e framework, or bake it in to the image. what's the best practice for installing packages on the images?
I believe updating is the best way to bake it into the image, and I also think that should be preferred over installing it each time. You could do the same with CNI. Thanks for working on this!
I updated the PR to install nsenter on the trusty images. As for cni, I still keep it in the node e2e framework to install it every time because 1) it only downloads from GCS and untar a file of ~14MB, 2) the tarball is maintained and updated by It'd be an extra burden to update the images just for cni updates.
OK with the plan.

Seeing the error message again after switching to kubenet+hairpin-veth do you want to take a look at the logs?
let me take a look
urrrh... we hit it again. Switching back to
There's a difference in the above kernel log; the device is not a interface but rather an interface (presumably the one within a container's net ns since there is no eth0 on the host). I think all the prior instances of this were veths on the host.
True. This one is slightly different. I think deadlock on interface also showed up before.
Haven't seen any new occurrence since landed. We'll know in a day or so whether this problem has been fixed.
AFAIK, there are no new failures after landed. It should be safe to let this issue stay closed.
unfortunately I hit the bug with hairpinmode sets to promiscuous-bridge kubelet v1.3.0, linux kernel 4.6.3 cbr0 has already enter the promiscuous mode, all veth hairpinmode = 1 start with parameter
can you ? what distro are you running?
also are you using a network-plugin, overlay (like flannel), which cloudprovider?
Are the SELinux warnings normal ?
Maybe a little out of topic. Are you running K8s on GCE? Do you have the node problem detector running in your cluster?
all veths hairpin_mode = 1 my distro is coreos 1068.6.0 I don't use any network-plugins, and no cloudprovider support here, controller-manager start with kubelet setup after address allocated by controller-manager, all pods connected each other with static routes which controlled by ospfd running inside docker. I can confirm the selinux warnings not related to the bug.
not running on GCE. no detector running inside.
Alright, just want to verify whether the NPD works as expected.
here is the logs of kubelet startup.
That's the problem, the reason we ran into this to begin with is because of hairpin mode. Apparently it is still getting set, or isn't getting erased in your environment. I don't think we erase existing values.
that's wired.

I checked all my nodes, each veth hairpin_mode is 1. it seems it was set to 1 regardless of --configure-cbr0.
there is no existing veth before kubelet create bridge , docker could not start without cbr0 which created by kubelet.
hairpin_mode is always enabled if the device is a bridge interface,
Trust me it's not on my (and our other e2e test) clusters, or we'd know because tests would fail, and teh default setup is docker running on cbr0 and using veths. In 1.3 we started using kubenet network plugin by default, but hairpin flag wasn't set since 1.2. I can trace the code path down when I have some time, but I suspect it's something to do with the way coreos is setup.
I trust you, I promise. I did some tests. None of results told me the flag is set by kubelet. I could not figure out what was happend to these veths. finally, I duplicate to , the only difference is hairepinEnable default to "0", and invoke the hairpinunset.SetUpContainerPid when false. It works! But, if I create veth by hand, the default hairpin_mode is 0 , oh my god. It will not happen in my test environment, only in production env
will enable hairpin_mode That's my fault, I should not set it to false because kubernetes do not use docker's network model.


Question: proxymetrics.RegisterMetrics() } else if proxyMode == proxyModeIPVS { klog.V(0).Info("Using ipvs Proxier.") <del> if utilfeature.DefaultFeatureGate.Enabled(features.IPv6DualStack) { </del> <ins> if dualStack { </ins> klog.V(0).Info("creating dualStackProxier for ipvs.") <del> // Create iptables handlers for both families, one is already created // Always ordered as IPv4, IPv6 var ipt [2]utiliptables.Interface if iptInterface.IsIPv6() { ipt[1] = iptInterface ipt[0] = utiliptables.New(execer, utiliptables.ProtocolIPv4) } else { ipt[0] = iptInterface ipt[1] = utiliptables.New(execer, utiliptables.ProtocolIPv6) } </del> nodeIPs := nodeIPTuple(config.BindAddress) // Always ordered to match []ipt

(update: likely this is benign, but it seems really weird when your reading kube proxy logs) in healthy, IPTABLES jobs, the various ip6tables-restore commands are happy....MEANWHILE in ipvs jobs (healthy, or non healthy, we see that some of the iptables dont exist which makes sense... why is ipvs kube proxy tyrying to do iptables-restore operations on the table ? why is it trying to do iptables stuff at all ? It seems like we get error messages when running ipv6. not sure why though? in particular i found these errors when looking at the test failures for You wouldnt see error messages for ip6tables-restore when running ipvs tests. This may be my own ignorance however, i.e. maybe this is normal. I guess every job has this (wether or not the job passes or fails) ? Here is an example from a passing test.... Similar logs from a failing tests that i originally saw this message ... has this failure, it is reproduced via the corresponding ci job ... the logs in the kube-proxy for one of the nodes look like this: ill add more links as i collect more data. Kubernetes version (use ): prow (main branch as of feb 12 2021)
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig network
So, this happens in HEALTHY jobs to, so i guess its not a driver for an actual bug.... but its still weird to me :) ... updating original issue...
any idea on why those ipv6 errors are printed in ipvs code ?
The proxier is configured in dual-stack mode in the CI Since we enabled the dual-stack gate recently, this is the expected behaviour
I don't get similar printouts myself so I can't say anything for sure. By first check the old re-occuring (and very annoying) "KUBE-MARK-DROP" problem.

This chain is usually created by "kubeadm" but so far kubeadm is unaware of dual-stack and has only created this chain for ipv4 (in iptables). Then it needs to be created for IPv6 before kube-proxy is started; In my test cluster I always do this in a prep-stage before starting K8s.
The ci nodes doesn't have the kernel ip6tables nat modules loaded, hence they fail all the operation related
hmmm ok so is the issue here that we need to load that module in the CI jobs?
signing here because I want to at least find why ip6tables restore is called on kube-proxy ipvs things :D
ipvs does uses iptables too I think that kube-proxy should implement something similar to what implemented in kubelet The nat table for IPv6 was in 3.7 and it seems that it may be not present in some systems. /cc
/assign
I think it's mostly just about whether IPv6 is enabled, not about the kernel version.
IIRC this specific error in the CI is about the nat module only Anyway, you fixed it :clap: /unassign /assign


Question: $host_kubectl create secret generic ${name} --from-file="${dir}/kubeconfig" --namespace="${FEDERATION_NAMESPACE}" done <del> $template "${manifests_root}/federation-apiserver-"{deployment,secrets}".yaml" | $host_kubectl create -f - $template "${manifests_root}/federation-controller-manager-deployment.yaml" | $host_kubectl create -f - </del> <ins> for file in federation-etcd-pvc.yaml federation-apiserver-{deployment,secrets}.yaml federation-controller-manager-deployment.yaml; do $template "${manifests_root}/${file}" | $host_kubectl create -f - done </ins> # Update the users kubeconfig to include federation-apiserver credentials. CONTEXT=federation-cluster

As mentioned in , the federation etcd will lose its state any time the pod dies, unless we modify to add a PVC. We'll also need to create a PV and modify the turnup script to create the backing volume in the cloud provider.
FYI I believe this is a showstopper.
OK. implied I could use to get this provisioned on-demand. AFAICT, , and since it makes our life easier I am trying to get this to work in my test cluster, but so far no joy. Any obvious mistake?
I needed to not specify a .
whew! , Matt Liggett wrote:


Question: if utilfeature.DefaultFeatureGate.Enabled(features.ResilientWatchCacheInitialization) { // The tests assume that Get/GetList/Watch calls shouldn't fail. // However, 429 error can now be returned if watchcache is under initialization. <del> // To avoid rewriting all tests, we wait for watcache to initialize. if err := cacher.ready.wait(ctx); err != nil { </del> <ins> // To avoid rewriting all tests, we wait for watchcache to initialize. if err := cacher.Wait(ctx); err != nil { </ins> t.Fatal(err) } }

pull-kubernetes-unit TestStoreListResourceVersion in seems like July 7th 2024: ! Don't know, here is the log: No response /sig
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig api-machinery
See also
/sig etcd
/assign /cc
Bisected this to cc
I'm not sure if this is the proper way to fix it, but setting a limit in the ListOptions deflakes the test. The flakiness comes from the fact that the cache is sometimes not initialized in time which causes it to return a "too many requests". Perhaps in tests we could wait till the cache is ready in order to proceed?
Right, setting a limit when a resource version is set would delegate the list to etcd when cache is not initialized based on the KEP description. It is due to this type of request is not that costly to etcd. Looks like it is exactly how the wait cache ready is in See below So I think we should implement this same thing in when initializing the .
Agree - waiting for cacher to be initialized when creating it here is a better fix for it:
I thought about this as well, but waiting on is not possible outside of the cacher module, since it is a private field. I'm thinking to either add an option to cacherstorage.Config and make wait or add a new public method to . Do you guys have a better idea? Thanks.
Exposing Wait(context) method sounds fine
I have pushed the changes we discussed. Thank you all for your guidance.


Question: cmd.SetStdin(bytes.NewBuffer(data)) b, err := cmd.CombinedOutput() if err != nil { <del> return fmt.Errorf("%v (%s)", err, b) </del> <ins> pErr, ok := parseRestoreError(string(b)) if ok { return pErr } return fmt.Errorf("%w: %s", err, b) </ins> } return nil }

my k8s cluster version is 1.17.x this cluster has 10 master only 2 nodes get problem. it make my svc sometime can usesometime can timeout what can i do can resolve this problem ? this 2 node's svc is nodeportType
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
you can increase kube-proxy verbosity adding flag and you should get more information about the cause of the error Kubernetes 1.17 is EOL, so you should upgrade to 1.19 at least
/sig network
hello bromy cluster is use kubeadm init .so what should i do .i know this is use configmap but in this i can not found how edit this configmap is availableabout upgrade this cluster .this time is impossible
about stack. i only get this log.
To increase the verbosity on kube-proxy on a kubeadm deployment you can do and wait until pods are restarted. Despite that, if there is a bug, it will not be backported to 1.17, so you have to go to a supported version first
I swear we code to dump the ruleset when this happens, but I don't see it any more.
I'm going to re-purpose this as a feature :)
Hi, I would like to pick this up. Can someone give me some code pointers on where to start?
Let me know if I can help in anyway.
/assign
I'd start with - It's worth thinking about the cleanest solution here, but maybe something like: When you get an error from exec, parse the error for and if it matches, return a . Then let the callers check for a by interface assertion, extract the Line(), extract their input's data at -3 and +3 from that line and log it. E.g. You'll proably want a helper func to do that, which could live in that same iptables package. And a test.
/assign
hi everyone! Is anyone doing this?

I am willing to do this
at least I didn't see any PR from , in case of collision please sync between both to avoid duplicate efforts
hi did you work on it?
Hi apologies I was busy with some other work. You must please continue.
thanks
/assign
I'm working on this now, but I'm stuck in here, I got the line, and then how I extract at -3 and +3 from that line?
you may need to write a helper function to extract them from the logs. my 2 cents.. here is my version for reference. you could add some tests, to cover the boundary conditions. to play around :)


Question: }, { cfg: &kubeadmapi.MasterConfiguration{ <del> Networking:        kubeadmapi.Networking{PodSubnet: "bar"}, </del> <ins> Networking:        kubeadmapi.Networking{PodSubnet: "10.0.1.15/16"}, </ins> CertificatesDir:   testCertsDir, KubernetesVersion: "v1.7.0", },

/kind bug What happened: Currently, if the user sets the cidr with --pod-network-cidr with kubeadm, the kube-controller-manager is passed the cidr with the --cluster-cidr arg. --node-cidr-mask-size is not passed to the kube-controller-manager and the controller-manager used the default value of 24 The default value of 24 if good for IPv4, but for IPv6 the controller-manager throws an error because the node-cidr-mask-size is incorrect. What you expected to happen: The user should be able to specify the node-cidr-mask-size for IPv6 How to reproduce it (as minimally and precisely as possible): Use Kubeadm specify an IPv6 address with --pod-network-cidr. with a subnet greater than 24. For example kubeadm init --pod-network-cidr=2001:101::0/64 Anything else we need to know?: Environment: IPv6 Kubernetes version (use ): Cloud provider or hardware configuration**: OS (e.g. from /etc/os-release): Kernel (e.g. ): Install tools: Others:
There are no sig labels on this issue. Please by: a sig: e.g., to notify the contributor experience sig, OR the label manually: e.g., to apply the label Note: Method 1 will trigger an email to the group. You can find the group list and label list . The in the method 1 has to be replaced with one of these: bugs, feature-requests, pr-reviews, test-failures, proposals
/area ipv6 /area kubeadm
/assign rpothier


Question: ctx context.Context, key string, destination runtime.Object, ignoreNotFound bool, preconditions *storage.Preconditions, tryUpdate storage.UpdateFunc, dryRun bool, cachedExistingObject runtime.Object) error { if dryRun { <del> err := s.Storage.Get(ctx, key, storage.GetOptions{IgnoreNotFound: ignoreNotFound}, destination) </del> <ins> var current runtime.Object v, err := conversion.EnforcePtr(destination) if err != nil { return fmt.Errorf("unable to convert output object to pointer: %v", err) } if u, ok := v.Addr().Interface().(runtime.Unstructured); ok { current = u.NewEmptyInstance() } else { current = reflect.New(v.Type()).Interface().(runtime.Object) } err = s.Storage.Get(ctx, key, storage.GetOptions{IgnoreNotFound: ignoreNotFound}, current) </ins> if err != nil { return err } <del> err = preconditions.Check(key, destination) </del> <ins> err = preconditions.Check(key, current) </ins> if err != nil { return err } <del> rev, err := s.Versioner().ObjectResourceVersion(destination) </del> <ins> rev, err := s.Versioner().ObjectResourceVersion(current) </ins> if err != nil { return err } <del> updated, _, err := tryUpdate(destination, storage.ResponseMeta{ResourceVersion: rev}) </del> <ins> updated, _, err := tryUpdate(current, storage.ResponseMeta{ResourceVersion: rev}) </ins> if err != nil { return err }

Apparently, the bug happens in 1.26.3 but not in 1.27.1 so maybe it's been fixed somewhere. Here are the steps to reproduce: Note that the metrics list is atomic, so the list should be entirely replaced every-time, not merged/appended. /wg api-expression /triage accepted /priority important-soon
I tried to add a printf of the json'd merged object right before leaving the ssa code, and the field isn't duplicated. So something else must be trying to merge somehow!?
I also checked that it's not kubectl doing anything crazy, the response from the server includes the duplicated field.
Checked both before and after admission, object seems to be properly merged.
Ah, something not mentioned in the bug, applying without dry-run doesn't trigger the same behavior.
storage version changed to v2 in 1.27 () HPA conversion is really weird (taking v2 fields and putting them in annotations in v1, then shipping annotations back into appended items when converting back to v2)... I'd suspect something in the storage → conversion path
Right, the exact conversions that we do are different in dry-run vs non-dry-run because we have to storage version. I'm still investigating the conversion path to see if anything seems shady and I've made progress on that.
OK, this is the trace that leads to the internal HorizontalPodAutoscaler object to have the metrics field duplicated is this one: So definitely in the path to dry-run. The following test reproduces the error, as it fails with two identical metrics. Problem is probably that 1. the "out" object isn't cleared before for dry-run. 2. conversion doesn't detect the duplication/remove previous entry from out. I'll check to see how the non-dry-run version clears the out object.
Did anyone find a workaround for this bug? Currently trying to compare equality between two objects after a dry-run to populate default fields.


Question: expectedIptablesChains: netlinktest.ExpectedIptablesChain{ string(KubeNodePortChain): {{ JumpChain: string(KubeMarkMasqChain), MatchSet: kubeNodePortSetUDP, <ins> }, { JumpChain: "ACCEPT", MatchSet: kubeHealthCheckNodePortSet, </ins> }}, string(kubeServicesChain): {{ <ins> JumpChain: string(KubeMarkMasqChain), MatchSet: kubeClusterIPSet, }, { </ins> JumpChain: string(KubeNodePortChain), MatchSet: "", <ins> }, { JumpChain: "ACCEPT", MatchSet: kubeClusterIPSet, </ins> }}, }, },

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --ipvs mode didn't preserve client IP for NodePort Service with protocol SCTP and ExternalTrafficPolicy Local. iptables mode works fine. ipvs mode with TCP and UDP protocol works fine. Client IP for NodePort Service should be preseved when ExternalTrafficPolicy is Local regardless of the mode and the protocol. kube-proxy mode to ipvs a NodePort service with protocol SCTP and ExternalTrafficPolicy Local the NodePort service, the client IP was not preserved This is caused by wrong order of iptables rules that match kubeNodePortLocalSetSCTP and kubeNodePortSetSCTP: Kubernetes version (use ): v1.22.0 Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others:
/sig network /assign
/triage accepted


Question: // Map node's condition to Taints. var taints []v1.Taint for _, condition := range node.Status.Conditions { <del> if taint, found := nodeConditionToTaintKeyStatusMap[condition.Type]; found { if condition.Status == taint.noScheduleStatus { </del> <ins> if taintMap, found := nodeConditionToTaintKeyStatusMap[condition.Type]; found { if taintKey, found := taintMap[condition.Status]; found { </ins> taints = append(taints, v1.Taint{ <del> Key:    taint.taintKey, </del> <ins> Key:    taintKey, </ins> Effect: v1.TaintEffectNoSchedule, }) }

In autoscaling tests, we verify that when a node suddenly disappears from the cluster, Cluster Autoscaler will add another one to replace it. With TaintNodesByCondition (enabled by default in ), this is now failing: https://k8s- Every time a node disappears, a subset of pods become NodeLost and aren't rescheduled on a new node. /cc /sig node
I can work on this.
just got merged. Let's check result of autoscaling testings tomorrow: https://k8s-
latest autoscaling testing is green. I think we can close this issue.
Thanks for fixing this!


Question: if [[ "${NUM_NODES}" -gt "500" ]]; then typha_count=5 fi <ins> if [[ "${NETWORK_POLICY_PROVIDER:-}" != "calico" ]]; then # We're not configured to use Calico, so don't start any Typhas. typha_count=0 fi </ins> echo "${typha_count}" }

Example: https://k8s- cc - Maybe related to /sig network /area platform/gke /kind bug /priority failing-test
/assign
/assign
I guess DaemonSet's behave differently the deployments wrt to scheduling. So in the DaemonSet case we return no nodes to run on since no labels match. In the deployment case (Typha) we want to run at least one thing and cannot so it sits in pending. Maybe the default for typha should be replicas 0? And then we can set the replicas at runtime?
Actually maybe leave it as is for now but in the e2e test maybe we just set it to 0?
Discussed with I think the right fix is . We should return typhacount=0 if NETWORKPOLICY_PROVIDER is not set to .
The tests are still failing with the same error: https://k8s-
I think the problem is here: if [[ "${NETWORKPOLICYPROVIDER:-}" == "calico" ]]; then setup-addon-manifests "addons" "calico-policy-controller" ..... sed -i -e "s I think we need to set that count to CALICOTYPHAREPLICAS_ regardless of provider
/reopen
ping ...
working on the fix that suggested in PR Tests are currently failing but haven't gotten to the bottom of it yet.
This is fixed now.