Question: inkscape:groupmode="layer" id="layer1" transform="translate(-6.1051208,-305.36552)"> <del> <text xml:space="preserve" </del> <ins> <g </ins> style="font-size:40px;font-style:normal;font-weight:normal;line-height:125%;letter-spacing:0px;word-spacing:0px;fill:#326de5;fill-opacity:1;stroke:none;font-family:Sans" <del> x="6.6551208" y="420.12552" id="text4373" sodipodi:linespacing="125%" inkscape:export-filename="/home/thockin/src/go/src/k8s.io/kubernetes/logo/name_blue.png" inkscape:export-xdpi="200.38" inkscape:export-ydpi="200.38"><tspan sodipodi:role="line" id="tspan4375" x="6.6551208" y="420.12552" style="font-size:135px;font-style:normal;font-variant:normal;font-weight:bold;font-stretch:normal;fill:#326de5;fill-opacity:1;font-family:Ubuntu;-inkscape-font-specification:Ubuntu Bold">kubernetes</tspan></text> </del> <ins> id="text4373"> <path d="m 36.220121,374.76552 c 1.979968,-2.15995 4.004966,-4.40995 6.075,-6.75 2.159962,-2.33995 4.22996,-4.63494 6.21,-6.885 1.979956,-2.33994 3.824954,-4.54494 5.535,-6.615 1.799951,-2.06993 3.329949,-3.86993 4.59,-5.4 l 23.895,0 c -4.770071,5.49006 -9.450067,10.75506 -14.04,15.795 -4.500057,4.95005 -9.450053,10.08004 -14.85,15.39 2.69995,2.43004 5.489947,5.35503 8.37,8.775 2.879942,3.33003 5.669939,6.79502 8.37,10.395 2.699933,3.60002 5.174931,7.20001 7.425,10.8 2.249926,3.60001 4.139925,6.885 5.67,9.855 l -23.085,0 c -1.440052,-2.34 -3.105051,-4.905 -4.995,-7.695 -1.800047,-2.87999 -3.735045,-5.71499 -5.805,-8.505 -2.070041,-2.87998 -4.275039,-5.62498 -6.615,-8.235 -2.250034,-2.60997 -4.500032,-4.81497 -6.75,-6.615 l 0,31.05 -20.115,0 0,-101.52 20.115,-3.24 0,59.4" style="font-size:135px;font-variant:normal;font-weight:bold;font-stretch:normal;fill:#326de5;font-family:Ubuntu;-inkscape-font-specification:Ubuntu Bold" id="path2985" /> <path d="m 153.44864,417.69552 c -3.42007,0.99 -7.83007,1.89 -13.23,2.7 -5.40005,0.9 -11.07005,1.35 -17.01,1.35 -6.03004,0 -11.07003,-0.81 -15.12,-2.43 -3.96002,-1.62 -7.11002,-3.87 -9.450004,-6.75 -2.340012,-2.96999 -4.005011,-6.47999 -4.995,-10.53 -0.990009,-4.04998 -1.485008,-8.50497 -1.485,-13.365 l 0,-39.555 20.115004,0 0,37.125 c -3e-5,6.48003 0.85497,11.16002 2.565,14.04 1.70996,2.88002 4.90496,4.32001 9.585,4.32 1.43995,10e-6 2.96995,-0.045 4.59,-0.135 1.61995,-0.17999 3.05995,-0.35999 4.32,-0.54 l 0,-54.81 20.115,0 0,68.58" style="font-size:135px;font-variant:normal;font-weight:bold;font-stretch:normal;fill:#326de5;font-family:Ubuntu;-inkscape-font-specification:Ubuntu Bold" id="path2987" /> <path d="m 217.95754,384.08052 c -5e-5,-12.95995 -4.77005,-19.43995 -14.31,-19.44 -2.07004,5e-5 -4.14003,0.27005 -6.21,0.81 -1.98003,0.54005 -3.60003,1.21505 -4.86,2.025 l 0,36.585 c 0.98997,0.18001 2.24997,0.36001 3.78,0.54 1.52997,0.09 3.19497,0.13501 4.995,0.135 5.48996,10e-6 9.62995,-1.88998 12.42,-5.67 2.78995,-3.77998 4.18495,-8.77497 4.185,-14.985 m 20.52,0.54 c -7e-5,5.67003 -0.85507,10.80002 -2.565,15.39 -1.62007,4.59001 -4.00507,8.50501 -7.155,11.745 -3.15006,3.24 -7.02006,5.76 -11.61,7.56 -4.59005,1.71 -9.81004,2.565 -15.66,2.565 -2.43003,0 -4.99503,-0.135 -7.695,-0.405 -2.61003,-0.18 -5.22002,-0.45 -7.83,-0.81 -2.52002,-0.36 -4.95002,-0.765 -7.29,-1.215 -2.34001,-0.54 -4.41001,-1.08 -6.21,-1.62 l 0,-99.225 20.115,-3.24 0,35.37 c 2.24997,-0.98993 4.58997,-1.75493 7.02,-2.295 2.42996,-0.53993 5.03996,-0.80993 7.83,-0.81 5.03995,7e-5 9.49495,0.90007 13.365,2.7 3.86994,1.71007 7.10994,4.18506 9.72,7.425 2.60993,3.24006 4.58993,7.15505 5.94,11.745 1.34993,4.50004 2.02493,9.54004 2.025,15.12" style="font-size:135px;font-variant:normal;font-weight:bold;font-stretch:normal;fill:#326de5;font-family:Ubuntu;-inkscape-font-specification:Ubuntu Bold" id="path2989" /> <path d="m 250.56215,385.16052 c 0,-6.29996 0.945,-11.78995 2.835,-16.47 1.97999,-4.76994 4.54499,-8.72994 7.695,-11.88 3.14998,-3.14993 6.74998,-5.53493 10.8,-7.155 4.13997,-1.61993 8.36997,-2.42993 12.69,-2.43 10.07995,7e-5 18.04494,3.10507 23.895,9.315 5.84993,6.12006 8.77493,15.16505 8.775,27.135 -7e-5,1.17003 -0.0451,2.47503 -0.135,3.915 -0.0901,1.35003 -0.18007,2.56503 -0.27,3.645 l -45.63,0 c 0.44997,4.14002 2.38497,7.42502 5.805,9.855 3.41997,2.43002 8.00996,3.64501 13.77,3.645 3.68995,10e-6 7.28995,-0.31499 10.8,-0.945 3.59994,-0.71998 6.52494,-1.57498 8.775,-2.565 l 2.7,16.335 c -1.08007,0.54 -2.52006,1.08 -4.32,1.62 -1.80006,0.54 -3.82506,0.99 -6.075,1.35 -2.16005,0.45 -4.50005,0.81 -7.02,1.08 -2.52005,0.27 -5.04004,0.405 -7.56,0.405 -6.39004,0 -11.97003,-0.945 -16.74,-2.835 -4.68002,-1.89 -8.59502,-4.455 -11.745,-7.695 -3.06001,-3.32999 -5.35501,-7.24499 -6.885,-11.745 -1.44,-4.49998 -2.16,-9.35997 -2.16,-14.58 m 47.25,-7.695 c -0.09,-1.70996 -0.40505,-3.37496 -0.945,-4.995 -0.45005,-1.61995 -1.21505,-3.05995 -2.295,-4.32 -0.99005,-1.25995 -2.29505,-2.29495 -3.915,-3.105 -1.53004,-0.80995 -3.46504,-1.21494 -5.805,-1.215 -2.25004,6e-5 -4.18503,0.40505 -5.805,1.215 -1.62003,0.72005 -2.97003,1.71005 -4.05,2.97 -1.08003,1.26005 -1.93503,2.74505 -2.565,4.455 -0.54003,1.62004 -0.94502,3.28504 -1.215,4.995 l 26.595,0" style="font-size:135px;font-variant:normal;font-weight:bold;font-stretch:normal;fill:#326de5;font-family:Ubuntu;-inkscape-font-specification:Ubuntu Bold" id="path2991" /> <path d="m 375.57004,366.93552 c -1.80005,-0.44995 -3.91505,-0.89995 -6.345,-1.35 -2.43004,-0.53995 -5.04004,-0.80995 -7.83,-0.81 -1.26003,5e-5 -2.79003,0.13505 -4.59,0.405 -1.71003,0.18005 -3.01503,0.40505 -3.915,0.675 l 0,54.27 -20.115,0 0,-67.23 c 3.59999,-1.25993 7.82999,-2.42993 12.69,-3.51 4.94998,-1.16993 10.43997,-1.75493 16.47,-1.755 1.07996,7e-5 2.38496,0.0901 3.915,0.27 1.52996,0.0901 3.05996,0.27007 4.59,0.54 1.52995,0.18007 3.05995,0.45007 4.59,0.81 1.52995,0.27007 2.83495,0.63007 3.915,1.08 l -3.375,16.605" style="font-size:135px;font-variant:normal;font-weight:bold;font-stretch:normal;fill:#326de5;font-family:Ubuntu;-inkscape-font-specification:Ubuntu Bold" id="path2993" /> <path d="m 389.72817,351.54552 c 3.41998,-0.98993 7.82998,-1.88993 13.23,-2.7 5.39997,-0.89993 11.06996,-1.34993 17.01,-1.35 6.02995,7e-5 11.02495,0.81007 14.985,2.43 4.04994,1.53007 7.24494,3.73507 9.585,6.615 2.33993,2.88006 4.00493,6.30006 4.995,10.26 0.98993,3.96005 1.48493,8.37004 1.485,13.23 l 0,40.095 -20.115,0 0,-37.665 c -5e-5,-6.47996 -0.85505,-11.06995 -2.565,-13.77 -1.71005,-2.69995 -4.90505,-4.04995 -9.585,-4.05 -1.44004,5e-5 -2.97004,0.0901 -4.59,0.27 -1.62003,0.09 -3.06003,0.22505 -4.32,0.405 l 0,54.81 -20.115,0 0,-68.58" style="font-size:135px;font-variant:normal;font-weight:bold;font-stretch:normal;fill:#326de5;font-family:Ubuntu;-inkscape-font-specification:Ubuntu Bold" id="path2995" /> <path d="m 465.98207,385.16052 c 0,-6.29996 0.945,-11.78995 2.835,-16.47 1.97999,-4.76994 4.54499,-8.72994 7.695,-11.88 3.14998,-3.14993 6.74998,-5.53493 10.8,-7.155 4.13997,-1.61993 8.36997,-2.42993 12.69,-2.43 10.07995,7e-5 18.04495,3.10507 23.895,9.315 5.84993,6.12006 8.77493,15.16505 8.775,27.135 -7e-5,1.17003 -0.0451,2.47503 -0.135,3.915 -0.0901,1.35003 -0.18007,2.56503 -0.27,3.645 l -45.63,0 c 0.44998,4.14002 2.38497,7.42502 5.805,9.855 3.41997,2.43002 8.00996,3.64501 13.77,3.645 3.68995,10e-6 7.28995,-0.31499 10.8,-0.945 3.59994,-0.71998 6.52494,-1.57498 8.775,-2.565 l 2.7,16.335 c -1.08006,0.54 -2.52006,1.08 -4.32,1.62 -1.80006,0.54 -3.82506,0.99 -6.075,1.35 -2.16005,0.45 -4.50005,0.81 -7.02,1.08 -2.52004,0.27 -5.04004,0.405 -7.56,0.405 -6.39003,0 -11.97003,-0.945 -16.74,-2.835 -4.68002,-1.89 -8.59501,-4.455 -11.745,-7.695 -3.06001,-3.32999 -5.35501,-7.24499 -6.885,-11.745 -1.44,-4.49998 -2.16,-9.35997 -2.16,-14.58 m 47.25,-7.695 c -0.0901,-1.70996 -0.40505,-3.37496 -0.945,-4.995 -0.45005,-1.61995 -1.21505,-3.05995 -2.295,-4.32 -0.99005,-1.25995 -2.29504,-2.29495 -3.915,-3.105 -1.53004,-0.80995 -3.46504,-1.21494 -5.805,-1.215 -2.25003,6e-5 -4.18503,0.40505 -5.805,1.215 -1.62003,0.72005 -2.97003,1.71005 -4.05,2.97 -1.08003,1.26005 -1.93502,2.74505 -2.565,4.455 -0.54002,1.62004 -0.94502,3.28504 -1.215,4.995 l 26.595,0" style="font-size:135px;font-variant:normal;font-weight:bold;font-stretch:normal;fill:#326de5;font-family:Ubuntu;-inkscape-font-specification:Ubuntu Bold" id="path2997" /> <path d="m 547.51996,331.43052 20.115,-3.24 0,20.925 24.165,0 0,16.74 -24.165,0 0,24.975 c -2e-5,4.23002 0.71997,7.60502 2.16,10.125 1.52997,2.52002 4.54497,3.78001 9.045,3.78 2.15996,10e-6 4.36496,-0.17999 6.615,-0.54 2.33996,-0.44998 4.45495,-1.03498 6.345,-1.755 l 2.835,15.66 c -2.43005,0.99 -5.13005,1.845 -8.1,2.565 -2.97004,0.72 -6.61504,1.08 -10.935,1.08 -5.49003,0 -10.03502,-0.72 -13.635,-2.16 -3.60002,-1.53 -6.48001,-3.6 -8.64,-6.21 -2.16001,-2.69999 -3.69001,-5.93999 -4.59,-9.72 -0.81,-3.77998 -1.215,-7.96498 -1.215,-12.555 l 0,-59.67" style="font-size:135px;font-variant:normal;font-weight:bold;font-stretch:normal;fill:#326de5;font-family:Ubuntu;-inkscape-font-specification:Ubuntu Bold" id="path2999" /> <path d="m 602.03676,385.16052 c 0,-6.29996 0.94499,-11.78995 2.835,-16.47 1.97999,-4.76994 4.54499,-8.72994 7.695,-11.88 3.14998,-3.14993 6.74998,-5.53493 10.8,-7.155 4.13997,-1.61993 8.36997,-2.42993 12.69,-2.43 10.07995,7e-5 18.04494,3.10507 23.895,9.315 5.84993,6.12006 8.77493,15.16505 8.775,27.135 -7e-5,1.17003 -0.0451,2.47503 -0.135,3.915 -0.0901,1.35003 -0.18007,2.56503 -0.27,3.645 l -45.63,0 c 0.44997,4.14002 2.38497,7.42502 5.805,9.855 3.41997,2.43002 8.00996,3.64501 13.77,3.645 3.68995,10e-6 7.28995,-0.31499 10.8,-0.945 3.59994,-0.71998 6.52494,-1.57498 8.775,-2.565 l 2.7,16.335 c -1.08007,0.54 -2.52006,1.08 -4.32,1.62 -1.80006,0.54 -3.82506,0.99 -6.075,1.35 -2.16005,0.45 -4.50005,0.81 -7.02,1.08 -2.52005,0.27 -5.04004,0.405 -7.56,0.405 -6.39004,0 -11.97003,-0.945 -16.74,-2.835 -4.68002,-1.89 -8.59502,-4.455 -11.745,-7.695 -3.06001,-3.32999 -5.35501,-7.24499 -6.885,-11.745 -1.44001,-4.49998 -2.16,-9.35997 -2.16,-14.58 m 47.25,-7.695 c -0.09,-1.70996 -0.40505,-3.37496 -0.945,-4.995 -0.45005,-1.61995 -1.21505,-3.05995 -2.295,-4.32 -0.99005,-1.25995 -2.29505,-2.29495 -3.915,-3.105 -1.53004,-0.80995 -3.46504,-1.21494 -5.805,-1.215 -2.25004,6e-5 -4.18503,0.40505 -5.805,1.215 -1.62003,0.72005 -2.97003,1.71005 -4.05,2.97 -1.08003,1.26005 -1.93503,2.74505 -2.565,4.455 -0.54003,1.62004 -0.94503,3.28504 -1.215,4.995 l 26.595,0" style="font-size:135px;font-variant:normal;font-weight:bold;font-stretch:normal;fill:#326de5;font-family:Ubuntu;-inkscape-font-specification:Ubuntu Bold" id="path3001" /> <path d="m 704.63465,405.68052 c 3.68997,10e-6 6.29997,-0.35999 7.83,-1.08 1.52996,-0.71998 2.29496,-2.11498 2.295,-4.185 -4e-5,-1.61998 -0.99004,-3.01498 -2.97,-4.185 -1.98003,-1.25998 -4.99503,-2.65497 -9.045,-4.185 -3.15002,-1.16997 -6.03002,-2.38497 -8.64,-3.645 -2.52001,-1.25997 -4.68001,-2.74497 -6.48,-4.455 -1.80001,-1.79996 -3.19501,-3.91496 -4.185,-6.345 -0.99001,-2.42996 -1.48501,-5.35495 -1.485,-8.775 -10e-6,-6.65994 2.47499,-11.92494 7.425,-15.795 4.94998,-3.86993 11.74498,-5.80493 20.385,-5.805 4.31996,7e-5 8.45996,0.40507 12.42,1.215 3.95995,0.72007 7.10995,1.53007 9.45,2.43 l -3.51,15.66 c -2.34005,-0.80995 -4.90505,-1.52995 -7.695,-2.16 -2.70004,-0.62994 -5.76004,-0.94494 -9.18,-0.945 -6.30003,6e-5 -9.45002,1.75505 -9.45,5.265 -2e-5,0.81005 0.13497,1.53005 0.405,2.16 0.26997,0.63005 0.80997,1.26005 1.62,1.89 0.80997,0.54005 1.88997,1.17005 3.24,1.89 1.43997,0.63004 3.23997,1.35004 5.4,2.16 4.40996,1.62004 8.05496,3.24004 10.935,4.86 2.87995,1.53004 5.12995,3.24003 6.75,5.13 1.70995,1.80003 2.87994,3.82503 3.51,6.075 0.71994,2.25002 1.07994,4.86002 1.08,7.83 -6e-5,7.02001 -2.65506,12.33001 -7.965,15.93 -5.22004,3.6 -12.64504,5.4 -22.275,5.4 -6.30002,0 -11.56502,-0.54 -15.795,-1.62 -4.14001,-1.08 -7.02,-1.98 -8.64,-2.7 l 3.375,-16.335 c 3.41999,1.35002 6.92999,2.43002 10.53,3.24 3.59998,0.72001 7.15498,1.08001 10.665,1.08" style="font-size:135px;font-variant:normal;font-weight:bold;font-stretch:normal;fill:#326de5;font-family:Ubuntu;-inkscape-font-specification:Ubuntu Bold" id="path3003" /> </g> </ins> </g> </svg>

The logo (, ) font does not match the others. I believe the png/pdf versions are correct.
What other ones? It should be the same font - they look the same to me...


Question: ConfigSyncPeriod metav1.Duration } <del> // Currently two modes of proxying are available: 'userspace' (older, stable) or 'iptables' // (newer, faster). If blank, use the best-available proxy (currently iptables, but may // change in future versions).  If the iptables proxy is selected, regardless of how, but // the system's kernel or iptables versions are insufficient, this always falls back to the // userspace proxy. </del> <ins> // Currently, four modes of proxying are available total: 'userspace' (older, stable), 'iptables' // (newer, faster), 'ipvs', and 'kernelspace' (Windows only, newer). // // If blank, use the best-available proxy (currently iptables, but may change in // future versions). If the iptables proxy is selected, regardless of how, but // the system's kernel or iptables versions are insufficient, this always falls // back to the userspace proxy. </ins> type ProxyMode string const ( <del> ProxyModeUserspace ProxyMode = "userspace" ProxyModeIPTables  ProxyMode = "iptables" ProxyModeIPVS      ProxyMode = "ipvs" </del> <ins> ProxyModeUserspace   ProxyMode = "userspace" ProxyModeIPTables    ProxyMode = "iptables" ProxyModeIPVS        ProxyMode = "ipvs" ProxyModeKernelspace ProxyMode = "kernelspace" </ins> ) // IPVSSchedulerMethod is the algorithm for allocating TCP connections and

Is this a BUG REPORT or FEATURE REQUEST?: /kind bug What happened: When running , and passing the flag which was introduced in , the following error occurs: What you expected to happen: The proxy should work as it used to, supporting this mode. How to reproduce it (as minimally and precisely as possible): ./kube- --proxy-mode=kernelspace Anything else we need to know?: This was introduced after the validation was refactored in which regressed validation for . IPVS validation (and thus support) was later , but we need another mode for Windows. Environment: Kubernetes version (use ): v1.10.0-alpha.0.684+ Cloud provider or hardware configuration: n/a OS (e.g. from ): Windows Server Core 1709 Kernel (e.g. ): Install tools: Others:
/sig windows /sig network
Tentatively adding milestone so we can track getting consensus on release-blocking status.
/priority critical-urgent
[MILESTONENOTIFIER] Milestone Issue Needs Approval Action required: This issue must have the label applied by a SIG maintainer. <details<summaryIssue Labels</summary- : Issue will be escalated to these SIGs if needed. : Never automatically move issue out of a release milestone; continually escalate to contributor and SIG through all available channels. : Fixes a bug discovered during the current release. </details<details<summaryHelp</summary<ul<li<a href=""Additional instructions</a</li<li<a href=""Commands for setting labels</a</li</ul</details


Question: # Get default service account credentials of the VM. GCE_METADATA_INTERNAL="http://metadata.google.internal/computeMetadata/v1/instance" function get-credentials { <del> curl --fail --retry 5 --retry-delay 3 ${CURL_RETRY_CONNREFUSED} --silent --show-error "${GCE_METADATA_INTERNAL}/service-accounts/default/token" -H "Metadata-Flavor: Google" -s | python -c  </del> <ins> curl --fail --retry 5 --retry-delay 3 ${CURL_RETRY_CONNREFUSED} --silent --show-error "${GCE_METADATA_INTERNAL}/service-accounts/default/token" -H "Metadata-Flavor: Google" -s | ${PYTHON} -c  </ins> 'import sys; import json; print(json.loads(sys.stdin.read())["access_token"])' }

What happened: Pick an OS image where python is python3 and the cluster will not come up. What you expected to happen: Cluster should come up. How to reproduce it (as minimally and precisely as possible): Switch the default OS image to use an OS which has python3 as the default python. Run cluster/kube- for GCE. Anything else we need to know?: Environment: Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others:
/sig cloud-provider
/assign


Question: var listener net.Listener addresses := hcs.nodeAddresses.List() <del> hcI.listeners = make([]net.Listener, 0, len(addresses)) </del> hcI.httpServers = make([]httpServer, 0, len(addresses)) // for each of the node addresses start listening and serving

When a LoadBalancer Service with Local ExternalTrafficPolicy was deleted normally, an error log "use of closed network connection" was always generated in kube-proxy log, which confused people that there was something wrong: Do not generate error log when it's expected. a LoadBalancer Service with Local ExternalTrafficPolicy it No response <detailsv1.26.0 </details<details</details<details</details<details</details<details</details<details</details
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig network


Question: } // Stop is the function that can stop the gRPC server. <ins> // Can be called concurrently, more than once, and is safe to call // without a prior Start. </ins> func (m *ManagerImpl) Stop() error { m.mutex.Lock() defer m.mutex.Unlock()

Is this a BUG REPORT or FEATURE REQUEST?: /kind bug What happened: I'm running into a test failure during "make test". To get more output and to avoid issue , I am using: What you expected to happen: The test should pass. How to reproduce it (as minimally and precisely as possible): It happens for me only when running all tests, but then it happens reliably. When limiting the test run with , it works. The failing calls are not surprising, because got unlinked after binding to it. Those might come from 's : it removes everything in the socket dir, which happens to be . For comparison, here's a successful run: Anything else we need to know?: I'm wondering whether it is guaranteed that different tests in the same .go file run serialized. Does anyone know for sure? I'm asking because all tests in use the same socket directory and thus would conflict with each other if they were to run in parallel. Note that the and calls in the failed test are for different pids. I've not checked whether those are truly different processes (as in fork+exec) or worker threads (clone). Environment: Kubernetes version (use ): master branch, commit OS (e.g. from /etc/os-release): Debian Stretch (9.0) Kernel (e.g. ): 4.9.65-3+deb9u2
After fixing the /tmp dir handling with PR I can no longer reproduce this issue. It might be worth keeping this issue open nevertheless, because the issue has not been explained completely and there might be some remaining problem lurking somewhere.
I think you originally wrote this test. Perhaps you can comment here and comment on PR ?
/cc This test has been re-written multiple times, I'll take a look ASAP :)
/sig node
Will take a look ASAP.
I experienced the same. What's interesting is I can get it to pass when running the devicemanager suite, but only about half the time.
/area hw-accelerators
Thanks a lot for reporting the issue! I think your analysis is right. Looks like the problem is because the devicemanager unit tests are interfering with each other. I will make a change to have each unit test in the file to use a separate test tmp dir.
I'm not exactly sure why, though.

Do they really run in parallel? According to my understanding of testing.Parallel, one has to explicitly request parallel execution, and the tests don't do that. When they run sequentially, then it isn't obvious why the tests interfere and thus using per-test socket dirs would be just a band-aid. Please have a look at PR , it has such a change.
Ah sorry I didn't realize you were already working on a fix. I think go test by default run tests in parallel, so we need to make sure each unit test is self contained. That also explains why you hit this problem more easily when running more tests. I think we perhaps need to modify setup() to make sure each newly created manager uses a separate test dir. Maybe you can take a look at my WIP change and see whether you can merge some of its changes into PR ?
I suggest you take a look at the documentation: On Fri, 9 Feb 2018, 22:34 Jiaying Zhang, <:
On Sat, 2018-02-10 at 05:31 +0000, Vikas Choudhary (vikasc) wrote: True, but which unit tests run concurrently? To answer that, I re-read the "testing" documentation and ran a quick experiment. That confirmed that individual tests inside the same test binary run sequentially. A test function has to explicitly call t.Parallel() before the next test starts, and that test blocks unless it also calls t.Parallel(). In other words, using the same /tmp/deviceplugin between all test functions should have been okay. I suspect it goes wrong sometimes because one test function starts a goroutine, does not wait for its completion, and that goroutine then interferes with the next test function when using the same dir, depending on timing. Running other test binaries in parallel and thus increasing system load probably also matters, together with overall number of CPUs. I think merging PR makes sense because a dynamic test directory is more flexible and enables running the tests in parallel (which could be a separate PR or be done as part of PR ), but I also think that it would be good if someone with a better understanding of the code that gets tested here could look into this race condition. [(quite) a bit later...] I chased down the spurious unlinkat() to UnixStream.Close for the socket, which unlinks the socket path [1].

By wrapping the listen socket such that a custom Close logs the call and a stack trace, I now have the smoking gun. [1] With some extra logging one can see that the that was created for TestNewManagerImplStart gets closed and unlinked while TestDevicePluginReRegistration already runs. The goroutine which runs too long is the from was called, but didn't wait for the goroutine to actually stop: That now begs the question: what is the right way to stop a grpc server? Below is the change I made to to track when it closes the socket:
On Tue, 2018-02-13 at 15:57 +0100, Patrick Ohly wrote: Server.Stop is the right way, it just doesn't fully stop the server. I consider this an issue in gRPC, therefore I have created https://github .com/grpc/grpc-go/issues/1861
Is it possible that defer statements are executed at the same time another test might start? EDIT: NVM looks like that's not the case
On Tue, 2018-02-13 at 07:39 -0800, Renaud Gaubert wrote: No, defer statements are guaranteed to run before a function returns. I think they can even change the return code and/or catch a panic.
thanks so much for your detailed debugging analysis and details about the tools you used (e.g. using filtered strace). I'm fairly new to golang and I've been messing around with the k8s source locally trying to get a feel for the project and work towards contributing, and I was debugging this test on my own but flailing against how to instrument the running tests. very much appreciated and nice work!


Question: defer stopFn() // Run scheduler. <del> informerFactory.Start(ctx.Done()) informerFactory.WaitForCacheSync(ctx.Done()) go scheduler.Run(ctx) </del> <ins> informerFactory.Start(tCtx.Done()) informerFactory.WaitForCacheSync(tCtx.Done()) go scheduler.Run(tCtx) </ins> // Send pods to be scheduled. for _, p := range tc.waitSchedulingPods { <del> _, err = fakeClient.CoreV1().Pods("").Create(ctx, p, metav1.CreateOptions{}) </del> <ins> _, err = fakeClient.CoreV1().Pods("").Create(tCtx, p, metav1.CreateOptions{}) </ins> if err != nil { t.Fatal(err) }

/kind failing-test /kind flake First seen in an unrelated PR, but also occurs in periodic jobs. /sig scheduling
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/assign This feature is reverted once, and I fixed the flake, I'm cherry-pick that back again.


Question: } func (cfgCtlr *configController) updateObservations() { <del> cfgCtlr.lock.Lock() defer cfgCtlr.lock.Unlock() </del> <ins> cfgCtlr.lock.RLock() defer cfgCtlr.lock.RUnlock() </ins> for _, plc := range cfgCtlr.priorityLevelStates { if plc.queues != nil { plc.queues.UpdateObservations()

Two flakes recently: They fail with error like: or /assign
/sig scalability
In logs I have found the following slow calls: It's a bug that we use exempt priority level for clusterloader: I created It's quite worrisome that APF handler takes 1s for exempt request.
I checked a APF code and I think the flow for exempt requests is: genericfilters.WithPriorityAndFairness() workEstimate := workEstimator.EstimateWork(r) fcIfc.Handle() cfgCtlr.startRequest() () looks for matching flowschemas in cfgCtrl.flowSchemas returns for exempt for other requests it calculates sha256 for flowDistinguisher and queues request req.Finish updates metrics calls execFn updates headers handler.ServeHTTP which stops the timer for priorityandfairness For me, the most likely reason for slowness is lock contention on We know that the test is super cpu-intensive in that part and other requests are e.g. calculating sha256 keeping that lock. A potential improvement would be to change this lock to RWLock or return the lock before starting calculating sha256 (if we believe that this is the most expensive operation there). /cc
/assign
I think the lock-contention is even worse than what you wrote, as e.g.: is happening under this lock (which is also dispatching requests). I will take a deeper look later this week, but I think that switching to RWLock might do the job without any invasive changes.
may trigger the execution of some requests that were waiting, but they are not executed in the same goroutine. Yes, I think changing that Mutex to a RWMutex would be an easy way to greatly reduce contention.
I agree switching to RWMutex is likely to help a lot, if that's not enough I think with a little work we could e.g. switch to s.
(perhaps because that's what my code originally did? Reviewers thought the retirement dance was too tricky.)
/triage accepted /priority important-soon I will work on the fix.
/sig api-machinery
Wow! Thanks a lot for rapid help here! Let's wait for the next test run to see if it helped. /reopen
Reopened this issue. <detailsIn response to : Wow! Thanks a lot for rapid help here! Let's wait for the next test run to see if it helped.

/reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Yes, can you please report the filter latency regardless of whether it is bad or good? I would like to know how much of a difference that PR made.
Sure, will report the latency change in both cases.
I compared (the one where we saw issue) (first with the lock improvements) I tried to find requests that: use exempt PL (so shouldn't be throttled) has fl_priorityandfairness, key which means that priority and fairness filter took more than 100ms Sample matching request: In the old run, I see ~6k such requests, with the locking improvements there are no such requests. The latency of PATCH jobs seems to improve significantly: ! So overall it looks great. The only worrisome metric I see is that actual P&F throttling for other requests has grown: With locking improvements: ! Old: ! E.g. 'node-high' has now throttling latency of ~1.5s in spikes where in the previous test run it was more like ~800ms. I don't think the lock improvements touched this directly. I expect that this is either: per-test variance (if we rerun the old run again we would see different throttling latencies as well) we throttle exempt requests less (as expected) giving less resources to other requests. Anyway, I don't think this change is meaningful as per-test latency aggregates for PATCH nodes/status and PUT leases (the most common requests in node-high) look ~same as previously: ! ! I think we can close this for now as the issue seems to be resolved. Thanks a lot for your help here! /close
Closing this issue. <detailsIn response to : I compared In the old run, I see ~6k such requests, with the locking improvements there are no such requests. The latency of PATCH jobs seems to improve significantly: ! ! ! I expect that this is either: ! ! /close </details
: thanks for reporting!


Question: func (o *createContextOptions) complete(cmd *cobra.Command) error { args := cmd.Flags().Args() if len(args) != 1 { <del> cmd.Help() return fmt.Errorf("Unexpected args: %v", args) </del> <ins> return helpErrorf(cmd, "Unexpected args: %v", args) </ins> } o.name = args[0]

The '-context' does not seem to add any value
add priority and team please.
Blocked on
/sig cli


Question: } if _, exists := configMap.Data[keyName]; exists { <del> return fmt.Errorf("cannot add key %s, another key by that name already exists in data: %v", keyName, configMap.Data) </del> <ins> return fmt.Errorf("cannot add key %q, another key by that name already exists in Data for ConfigMap %q", keyName, configMap.Name) </ins> } if _, exists := configMap.BinaryData[keyName]; exists { <del> return fmt.Errorf("cannot add key %s, another key by that name already exists in binaryData: %v", keyName, configMap.BinaryData) </del> <ins> return fmt.Errorf("cannot add key %q, another key by that name already exists in BinaryData for ConfigMap %q", keyName, configMap.Name) </ins> } return nil }

What happened: What you expected to happen: Human readable error output. Anything else we need to know?: The same is happening when in duplicate key while creating a with binaries. !!! The error output will be all the of the binary itself !!! Environment: Kubernetes version (use ):
/sig cli
/cc


Question: // (i.e. with SELinuxMountReadWriteOncePod disabled) defer featuregatetesting.SetFeatureGateDuringTest(t, utilfeature.DefaultFeatureGate, features.SELinuxMountReadWriteOncePod, false)() // create dswp <del> dswp, fakePodManager := prepareDswpWithVolume(t) </del> <ins> dswp, fakePodManager, _ := prepareDswpWithVolume(t) </ins> // create pod fakeOuterVolumeName := "dswp-test-volume-name"

We observed that while a container is in the process of being torn down, but not completed, the volume gets unmounted and then subsequently detached, causing the fs to enter read-only mode. This happened after kubelet restarted. We observed that on the restart, the volume was not to DSW because the Pod was marked for deletion. In the normal case, we only remove the pod from DSW when the containers have finished teardown: However, in the reconstruction case, it looks like if a pod has started termination, but not necessarily completed, then we don't add the pod to DSW: We probably need to be consistent and change the add pod criteria to also add even if it's started terminating. The side effect is that in the normal case we may do unnecessary mounts if a pod is quickly created and deleted. We can also consider a flag to only use the stricter check during reconstruction. What happened in our env was that kubelet was crash looping, and then all the pods on the node were deleted. It looks like there was a good 30 seconds between volume unmount and container teardown. I think this may be easily reproducible if the pod has a long graceful termination period and the container ignores the stop signal. No response <details22 </details<details</details<details</details<details</details<details</details<details</details
/triage accepted
Yeah this looks problematic. What was the exact kube version btw? So it was possible that container is still writing to the volume? Do we know why fs enters read-only mode when this happens? I know the behaviour of kubelet is problematic regardless, but still. I think adding volumes belonging to terminating pod doesn't sound so bad. I wonder if we could change this to .
Does this mean instead of checking ShouldPodContainersBeTerminating to decide if we add the pod or not, we should check IsPodTerminationRequested or not to decide? Or we should use the same check ShouldPodRuntimeBeRemoved with reconstruction(this one is more consistent)? pointed out that there were already scaling challenges with volume manager processing pods it doesn't need to.
For reconstruction, we need to add the volume in DSW if there is a chance that containers are running. So is the best call to determine that.
We observed a similar issue but could only reproduce it with emptydir volumes. Do you think is the same issue?

/assign sunnylovestiramisu
thanks for the report. I think this issue could impact emptydir volumes too.
Is there any recommended way to make a pod having a long graceful termination period? Tried with some simple pods but it just terminated right away.
You can use the sample pod and instructions from here
cc
Here is my repro pod and steps: the pod shell, another window (window 2), ssh to the node another window (window 3), window 2, in window 1, exec back into the pod . The file will be gone. try . It will return When looking at kubelet logs, you'll see the log messages that indicate the volume was reconstructed without being to dsw. and UnmountVolume happen soon after: Pod lifecycle logs:
Tested locally via kind cluster. /k master is showing error of touch: cannot touch '/scratch/test': No such file or directory. cluster with the fix in this pr shows no error. And running shows no result in the log.


Question: } csiAttacher := attacher.(*csiAttacher) <ins> // FIXME: We need to ensure this goroutine exits in the test. </ins> go func(spec *volume.Spec, nodename string, fail bool) { attachID, err := csiAttacher.Attach(spec, types.NodeName(nodename)) if fail != (err != nil) {

Which test(s) are flaking: TestAttacherWithCSIDriver Reason for failure: This is flaking rarely enough that it is not caught by our CI jobs, which currently tolerate up to 2 unit test failures per run (!). With that toleration removed in , this flake has been seen (). Reproducible with the following steps: a test binary for the affected package: the affected test using the tool (setting the timeout to a generous timeout relative to the normal test runtime of .02 seconds): failures are seen, look at the logs to see the details about the error: /sig storage cc
/assign can you help investigate this flake?
I have found the root cause. /assign
filed to address this issue.


Question: <del> {% set cluster_state = 'new' -%} {% set host_name = pillar.get('master_name', '') -%} {% set etcd_cluster_array = pillar.get('initial_etcd_cluster', '').split(',') -%} {% set etcd_cluster = '' -%} {# We use vars dictionary to pass variables set inside the for loop, because jinja defines new variables inside the for loop that hide variables from the outside. #} {% set vars = {'etcd_cluster': '', 'cluster_state': cluster_state} -%} {% for host in etcd_cluster_array -%} {% if etcd_cluster != '' -%} {% set cluster_state = 'existing' -%} {% set etcd_cluster = etcd_cluster ~ ',' -%} {% endif -%} {% set etcd_cluster = etcd_cluster ~ 'etcd-' ~ host ~ '=http://' ~ host ~ ':' ~ server_port -%} {% do vars.update({'etcd_cluster': etcd_cluster, 'cluster_state': cluster_state}) -%} {% endfor -%} {% set etcd_cluster = vars.etcd_cluster -%} {% set cluster_state = vars.cluster_state -%} </del> { "apiVersion": "v1", "kind": "Pod",

test grid: Data as of June 3rd, 9:00 PM Eastern. Latest data can be found here: Additional Context:
/sig release
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
cc Do we want to keep this job and fix it, or do something different considering the debs/rpms efforts?
I think we should keep this job as long as we build packages for Google infra (those are using different specs then packages for OBS).
fixed it for debs: The corresponding RPM job is also faster now:


Question: ) func TestTimeBudget(t *testing.T) { <del> budget := &timeBudget{ </del> <ins> budget := &timeBudgetImpl{ </ins> budget:    time.Duration(0), maxBudget: time.Duration(200), }

<!-- Please only use this template for submitting reports about flaky tests or jobs (pass or fail with no underlying change in code) in Kubernetes CI --Which jobs are flaking: bazel-test-master (periodic-kubernetes-bazel-test-master ) Which test(s) are flaking: gotest: Testgrid link: Reason for failure: Anything else we need to know: links to appreciated links to specific failures in spyglass appreciated <!-- Please see the deflaking doc () for more guidance! --cc
/sig api-machinery
This is continuing to be flaky with more recent failure:
/assign Would you mind taking a look here? thanks! /triage accepted
- can we somehow uncover when this started happening? This test is very old and it seems it started happening very recently, so it would be helpful to see when this started happening...
Triage shows flakes starting ~Sept 30: Before Sept 30 it seems the job was flaking much less frequently.
Might be related to I will take a look.
It seems it's not related to that PR (neither to the previous: ) As there were already failures like that on Sep 24th (before this one was merged). I will take a deeper look.
OK - it's not related to any changes mentioned before. I can reproduce this particular failure by simply adding somewhere in this block: So it's probably connected to cpu-starvation at the test level and the solution is to switch to less aggressive timeouts. Will send out a PR.
Actually - it appeared to be slightly more subtle then that. But I already fully understand the problem. It's purely test issue, but require a bit more changes (and I won't get to it today/tomorrow). I will send a fix for that early next week.
- if that changes anything from your tracking perspective ^^
No board changes yet - the next stage is "Observing" once this issue gets closed :) If you have time - I'd love to pick your brain/understand the error more thoroughly though! Hoping you drop a good explanation in your PR next week
Thanks for sharing those insights, on your discovery of the solution to this bug.
It seems the PR didn't address that fully. Looking.
out for review


Question: } } <del> // chainExists is used internally by Monitor; none of the public Interface methods can be // used to distinguish "chain exists" from "chain does not exist" with no side effects func (runner *runner) chainExists(table Table, chain Chain) (bool, error) { </del> <ins> // ChainExists is part of Interface func (runner *runner) ChainExists(table Table, chain Chain) (bool, error) { </ins> fullArgs := makeFullArgs(table, chain) runner.mu.Lock() defer runner.mu.Unlock() <del> trace := utiltrace.New("iptables Monitor CANARY check") </del> <ins> trace := utiltrace.New("iptables ChainExists") </ins> defer trace.LogIfLong(2 * time.Second) _, err := runner.run(opListChain, fullArgs)

(update: likely this is benign, but it seems really weird when your reading kube proxy logs) in healthy, IPTABLES jobs, the various ip6tables-restore commands are happy....MEANWHILE in ipvs jobs (healthy, or non healthy, we see that some of the iptables dont exist which makes sense... why is ipvs kube proxy tyrying to do iptables-restore operations on the table ? why is it trying to do iptables stuff at all ? It seems like we get error messages when running ipv6. not sure why though? in particular i found these errors when looking at the test failures for You wouldnt see error messages for ip6tables-restore when running ipvs tests. This may be my own ignorance however, i.e. maybe this is normal. I guess every job has this (wether or not the job passes or fails) ? Here is an example from a passing test.... Similar logs from a failing tests that i originally saw this message ... has this failure, it is reproduced via the corresponding ci job ... the logs in the kube-proxy for one of the nodes look like this: ill add more links as i collect more data. Kubernetes version (use ): prow (main branch as of feb 12 2021)
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig network
So, this happens in HEALTHY jobs to, so i guess its not a driver for an actual bug.... but its still weird to me :) ... updating original issue...
any idea on why those ipv6 errors are printed in ipvs code ?
The proxier is configured in dual-stack mode in the CI Since we enabled the dual-stack gate recently, this is the expected behaviour
I don't get similar printouts myself so I can't say anything for sure. By first check the old re-occuring (and very annoying) "KUBE-MARK-DROP" problem.

This chain is usually created by "kubeadm" but so far kubeadm is unaware of dual-stack and has only created this chain for ipv4 (in iptables). Then it needs to be created for IPv6 before kube-proxy is started; In my test cluster I always do this in a prep-stage before starting K8s.
The ci nodes doesn't have the kernel ip6tables nat modules loaded, hence they fail all the operation related
hmmm ok so is the issue here that we need to load that module in the CI jobs?
signing here because I want to at least find why ip6tables restore is called on kube-proxy ipvs things :D
ipvs does uses iptables too I think that kube-proxy should implement something similar to what implemented in kubelet The nat table for IPv6 was in 3.7 and it seems that it may be not present in some systems. /cc
/assign
I think it's mostly just about whether IPv6 is enabled, not about the kernel version.
IIRC this specific error in the CI is about the nat module only Anyway, you fixed it :clap: /unassign /assign


Question: // TODO: Get rid of [DisabledForLargeClusters] tag when issue #56138 is fixed. It("should be able to switch session affinity for LoadBalancer service with ESIPP on [Slow] [DisabledForLargeClusters]", func() { <ins> // L4 load balancer affinity `ClientIP` is not supported on AWS ELB. framework.SkipIfProviderIs("aws") </ins> svc := getServeHostnameService("service") svc.Spec.Type = v1.ServiceTypeLoadBalancer svc.Spec.ExternalTrafficPolicy = v1.ServiceExternalTrafficPolicyTypeLocal

<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . If the matter is security related, please disclose it privately via --Is this a BUG REPORT or FEATURE REQUEST?: What happened: What you expected to happen: The test should either succeed or be skipped on AWS How to reproduce it (as minimally and precisely as possible): Run this test on an AWS cluster with the AWS cloud provider configured Anything else we need to know?: The test sets the service's session affinity to here: The AWS cloud provider specifically only supports : Environment: Kubernetes version (use ): Cloud provider or hardware configuration: AWS OS (e.g. from /etc/os-release): Kernel (e.g. ): Install tools: Others:
cc
/cc
Thanks for the investigation, given the information so far, we should just skip this test on AWS. Will prepare a PR soon.


Question: echo "$ brew install md5sha1sum bash jq" fi echo <ins> exit 9 </ins> fi # This variable can be injected, as in the verify script.

<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . If the matter is security related, please disclose it privately via --Is this a BUG REPORT or FEATURE REQUEST?: What happened: tried to update godeps on mac OSX Eventual error out with osx bash is old and doesn't know what the flag is for declare. What you expected to happen: successfully finish running the scripts How to reproduce it (as minimally and precisely as possible): see above Anything else we need to know?: Environment: OS (e.g. from /etc/os-release): mac os 10.13.5
There are no sig labels on this issue. Please add a sig label. <detailsA sig label can be by either: a sig: e.g., to notify the contributor experience sig, OR the label manually: e.g., to apply the label Note: Method 1 will trigger an email to the group. See the . The in method 1 has to be replaced with one of these: bugs, feature-requests, pr-reviews, test-failures, proposals Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/cc since you use a mac
/cc
What version of OSX and what version of bash are you using?
The licenses script is important but horrible.
worked last time I tried (some days ago, on 10.13.5.
Default bash version in OSX is pretty old, you can check your current version with The option is only available on version 4 or newer, so you have to install a new bash version The shebang line in the script will use the first bash in your PATH, which should be the new one.
so this error is actually being thrown by . there's actually a version check in that script to ensure you're running bash 4 () I'm curious though.. because you have declare, but just not the -A flag... but somehow you're passing the version check, which I know fails on bash 3 (just tested it). Seems weird.
I think that the problem is that there is no inside the sentence so it warns the user but it continues running.
huh, you're right. I do have bash 4. I think I have an incorrect root cause. I made assumptions. My apologies. I'll try and take another look at it next week.


Question: By("verifying PD is present in node0's VolumeInUse list") framework.ExpectNoError(waitForPDInVolumesInUse(nodeClient, diskName, host0Name, nodeStatusTimeout, true /* should exist*/)) <del> if nodeOp == deleteNode { </del> <ins> if disruptOp == deleteNode { </ins> By("getting gce instances") gceCloud, err := framework.GetGCECloud() framework.ExpectNoError(err, fmt.Sprintf("Unable to create gcloud client err=%v", err))

When pod is evicted, pod is terminated. After this change , attach/detach controller should be able to detach the pod correctly so that new pod which uses the same volume will start up. We need to have an e2e test for this scenario.
what environment is good for this e2e test? GCE or GKE or AWS? I see we have which has func, Can you comment on why is insufficient to cover this issue?
does the pod need to be evicted? Or would be ok to delete the pod and then check that the PD is detached. Then create a new pod using the same PD and test that it attaches?
[MILESTONENOTIFIER] Milestone Removed Important: This issue was missing labels required for the v1.9 milestone for more than 3 days: kind: Must specify exactly one of , or . <details<summaryHelp</summary<ul<li<a href=""Additional instructions</a</li<li<a href=""Commands for setting labels</a</li</ul</details
ping ^^
sorry for the delay. Here we want to test the detach behavior of evicted pods. So delete a pod directly is different from it. When Pod is evicted, the containers will be killed and the pod status is terminated. But the pod api object is not deleted from API server. Before PR , such evicted pods will not trigger detach operation because the pod api object is not deleted from api server. So the purpose of the test is to make sure detach operation is triggered for such evicted pods.
Thanks Do you have a suggestion for how to force a pod to be evicted?
You could have the pod do some memory intensive application
ah, like my use of in the io-streaming pr :-)
- Here's another option to cause eviction: .


Question: kubelet.kubeClient = nil // ensure only the heartbeat client is used kubelet.hostname = testKubeletHostname <del> existingNode := v1.Node{ ObjectMeta: metav1.ObjectMeta{Name: testKubeletHostname, Annotations: make(map[string]string)}, Spec:       v1.NodeSpec{}, } // TODO : is it possible to mock validateNodeIP() to avoid relying on the host interface addresses ? addrs, err := net.InterfaceAddrs() assert.NoError(t, err) for _, addr := range addrs { var ip net.IP switch v := addr.(type) { case *net.IPNet: ip = v.IP case *net.IPAddr: ip = v.IP } if ip != nil && !ip.IsLoopback() && ip.To4() != nil { kubelet.nodeIP = ip break } } assert.NotNil(t, kubelet.nodeIP) fakeCloud := &fakecloud.FakeCloud{ Addresses: []v1.NodeAddress{ { Type:    v1.NodeExternalIP, Address: "132.143.154.163", </del> <ins> cases := []struct { name              string nodeIP            net.IP nodeAddresses     []v1.NodeAddress expectedAddresses []v1.NodeAddress shouldError       bool }{ { name:   "A single InternalIP", nodeIP: net.ParseIP("10.1.1.1"), nodeAddresses: []v1.NodeAddress{ {Type: v1.NodeInternalIP, Address: "10.1.1.1"}, {Type: v1.NodeHostName, Address: testKubeletHostname}, </ins> }, <del> { Type:    v1.NodeExternalIP, Address: kubelet.nodeIP.String(), </del> <ins> expectedAddresses: []v1.NodeAddress{ {Type: v1.NodeInternalIP, Address: "10.1.1.1"}, {Type: v1.NodeHostName, Address: testKubeletHostname}, </ins> }, <del> { Type:    v1.NodeInternalIP, Address: "132.143.154.164", </del> <ins> shouldError: false, }, { name:   "NodeIP is external", nodeIP: net.ParseIP("55.55.55.55"), nodeAddresses: []v1.NodeAddress{ {Type: v1.NodeInternalIP, Address: "10.1.1.1"}, {Type: v1.NodeExternalIP, Address: "55.55.55.55"}, {Type: v1.NodeHostName, Address: testKubeletHostname}, </ins> }, <del> { Type:    v1.NodeInternalIP, Address: kubelet.nodeIP.String(), </del> <ins> expectedAddresses: []v1.NodeAddress{ {Type: v1.NodeInternalIP, Address: "10.1.1.1"}, {Type: v1.NodeExternalIP, Address: "55.55.55.55"}, {Type: v1.NodeHostName, Address: testKubeletHostname}, </ins> }, <del> { Type:    v1.NodeInternalIP, Address: "132.143.154.165", </del> <ins> shouldError: false, }, { // Accommodating #45201 and #49202 name:   "InternalIP and ExternalIP are the same", nodeIP: net.ParseIP("55.55.55.55"), nodeAddresses: []v1.NodeAddress{ {Type: v1.NodeInternalIP, Address: "55.55.55.55"}, {Type: v1.NodeExternalIP, Address: "55.55.55.55"}, {Type: v1.NodeHostName, Address: testKubeletHostname}, </ins> }, <del> { Type:    v1.NodeHostName, Address: testKubeletHostname, </del> <ins> expectedAddresses: []v1.NodeAddress{ {Type: v1.NodeInternalIP, Address: "55.55.55.55"}, {Type: v1.NodeExternalIP, Address: "55.55.55.55"}, {Type: v1.NodeHostName, Address: testKubeletHostname}, </ins> }, <ins> shouldError: false, </ins> }, <del> Err: nil, } kubelet.cloud = fakeCloud kubelet.cloudproviderRequestParallelism = make(chan int, 1) kubelet.cloudproviderRequestSync = make(chan int) kubelet.cloudproviderRequestTimeout = 10 * time.Second kubelet.setNodeAddress(&existingNode) expectedAddresses := []v1.NodeAddress{ </del> { <del> Type:    v1.NodeExternalIP, Address: kubelet.nodeIP.String(), </del> <ins> name:   "An Internal/ExternalIP, an Internal/ExternalDNS", nodeIP: net.ParseIP("10.1.1.1"), nodeAddresses: []v1.NodeAddress{ {Type: v1.NodeInternalIP, Address: "10.1.1.1"}, {Type: v1.NodeExternalIP, Address: "55.55.55.55"}, {Type: v1.NodeInternalDNS, Address: "ip-10-1-1-1.us-west-2.compute.internal"}, {Type: v1.NodeExternalDNS, Address: "ec2-55-55-55-55.us-west-2.compute.amazonaws.com"}, {Type: v1.NodeHostName, Address: testKubeletHostname}, }, expectedAddresses: []v1.NodeAddress{ {Type: v1.NodeInternalIP, Address: "10.1.1.1"}, {Type: v1.NodeExternalIP, Address: "55.55.55.55"}, {Type: v1.NodeInternalDNS, Address: "ip-10-1-1-1.us-west-2.compute.internal"}, {Type: v1.NodeExternalDNS, Address: "ec2-55-55-55-55.us-west-2.compute.amazonaws.com"}, {Type: v1.NodeHostName, Address: testKubeletHostname}, }, shouldError: false, </ins> }, { <del> Type:    v1.NodeInternalIP, Address: kubelet.nodeIP.String(), </del> <ins> name:   "An Internal with multiple internal IPs", nodeIP: net.ParseIP("10.1.1.1"), nodeAddresses: []v1.NodeAddress{ {Type: v1.NodeInternalIP, Address: "10.1.1.1"}, {Type: v1.NodeInternalIP, Address: "10.2.2.2"}, {Type: v1.NodeInternalIP, Address: "10.3.3.3"}, {Type: v1.NodeExternalIP, Address: "55.55.55.55"}, {Type: v1.NodeHostName, Address: testKubeletHostname}, }, expectedAddresses: []v1.NodeAddress{ {Type: v1.NodeInternalIP, Address: "10.1.1.1"}, {Type: v1.NodeExternalIP, Address: "55.55.55.55"}, {Type: v1.NodeHostName, Address: testKubeletHostname}, }, shouldError: false, </ins> }, { <del> Type:    v1.NodeHostName, Address: testKubeletHostname, </del> <ins> name:   "An InternalIP that isn't valid: should error", nodeIP: net.ParseIP("10.2.2.2"), nodeAddresses: []v1.NodeAddress{ {Type: v1.NodeInternalIP, Address: "10.1.1.1"}, {Type: v1.NodeExternalIP, Address: "55.55.55.55"}, {Type: v1.NodeHostName, Address: testKubeletHostname}, }, expectedAddresses: nil, shouldError:       true, </ins> }, } <del> assert.True(t, apiequality.Semantic.DeepEqual(expectedAddresses, existingNode.Status.Addresses), "%s", diff.ObjectDiff(expectedAddresses, existingNode.Status.Addresses)) </del> <ins> for _, testCase := range cases { // testCase setup existingNode := v1.Node{ ObjectMeta: metav1.ObjectMeta{Name: testKubeletHostname, Annotations: make(map[string]string)}, Spec:       v1.NodeSpec{}, } kubelet.nodeIP = testCase.nodeIP fakeCloud := &fakecloud.FakeCloud{ Addresses: testCase.nodeAddresses, Err:       nil, } kubelet.cloud = fakeCloud kubelet.cloudproviderRequestParallelism = make(chan int, 1) kubelet.cloudproviderRequestSync = make(chan int) kubelet.cloudproviderRequestTimeout = 10 * time.Second kubelet.nodeIPValidator = func(nodeIP net.IP) error { return nil } // execute method err := kubelet.setNodeAddress(&existingNode) if err != nil && !testCase.shouldError { t.Errorf("Unexpected error for test %s: %q", testCase.name, err) continue } else if err != nil && testCase.shouldError { // expected an error continue } // Sort both sets for consistent equality sortNodeAddresses(testCase.expectedAddresses) sortNodeAddresses(existingNode.Status.Addresses) assert.True( t, apiequality.Semantic.DeepEqual( testCase.expectedAddresses, existingNode.Status.Addresses, ), fmt.Sprintf("Test %s failed %%s", testCase.name), diff.ObjectDiff(testCase.expectedAddresses, existingNode.Status.Addresses), ) } } // sortableNodeAddress is a type for sorting []v1.NodeAddress type sortableNodeAddress []v1.NodeAddress func (s sortableNodeAddress) Len() int { return len(s) } func (s sortableNodeAddress) Less(i, j int) bool { return (string(s[i].Type) + s[i].Address) < (string(s[j].Type) + s[j].Address) } func (s sortableNodeAddress) Swap(i, j int) { s[j], s[i] = s[i], s[j] } func sortNodeAddresses(addrs sortableNodeAddress) { sort.Sort(addrs) </ins> } func TestUpdateNewNodeStatus(t *testing.T) {

<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . If this may be security issue, please disclose it privately via --Is this a BUG REPORT or FEATURE REQUEST?: /kind bug What happened: When using the flag on the kubelet, the node's list no longer reports , , addresses. What you expected to happen: That I can explicitly set to an IP, but the other addresses will still show up. How to reproduce it (as minimally and precisely as possible): Create a new kops cluster, modify the kubelet to use the flag with the eth0 on the node. Run Anything else we need to know?: This change will probably take place in  This came up as a result of Environment: Kubernetes version (use ): 1.9.x Cloud provider or hardware configuration: AWS OS (e.g. from /etc/os-release): any Kernel (e.g. ): any Install tools: kops /sig network /sig node I'll make a PR for this change shortly


Question: } func allowsConsolidation(rule network.SecurityRule) bool { <del> return strings.HasPrefix(*rule.Name, "shared") </del> <ins> return strings.HasPrefix(to.String(rule.Name), "shared") </ins> } func findConsolidationCandidate(rules []network.SecurityRule, rule network.SecurityRule) (int, bool) { for index, r := range rules { if allowsConsolidation(r) { <del> if strings.EqualFold(*r.Name, *rule.Name) { </del> <ins> if strings.EqualFold(to.String(r.Name), to.String(rule.Name)) { </ins> return index, true } }

<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . If this may be security issue, please disclose it privately via --Is this a BUG REPORT or FEATURE REQUEST?: /kind bug What happened: controller manager crash looped with a seg fault after failing to create a load balancer for the kubernetes dashboard. The calls to create the load balancer were failing with a 400 What you expected to happen: controller manager to provision the resources or to fail without a segfault and a clearer error message saying what the error creating the resource was. How to reproduce it (as minimally and precisely as possible): Just created a cluster with an add on for the dashboard. Anything else we need to know?: logs from controller manager: I0423 12:50:28. ] Starting range CIDR allocator I0423 12:50:28. ] Waiting for caches to sync for cidrallocator controller I0423 12:50:28. ] Starting NoExecuteTaintManager I0423 12:50:28. ] Initializing eviction metric for zone: westeurope: :0 W0423 12:50:28. ] Missing timestamp for Node aks-nodepool1--0. Assuming now as a timestamp. I0423 12:50:28. ] Controller detected that zone westeurope: :0 is now in state Normal. I0423 12:50:28. ] Event(v1.ObjectReference{Kind:"Node", Namespace:"", Name:"aks-nodepool1--0", UID:"-4476-11e8-9b1c-", APIVersion:"", ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: 'RegisteredNode' Node aks-nodepool1--0 event: Registered Node aks-nodepool1--0 in Controller I0423 12:50:28. ] Caches are synced for ClusterRoleAggregator controller I0423 12:50:28. ] Caches are synced for endpoint controller I0423 12:50:28. ] Caches are synced for attach detach controller E0423 12:50:28. ]

Error creating spec for volume "mssqldb", pod "feature-test3"/"mssqlserver--r5r7l": error processing PVC "feature-test3"/"mssql-data": PVC feature-test3/mssql-data has non-bound phase ("Pending") or empty pvc.Spec.VolumeName ("") I0423 12:50:28. ] Caches are synced for deployment controller I0423 12:50:28. ] Caches are synced for persistent volume controller I0423 12:50:28. ] Caches are synced for ReplicationController controller I0423 12:50:28. ] Caches are synced for TTL controller I0423 12:50:28. ] Caches are synced for cidrallocator controller E0423 12:50:28. ] Failed to process service kube-system/kubernetes-dashboard. Retrying in 5s: error getting LB for service kube-system/kubernetes-dashboard: Service(kube-system/kubernetes-dashboard) - Loadbalancer not found I0423 12:50:28. ] Event(v1.ObjectReference{Kind:"Service", Namespace:"kube-system", Name:"kubernetes-dashboard", UID:"-4476-11e8-9b1c-", APIVersion:"v1", ResourceVersion:"193", FieldPath:""}): type: 'Warning' reason: 'CreatingLoadBalancerFailed' Error creating load balancer (will retry): error getting LB for service kube-system/kubernetes-dashboard: Service(kube-system/kubernetes-dashboard) - Loadbalancer not found I0423 12:50:28. ] Event(v1.ObjectReference{Kind:"Service", Namespace:"feature-initialproject", Name:"identityserver-service", UID:"-4485-11e8-9b1c-", APIVersion:"v1", ResourceVersion:"7879", FieldPath:""}): type: 'Normal' reason: 'EnsuringLoadBalancer' Ensuring load balancer I0423 12:50:28. ] Caches are synced for namespace controller I0423 12:50:28. ] Caches are synced for service account controller I0423 12:50:28. ] Caches are synced for resource quota controller I0423 12:50:28. ] Caches are synced for resource quota controller I0423 12:50:28. ]

Caches are synced for garbage collector controller I0423 12:50:28. ] Garbage collector: all resource monitors have synced. Proceeding to collect garbage I0423 12:50:28. ] Caches are synced for garbage collector controller I0423 12:50:28. ] Caches are synced for stateful set controller I0423 12:50:28. ] Caches are synced for disruption controller I0423 12:50:28. ] Sending events to api server. I0423 12:50:29. ] selectLoadBalancer: cluster(kubernetes) service(feature-initialproject/identityserver-service) isInternal(true) - availabilitysetsnames [nodepool1-availabilitySet-] E0423 12:50:30. ] Observed a panic: "invalid memory address or nil pointer dereference" (runtime error: invalid memory address or nil pointer dereference) /usr/local/go/src/runtime/asmamd64.s:509 /usr/local/go/src/runtime/asmamd64.s:2337 panic: runtime error: invalid memory address or nil pointer dereference [recovered] panic: runtime error: invalid memory address or nil pointer dereference [signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x1d5772d] goroutine 1338 [running]: (0x0, 0x0, 0x0) +0x111 panic(0x40b2e00, 0xa603590) +0x283 (Cloud).getServiceLoadBalancerStatus(0xc4206cdc00, 0xc422b401e0, 0xc4208f6b90, 0xc422b401e0, 0xc4210c0388, 0x1) +0x30d (Cloud).EnsureLoadBalancer(0xc4206cdc00, 0x496be8b, 0xa, 0xc422b401e0, 0xc4210c0388, 0x1, 0x1, 0xbeaf93852d9a680b, 0x2fe0fa757, 0xa9f0d40) +0x2ea (ServiceController).ensureLoadBalancer(0xc420f56000,

0xc422b401e0, 0xc422b401e0, 0x4961da9, 0x6) +0xcc (ServiceController).createLoadBalancerIfNeeded(0xc420f56000, 0xc421fd81e0, 0x2d, 0xc422b401e0, 0xc422c8dc40, 0xc422c8dc78, 0x1265492) +0x20e (ServiceController).processServiceUpdate(0xc420f56000, 0xc4217bcc60, 0xc422b401e0, 0xc421fd81e0, 0x2d, 0x0, 0x0, 0x0) +0xeb (ServiceController).syncService(0xc420f56000, 0xc421fd81e0, 0x2d, 0x0, 0x0) +0x3aa (ServiceController).worker.func1(0xc420f56000) +0xd9 (ServiceController).worker(0xc420f56000) +0x2b (ServiceController).()-fm() +0x2a (0xc420a04570) +0x5e (0xc420a04570, 0x3b9aca00, 0x0, 0x1, 0xc4208fa0c0) +0xbd (0xc420a04570, 0x3b9aca00, 0xc4208fa0c0) +0x4d created by (ServiceController).Run +0x20c Environment: Kubernetes version (use ): 1.9.1 Cloud provider or hardware configuration: Azure, AKS OS (e.g. from /etc/os-release): Ubunutu Kernel (e.g. ): xenial Install tools:AKS Others:
not sure what sig should be labeled in on this. Seems like a bug in sig-azure and whoever owns the piece getting the nil pointer/panic
what's the k8s version?
1.9.1
- I suspect that the error is here. This can be either one of two things A bug in SDK serialization (LB does have a private ip) but it does not get serialized correctly.

A bug in our assumption that that each internal LB ip fe config will have a valid internal IP. Can you dump the model for this internal LB here? Also -- did this error eventually recover? (it could be a race condition between setting ip and getting ip)
this was actually a customer cluster and they deleted it before I got a chance to get things like the model/what error code on the 400 was. And it didn't ever recover.
according to the evente table it was an internal loadbalancer here is the link to it --https://jarvis- maybe it the Details logged Can help you further
I was able to get more details from logs, {"error":{"code":"LoadBalancerFrontendIPConfigurationCountLimitReached","message":"Basic sku load balancer /subscriptions/redacted/resourceGroups/redacted/providers/Microsoft.Network/loadBalancers/kubernetes-internal cannot have more than 10 FrontendIPConfigurations.","details":[]}}
Thanks - So there are two errors here. One the user can not use more than 10 IPs. The fact that we panic on this. i believe the fix is to log and fail the operation. /assign
The issue has been fixed in and has been included in v1.9.2. Could you upgrade your cluster (e.g. to latest v1.9.7) and check whether it fixes the issue?
I'm closing this issue since it's been resolved. /close
I am seeing this on 1.10.x and 1.11.x clusters: 10.6 11.2
The PR has already been included in v1.11.x and v1.10.x. However, there's still other potential issues. Let me reopen this issue and file another PR to fix that /reopen
Reopening this issue. <detailsIn response to : The PR has already been included in v1.11.x and v1.10.x. However, there's still other potential issues. Let me reopen this issue and file another PR to fix that Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Added this to v1.12 since this is a critical urgent bug.


Question: --image "${MASTER_IMAGE}"  --tags "${MASTER_TAG}"  --network "${NETWORK}"  <del> --scopes "storage-ro" "compute-rw"  </del> <ins> --scopes "storage-ro,compute-rw"  </ins> --can-ip-forward  --metadata-from-file  <del> "startup-script=${KUBE_ROOT}/cluster/gce/configure-vm.sh"  "kube-env=${KUBE_TEMP}/master-kube-env.yaml"  --disk name="${MASTER_NAME}-pd" device-name=master-pd mode=rw boot=no auto-delete=no </del> <ins> "startup-script=${KUBE_ROOT}/cluster/gce/configure-vm.sh,kube-env=${KUBE_TEMP}/master-kube-env.yaml"  --disk "name=${MASTER_NAME}-pd,device-name=master-pd,mode=rw,boot=no,auto-delete=no" </ins> } # TODO(mbforbes): Make $1 required.

This happens over and over. Scary messages. WARNING: Lists should be separated by commas, try "--allow=tcp,udp,icmp,esp,ah,sctp". If you intend to use udp, icmp, esp, ah, sctp for positional arguments, put them before the flag. WARNING: Lists should be separated by commas, try "--scopes=storage-ro,compute-rw". If you intend to use compute-rw for positional arguments, put them before the flag. WARNING: Lists should be separated by commas, try "--metadata-from-file=startup-script=/usr/local/google/home/thockin/src/kubernetes/hack/e2e-internal/../../cluster/../cluster/gce/../../cluster/../cluster/gce/configure-,kube-env=/tmp/kubernetes.MURmzX/master-kube-". If you intend to use kube-env=/tmp/kubernetes.MURmzX/master-kube- for positional arguments, put them before the flag. WARNING: Lists should be separated by commas, try "--disk=name=e2e-test-thockin-master-pd,device-name=master-pd,mode=rw,boot=no,auto-delete=no". If you intend to use device-name=master-pd, mode=rw, boot=no, auto-delete=no for positional arguments, put them before the flag.
That's a new one. gcloud started getting a little more anal here. Exciting. cc
They changed list argument syntax a while back, but the warnings are new. Time to update startup scripts I guess.


Question: // TODO(nikhiljindal): Check the instance group annotation value and verify with a multizone cluster. }) <del> It("should be able to switch between HTTPS and HTTP2 modes", func() { </del> <ins> // TODO: remove [Unreleased] tag to once the new GCE API GO client gets revendored in ingress-gce repo It("should be able to switch between HTTPS and HTTP2 modes [Unreleased]", func() { </ins> httpsScheme := "request_scheme=https" By("Create a basic HTTP2 ingress")

Which jobs are failing:   Which test(s) are failing: The jobs are timing out, but just before that, the following tests failed: (in ) (in ) Since when has it been failing: late 2019-03-07 Testgrid link: Reason for failure: Timeout related to the L7 controller: /sig network /priority critical-urgent (failing consistently in release blocking jobs, blocking other tests entirely from running) cc cc
/milestone v1.14
could this be a gcp issue?
also cc'ing the sig-network chairs; could you help to triage this please?
Seems like Ingress controller kept panicing during the test (): /cc
Seems like has filed
this will likely mitigate the CI failures
Assigning to freehan and kubernetes/ingress-gce 's reviewer /assign
Can we move this to "Under investigation" on the CI project board?
^^
/triage unresolved
Hey I noticed the label - does that mean that it won't be resolved (as is documented ) or that it hasn't been yet?
No this is a misunderstanding on my part with what this label meant. I thought it signified "open but unassigned" . /remove-label triage/unresolved
/remove-triage unresolved triage labels seem to be a bit old, they should be revamped
and are still failing consistently, nearly all other are flaking frequently at the moment. /reopen
Reopened this issue. <detailsIn response to : Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
master-blocking runs are clearing out, but 1.14-blocking seem to still include the http2 test ( is the first that includes in )
Took a quick look, turned out job "ci-kubernetes-e2e-gce-cos-k8sbeta-ingress" doesn't skip test with the tag. The job config is auto-generated: or how can we add flag to that job?
hummmmmmm when did that tag becomes a thing?!
I think it is here We introduced that filter for Ingress-GCE tests only because we needed a way to turn off tests on specific branches.
Thanks both, let me send a PR to update that.
Why not use the feature tag?

Introduce a new tag that need to be skipped by default will presumably break everything :-)
[feature:unreleased], instead of [unreleased]
That sounds like a reasonable approach, but it means we will need to update most of the ingress tests & jobs (more on-going PRs). Could we update that job to skip for now and file an issue to cleanup the tag after code freeze? ( Or maybe will happen first:) )
I have a feeling this unrelease tag is running in other jobs.. which branch has this tag?
The unrelease tag always co-presents with tag. So as long as we patch those ingress jobs it should be fine. is updating the remaining jobs: k8s-stable3 k8s-stable2 k8s-stable1 k8s-beta
+1 to MrHohn We have the filter on every job config. Each individual test has the unreleased tag based on what branch it needs to be tested on. So if none of the actual tests have "Unreleased" nothing gets skipped. Ref:
:+1:
that explains a lot of things 14-blocking is now starting to pass; with the https/http2 test removed: !
All clear, closing. /close
Closing this issue. <detailsIn response to : All clear, closing. Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: } func (plugin *fcPlugin) SupportsMountOption() bool { <del> return false </del> <ins> return true </ins> } func (plugin *fcPlugin) SupportsBulkVolumeVerification() bool {

<!-- Please only use this template for submitting enhancement requests --What would you like to be : Currently many in-tree storage plugins, including the iSCSI plugin, support mount options. The FC plugin does not. I have poked around the code enough to know where the plugin reports that mount options are not supported, but it isn't clear to me WHY they aren't supported. Why is this needed: Mount options can drastically change the performance and data persistence characteristics of a filesystem. For example, "noatime" can reduce metadata operations on frequently read data or "sync" can ensure all writes go immediately down to a backing storage array. I am working on a deployment across an FC backed cluster for which the the containerized application benefits from specific mount options, and I would guess other users might find similar use cases. It seems to me that if mount options are supportable on iSCSI, they should be equally supportable on FC, but I certainly recognize the possibility that I may be missing something. Otherwise, I'm guessing they simply haven't been a priority?
Reiterating the mentions to trigger a notification: <detailsIn response to : Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/good-first-issue /help
This request has been marked as suitable for new contributors. Please ensure the request meets the requirements listed . If this request no longer meets these requirements, the label can be removed by commenting with the command. <detailsIn response to : /good-first-issue /help Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Yeah, we just miss a volunteer that would implement this feature. It should be relatively easy and straightforward, following iscsi as an example.
I would like to work on it! As this my first issue I would like to get some help.
/assign sathish-thangaraj
GitHub didn't allow me to assign the following users: sathish-thangaraj. Note that only and repo collaborators can be assigned and that issues/PRs can only have 10 assignees at the same time.

For more information please see <detailsIn response to : /assign sathish-thangaraj Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
that's great! I was going to write detailed info how to fix this, however, I found out that mount options should work, at least in Kubernetes 1.13, see this commit: So there is a simple bug somewhere. Looking around the code, this code smells fishy: . Can you check with "BlockVolume" feature turned off in kubelet, i.e. ? Mount options should work there. And then it should be fairly easy to fix it in .
It definitely looks like there's an issue there. I don't think the kubelet ever made it that far in my initial testings, as mounting (with mountOptions listed in the PersistentVolume object) failed with the following message: I assume this is related to the following code: Is there a way to disable a check like this without rebuilding all of Kubernetes?
Good catch! No. IMO it should be enough to rebuild kubelet here and it's fast and easy: and find it somewhere in .
Thanks for the help so far I flipped the SupportsMountOption() bool to "true" and the BlockVolume feature gate to false. I was then able to mount an ext4 volume with the "noatime" and "nodiratime" option (confirmed on the worker node). However, I attempted to use the "nobarrier" option but the worker node mounted the volume without it. Finally, I attempted to use a completely made-up option ("hello"). The volume did not fail to mount (as is suggested in the Kubernetes documentation. Instead, it mounted without the "hello" options. I am attempting to locate useful logs to try to understand why some mount options are ignored or why incorrect mount options aren't causing failures, but the kubelet logs haven't been helpful thus far. Unfortunately I don't currently have any easy access to any iSCSI equipment for comparison.
anyone working on this?
is there anywhere that I could help on the issue?
First time contributor here - I just changed 2 lines of code to make FC similar to other volume types (eg. iSCSI).

I'm not sure about the mount option validation that mentions. Should that be a separate issue?
Apologies for the extreme delay in keeping up with this issue. I believe the issue with ignored mount options may be more generally related to the way the Linux mounter (/pkg/utils/mount/mount-) handles mount options, although I don't have enough experience with other volume plugins to speak authoritatively on the issue. I recommend we try to get the basic changes suggested above into the code, as my limited testing shows them to be effective for at least some mount options. I'd try to submit a pull request myself, but I don't want to take it from as they've already put forth the work to get it started.
To clarify my previous comment, during Attach, the FC volume plugin mounts an FC volume to a global path with no mount options. During Mount, the Linux mounter bind mounts (-o bind) and bind remounts (-o bind,remount) an FC volume to a Pod specific path with mount options. However, the bind remount action is only capable of applying a limited number of mount options (those not on the list are silently ignored). In order to expand our mount option capabilities, we would (I believe) need to use mount options in the Attach phase.
First time contributor, taking this one /assign
you are, of course, welcome to work on this issue. However, I wanted to let you know that I made code changes back in November that (I believe) address my previous comments. I only received approval from my employer in the last couple of days to make the contributions, or I would have submitted them already. I plan to clean things up and get a PR in sometime this afternoon. I am also a first time contributor, so we'll see how it goes. /assign
No problem at all dude. You take this one since you already made the changes. On Thu, 23 Jan 2020 at 18:47 Eric Weber <:


Question: volumeName, err := framework.CreatePDWithRetry() Expect(err).NotTo(HaveOccurred()) defer func() { <del> framework.ExpectNoError(framework.DeletePDWithRetry(volumeName)) </del> <ins> // - Get NodeName from the pod spec to which the volume is mounted. // - Force detach and delete. pod, err := f.PodClient().Get(config.Prefix+"-client", metav1.GetOptions{}) Expect(err).NotTo(HaveOccurred(), "Failed getting pod %q.", config.Prefix+"-client") detachAndDeletePDs(volumeName, []types.NodeName{types.NodeName(pod.Spec.NodeName)}) </ins> }() defer func() {

https://k8s-
See flakes on . This test needs to be quarantined until it's deflaked.
This test has been moved in the flaky suite ()
Could you please help triage and fix this test? CC
and I are looking into it...
After comparing the relevant passing and failing test logs in the Flaky suite, this appears to just be a timing issue. doesn't give the PD enough time to detach (5 min timeout). Adding above the delete allows a 10 minute detach window before deletion. This is the pattern we've used in other GCEPD tests and have yet to see similar flakes. I'll have a PR up in a bit to add the detach wait. If there's not more flakes for a week after it's merged, I'll submit a PR to pull the Flaky tag.
So I notice this happens when I upgrade a deployment that moves the pod to another node. I can pretty much recreate it at will now. Is the solution to wait 5 minutes before allowing a pod to be created after it was deleted?
After checking the log, there is something wrong at the node side when reconciler tries to unmount the device (global mount path) because there are still some references to the mount. But unmounting local mount point is successfully. Need to investigate more about this issue.


Question: NumberAvailable:        1, }, <del> msg:  "Waiting for rollout to finish: 1 of 2 updated pods are available...n", </del> <ins> msg:  "Waiting for daemon set "foo" rollout to finish: 1 of 2 updated pods are available...n", </ins> done: false, }, {

Is this a BUG REPORT or FEATURE REQUEST?: /kind feature What happened: In our CI, we run multiple times in parallel when we update multiple deployments. This lets us see the status of all deployments as they run. When a deployment succeeds, the final message tells us the name of the deployment. However, the intermediate "Waiting for rollout to finish" messages do not say the name of the deployment. This means that in order to look at multiple rollouts in parallel, we need to manually pipe through another command to prefix it with the rollout name. What you expected to happen: Each line to mention the rollout name. How to reproduce it (as minimally and precisely as possible): Update any deployment, run , note that intermediate status names don't contain the deployment name. Anything else we need to know?: Arguably another fix would be to allow you to pass multiple rollouts to and have kubectl handle polling multiple deployments in parallel. Environment: This is still true on master by code inspection.
Reiterating the mentions to trigger a notification: <detailsIn response to : Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: --v $(KUBE_VERBOSE)                                          --logtostderr                                                -i "$$pkgs"                                                  <del> --extra-peer-dirs $$(echo $(addprefix $(PRJ_SRC_PATH)/, $(DEFAULTER_DIRS)) | sed 's/ /,/g')  </del> <ins> --extra-peer-dirs $$(echo $(DEFAULTER_EXTRA_PEER_PKGS) | sed 's/ /,/g')  </ins> -O $(DEFAULTER_BASENAME)                                     "$$@";                                                       fi

In the codegenerator we have: So basically we process only packages that are explicitly requested by the user and ignore ones that are imported. So in order to have the conversion func imported by the generator, the package where it lives has to be explicitly imported. So we should basically always pass all input packages to conversion generator. This is problematic, when regenerating conversions, we simply pass to input directories only those that were changed in the meantime: We should fix that - probably by passing all CONVERSIONDIRS as inputs to conversion-gen /kind-bug
I think this caused Not sure if was related yet.
It's the other way around - this is the main bug and root cause and the other that you linked is a consequence.
Also - FYI
I actually already started looking into it - so let me assign this.
is out for review
/triage accepted


Question: func (pf *fakePortForwarder) PortForward(name string, uid types.UID, port uint16, stream io.ReadWriteCloser) error { defer stream.Close() <del> var wg sync.WaitGroup // client -> server wg.Add(1) go func() { defer wg.Done() // copy from stream into a buffer received := new(bytes.Buffer) io.Copy(received, stream) // store the received content pf.lock.Lock() pf.received[port] = received.String() pf.lock.Unlock() }() // server -> client wg.Add(1) go func() { defer wg.Done() </del> <ins> // read from the client received := make([]byte, len(pf.expected[port])) n, err := stream.Read(received) if err != nil { return fmt.Errorf("error reading from client for port %d: %v", port, err) } if n != len(pf.expected[port]) { return fmt.Errorf("unexpected length read from client for port %d: got %d, expected %d. data=%q", port, n, len(pf.expected[port]), string(received)) } </ins> <del> // send the hardcoded data to the stream io.Copy(stream, strings.NewReader(pf.send[port])) }() </del> <ins> // store the received content pf.lock.Lock() pf.received[port] = string(received) pf.lock.Unlock() </ins> <del> wg.Wait() </del> <ins> // send the hardcoded data to the client io.Copy(stream, strings.NewReader(pf.send[port])) </ins> return nil }

I wanted to drop a note that I haven't forgotten about this issue. I have spent some time investigating and have yet to determine why some test runs result in the data not being forwarded correctly. I hope to resume digging into this soon, but I've got some planning tasks I need to complete first.
I have determined the root cause of this. In the test case, it: a port forwarding session, listening on a local port a TCP connection to the local port a goroutine copies data from the local port to the remote a goroutine copies data from the remote to the local port data to the TCP connection data from the TCP connection port forwarding, which results in the client streams being closed The problem is that the data written in step 3 is only to the local TCP port. There's no guarantee that the goroutine in 2.i copies the data to the remote before step 5 happens. In fact, if you reverse steps 3 () and 4 () so the write happens immediately before you close the stopChan, this test fails very quickly. I am thinking about how to fix this...
Not done yet:
Saw this again today:
- this was a different failure than this one. I've just opened for it and closing this one


Question: assert.Empty(t, containers) } <ins> // TestKillContainer tests killing the container in a Pod. func TestKillContainer(t *testing.T) { _, _, m, _ := createTestRuntimeManager() tests := []struct { caseName            string pod                 *v1.Pod containerID         kubecontainer.ContainerID containerName       string reason              string gracePeriodOverride int64 succeed             bool }{ { caseName: "Failed to find container in pods, expect to return error", pod: &v1.Pod{ ObjectMeta: metav1.ObjectMeta{UID: "pod1_id", Name: "pod1", Namespace: "default"}, Spec:       v1.PodSpec{Containers: []v1.Container{{Name: "empty_container"}}}, }, containerID:         kubecontainer.ContainerID{Type: "docker", ID: "not_exist_container_id"}, containerName:       "not_exist_container", reason:              "unknown reason", gracePeriodOverride: 0, succeed:             false, }, } for _, test := range tests { err := m.killContainer(test.pod, test.containerID, test.containerName, test.reason, &test.gracePeriodOverride) if test.succeed != (err == nil) { t.Errorf("%s: expected %v, got %v (%v)", test.caseName, test.succeed, (err == nil), err) } } } </ins> // TestToKubeContainerStatus tests the converting the CRI container status to // the internal type (i.e., toKubeContainerStatus()) for containers in // different states.

<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . --Is this a BUG REPORT or FEATURE REQUEST?: What happened: I tried to use virtlet with Kubernetes release 1.6.6, but a pod with a VM in it gets stuck in "ContainerCreating" state. This problem does not happen with Kubernetes release 1.6.3. The log of the responsible kubelet shows a panic when trying to restart the container. I do not know why the container is being restarted, but that should not cause a panic in any case. Here is an excerpt from the kubelet's log: It is easy to see a code path that can cause this panic. The offending line in is this: and there is indeed a way for to be nil at this point. Earlier in that function we see the line and there is no intervening statement that checks whether is ; ends with . What you expected to happen: I expected the virtlet to run, like it does with Kubernetes release 1.6.3. How to reproduce it (as minimally and precisely as possible): That would take some work, I hope the diagnosis above makes this moot. Anything else we need to know?: Environment: Kubernetes version (use ):Cloud provider or hardware configuration**: "bare metal" (really VMs, but who cares?) OS (e.g. from /etc/os-release): Ubuntu 16.04.2 LTS Kernel (e.g. ): Linux n11rdu1ctrl 4.4.0-34-generic -Ubuntu SMP Wed Jul 27 16:06:39 UTC 2016 x8664 x8664 x8664 GNU/Linux Install tools: I installed Kubernetes and virtlet using some local Ansible Others: I am using a local fork of virtlet
There are no sig labels on this issue. Please by:<br(1) mentioning a sig: <bre.g., for API Machinery<br(2) specifying the label manually: <bre.g., for sig/scalability<br<brNote: method (1) will trigger a notification to the team. You can find the team list and label list
/sig node
Reiterating the mentions to trigger a notification: <detailsIn response to : </details
, did you do any update action against that Pods?

I upped the kubelet's to 4, and got an interesting clue when I tried the experiment again. Here is an excerpt from the more verbose log. Note the empty container name in the "Killing unwanted container" entry. That telling log entry comes from these lines in which is soon followed by .
I only created the pod, no updates.
Is there a running container with id something like when the panic occurs? A quick fix is to add checking whether containerSpec is nil as you mentioned. But it's weird that gets a container without name.
Did virtlet set correctly? Seems it is "", which caused the problem.
Create a PR () that checking containerSpec to avoid panic. And it seems there's no UT case for , I'll create related PRs for
The actual Docker containers are all gone now. But here is what I got from shortly before opening this issue: I will look into the question asked.


Question: }, } <del> leaseTime := uint64(s.leaseTime) </del> <ins> // leaseTime needs to be in seconds leaseTime := uint64(s.leaseTime / time.Second) </ins> // NB: GuaranteedUpdate does not perform the store operation unless // something changed between load and store (not including resource

/kind bug The lease reconciler creates the kubernetes endpoint with a prefix of kube-apiserver-endpoint, however that is not the prefix that the rest of kube uses. The end result is that the kubernetes service doesn't have an endpoint, and kubectl won't list the kubernetes endpoint. The lease reconciler also creates lease files with a really long lease time instead of the default 15 seconds.
/cc
cc xref
kube-apiserver-endpoint is used for the lease itself, from what I remember. I don't think that's an issue by itself.
Oh but it does look like the endpointRegistry for the lease reconciler is misconfigured to use that
i'll look into it
what is the name that should be used?
services/endpoints afaik -
Yeah, making that change locally fixed it for me
But I'm not seeing the masterleases/apiServerIPInfo data in etcd, or I'm issuing the wrong command to find it
I had that same problem. The lease timeout is set way high, but for some reason that resulted in the lease files being created for only a very short time. I'm not sure why. Playing with the ttl amount got the lease files to stay around....forever. I've got a PR I'm about to push to fix both issues.
FYI has a PR for this now
cc


Question: REGISTRY?=gcr.io/google-containers IMAGE?=debian-hyperkube-base <del> TAG=0.2 </del> <ins> TAG=0.3 </ins> ARCH?=amd64 CACHEBUST?=1

/kind bug What happened: When a cluster is provisioned using a containerised kubelet using the following images: Pods with PVCs are not created because kubelet cannot mount the disks. If you look at events, you can see that it's missing some binaries: This executable is . This problem can be resolved by mounting paths from the host into the rkt container, specifically and . Is this expected? mentioned in , that mounting operations are moving out from the main process into containers. Is this part of that effort? This caught me quite unexpected, and I didn't find any docs or changelog entries which indicated additional host mounts were necessary. If it's deliberate, I can close this until we perhaps document the breaking change. Is there a good place where we can document this? If it's not, I'll leave this open as a tracking issue. What you expected to happen: The hyperkube image to work without lots of host mounts. How to reproduce it (as minimally and precisely as possible): Run kubelet with rkt in one of the above listed images (see CoreOS' ). Environment: Kubernetes version (use ): Cloud provider or hardware configuration**: OpenStack OS (e.g. from /etc/os-release):Kernel (e.g. ):Install tools: tectonic-installer
/sig storage /sig node
Seems the base hyperkube image was changed: done in /cc
The pkg/util/mount/mount_linux source defaults to ext4 if a fstype is not selected. It makes sense to restore the e2fsprogs package within the hyperkube image.
cc cc cc This seems like something that should be in the 1.8 milestone. Can someone who owns the hyperkube image comment on that?
[MILESTONENOTIFIER] Milestone Labels Complete Issue label settings: sig/node sig/storage: Issue will be escalated to these SIGs if needed. priority/important-soon: Escalate to the issue owners and SIG owner; move out of milestone after several unsuccessful escalation attempts. kind/bug: Fixes a bug discovered during the current release. <detailsAdditional instructions available <a href=""here</aThe commands available for adding these labels are documented <a href=""here</a</details
Containerization of mount is not going to happen until 1.9 at the earliest.

So if hyperkube is broken, it should be fixed before ship.
"modprobe" also went missing in 1.8 hyperkube images... Both kubelet and kube-proxy need it.
Are you noticing any buggy behaviour on those components without it? Would you mind creating an issue and/or PR to restore that dep
Done: And done: :)


Question: flag.StringVar(&cloudConfig.Cluster, "gke-cluster", "", "GKE name of cluster being used, if applicable") flag.StringVar(&cloudConfig.NodeInstanceGroup, "node-instance-group", "", "Name of the managed instance group for nodes. Valid only for gce, gke or aws. If there is more than one group: comma separated list of groups.") flag.StringVar(&cloudConfig.Network, "network", "e2e", "The cloud provider network for this e2e cluster.") <del> flag.IntVar(&cloudConfig.NumNodes, "num-nodes", -1, "Number of nodes in the cluster") </del> <ins> flag.IntVar(&cloudConfig.NumNodes, "num-nodes", DefaultNumNodes, fmt.Sprintf("Number of nodes in the cluster. If the default value of '%q' is used the number of schedulable nodes is auto-detected.", DefaultNumNodes)) </ins> flag.StringVar(&cloudConfig.ClusterIPRange, "cluster-ip-range", "10.64.0.0/14", "A CIDR notation IP range from which to assign IPs in the cluster.") flag.StringVar(&cloudConfig.NodeTag, "node-tag", "", "Network tags used on node instances. Valid only for gce, gke") flag.StringVar(&cloudConfig.MasterTag, "master-tag", "", "Network tags used on master instances. Valid only for gce, gke")

A) seems to be required by some tests. we need to decide if jobs need to pass explicitly or let the e2e framework detect how many worker nodes a cluster has. the conformance image has no notion of yet. B) the conformance image runs tests even in parallel mode: if "y" is set in the YAML for if we assume that tests should not run in parallel mode, this means we need to pass from the runner. C) should we switch the conformance image to ? currently it's . we need to check if this breaks anything. discussion: /area test /kind design /kind bug /assign
/sig testing
PRs - disable Parallel mode for conformance image (and test):
RE: the introduction of ` in this test broke not only the kubeadm tests but also the conformance image and sonobuoy. previously they did not require and we are now forced to detect the number of nodes outside of the e2e testing framework. possibly this test was the first conformance test that uses as a aggressive failure condition. there are two options here: A) adapt all prow jobs, conformance image and sonobuoy, etc to explicitly pass . this ensures that future test will rely on the number the user provides and expects. we already started going this route by providing mechanics for detecting schedulable nodes, per-solution. B) auto-detect "ready", schedulable nodes in the e2e framework as a start number before, say a daemon set test runs and compare outcomes. this way a test will run on whatever number of nodes are currently available. cc
/area conformance /milestone v1.14
Line should be a explicit failure, not a
i like option B!
related to
Note that this test was NOT new, the other bug is ignoring skipped conformance tests. Ideally that shouldn't be possible.
/assign
GitHub didn't allow me to assign the following users: johnSchnake, stevesloka. Note that only and repo collaborators can be assigned and that issues/PRs can only have 10 assignees at the same time. For more information please see <detailsIn response to : /assign Instructions for interacting with me using PR comments are available .

If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
I'm reluctant to make a sweeping change for 1.14 other then a band aide inside of test-infra. Ideally I'd like to assign this issue to the test-commons group to solve in 1.15. Thoughts?
Agree that (A) (passing num-nodes explicitly) isnt a good option since it breaks other users who dont know about it in the first place and that this is well within the capabilities of the framework. So for option (B) during startup of the framework we need to grab the nodes and count the number that are schedulable and not tainted as master nodes, correct? Save that into a TestContext variable and then reference it elsewhere as needed? Is the plan to reuse the existing value? I'm not at this time familiar with that value and whether it is needed in some other capacity and shouldn't be auto-set. w.r.t. 1.14 vs 1.15; I don't have too strong a preference. I do generally prefer to let things bake a little longer inside of a release but it seems like if just the node count solves the problem its a small, scoped change that could make it in.
From slack: If num-nodes is not specified query on startup and over-ride the test context variable.
a control plane node can be tainted as "master", but still allow workloads. search for on all nodes, and exclude those that have it from the count. this is fed the flag, which is -1 by default. if the user didn't add a value my suggestion is to just update with the detected count. also possibly add notes about that in the flag description. future tests might decide to use this again (we already have some) and it will break if the user doesn't pass a valid number from the CLI flag. the "auto-detect if -1" logic seems reasonable and backwards compatible. how well our detection will work in complicated taint / toleration scenarios is another question. "Code Thaw" starts on the 19th of march. if we merge a change in master before that the change will also be picked by 1.14. if we patch this later the change has to be cherry picked for 1.14.


Question: /srv/kubernetes/abac-authz-policy.jsonl: file.managed: - source: salt://kube-apiserver/abac-authz-policy.jsonl <ins> - template: jinja </ins> - user: root - group: root - mode: 600

abac-authz- hardcodes "admin" instead of using the $KUBE_USER variable. This means that "admin" is the only basic-auth username that will work with that abac config.
To workaround until it is fixed, you can add a line to the master's abac-authz- file: and then restart the kube-apiserver (docker kill it, and let it restart).


Question: err = gce.ensureInternalBackendService(bsName, "description", svc.Spec.SessionAffinity, cloud.SchemeInternal, "TCP", igLinks, "") require.NoError(t, err) <del> // Update the BackendService with new Instances </del> <ins> // Update the BackendService with new InstanceGroups </ins> if tc.mockModifier != nil { tc.mockModifier(gce.c.(*cloud.MockGCE)) } <del> newNodeNames := []string{"new-test-node-1", "new-test-node-2"} err = gce.ensureInternalBackendServiceGroups(bsName, newNodeNames) </del> <ins> newIGLinks := []string{"new-test-ig-1", "new-test-ig-2"} err = gce.ensureInternalBackendServiceGroups(bsName, newIGLinks) </ins> if tc.mockModifier != nil { assert.Error(t, err) return

Seeing several of the tests in ./pkg/cloudprovider/providers/gce flake 1 in 5-10 runs when run locally (I am logged in to gcloud). Is it possible one of the APIs is not correctly shimmed and is fetching real data?
you were the last folks in here - why would this and other internal_tests in here be flaking?
Happens in both master and 1.11 release branches
/cc cheftako