Friday, 2018-11-30

openstackgerritClint 'SpamapS' Byrum proposed openstack-infra/nodepool master: Amazon EC2 driver  https://review.openstack.org/53555800:10
*** j^2 has quit IRC00:14
*** threestrands has joined #zuul00:18
clarkbcorvus: super minor thing in https://review.openstack.org/#/c/615356/5 so I +2'd but may be worth changing that if another patchset happens00:28
corvusclarkb: good point -- though i intended to resolve that in the other direction00:30
openstackgerritJames E. Blair proposed openstack-infra/zuul master: Remove nodeid argument from updateNode  https://review.openstack.org/62104700:32
corvusclarkb: ^00:32
clarkbthat also works00:32
corvuseither one is better than both :)  and that's a bit more consistent i think00:33
clarkbexciting that we could have the feature running tomorrow00:37
clarkbalso I should probably brainstorm a nice way of explaining it for infra users that had previously been taking advantage of the allocation scheme00:38
corvusclarkb: the word "fair" comes to mind :)00:39
clarkbcorvus: yup. I half expect some sort of "but we deserve preferential treatment because $raisins" but that may be me over worrying about it00:39
clarkbmaking our zuul instance viable for the small/less active projects is incredibly important imo00:40
clarkbI should've paid more attention to whether or not the nodepool side will handle requests with and without a relative priority00:41
clarkbcorvus: ^ if nodepool doesn't handle that maybe we should have it default to a value so that you can have mixed nodepool and zuul versions deployed?00:42
clarkbok default is 0 I guess it has to be since this is an optional feature. Yay for not needing to do upgrades in one go or in locksetp00:46
pabelangerclarkb: Yah, I'm kinda of intersted to see how much of an impact it makes00:51
ianwdoes anyone know how to run ansible-test without it creating a million docker things01:30
ianwansible-test integration --tox seems like it might, but ... it doesn't01:31
pabelangerianw: ansible-test from ansible/ansible?01:34
ianwyeah01:35
pabelangeryah, it is pretty tough locally01:35
ianwERROR: Command "docker run -d --name vcenter-simulator -p 80:80 -p 443:443 -p 8080:8080 -p 8989:8989 -p 5000:5000 quay.io/ansible/vcenter-test-container:1.4.0" returned exit status 125.01:35
pabelangerI usually just push up to github PR and wait for results01:35
ianwi mean ... seriously why do i need this ...01:35
clarkbianw: gotta make sure you didn't break vmware :P01:35
pabelangerthere is plans to do it via molecule01:36
pabelangerbut, that is a while off01:36
clarkbianw: I'm going to guess you've already bound port 80, 443 or 8080?01:36
ianwcertainly not intentionally, but this is about the nth docker container it's downloaded01:38
pabelangeryah, it swings for the fence on what it tests01:40
ianwhrm, it's created a container quay.io/ansible/cloudstack-test-container:1.2.001:43
*** bhavikdbavishi has joined #zuul01:43
ianwwhich has bound the ports, then it tries to bind this vcentre one to the same ports01:44
clarkb01:45
clarkbwhoops01:45
clarkbwhoops to both my failed message and double bind01:45
openstackgerritMerged openstack-infra/nodepool master: Support relative priority of node requests  https://review.openstack.org/62095401:58
ianwok, fyi "ansible-test integration --tox --python 3.6 include_import" seems to work to be tox-only01:58
*** rlandy has quit IRC03:31
*** dkehn has quit IRC03:38
*** dkehn has joined #zuul03:50
openstackgerritBrendan proposed openstack-infra/zuul master: Fix "reverse" Depends-On detection with new Gerrit URL schema  https://review.openstack.org/62083804:03
openstackgerritMerged openstack-infra/zuul-jobs master: upload-logs-swift: Cleanup temporary directories  https://review.openstack.org/59234004:49
openstackgerritMerged openstack-infra/zuul-jobs master: upload-logs-swift: Make indexer more generic  https://review.openstack.org/59285204:50
*** threestrands has quit IRC05:20
*** pcaruana has quit IRC05:35
openstackgerritMerged openstack-infra/zuul master: Add support for zones in executors  https://review.openstack.org/54919706:00
*** dkehn has quit IRC06:44
*** quiquell|off is now known as quiquell07:03
*** pcaruana has joined #zuul07:22
*** bhavikdbavishi has quit IRC07:42
openstackgerritMerged openstack-infra/zuul master: More strongly recommend the simple reverse proxy deployment  https://review.openstack.org/62096907:56
*** gtema has joined #zuul07:56
openstackgerritMerged openstack-infra/zuul master: Add gearman stats reference  https://review.openstack.org/62019207:57
*** jpena|off is now known as jpena08:01
*** bhavikdbavishi has joined #zuul08:05
quiquellGood morning08:10
quiquellI see zuul project is using pbrx-build-container-images to build docker images08:11
quiquellWhere do you push them to dockerhub ?08:11
quiquelltobiash, corvus: ^ ?08:11
quiquellthis is it ? http://git.openstack.org/cgit/openstack-infra/project-config/tree/zuul.d/jobs.yaml#n126108:12
*** hashar has joined #zuul08:27
*** hashar has quit IRC09:53
*** bjackman has joined #zuul10:10
*** electrofelix has joined #zuul10:44
*** pcaruana has quit IRC10:44
*** pcaruana has joined #zuul10:50
*** rfolco is now known as rfolco_doctor11:09
*** bhavikdbavishi has quit IRC11:11
*** dkehn has joined #zuul11:26
*** gtema has quit IRC11:38
*** gtema has joined #zuul11:49
tobiashquiquell: look here: https://hub.docker.com/r/zuul/11:51
quiquelltobiash: found who it done, pbrx just create them at your local docker registry11:52
tobiashquiquell: ok11:52
openstackgerritTobias Henkel proposed openstack-infra/zuul master: Set relative priority of node requests  https://review.openstack.org/61535611:52
quiquelltobiash: Those images re only used by the job that test the admin example ?11:52
tobiashI think so11:53
quiquelltobiash: they don't have lavels with version or anything11:53
quiquelllabels11:53
tobiashcurrently we only push to latest on each change that gets merged11:53
quiquelltobiash: Do they still has some kind of hash number to get other than latest from dockerhub ?11:55
quiquelltobiash: like docker image hash11:55
tobiashquiquell: I don't know11:56
tobiashmordred: ^11:56
openstackgerritTobias Henkel proposed openstack-infra/nodepool master: Ensure that completed handlers are removed frequently  https://review.openstack.org/61002912:07
openstackgerritTobias Henkel proposed openstack-infra/zuul master: Remove nodeid argument from updateNode  https://review.openstack.org/62104712:11
*** pcaruana has quit IRC12:11
openstackgerritTobias Henkel proposed openstack-infra/zuul master: Report tenant and project specific resource usage stats  https://review.openstack.org/61630612:24
*** rfolco_doctor is now known as rfolco12:34
quiquelltobiash: another one, If I add a 'merger' node it will reduce latency starting jobs ?12:40
quiquelltobiash: right now I have only an executor12:40
*** jpena is now known as jpena|lunch12:41
tobiashquiquell: that depends on the queue of the mergers, typically it has no real impact on starting jobs, but it might improve reconfigure and startup times of the scheduler12:41
tobiashwell, that also depends on the number of executors you have12:42
quiquelltobiash: just one, as we are only running one job at a time12:42
tobiashquiquell: but it's best of you look at the merger queue statistics to judge on that12:42
quiquelltobiash: where is that ?12:42
quiquellstatd ?12:42
tobiashquiquell: you need to configure statsd reporting and run  e.g. a graphite/grafana or statsd_exporter/prometheus/grafana12:43
quiquelltobiash: ack, thanks !12:43
tobiashquiquell: http://grafana.openstack.org/d/T6vSHcSik/zuul-status?orgId=112:43
tobiashlook for the merger queue12:43
tobiashbut there are much more interesting metrics too12:44
quiquelltobiash: scheduler uses merger too ?12:44
tobiashyes, for config gathering and merge checks12:44
tobiashquiquell: running a job actually does the merging twice, the scheduler does it and gets the config from that and the executor does it again12:45
tobiashquiquell: the first one can be done by the mergers, the second can only be done directly on the executor12:46
quiquelltobiash: so the first one can be parallelize with multiple merger nodes ?12:46
quiquelltobiash: like merge per project or something like his ?12:46
tobiashquiquell: yes12:47
quiquelltobiash: cool nice12:47
tobiashquiquell: when zuul does a reconfiguration (e.g. when the scheduler starts) it gathers the config of all branches of all projects. This work is done and by the mergers.12:48
*** hashar has joined #zuul12:54
*** bjackman has quit IRC13:09
*** rlandy has joined #zuul13:36
pabelangercorvus: clarkb: mordred: do you mind a review and +3 on https://review.openstack.org/620989/ documentation update for executor-zone13:37
*** jpena|lunch is now known as jpena13:40
*** dkehn has quit IRC13:50
*** EmilienM is now known as EvilienM13:58
tobias-urdinhm when gating with zuul, it should try to merge my change but throws the "this change or one of it cross-repo deps, please rebase and upload new ps"14:00
tobias-urdinshouldn't it try to merge it by rebasing it automatically if there parent is no the same as the change?14:00
tobias-urdinthis is with noop jobs btw, but i was thinking should it take by change, rebase, run noop jobs and merge if successful14:01
pabelangertobias-urdin: are you using gerrit or github?14:03
tobias-urdingerrit14:06
tobias-urdinso 1. upload first change 2. upload second change (not with first change as parent) 3. merge first change 4. merge second change and it throws that14:07
tobias-urdinall with noop jobs if that makes any difference14:07
pabelangertobias-urdin: you should be able to check zuul logs to see what is happening, are you using the merge-check pipeline by change?14:07
pabelangerchance*14:07
tobias-urdini'll check sec14:07
pabelangerdoes gerrit report the 2 patches are in conflict of each other?14:08
tobias-urdinno, i can remove zuul's -2 and submit it manually14:08
mordredquiquell: no - we haven't done much with labels or tags for them yet - that's on the todo list14:08
mordredquiquell: (referring to the docker images)14:08
quiquellmordred: ack14:09
tobias-urdinpabelanger: scheduler logs says: zuul.Pipeline.binero.gate: Unable to merge change <Change 0x7fd124656c88 project-name-here 3774,1>14:11
pabelangertobias-urdin: http://git.openstack.org/cgit/openstack-infra/project-config/tree/zuul.d/projects.yaml / http://git.openstack.org/cgit/openstack-infra/openstack-zuul-jobs/tree/zuul.d/project-templates.yaml#n3 is an example of using merge-check pipeline, which should help to ensure all patches can be merged. Hopefully preventing that from happening it gate14:11
tobias-urdinhere is pipelines http://paste.openstack.org/show/736482/ should be pretty much same as openstack14:11
tobias-urdinhttp://paste.openstack.org/show/736482/14:11
pabelangertobias-urdin: does the zuul-merger show anything specific for that change?14:12
tobias-urdinchecking14:15
*** etp has quit IRC14:18
tobias-urdinpabelanger: sorry for the delay but i had to test with new commits, it works now, zuul-merger was not running :) thanks!14:21
*** etp has joined #zuul14:22
pabelangeryay14:24
tobias-urdinout of curiosity should i have the merge-check part? i only have a empty base job and noop jobs for all project right now until i start migrate away from jenkins14:25
*** dkehn has joined #zuul14:26
openstackgerritMerged openstack-infra/nodepool master: OpenStack: count leaked nodes in unmanaged quota  https://review.openstack.org/62104014:29
openstackgerritMerged openstack-infra/nodepool master: OpenStack: store ZK records for launch error nodes  https://review.openstack.org/62104314:29
pabelangerYay!^14:30
tobiashpabelanger, corvus: does that solve your problems from yesterday or do we still need a discussion about non fatal quota errors?14:35
*** bhavikdbavishi has joined #zuul14:39
*** pwhalen has joined #zuul14:40
*** bhavikdbavishi has quit IRC14:47
*** bhavikdbavishi has joined #zuul14:47
pabelangertobiash: I think it will solve my issue yes, I was able to clean up the ERROR nodes by manaually deleting them today. Now that nodepool will see them, I expect this issue to have been fixed14:48
tobiashok14:48
pabelangerwill have to wait until we tag a release and update SF.io nodepool, but will keep an eye on quota14:49
pabelangerthe good news is, our quota on the providers is very small, so it doesn't take much to max out14:49
pabelangerI pinged #openstack-infra to see if we are planning to restart nodepool today, in hopes of us tagging sooner14:50
tobiashpabelanger: a compromise could be that we don't pause on quota error but unwind and unlock the node-request. That way it could be taken by a different (or the same provider) again without failing the request completely.14:51
pabelangertobiash: ack, lets see what corvus / Shrews thinks14:52
Shrewspabelanger: think about what?14:55
pabelangerShrews: there was a discussion yesterday about how providers deal with nodes hitting quota errors, IIRC the provider will be paused, and not allow another provider to allocate the request14:59
pabelangerin my case, it wedged jobs.  However clarkb and corvus addressed the issue of bad quota in ^ above14:59
Shrewspabelanger: right, but those appear to have merged15:00
pabelangeryup, tobiash was just asking if we still needed to discuss it15:00
pabelangerI wasn't sure, and defering to experts :)15:00
pabelangerI've noticed a few times now that zuul is failing in gate with sql tests: http://logs.openstack.org/89/620989/1/gate/tox-py35/894c4e6/testr_results.html.gz15:08
pabelangerI thought it might be a slow node, but maybe not sure now15:08
*** chandan_kumar is now known as chkumar|off15:14
openstackgerritMerged openstack-infra/zuul master: Set relative priority of node requests  https://review.openstack.org/61535615:24
*** cdent has joined #zuul15:36
*** quiquell is now known as quiquell|off15:39
tobiashand we only get stringexception...15:39
corvuspabelanger: yeah, it's pret suspicious since we've been merging sql changes15:47
tobiashbut I think I saw these already before sometimes so maybe the merged sql changes made these tests running a little longer15:49
tobiashand they timout more often now15:49
corvusthat could certainly be it15:50
corvusthey don't use scenarios -- some of them have 4 sql connections, each of which does its own schema upgrades15:51
*** cdent has left #zuul15:56
*** j^2 has joined #zuul16:17
*** hashar has quit IRC16:23
*** pcaruana has joined #zuul16:26
*** bhavikdbavishi has quit IRC16:45
*** bhavikdbavishi has joined #zuul16:45
*** jpena is now known as jpena|off16:59
*** gtema has quit IRC17:04
*** pcaruana has quit IRC17:04
openstackgerritMerged openstack-infra/zuul master: Clarify executor zone documentation  https://review.openstack.org/62098917:24
*** njohnston has joined #zuul17:38
openstackgerritMerged openstack-infra/zuul master: Remove STATE_PENDING  https://review.openstack.org/62028417:38
*** openstackgerrit has quit IRC17:51
SpamapSHrm...18:20
SpamapSmy test for the AWS driver seems to be racey. :-/18:20
SpamapSGuessing it's because the instance starts _immediately_ instead of 5-10 seconds later as in reality. :-P18:21
*** dkehn_ has joined #zuul18:24
*** dkehn has quit IRC18:26
*** dkehn_ is now known as dkehn18:26
*** pwhalen has quit IRC18:26
tobiashsounds like you're having fun18:36
*** bhavikdbavishi has quit IRC18:41
*** electrofelix has quit IRC18:59
*** openstackgerrit has joined #zuul19:18
openstackgerritMonty Taylor proposed openstack-infra/nodepool master: Block 0.19.0 of openstacksdk  https://review.openstack.org/62127219:18
corvustobiash: ^19:20
pabelangercorvus: mordred: isn't 0.20.0 that is broken?19:20
corvusShrews seems to think so :)19:21
openstackgerritMonty Taylor proposed openstack-infra/nodepool master: Block 0.20.0 of openstacksdk  https://review.openstack.org/62127219:21
mordredpabelanger: yup19:21
mordredpabelanger: I mean - 0.20.0 works ... the problem is that it works too well19:22
pabelangerack19:22
corvusShrews, clarkb, pabelanger, mordred: i'm going to direct-enqueue that.  it means we won't see the devstack functional test results for it, but i think we can risk it.  agree?19:23
Shrewscorvus: yeah19:23
clarkbyes it is just a requirements change we'll know from unittests if things fail to install19:23
corvusand it ran with that version previously :)19:24
mordredyah19:24
Shrews3 +A is super-mega-approval?19:26
tobiashlooks like y'all are having fun this friday19:27
Shrewstobiash: are you running with the new caching stuff in production?19:27
Shrewsor at least a dev env?19:27
tobiashShrews: yes, since weeks19:27
Shrewstobiash: cool19:27
tobiashI run most of my patches that are in review in production19:30
mordredShrews: I believe tobiash would like for us to try out the port cleanup patch revert revert in our production this time though :)19:31
tobiashlol19:31
tobiash;)19:31
tobiashbtw, we now reached roundabout a third of the size of the openstack deployment19:33
clarkbtobiash: nice, its great to see the tool scale up for others too (since that is one of the benefits we try to convince others of)19:35
corvusnow that crazy-travel-season is over, i'd like to send out a project status email next week; can folks update https://etherpad.openstack.org/p/zuul-update-email with what they're up to?19:47
corvusi'll probably send it on tuesday to give folks plenty of time19:47
corvusShrews, pabelanger, tristanC, mordred, tobiash, SpamapS, everyone else: ^19:47
mordredcorvus: I assume "stuff" isn't the right answer?19:48
Shrewsis "vacation" worth noting  :)19:48
corvusmordred: yes, we *do* give out "participation" awards.  ;)19:48
tobiashcorvus: do you like to start fresh with this week or rather have a summary since the last status?19:49
corvustobiash: mostly focused on current activities i think (but if there's still an ongoing effort we haven't mentioned, we should include that)19:50
tobiashok19:50
clarkbmight also be good to push the governance thing? I think people are antsy for that post summit19:51
corvusi think it mostly helps people keep up with what's going on and how they can participate; doesn't need to be a changelog19:51
corvusclarkb: yep, i'll start on that next week too19:51
corvus(it didn't feel like we had a quorum at the beginningn of the week, so i deferred)19:51
ssbarnea|rovercan zuul trick for displaying live output when running ansible tasks can be used with normal (cli) ansible or not really?19:53
tobiashhrm, 616306 conflicts already the second time today :/19:53
tobiashssbarnea|rover: in theory yes, but in practice this could be tricky19:54
ssbarnea|rovertobiah: i have ipmi based provisioning and I want to tail the serial-console (sol) till the host is up and running.19:55
ssbarnea|roverrunning something for ~10 minutes which reboots the machine twice without any feedback is far from optimal experience.19:55
mordredssbarnea|rover: I have a todo list task to port it in to ansible proper19:57
mordredssbarnea|rover: but before that, there is a half-finished project to rework how it works in zuul - so that it might be more acceptable in ansible itself19:57
mordred(it's a bit hacky and very zuul-specific right now)19:57
tobiashssbarnea|rover: it currently only works with shell and command tasks and ssh connection19:58
ssbarnea|rovermordred: ok, in this case I will wait for you. ping me if you need beta testing. i will just save the output to a file for the moment.19:58
ssbarnea|roverfor me is more than enough, i only needed shell.19:58
mordredssbarnea|rover: thanks! will do19:58
ssbarnea|rovermordred: if you manage to make it work you could become (more) famous! that being the #1 ansible feature request.19:59
mordredssbarnea|rover: I don't know if that's motivation or de-motivation :)20:00
openstackgerritTobias Henkel proposed openstack-infra/zuul master: Report tenant and project specific resource usage stats  https://review.openstack.org/61630620:03
openstackgerritTobias Henkel proposed openstack-infra/zuul master: Report tenant and project specific resource usage stats  https://review.openstack.org/61630620:04
openstackgerritMerged openstack-infra/nodepool master: Block 0.20.0 of openstacksdk  https://review.openstack.org/62127220:08
tobiashmordred: I'd be happy about a second review on 590761 (should be an easy one)20:22
mordredtobiash: done!20:26
tobiashmordred: thank you :)20:26
Shrewstobiash: i think the "Updating stats" debug log message happens a little more than i like. maybe we should drop that?20:27
tobiashShrews: fine for me20:28
openstackgerritTobias Henkel proposed openstack-infra/nodepool master: Remove updating stats debug log  https://review.openstack.org/62128320:29
tobiashShrews, pabelanger ^20:29
*** openstackgerrit has quit IRC20:50
*** openstackgerrit has joined #zuul20:50
openstackgerritTobias Henkel proposed openstack-infra/nodepool master: Fix leak detection in unmanaged quota calculation  https://review.openstack.org/62128620:50
*** rlandy has quit IRC21:08
openstackgerritJames E. Blair proposed openstack-infra/nodepool master: Log exceptions in cache listener events  https://review.openstack.org/62129221:28
openstackgerritMerged openstack-infra/nodepool master: Fix leak detection in unmanaged quota calculation  https://review.openstack.org/62128621:34
openstackgerritJames E. Blair proposed openstack-infra/nodepool master: Log exceptions in cache listener events  https://review.openstack.org/62129221:48
openstackgerritMerged openstack-infra/nodepool master: Log exceptions in cache listener events  https://review.openstack.org/62129222:17
openstackgerritJames E. Blair proposed openstack-infra/nodepool master: Log exceptions deleting ZK nodes  https://review.openstack.org/62130122:21
openstackgerritTobias Henkel proposed openstack-infra/nodepool master: Don't update caches with empty zNodes  https://review.openstack.org/62130522:28
*** njohnston has quit IRC22:49
openstackgerritJames E. Blair proposed openstack-infra/nodepool master: Add relative priority to request list  https://review.openstack.org/62131423:06
openstackgerritJames E. Blair proposed openstack-infra/zuul master: Only count live items for relative priority  https://review.openstack.org/62131523:09
openstackgerritMerged openstack-infra/nodepool master: Remove updating stats debug log  https://review.openstack.org/62128323:28
*** threestrands has joined #zuul23:28

Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!