| *** calebb has joined #openstack-infra | 00:01 | |
| *** markvoelker has quit IRC | 00:08 | |
| *** dingyichen has joined #openstack-infra | 00:10 | |
| openstackgerrit | Merged openstack-infra/irc-meetings master: Update docs team mtg time and channel https://review.openstack.org/515783 | 00:15 |
|---|---|---|
| *** thorst has joined #openstack-infra | 00:18 | |
| *** fultonj has joined #openstack-infra | 00:19 | |
| *** thorst has quit IRC | 00:20 | |
| *** sflanigan has joined #openstack-infra | 00:21 | |
| *** sflanigan has joined #openstack-infra | 00:21 | |
| *** bobh has joined #openstack-infra | 00:30 | |
| *** gildub has joined #openstack-infra | 00:40 | |
| *** Jeffrey4l has quit IRC | 00:42 | |
| *** Jeffrey4l has joined #openstack-infra | 00:42 | |
| *** salv-orlando has joined #openstack-infra | 00:46 | |
| openstackgerrit | Merged openstack-infra/tripleo-ci master: Remove dlrnapi report at the end of the run https://review.openstack.org/512590 | 00:50 |
| *** salv-orlando has quit IRC | 00:51 | |
| *** huanxie has joined #openstack-infra | 00:51 | |
| mnaser | how long does it take for a dependent change to get queue'd into the gate | 00:51 |
| *** LindaWang has joined #openstack-infra | 00:51 | |
| mnaser | changes in question =- https://review.openstack.org/#/q/topic:bindep/puppet+branch:stable/pike+label:Verified%252B1 | 00:52 |
| *** cuongnv has joined #openstack-infra | 01:02 | |
| *** markvoelker has joined #openstack-infra | 01:05 | |
| *** catintheroof has joined #openstack-infra | 01:08 | |
| *** kiennt26 has joined #openstack-infra | 01:09 | |
| *** bobh has quit IRC | 01:10 | |
| openstackgerrit | Merged openstack/diskimage-builder master: Zuul: add file extension to playbook path https://review.openstack.org/516129 | 01:14 |
| *** daidv has joined #openstack-infra | 01:15 | |
| *** liusheng has quit IRC | 01:20 | |
| *** xingchao has joined #openstack-infra | 01:21 | |
| *** catintheroof has quit IRC | 01:22 | |
| *** liujiong has joined #openstack-infra | 01:29 | |
| *** markvoelker has quit IRC | 01:38 | |
| *** namnh has joined #openstack-infra | 01:38 | |
| *** zzzeek has quit IRC | 01:40 | |
| openstackgerrit | Ian Wienand proposed openstack-infra/system-config master: Add new backup hosts to puppet https://review.openstack.org/516148 | 01:42 |
| openstackgerrit | Ian Wienand proposed openstack-infra/system-config master: Backup zuulv3 with bup https://review.openstack.org/516157 | 01:42 |
| *** xingchao has quit IRC | 01:45 | |
| ianw | infra-root: ^ i've just kind of run with the idea of a new backup server. it's currently rsyncing the old /opt/backups ... i figured it would not be a bad idea to actually read over all the data. i also doulbed the inodes on the new /opt/backups (it wasn't even close, but it seems worth it) | 01:45 |
| *** xingchao has joined #openstack-infra | 01:47 | |
| *** daidv has quit IRC | 01:50 | |
| *** daidv has joined #openstack-infra | 01:51 | |
| openstackgerrit | Ian Wienand proposed openstack-infra/system-config master: Add new backup host to puppet https://review.openstack.org/516148 | 01:53 |
| openstackgerrit | Ian Wienand proposed openstack-infra/system-config master: Backup zuulv3 with bup https://review.openstack.org/516157 | 01:53 |
| *** xingchao has quit IRC | 01:56 | |
| *** gildub has quit IRC | 01:58 | |
| *** xingchao has joined #openstack-infra | 02:00 | |
| openstackgerrit | Ian Wienand proposed openstack-infra/system-config master: [DNM] remove ci-backup-rs-ord.openstack.org https://review.openstack.org/516159 | 02:04 |
| *** bobh has joined #openstack-infra | 02:06 | |
| *** thorst has joined #openstack-infra | 02:21 | |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/os-testr master: Updated from global requirements https://review.openstack.org/503645 | 02:22 |
| *** namnh has quit IRC | 02:23 | |
| *** daidv has quit IRC | 02:23 | |
| *** daidv has joined #openstack-infra | 02:24 | |
| *** namnh has joined #openstack-infra | 02:24 | |
| *** thorst has quit IRC | 02:25 | |
| *** liusheng has joined #openstack-infra | 02:31 | |
| openstackgerrit | Ian Wienand proposed openstack-infra/system-config master: puppetmaster: rotate /var/log/puppet_run_all_infracloud.log https://review.openstack.org/516165 | 02:33 |
| *** markvoelker has joined #openstack-infra | 02:35 | |
| *** mriedem has quit IRC | 02:38 | |
| openstackgerrit | Ian Wienand proposed openstack-infra/system-config master: Add new backup host to puppet https://review.openstack.org/516148 | 02:42 |
| openstackgerrit | Ian Wienand proposed openstack-infra/system-config master: Backup zuulv3 with bup https://review.openstack.org/516157 | 02:42 |
| openstackgerrit | Ian Wienand proposed openstack-infra/system-config master: [DNM] remove ci-backup-rs-ord.openstack.org https://review.openstack.org/516159 | 02:42 |
| *** felipemonteiro_ has joined #openstack-infra | 02:44 | |
| *** salv-orlando has joined #openstack-infra | 02:47 | |
| *** salv-orlando has quit IRC | 02:51 | |
| *** annp has joined #openstack-infra | 02:53 | |
| *** xingchao has quit IRC | 03:03 | |
| *** gongysh has joined #openstack-infra | 03:06 | |
| *** bobh has quit IRC | 03:08 | |
| *** markvoelker has quit IRC | 03:09 | |
| *** ramishra has joined #openstack-infra | 03:12 | |
| ramishra | hi guys, can someone look at https://review.openstack.org/#/c/515734 please, would unblock heat gate | 03:15 |
| ramishra | ianw: Hi! ^^^ | 03:15 |
| *** gildub has joined #openstack-infra | 03:18 | |
| *** namnh has quit IRC | 03:18 | |
| *** daidv has quit IRC | 03:18 | |
| *** daidv has joined #openstack-infra | 03:19 | |
| *** namnh has joined #openstack-infra | 03:19 | |
| *** felipemonteiro_ has quit IRC | 03:21 | |
| openstackgerrit | sebastian marcet proposed openstack-infra/openstackid-resources master: Error on Google Facade https://review.openstack.org/516174 | 03:32 |
| openstackgerrit | Merged openstack-infra/openstackid-resources master: Error on Google Facade https://review.openstack.org/516174 | 03:32 |
| *** daidv has quit IRC | 03:40 | |
| *** daidv has joined #openstack-infra | 03:41 | |
| *** xingchao has joined #openstack-infra | 03:43 | |
| *** pahuang has quit IRC | 03:44 | |
| *** udesale has joined #openstack-infra | 03:45 | |
| ianw | ramishra: it is no longer configurable? or just mysql is removed? | 03:46 |
| *** rcernin has quit IRC | 03:47 | |
| *** rcernin_ has joined #openstack-infra | 03:47 | |
| *** dave-mccowan has quit IRC | 03:48 | |
| SpamapS | ianw: it's gone IIRC | 03:48 |
| *** pahuang has joined #openstack-infra | 03:52 | |
| *** gildub has quit IRC | 03:54 | |
| ianw | from a bit of archaeology it's unclear why it's set there anyway, so removal LGTM, but should be double checked | 03:54 |
| *** xingchao has quit IRC | 04:01 | |
| ramishra | ianw: default backend is gnocchi, mysql is not there amymore | 04:01 |
| ianw | right, i was just more worried about pulling out the variable if people still use it, seems like "none" is still a valid option too | 04:02 |
| ramishra | ianw: I don't know the history why it was overidden in devstack-gate (may be mongo was not a suitble option), someone can set it directly in jobs without using the varibale here though | 04:05 |
| ianw | right, which is the conclusion i came too in my comment as well ;) | 04:06 |
| *** markvoelker has joined #openstack-infra | 04:06 | |
| ramishra | ianw: OK:) | 04:07 |
| *** ykarel has joined #openstack-infra | 04:10 | |
| *** namnh has quit IRC | 04:13 | |
| *** daidv has quit IRC | 04:13 | |
| *** daidv has joined #openstack-infra | 04:14 | |
| *** namnh has joined #openstack-infra | 04:14 | |
| *** Dinesh_Bhor has joined #openstack-infra | 04:14 | |
| *** armaan has joined #openstack-infra | 04:19 | |
| *** thorst has joined #openstack-infra | 04:22 | |
| *** thorst has quit IRC | 04:27 | |
| *** rkukura has quit IRC | 04:27 | |
| *** huanxie has quit IRC | 04:27 | |
| *** dsariel_ has quit IRC | 04:30 | |
| *** janki has joined #openstack-infra | 04:35 | |
| *** armaan has quit IRC | 04:37 | |
| *** markvoelker has quit IRC | 04:39 | |
| *** gongysh has quit IRC | 04:43 | |
| *** salv-orlando has joined #openstack-infra | 04:48 | |
| *** salv-orlando has quit IRC | 04:52 | |
| *** jaosorior has joined #openstack-infra | 04:54 | |
| *** huanxie has joined #openstack-infra | 04:58 | |
| *** udesale__ has joined #openstack-infra | 04:59 | |
| *** udesale has quit IRC | 04:59 | |
| *** armaan has joined #openstack-infra | 04:59 | |
| *** wolverineav has quit IRC | 05:17 | |
| *** wolverineav has joined #openstack-infra | 05:18 | |
| *** wolverin_ has joined #openstack-infra | 05:25 | |
| *** wolverineav has quit IRC | 05:25 | |
| *** dhinesh has joined #openstack-infra | 05:27 | |
| *** markvoelker has joined #openstack-infra | 05:37 | |
| *** gongysh has joined #openstack-infra | 05:41 | |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/openstack-zuul-jobs master: Remove obsolete infra-publications jobs https://review.openstack.org/516011 | 05:45 |
| *** dsariel_ has joined #openstack-infra | 05:45 | |
| *** cshastri has joined #openstack-infra | 05:46 | |
| *** threestrands_ has joined #openstack-infra | 05:47 | |
| *** zhurong has joined #openstack-infra | 05:48 | |
| openstackgerrit | Merged openstack-infra/devstack-gate master: Don't set CEILOMETER_BACKEND in localrc https://review.openstack.org/515734 | 05:48 |
| *** threestrands has quit IRC | 05:50 | |
| *** otherwiseguy has quit IRC | 05:51 | |
| *** rnoriega has quit IRC | 05:52 | |
| *** haleyb has quit IRC | 05:53 | |
| *** wolverin_ has quit IRC | 05:59 | |
| *** dhinesh has quit IRC | 05:59 | |
| *** huanxie has quit IRC | 06:01 | |
| *** Kevin_Zheng has quit IRC | 06:01 | |
| kazsh | Hi team, could you someone help me out ? I got an zuul error "Unable to freeze job graph: Job requirements-check does not specify a run playbook" https://review.openstack.org/#/c/515591/ suppose this merge https://review.openstack.org/#/c/514625/ affects us.. before this merge, we could make it well.. :/ | 06:02 |
| *** slaweq has quit IRC | 06:05 | |
| *** slaweq has joined #openstack-infra | 06:06 | |
| *** salv-orlando has joined #openstack-infra | 06:08 | |
| *** markvoelker has quit IRC | 06:10 | |
| *** slaweq has quit IRC | 06:10 | |
| SamYaple | kazsh: just add a noop job with zuul | 06:14 |
| *** udesale__ has quit IRC | 06:14 | |
| SamYaple | kazsh: paste.openstack.org/show/624929/ | 06:16 |
| SamYaple | thats assuming there is nothing in project-config breaking it | 06:17 |
| SamYaple | didnt check yet | 06:18 |
| kazsh | SamYaple: thanks a lot! I got it your point but what I want to do is to run jobs... | 06:19 |
| SamYaple | kazsh: well you have to write them :) | 06:19 |
| SamYaple | what are you trying to run? ive got a fewminutes to get you a simple job run | 06:20 |
| kazsh | project config already has my change for heat-dashboard... I just want to run those :) | 06:21 |
| openstackgerrit | Chandan Kumar proposed openstack-infra/project-config master: Add post tarball gate to kuryr-tempest-plugin https://review.openstack.org/516186 | 06:21 |
| kazsh | SamYaple: http://git.openstack.org/cgit/openstack-infra/project-config/tree/zuul.d/projects.yaml | 06:21 |
| * SamYaple sighs | 06:21 | |
| SamYaple | ok good :) | 06:21 |
| SamYaple | was about to divein | 06:21 |
| SamYaple | i wonder if you are breaking because no .gitreview? | 06:22 |
| kazsh | ah yea I removed it temporary to check out the change on my local.. | 06:23 |
| *** thorst has joined #openstack-infra | 06:23 | |
| kazsh | oh my fault :( | 06:23 |
| openstackgerrit | Chandan Kumar proposed openstack-infra/project-config master: Add post tarball gate to magnum-tempest-plugin https://review.openstack.org/516187 | 06:23 |
| SamYaple | kazsh: nah i see even in your PS7 you had it sameerror | 06:24 |
| SamYaple | this might be related to the playbook file extentions change jeblairs been pushing | 06:24 |
| openstackgerrit | Chandan Kumar proposed openstack-infra/project-config master: Add post tarball gate to zaqar-tempest-plugin https://review.openstack.org/516186 | 06:24 |
| kazsh | SamYaple: oh ok, but for now I rollbacked it. | 06:25 |
| kazsh | & got same.. | 06:25 |
| SamYaple | yea you definetely need that, but not your issue i think | 06:25 |
| *** jtomasek has joined #openstack-infra | 06:26 | |
| openstackgerrit | Chandan Kumar proposed openstack-infra/project-config master: Add post tarball gate to neutron-tempest-plugin https://review.openstack.org/516188 | 06:27 |
| *** gildub has joined #openstack-infra | 06:27 | |
| *** krtaylor has quit IRC | 06:27 | |
| *** thorst has quit IRC | 06:27 | |
| *** hamzy has quit IRC | 06:28 | |
| *** jbadiapa has joined #openstack-infra | 06:29 | |
| SamYaple | check-requirements is the actual job.... where is requirements-check coming from | 06:29 |
| SamYaple | ah i see. just some confusing names. https://github.com/openstack/requirements/blob/master/.zuul.d/project-template.yaml#L10 | 06:31 |
| kazsh | yes it is | 06:31 |
| gongysh | who knows how zuul writes the devstack plugin orders? | 06:32 |
| SamYaple | kazsh: good(?) news is its not your repo | 06:32 |
| SamYaple | kazsh: https://github.com/openstack-infra/zuul-jobs/blob/fa896e2125cbf47a5fa8338b2bd819a72179ea58/zuul.yaml#L4 | 06:32 |
| SamYaple | kazsh: change would be needed to requirments repo to specify a run job | 06:33 |
| SamYaple | kazsh: trying to find when this broke | 06:33 |
| kazsh | SamYaple: thank you ! suppose other projects also are being affected.. | 06:36 |
| SamYaple | kazsh: testing that now | 06:36 |
| kazsh | :) | 06:36 |
| *** dhajare|mtg has joined #openstack-infra | 06:36 | |
| *** slaweq has joined #openstack-infra | 06:37 | |
| SamYaple | hmmmmm it seems not.. | 06:37 |
| kazsh | oh | 06:37 |
| SamYaple | still working through it. im not an expert, im just around :) | 06:38 |
| kazsh | no worries, thank you very much for your prompt work, really appreciated ;) | 06:38 |
| SamYaple | requirements very clearly doesnt specify a run playbook, https://github.com/openstack/requirements/blob/master/.zuul.d/project-template.yaml#L10 | 06:39 |
| SamYaple | and niether does its parent, unnittests https://github.com/openstack-infra/zuul-jobs/blob/fa896e2125cbf47a5fa8338b2bd819a72179ea58/zuul.yaml#L4 | 06:39 |
| SamYaple | so im a bit lost on how this is working out for other jobs | 06:40 |
| SamYaple | s/jobs/projects | 06:40 |
| *** gildub has quit IRC | 06:41 | |
| *** hamzy has joined #openstack-infra | 06:42 | |
| *** krtaylor has joined #openstack-infra | 06:43 | |
| SamYaple | kazsh: i feel this might be beyond me and more infra-root folks might be needed to help | 06:45 |
| *** salv-orlando has quit IRC | 06:46 | |
| SamYaple | i dont understand how any projects using that template areworking with thelack of a run plabook anywhere | 06:46 |
| AJaeger | kazsh, SamYaple, somehow we indeed broke this ;( | 06:48 |
| AJaeger | jeblair: getting the same on https://review.openstack.org/516195 | 06:48 |
| *** salv-orlando has joined #openstack-infra | 06:48 | |
| SamYaple | awesome. im not crazy | 06:48 |
| kazsh | :) | 06:48 |
| SamYaple | bad news is git blaming isnt helping because neither of those jobs have been modified recently | 06:48 |
| SamYaple | also AJaeger https://review.openstack.org/#/c/516191/ that started running jobs before i abanonded it | 06:49 |
| SamYaple | so im not sure exactly what is happening | 06:49 |
| AJaeger | and indeed requirements-check is broken, it tests nothing ;( | 06:50 |
| *** threestrands_ has quit IRC | 06:51 | |
| SamYaple | ahh i found it | 06:51 |
| SamYaple | https://review.openstack.org/#/c/516191/ | 06:51 |
| SamYaple | ohly breaks when modifying the requirements files | 06:51 |
| SamYaple | ok that makes more sense | 06:51 |
| AJaeger | kazsh: add to your change "Depends-On: I4b1140c20c68821eaf0849c9ee551ff9b1c27deb" and let's see whether it then works | 06:52 |
| AJaeger | tonyb, dirk, prometheanfire, please review https://review.openstack.org/516197 Actually run check-requirements | 06:53 |
| kazsh | I see, will do that | 06:53 |
| AJaeger | kazsh, SamYaple 516197 should be the fix | 06:53 |
| SamYaple | AJaeger: any idea of how this breakage occured? | 06:53 |
| SamYaple | that job hasnt been modifed in 10 days | 06:54 |
| SamYaple | neither has the template from what ican tell | 06:54 |
| AJaeger | SamYaple: stricter testing in zuul | 06:54 |
| AJaeger | SamYaple: so, it was always broken, we just never noticed ;( | 06:54 |
| SamYaple | ah. so zuul internals. well i feel better about my troubleshooting skills :) | 06:54 |
| SamYaple | i was really worried for a moment | 06:54 |
| kazsh | SamYaple: you are the expert from me :) | 06:56 |
| * SamYaple is scared | 06:57 | |
| SamYaple | what have i done :O | 06:57 |
| SamYaple | good luck kazsh! | 06:57 |
| kazsh | thanks mate ;) | 06:58 |
| *** spectr has joined #openstack-infra | 07:02 | |
| *** florianf has joined #openstack-infra | 07:06 | |
| *** jascott1 has joined #openstack-infra | 07:07 | |
| *** markvoelker has joined #openstack-infra | 07:07 | |
| tonyb | AJaeger: hehe, I did ask others for some testign to indicate it worked. | 07:08 |
| AJaeger | tonyb: you can self-test nowadays ;) https://review.openstack.org/516195 is one way of testing it... | 07:09 |
| tonyb | AJaeger: I get that. | 07:09 |
| *** slaweq has quit IRC | 07:10 | |
| kazsh | AJaeger, SamYaple: looks check-requirements recovered, got +1 from zuul https://review.openstack.org/#/c/515591/ | 07:11 |
| AJaeger | kazsh: and looking at the json output, it passed - see my comments on https://review.openstack.org/#/c/516197 | 07:13 |
| AJaeger | kazsh: glad to hear! | 07:13 |
| ykarel | AJaeger, can you confirm zuulv3 stable branch patch: https://review.openstack.org/#/c/514272/ | 07:15 |
| *** slaweq has joined #openstack-infra | 07:16 | |
| kazsh | AJaeger: thanks a lot as always ! | 07:16 |
| AJaeger | kazsh: thanks to SamYaple for good debugging! | 07:17 |
| *** rcernin_ has quit IRC | 07:17 | |
| kazsh | SamYaple: ;) appreciated your prompt effort to my problem, I was really happy to hear your response actually. | 07:19 |
| *** aviau has quit IRC | 07:20 | |
| *** aviau has joined #openstack-infra | 07:20 | |
| SamYaple | kazsh: glad to hear it! | 07:24 |
| chandankumar | AJaeger: hello | 07:25 |
| AJaeger | morning, chandankumar | 07:25 |
| chandankumar | AJaeger: Good morning | 07:25 |
| chandankumar | AJaeger: Regarding this review https://review.openstack.org/#/c/507038/ | 07:25 |
| AJaeger | chandankumar: what's unclear in my message? | 07:26 |
| chandankumar | AJaeger: i need to add 3 tempest jobs against neutron to test newly neutron tempest plugin and the same job will run for neutron as well as neutron tempest plugin | 07:26 |
| chandankumar | AJaeger: will i add the jobs with in neutron repo or openstack-zuul-jobs? | 07:26 |
| AJaeger | chandankumar: neutron owned repo | 07:27 |
| AJaeger | You can even add them to neutron-tempest-plugin | 07:27 |
| AJaeger | chandankumar: we have global namespace, so define them somewhere | 07:27 |
| AJaeger | The important part is not where to define jobs - you need to add the trigger in neutron's project section and that lives in neutron and project-config | 07:28 |
| chandankumar | AJaeger: ok, let me create the reviews. | 07:28 |
| chandankumar | AJaeger: Thanks :-) | 07:28 |
| AJaeger | So, create jobs in either repo - and add the project stanza in neutron, possible with a depends-on | 07:28 |
| AJaeger | chandankumar: if you have questions, just leave a message here and somebody might answer later... | 07:29 |
| AJaeger | chandankumar: and please abandon the project-config one ;) | 07:30 |
| *** tesseract has joined #openstack-infra | 07:33 | |
| *** wolverineav has joined #openstack-infra | 07:34 | |
| *** salv-orlando has quit IRC | 07:35 | |
| *** salv-orlando has joined #openstack-infra | 07:36 | |
| chandankumar | AJaeger: sure | 07:36 |
| *** gildub has joined #openstack-infra | 07:37 | |
| AJaeger | chandankumar: if you add the job definition in neutron repo, add the trigger as well in neutron repo. I'm happy to review if needed. You can just add this single job there for now... | 07:38 |
| *** vsaienk0 has joined #openstack-infra | 07:39 | |
| *** markvoelker has quit IRC | 07:40 | |
| *** aarefiev_afk is now known as aarefiev | 07:41 | |
| *** wolverineav has quit IRC | 07:41 | |
| *** gildub has quit IRC | 07:49 | |
| *** hashar has joined #openstack-infra | 07:49 | |
| AJaeger | infra-root, review.o.o throws 500s and unreachable errors. | 07:51 |
| SamYaple | can confirm | 07:53 |
| *** yamamoto has quit IRC | 07:55 | |
| *** pow_2_30 is now known as gibi | 07:57 | |
| AJaeger | and working again... strange | 07:58 |
| *** salv-orlando has quit IRC | 07:59 | |
| *** salv-orlando has joined #openstack-infra | 08:00 | |
| *** ykarel is now known as ykarel|lunch | 08:00 | |
| *** salv-orl_ has joined #openstack-infra | 08:02 | |
| *** salv-orlando has quit IRC | 08:04 | |
| *** gongysh has quit IRC | 08:05 | |
| *** pgadiya has joined #openstack-infra | 08:10 | |
| *** wolverineav has joined #openstack-infra | 08:14 | |
| *** ccamacho has joined #openstack-infra | 08:14 | |
| *** alexchadin has joined #openstack-infra | 08:15 | |
| *** wolverineav has quit IRC | 08:18 | |
| *** gildub has joined #openstack-infra | 08:22 | |
| *** slaweq has quit IRC | 08:22 | |
| *** rcernin has joined #openstack-infra | 08:22 | |
| *** thorst has joined #openstack-infra | 08:24 | |
| *** huanxie has joined #openstack-infra | 08:27 | |
| *** Liced has joined #openstack-infra | 08:27 | |
| *** jpena|off is now known as jpena | 08:27 | |
| *** udesale has joined #openstack-infra | 08:27 | |
| *** thorst has quit IRC | 08:28 | |
| *** armaan_ has joined #openstack-infra | 08:28 | |
| *** armaan__ has joined #openstack-infra | 08:30 | |
| *** armaan has quit IRC | 08:30 | |
| *** arxcruz|pto is now known as arxcruz | 08:30 | |
| ramishra | it seems patches are stuck in the check queue after finishing all jobs, probably some infra issue | 08:31 |
| ramishra | ianw: if you're still around ^^^ | 08:32 |
| AJaeger | ramishra: "Queue lengths: 855 events, 0 management events, 486 results." on the top, it's going of the queue right now. Please have patience | 08:32 |
| chandankumar | AJaeger: regarding this review https://review.openstack.org/#/c/516188/ anything to do more from myside | 08:32 |
| *** salv-orl_ has quit IRC | 08:32 | |
| AJaeger | chandankumar: no - but I'm not riewing currently... | 08:32 |
| chandankumar | AJaeger: no problem. | 08:32 |
| *** salv-orlando has joined #openstack-infra | 08:33 | |
| *** salv-orlando has quit IRC | 08:33 | |
| *** salv-orlando has joined #openstack-infra | 08:33 | |
| *** armaan_ has quit IRC | 08:33 | |
| *** d0ugal has quit IRC | 08:34 | |
| *** dingyichen has quit IRC | 08:35 | |
| *** d0ugal has joined #openstack-infra | 08:35 | |
| *** d0ugal has quit IRC | 08:35 | |
| *** d0ugal has joined #openstack-infra | 08:35 | |
| ramishra | AJaeger: do you see the results count reducing? I see it steadily increasing | 08:35 |
| *** d0ugal has quit IRC | 08:35 | |
| AJaeger | ramishra: it first increases... | 08:35 |
| AJaeger | ramishra: it collects all events Zuul has to handle and that grows | 08:36 |
| *** slaweq has joined #openstack-infra | 08:36 | |
| *** stakeda has joined #openstack-infra | 08:36 | |
| *** markvoelker has joined #openstack-infra | 08:37 | |
| ramishra | AJaeger: OK, but I see patches stuck for 30 mins after finishing. If that's expected then it's ok | 08:37 |
| *** Hal has joined #openstack-infra | 08:37 | |
| AJaeger | ramishra: as long as that count is going up, it's expected | 08:37 |
| *** Hal is now known as Guest548 | 08:38 | |
| Liced | Hi everyone. I'm asking if there is not a issue with translation job in zuul v3. Translation setup has been added for networking_bgpvpn https://review.openstack.org/#/c/509178/ and https://review.openstack.org/#/c/486349/ but the .pot file is never generated for zanata | 08:38 |
| AJaeger | Liced: where do you expect .pot files to be generated? | 08:39 |
| *** d0ugal has joined #openstack-infra | 08:40 | |
| *** d0ugal has quit IRC | 08:40 | |
| *** d0ugal has joined #openstack-infra | 08:40 | |
| *** d0ugal has quit IRC | 08:40 | |
| *** evrardjp has quit IRC | 08:40 | |
| *** evrardjp has joined #openstack-infra | 08:41 | |
| AJaeger | Liced: the pot files are synced as post jobs. So, when you merge a change on master branch, it will upload to our translation server. When did the changes merge and when was last merge for master in the repo? | 08:41 |
| Liced | AJaeger: I'm expecting a directory here http://tarballs.openstack.org/translation-source/ The last change was 10 days ago. | 08:42 |
| *** eyalb has joined #openstack-infra | 08:42 | |
| AJaeger | Liced: Ah, 10 days - that was when our translation update was broken. It's working since just after that. So, get another change merged to the repo and you should be fine ;) | 08:43 |
| Liced | ok | 08:43 |
| Liced | thanks | 08:44 |
| AJaeger | ramishra: it indeed works long, let's see. If an admin is around, it might be worth checking, otherwise patience ;) | 08:45 |
| *** pgadiya has quit IRC | 08:45 | |
| AJaeger | bbl | 08:46 |
| *** gongysh has joined #openstack-infra | 08:46 | |
| *** ccamacho has quit IRC | 08:48 | |
| *** d0ugal has joined #openstack-infra | 08:48 | |
| *** d0ugal has quit IRC | 08:48 | |
| *** d0ugal has joined #openstack-infra | 08:48 | |
| *** ccamacho has joined #openstack-infra | 08:48 | |
| *** alexchadin has quit IRC | 08:49 | |
| *** ociuhandu has joined #openstack-infra | 08:55 | |
| *** yamamoto has joined #openstack-infra | 08:55 | |
| *** pgadiya has joined #openstack-infra | 08:58 | |
| *** ykarel|lunch is now known as ykarel | 09:00 | |
| *** ifat_afek has joined #openstack-infra | 09:03 | |
| *** dhajare|mtg has quit IRC | 09:03 | |
| *** alexchadin has joined #openstack-infra | 09:04 | |
| *** yamamoto has quit IRC | 09:04 | |
| *** cshastri has quit IRC | 09:06 | |
| *** slaweq has quit IRC | 09:07 | |
| *** markvoelker has quit IRC | 09:10 | |
| openstackgerrit | Tovin Seven proposed openstack-infra/openstack-zuul-jobs master: Remove legacy pyeclib job https://review.openstack.org/514513 | 09:12 |
| openstackgerrit | Jens Harbott (frickler) proposed openstack-infra/project-config master: Unblock designate-dashboard gate https://review.openstack.org/516223 | 09:12 |
| eyalb | Hi, I have a review from the vitrage project https://review.openstack.org/#/c/516093/ but I don't see it in the zuul check column did recheck and nothing happened are there any issues with zuul ? | 09:12 |
| *** yamamoto has joined #openstack-infra | 09:13 | |
| *** tosky has joined #openstack-infra | 09:14 | |
| *** slaweq has joined #openstack-infra | 09:15 | |
| frickler | eyalb: zuul looks to be extremely slow at processing events, maybe even stuck, we will need to wait for some infra-root to show up and check in detail | 09:16 |
| eyalb | frickler: thanks | 09:17 |
| *** evrardjp has quit IRC | 09:22 | |
| *** evrardjp has joined #openstack-infra | 09:23 | |
| openstackgerrit | Jesse Pretorius (odyssey4me) proposed openstack-infra/openstack-zuul-jobs master: Remove remaining legacy openstack-ansible tests https://review.openstack.org/512635 | 09:24 |
| *** alexchadin has quit IRC | 09:25 | |
| *** alexchadin has joined #openstack-infra | 09:25 | |
| openstackgerrit | Jesse Pretorius (odyssey4me) proposed openstack-infra/openstack-zuul-jobs master: Remove remaining legacy openstack-ansible tests https://review.openstack.org/512635 | 09:25 |
| openstackgerrit | Jesse Pretorius (odyssey4me) proposed openstack-infra/openstack-zuul-jobs master: Remove remaining legacy openstack-ansible tests https://review.openstack.org/512635 | 09:27 |
| *** d0ugal has quit IRC | 09:28 | |
| *** gmann is now known as gmann_afk | 09:29 | |
| *** pgadiya has quit IRC | 09:31 | |
| *** tosky has quit IRC | 09:33 | |
| *** Liced has quit IRC | 09:34 | |
| *** tosky has joined #openstack-infra | 09:34 | |
| *** shardy has joined #openstack-infra | 09:34 | |
| *** vsaienk0 has quit IRC | 09:35 | |
| *** cshastri has joined #openstack-infra | 09:35 | |
| *** d0ugal has joined #openstack-infra | 09:37 | |
| *** wolverineav has joined #openstack-infra | 09:39 | |
| *** vsaienk0 has joined #openstack-infra | 09:41 | |
| *** tmorin has joined #openstack-infra | 09:43 | |
| *** pgadiya has joined #openstack-infra | 09:45 | |
| *** e0ne has joined #openstack-infra | 09:45 | |
| tmorin | hi infraroot, I did a +2 W+1 on https://review.openstack.org/#/c/514742 more than an hour ago, but nothing seems to happen (looking at zuulv3.openstack.org) ... I certainly can recheck or W-1/W+1, but isn't there an issue somewhere ? | 09:45 |
| *** edmondsw has joined #openstack-infra | 09:47 | |
| *** slaweq has quit IRC | 09:48 | |
| *** sdague has joined #openstack-infra | 09:50 | |
| *** edmondsw has quit IRC | 09:52 | |
| *** slaweq has joined #openstack-infra | 09:53 | |
| frickler | tmorin: yes, zuul is slow or stuck, waiting for infra-root to check | 09:53 |
| *** yamamoto has quit IRC | 09:54 | |
| *** slaweq_ has joined #openstack-infra | 09:59 | |
| *** gongysh has quit IRC | 10:00 | |
| tosky | yep, "Queue length: 2677 events" does not sound normal | 10:02 |
| *** zhurong has quit IRC | 10:05 | |
| *** cuongnv has quit IRC | 10:05 | |
| *** markvoelker has joined #openstack-infra | 10:07 | |
| *** kiennt26 has quit IRC | 10:10 | |
| *** liujiong has quit IRC | 10:13 | |
| *** gildub has quit IRC | 10:13 | |
| *** yamamoto has joined #openstack-infra | 10:16 | |
| *** annp has quit IRC | 10:17 | |
| *** udesale has quit IRC | 10:20 | |
| *** zhurong has joined #openstack-infra | 10:20 | |
| *** LindaWang has quit IRC | 10:22 | |
| *** yamamoto has quit IRC | 10:24 | |
| *** thorst has joined #openstack-infra | 10:25 | |
| *** slaweq has quit IRC | 10:27 | |
| *** slaweq has joined #openstack-infra | 10:29 | |
| *** thorst has quit IRC | 10:30 | |
| ianw | ok ... having a look | 10:30 |
| *** xinliang has quit IRC | 10:30 | |
| d0ugal | "3160 events" | 10:30 |
| d0ugal | climbing fast too :) | 10:30 |
| ianw | it's not out of memory, and it's not being particularly cpu hungry | 10:34 |
| *** lathiat has quit IRC | 10:35 | |
| ianw | it's doing stuff, just ... slowly | 10:35 |
| *** jascott1 has quit IRC | 10:36 | |
| d0ugal | tbf, that is how I am on Mondays. | 10:36 |
| *** kjackal_ has joined #openstack-infra | 10:37 | |
| *** LindaWang has joined #openstack-infra | 10:37 | |
| ianw | i've restart it ... | 10:38 |
| openstackgerrit | Jens Harbott (frickler) proposed openstack-infra/shade master: Fix regression for list_router_interfaces https://review.openstack.org/516247 | 10:38 |
| ianw | it seems suspicious the oldest things seemed to maybe be around the time AJaeger mentioned review.o.o having issues | 10:39 |
| ianw | that also seems ok? | 10:39 |
| d0ugal | yeah, gerrit seems fine now | 10:39 |
| SamYaple | 3/win 14 | 10:40 |
| frickler | ianw: the gerrit outage probably was the announced provider maintainance | 10:40 |
| frickler | seems to have gotten zuul trip up | 10:40 |
| *** markvoelker has quit IRC | 10:41 | |
| frickler | that was the announcement: 2017-10-26 15:32:52 UTC Provider maintenance is scheduled for 2017-10-30 between 06:00-09:00 UTC which may result in up to a 5 minute connectivity outage for the production Gerrit server's Trove database instance | 10:41 |
| *** boden has joined #openstack-infra | 10:41 | |
| *** xinliang has joined #openstack-infra | 10:42 | |
| tmorin | infraroot: http://zuulv3.openstack.org/ does not display anything useful except "status.json: Proxy Error" , is it just for me ? (that would be surprising, it was ok one hour ago...) | 10:43 |
| ianw | ok, just about finished reading config i think, should probably come alive now | 10:43 |
| tmorin | frickler: thanks for your reply, on a zuul restart, do we have to retrigger the jobs ? (was true with v2, I'm wondering about v3) | 10:44 |
| ianw | #status notice Zuul has been restarted due to an unexpected issue. Please recheck any jobs that were in progress | 10:45 |
| openstackstatus | ianw: sending notice | 10:45 |
| frickler | tmorin: it depends, sometimes it is possible to save the queue and refill it after the restart, doesn't look like that was possible in the current case | 10:46 |
| tmorin | frickler: ignore my question above :) | 10:46 |
| ianw | probably could have saved the queues, but anyway | 10:46 |
| -openstackstatus- NOTICE: Zuul has been restarted due to an unexpected issue. Please recheck any jobs that were in progress | 10:46 | |
| tmorin | ianw, frickler: anything that makes the restart more transparent is appreciated, but I guess you guys do what you can... | 10:46 |
| openstackstatus | ianw: finished sending notice | 10:48 |
| ianw | jeblair: ok, digging in a bit more in the logs, I'm pretty confident it stopped working at 2017-10-30 07:53:16,724 | 10:49 |
| ianw | http://paste.openstack.org/show/624950/ | 10:50 |
| *** eyalb has left #openstack-infra | 10:50 | |
| *** pbourke has quit IRC | 10:50 | |
| ianw | i think there might be an error case worth handling there, but it's a bit late for me to dig into it now :) | 10:51 |
| frickler | ianw: at least that timestamp seems to pretty exactly match the traffic drop here http://cacti.openstack.org/cacti/graph.php?action=view&local_graph_id=63987&rra_id=all | 10:52 |
| *** pbourke has joined #openstack-infra | 10:52 | |
| ianw | hmm, though are jobs processing? i wonder if the executors are unhappy too | 10:52 |
| *** florianf has quit IRC | 10:53 | |
| frickler | ianw: doesn't look like real progress is happening on the status page indeed | 10:53 |
| *** florianf has joined #openstack-infra | 10:55 | |
| *** ifat_afek has quit IRC | 10:56 | |
| *** pcaruana has joined #openstack-infra | 10:56 | |
| *** ldnunes has joined #openstack-infra | 10:57 | |
| *** erlon has joined #openstack-infra | 10:58 | |
| *** lathiat has joined #openstack-infra | 10:58 | |
| *** slaweq has quit IRC | 11:01 | |
| *** slaweq has joined #openstack-infra | 11:02 | |
| ianw | ok, exectuors restarted and things seems to be processing | 11:03 |
| ianw | i wonder if i should restart the scheduler for the ones that were inflight ... hmm | 11:03 |
| *** dmsimard is now known as dmsimard|afk | 11:05 | |
| ianw | i think i will, i don't think those jobs will ever complete | 11:05 |
| ianw | #status log restarted all zuul executors and restarted scheduler | 11:06 |
| openstackstatus | ianw: finished logging | 11:06 |
| ianw | give it a few minutes to reload config | 11:06 |
| *** smatzek has joined #openstack-infra | 11:07 | |
| *** nicolasbock has joined #openstack-infra | 11:07 | |
| ianw | jeblair: 2017-10-30 07:15:54,667 in ze01 logs looks pretty suspicious to me | 11:08 |
| ianw | http://paste.openstack.org/show/624954/ | 11:08 |
| ianw | ok, already things look better | 11:09 |
| ianw | definitely a series of failures seemed to cascade through around 07:00 | 11:09 |
| ianw | ok, my recheck seems to have worked ... if anyone else can recheck theirs and report for sanity that would be good | 11:12 |
| andreaf | jeblair, mordred: anything holding this off https://review.openstack.org/#/c/509233/ from merging? | 11:15 |
| prometheanfire | AJaeger: looking | 11:15 |
| andreaf | it would be good to have that in so other roles could start using it right away | 11:16 |
| *** namnh has quit IRC | 11:17 | |
| tosky | rechecked two jobs, I see them on the right queues | 11:18 |
| ianw | ok, jobs are running, stuff is merging (https://git.openstack.org/cgit/openstack/charm-ceph/commit/?id=b210d18d00e2b7e1c5b5b97ab403dd0951c90935 for example just merged) | 11:21 |
| *** jtomasek has quit IRC | 11:21 | |
| tosky | ianw: uhm, almost: 516079 (which already had +w) ended up into both check and gate queue | 11:22 |
| *** mat128 has joined #openstack-infra | 11:25 | |
| *** yamamoto has joined #openstack-infra | 11:25 | |
| *** mat128 has quit IRC | 11:25 | |
| ianw | 2017-10-30 11:18:10,895 INFO zuul.Pipeline.openstack.check: Adding change <Change 0x7fc5779ca0f0 516079,1> to queue <ChangeQueue check: openstack/sahara-image-elements> in <Pipeline check> | 11:26 |
| ianw | 2017-10-30 11:18:10,896 INFO zuul.Pipeline.openstack.infra-check: Adding change <Change 0x7fc5779ca0f0 516079,1> to queue <ChangeQueue infra-check: openstack/sahara-image-elements> in <Pipeline infra-check> | 11:26 |
| ianw | 2017-10-30 11:18:10,897 INFO zuul.Pipeline.openstack.gate: Adding change <Change 0x7fc5779ca0f0 516079,1> to queue <ChangeQueue gate: openstack/sahara-image-elements> in <Pipeline gate> | 11:26 |
| ianw | 2017-10-30 11:18:10,898 INFO zuul.Pipeline.openstack.check-tripleo: Adding change <Change 0x7fc5779ca0f0 516079,1> to queue <ChangeQueue check-tripleo: openstack/sahara-image-elements> in <Pipeline check-tripleo> | 11:26 |
| ianw | really not sure why. anyway, it will merge and just get another check comment | 11:27 |
| ianw | oh it has merged now | 11:27 |
| *** salv-orlando has quit IRC | 11:28 | |
| *** salv-orlando has joined #openstack-infra | 11:29 | |
| ianw | ok with that i'm out. east coaster's will be around soon i'd imagine anyway. goodnight! | 11:29 |
| *** ramishra has quit IRC | 11:30 | |
| *** alexchadin has quit IRC | 11:31 | |
| *** alexchadin has joined #openstack-infra | 11:32 | |
| *** yamamoto has quit IRC | 11:32 | |
| *** ramishra has joined #openstack-infra | 11:32 | |
| *** stakeda has quit IRC | 11:34 | |
| *** mhayden has quit IRC | 11:35 | |
| *** dave-mccowan has joined #openstack-infra | 11:35 | |
| *** slaweq has quit IRC | 11:36 | |
| *** mhayden has joined #openstack-infra | 11:36 | |
| *** markvoelker has joined #openstack-infra | 11:38 | |
| *** rhallisey has joined #openstack-infra | 11:38 | |
| *** yamamoto has joined #openstack-infra | 11:39 | |
| d0ugal | Thanks for the help! | 11:40 |
| *** ifat_afek has joined #openstack-infra | 11:42 | |
| *** slaweq has joined #openstack-infra | 11:45 | |
| *** hemna_ has joined #openstack-infra | 11:48 | |
| *** jtomasek has joined #openstack-infra | 11:51 | |
| mpeterson | hello! we are having problems with patchsets not being scheduled on Zuul (ie: https://review.openstack.org/#/c/516246 and https://review.openstack.org/#/c/515057/ ) | 11:56 |
| *** pgadiya has quit IRC | 11:58 | |
| mpeterson | oh, just done a buffer playback of the channel, I see there was a restart that lost the queue. nvm. | 11:59 |
| *** spectr has quit IRC | 11:59 | |
| *** salv-orlando has quit IRC | 11:59 | |
| *** dhajare has joined #openstack-infra | 11:59 | |
| *** spectr has joined #openstack-infra | 12:00 | |
| *** bhavik1 has joined #openstack-infra | 12:00 | |
| *** udesale has joined #openstack-infra | 12:03 | |
| *** thorst has joined #openstack-infra | 12:09 | |
| *** edmondsw has joined #openstack-infra | 12:09 | |
| *** markvoelker has quit IRC | 12:11 | |
| *** bhavik1 has quit IRC | 12:13 | |
| *** jaypipes has joined #openstack-infra | 12:15 | |
| *** dprince has joined #openstack-infra | 12:16 | |
| *** slaweq has quit IRC | 12:18 | |
| *** thorst_ has joined #openstack-infra | 12:19 | |
| *** thorst__ has joined #openstack-infra | 12:20 | |
| Dinesh_Bhor | ianw: Hi, Have you come across such zuul error before? "Unable to freeze job graph: Job requirements-check does not specify a run playbook" | 12:21 |
| *** jamesden_ has joined #openstack-infra | 12:21 | |
| Dinesh_Bhor | ianw: I am getting this on my patch: https://review.openstack.org/#/c/515649/ | 12:21 |
| *** ociuhandu has quit IRC | 12:23 | |
| *** thorst has quit IRC | 12:23 | |
| *** jamesdenton has quit IRC | 12:24 | |
| *** thorst_ has quit IRC | 12:24 | |
| *** slaweq has joined #openstack-infra | 12:27 | |
| *** ociuhandu has joined #openstack-infra | 12:28 | |
| *** markvoelker has joined #openstack-infra | 12:29 | |
| *** trown|outtypewww is now known as trown | 12:30 | |
| *** rlandy has joined #openstack-infra | 12:31 | |
| *** catintheroof has joined #openstack-infra | 12:33 | |
| *** hemna_ has quit IRC | 12:36 | |
| *** jascott1 has joined #openstack-infra | 12:36 | |
| *** hemna_ has joined #openstack-infra | 12:36 | |
| *** hashar has quit IRC | 12:37 | |
| *** jascott1 has quit IRC | 12:40 | |
| *** haleyb has joined #openstack-infra | 12:41 | |
| *** dtantsur|afk is now known as dtantsur | 12:42 | |
| *** shardy is now known as shardy_lunch | 12:42 | |
| *** yamamoto has quit IRC | 12:43 | |
| *** catintheroof has quit IRC | 12:47 | |
| *** catintheroof has joined #openstack-infra | 12:49 | |
| *** hemna_ has quit IRC | 12:49 | |
| *** zhurong has quit IRC | 12:50 | |
| frickler | Dinesh_Bhor: I think there is a proposed patch that should fix your error, let me see if I can find it | 12:50 |
| *** jpena is now known as jpena|lunch | 12:55 | |
| frickler | Dinesh_Bhor: you can try to add a depends-on to https://review.openstack.org/516197 to your patch, seems that helped for https://review.openstack.org/515591 at least | 12:57 |
| *** dhajare has quit IRC | 12:58 | |
| *** mriedem has joined #openstack-infra | 12:58 | |
| *** rcernin has quit IRC | 12:58 | |
| Dinesh_Bhor | frickler: ohh, okay, Thank you so much. | 12:59 |
| frickler | ianw: tosky: it is a known issue (or maybe feature) that a recheck may place a patch in both gate and check queue simultanously when it satisfies the gate requirements already, been seing that earlier | 12:59 |
| *** lindycoder has joined #openstack-infra | 13:00 | |
| *** slaweq has quit IRC | 13:00 | |
| *** salv-orlando has joined #openstack-infra | 13:00 | |
| tosky | frickler: I would say issue in this case | 13:02 |
| *** rnoriega has joined #openstack-infra | 13:02 | |
| *** otherwiseguy has joined #openstack-infra | 13:03 | |
| *** slaweq has joined #openstack-infra | 13:03 | |
| *** otherwiseguy has quit IRC | 13:05 | |
| *** rnoriega has quit IRC | 13:05 | |
| *** baoli has joined #openstack-infra | 13:05 | |
| *** salv-orlando has quit IRC | 13:06 | |
| *** janki has quit IRC | 13:06 | |
| *** otherwiseguy has joined #openstack-infra | 13:06 | |
| *** rnoriega has joined #openstack-infra | 13:07 | |
| *** felipemonteiro_ has joined #openstack-infra | 13:09 | |
| *** felipemonteiro has joined #openstack-infra | 13:11 | |
| *** hemna_ has joined #openstack-infra | 13:11 | |
| *** kgiusti has joined #openstack-infra | 13:11 | |
| *** gongysh has joined #openstack-infra | 13:12 | |
| *** gongysh has quit IRC | 13:12 | |
| *** tpsilva has joined #openstack-infra | 13:13 | |
| *** jcoufal has joined #openstack-infra | 13:13 | |
| *** lindycoder has quit IRC | 13:14 | |
| *** felipemonteiro_ has quit IRC | 13:14 | |
| *** kjackal_ has quit IRC | 13:14 | |
| *** camunoz has joined #openstack-infra | 13:18 | |
| *** pkovar has joined #openstack-infra | 13:23 | |
| pkovar | i read https://docs.openstack.org/infra/system-config/irc.html#meetbot but it's not very clear to me if i can run a mtg in a team channel if there's the openstack bot present | 13:25 |
| *** camunoz has quit IRC | 13:25 | |
| pkovar | can someone confirm please? | 13:25 |
| *** lbragstad has quit IRC | 13:25 | |
| AJaeger | pkovar: just try it ;) It should work... | 13:26 |
| pkovar | AJaeger: thanks | 13:28 |
| *** superdan is now known as dansmith | 13:28 | |
| *** pblaho has joined #openstack-infra | 13:28 | |
| *** apetrich has quit IRC | 13:30 | |
| *** salv-orlando has joined #openstack-infra | 13:30 | |
| *** smatzek has quit IRC | 13:30 | |
| *** smatzek has joined #openstack-infra | 13:30 | |
| *** apetrich has joined #openstack-infra | 13:30 | |
| *** smatzek has quit IRC | 13:31 | |
| *** jtomasek_ has joined #openstack-infra | 13:31 | |
| *** jtomasek has quit IRC | 13:32 | |
| *** lbragstad has joined #openstack-infra | 13:33 | |
| openstackgerrit | Merged openstack-infra/project-config master: Add openstack/monasca-specs repo https://review.openstack.org/515685 | 13:35 |
| *** slaweq has quit IRC | 13:36 | |
| *** mat128 has joined #openstack-infra | 13:38 | |
| *** yamamoto has joined #openstack-infra | 13:39 | |
| *** thorst__ has quit IRC | 13:40 | |
| *** kiennt26 has joined #openstack-infra | 13:40 | |
| *** dsariel_ has quit IRC | 13:41 | |
| *** gcb has joined #openstack-infra | 13:41 | |
| *** slaweq has joined #openstack-infra | 13:42 | |
| *** ihrachys has joined #openstack-infra | 13:42 | |
| openstackgerrit | Merged openstack-infra/openstack-zuul-jobs master: Remove legacy bifrost jobs https://review.openstack.org/515729 | 13:43 |
| *** ifat_afek has left #openstack-infra | 13:44 | |
| *** shardy_lunch is now known as shardy | 13:46 | |
| AJaeger | frickler, pabelanger could either of you review https://review.openstack.org/#/q/topic:ansible-lint+status:open , please? | 13:46 |
| *** rosmaita has joined #openstack-infra | 13:46 | |
| *** kgiusti has quit IRC | 13:50 | |
| *** kgiusti has joined #openstack-infra | 13:50 | |
| *** jcoufal_ has joined #openstack-infra | 13:54 | |
| *** jcoufal has quit IRC | 13:57 | |
| *** jpena|lunch is now known as jpena | 13:57 | |
| *** armax has joined #openstack-infra | 13:58 | |
| AJaeger | thanks, frickler | 13:59 |
| *** hashar has joined #openstack-infra | 13:59 | |
| *** eharney has joined #openstack-infra | 14:00 | |
| *** alexchadin has quit IRC | 14:00 | |
| *** hongbin has joined #openstack-infra | 14:00 | |
| *** salv-orlando has quit IRC | 14:02 | |
| *** smatzek has joined #openstack-infra | 14:03 | |
| *** smatzek has quit IRC | 14:03 | |
| *** smatzek has joined #openstack-infra | 14:04 | |
| *** smatzek has quit IRC | 14:04 | |
| *** smatzek has joined #openstack-infra | 14:05 | |
| openstackgerrit | YAMAMOTO Takashi proposed openstack-infra/system-config master: Collect devstack-early.txt https://review.openstack.org/516324 | 14:05 |
| *** marst has joined #openstack-infra | 14:11 | |
| openstackgerrit | Merged openstack-infra/project-config master: Simplify ansible-lint invocation https://review.openstack.org/516021 | 14:11 |
| openstackgerrit | Merged openstack-infra/project-config master: ansible-lint: Add tests/inventory and vars.yaml https://review.openstack.org/516020 | 14:11 |
| openstackgerrit | Merged openstack-infra/project-config master: ansible-lint: Silence shell warnings https://review.openstack.org/516025 | 14:11 |
| *** alexchadin has joined #openstack-infra | 14:13 | |
| *** lbragstad has quit IRC | 14:13 | |
| *** lbragstad has joined #openstack-infra | 14:13 | |
| *** rbrndt has joined #openstack-infra | 14:15 | |
| *** lbragstad has quit IRC | 14:18 | |
| *** pkovar has left #openstack-infra | 14:20 | |
| mnaser | any gertty users... ever run into really slow ops like itll be stuck on syncing change for way too long | 14:21 |
| *** lbragstad has joined #openstack-infra | 14:22 | |
| jeblair | mnaser: running tip of master? | 14:22 |
| mnaser | jeblair: no, pip install'd latest release | 14:22 |
| mnaser | gertty==1.5.0 | 14:23 |
| jeblair | mnaser: ah, master has a significant speed improvement syncing large changes | 14:23 |
| mnaser | jeblair: ok, ill upgrade and see how things are | 14:23 |
| *** salv-orlando has joined #openstack-infra | 14:24 | |
| *** makowals has quit IRC | 14:27 | |
| *** makowals has joined #openstack-infra | 14:29 | |
| *** felipemonteiro has quit IRC | 14:30 | |
| *** thorst has joined #openstack-infra | 14:30 | |
| *** thorst has quit IRC | 14:31 | |
| *** thorst has joined #openstack-infra | 14:31 | |
| *** LindaWang has quit IRC | 14:31 | |
| *** trown is now known as trown|brb | 14:31 | |
| *** LindaWang1 has joined #openstack-infra | 14:31 | |
| *** Apoorva has joined #openstack-infra | 14:33 | |
| openstackgerrit | Matt Riedemann proposed openstack-infra/openstack-zuul-jobs master: Restrict legacy-tempest-dsvm-nova-wsgi-full to stable/ocata https://review.openstack.org/516334 | 14:33 |
| *** LindaWang1 is now known as LindaWang | 14:33 | |
| *** udesale has quit IRC | 14:37 | |
| vdrok | AJaeger: looking at the ML, we now need to backport to stable not only project.yaml and playbooks, but also legacy-jobs.yaml? just checking :) | 14:39 |
| *** wewe0901 has quit IRC | 14:40 | |
| *** trown|brb is now known as trown | 14:43 | |
| *** slaweq has quit IRC | 14:46 | |
| *** zzzeek has joined #openstack-infra | 14:47 | |
| openstackgerrit | Merged openstack-infra/zuul-jobs master: add more debugging to the upload-pypi role https://review.openstack.org/514394 | 14:48 |
| *** slaweq has joined #openstack-infra | 14:49 | |
| clarkb | AJaeger: re gerrit 500s I see a large spike in http requests paired with a large spike in gerrit threads and a minor amount of garbage collection | 14:49 |
| clarkb | AJaeger: I think ti may have effectively been a DoS and apache was refusing to connect more connections | 14:49 |
| openstackgerrit | Petr Kovar proposed openstack-infra/irc-meetings master: Update channel for doc team meeting https://review.openstack.org/516340 | 14:52 |
| *** cshastri has quit IRC | 14:52 | |
| frickler | clarkb: wasn't that the provider maintainance with 5 minute trove downtime? | 14:54 |
| clarkb | oh could be and we piled up connections until apache wouldn't serve more | 14:54 |
| clarkb | 0600UTC ? | 14:54 |
| clarkb | that is when the connection spike happens according to java melody | 14:54 |
| frickler | that was the announcement: 2017-10-26 15:32:52 UTC Provider maintenance is scheduled for 2017-10-30 between 06:00-09:00 UTC which may result in up to a 5 minute connectivity outage for the production Gerrit server's Trove database instance | 14:55 |
| clarkb | ok I think that explains it then | 14:55 |
| fungi | should be able to correlate to database disconnect/error messages in gerrit's log, i expect, if you're concerned | 14:57 |
| *** gcb has quit IRC | 15:00 | |
| *** yamahata has joined #openstack-infra | 15:01 | |
| *** salv-orlando has quit IRC | 15:04 | |
| *** alexchadin has quit IRC | 15:04 | |
| *** salv-orlando has joined #openstack-infra | 15:06 | |
| *** esberglu has joined #openstack-infra | 15:07 | |
| *** esberglu has quit IRC | 15:07 | |
| *** esberglu has joined #openstack-infra | 15:07 | |
| openstackgerrit | Jens Harbott (frickler) proposed openstack-infra/shade master: Fix regression for list_router_interfaces https://review.openstack.org/516247 | 15:09 |
| *** ykarel has quit IRC | 15:11 | |
| openstackgerrit | Merged openstack-infra/project-config master: Use native v3 publishing for project-team-guide https://review.openstack.org/514053 | 15:21 |
| *** slaweq has quit IRC | 15:22 | |
| frickler | AJaeger: ^5 ;) | 15:22 |
| *** salv-orl_ has joined #openstack-infra | 15:23 | |
| AJaeger | thanks | 15:23 |
| *** salv-orlando has quit IRC | 15:27 | |
| *** makowals has quit IRC | 15:28 | |
| *** kiennt26 has quit IRC | 15:30 | |
| *** dmsimard|afk is now known as dmsimard | 15:30 | |
| *** makowals has joined #openstack-infra | 15:35 | |
| bswartz | next Ubuntu LTS is "Bionic Beaver" | 15:35 |
| fungi | wow | 15:35 |
| fungi | really? | 15:35 |
| fungi | wow | 15:35 |
| bswartz | #link http://www.omgubuntu.co.uk/2017/10/ubuntu-18-04-lts-called-bionic-beaver | 15:36 |
| dmsimard | clarkb: it doesn't look like elastic-recheck is happy, do we know what the issue is ? (sorry I've been a bit away) | 15:36 |
| *** felipemonteiro has joined #openstack-infra | 15:36 | |
| clarkb | dmsimard: the graph producing pages are functioning now. I think we may still have some bugs in the queries that make it think it is behind though | 15:36 |
| clarkb | dmsimard: I haven't looekd into the uncategorized pages any more yet | 15:36 |
| *** catintheroof has quit IRC | 15:37 | |
| *** felipemonteiro_ has joined #openstack-infra | 15:38 | |
| dmsimard | clarkb: okay, I'll try and see if I can figure something out | 15:38 |
| *** gyee has joined #openstack-infra | 15:38 | |
| *** yamamoto has quit IRC | 15:40 | |
| *** yamamoto has joined #openstack-infra | 15:41 | |
| *** felipemonteiro has quit IRC | 15:42 | |
| dmsimard | clarkb: hrm.. I see two issues: 1) Searching for something that should be very common like "PRE-RUN START" yields *one* result over the last 24hrs, 2) The filename for that one result is "job-output.txt.gz", not "job-output.txt". I'll poke to understand why. | 15:43 |
| clarkb | dmsimard: I think that may be a regression in how the gearman jobs are submitted | 15:43 |
| clarkb | we used to always drop the .gz (as it was implied deployment specific thing) | 15:43 |
| dmsimard | Yeah I'll look at the bits where we submit stuff to gearman | 15:44 |
| clarkb | so we may be able to do that in the zuul job that submits gearman records | 15:44 |
| * dmsimard wants ara reports even for successful jobs :( | 15:44 | |
| *** yamamoto has quit IRC | 15:45 | |
| *** yamamoto has joined #openstack-infra | 15:45 | |
| dmsimard | logtash submits to gearman are done after the logs are uploaded, it's not even in the json so I can't tell if from an ansible perspective there's errors, I guess you would see that on the executor | 15:47 |
| dmsimard | http://git.openstack.org/cgit/openstack-infra/project-config/tree/playbooks/base/post-logs.yaml | 15:47 |
| clarkb | I don't think there are errors | 15:48 |
| clarkb | we are just submitting the file name as having the .gz suffix | 15:48 |
| clarkb | rather than omitting it | 15:48 |
| dmsimard | clarkb: but here we define without .gz http://git.openstack.org/cgit/openstack-infra/project-config/tree/roles/submit-logstash-jobs/defaults/main.yaml | 15:49 |
| clarkb | dmsimard: ya but those are regexes | 15:49 |
| clarkb | python processes that to generate a list of files, and it is that python that needs to drop the .gz I think | 15:49 |
| dmsimard | oh | 15:49 |
| dmsimard | would logstash/elasticsearch be able to index the gzipped file in the first place or does it need to be extracted ? | 15:50 |
| dmsimard | iirc console.html were not originally gzipped, only like 10 days later by the log rotation | 15:50 |
| clarkb | logstash should handle it properly. The gearman job worker requests gzip encoded logs (and apache will serve without the .gz too) and if that fails attempts with the .gz added as well | 15:52 |
| dmsimard | from quick googling, it looks like there's a specific logstash plugin for reading gzipped content | 15:52 |
| dmsimard | oh, that's right | 15:53 |
| *** slaweq has joined #openstack-infra | 15:53 | |
| clarkb | I think the only thing that has to change is dropping the .gz suffix in the job request sent to the gearman workers (and the job-output.txt for the change that updates the submitter should get indexed properly if that is the case) | 15:53 |
| clarkb | dmsimard: in this case we have a daemon in front of logstash handling the "get logs based on gearman job and inject them into logstash tcp port" so all that is handled for it | 15:54 |
| dmsimard | http://git.openstack.org/cgit/openstack-infra/log_processor right ? | 15:55 |
| clarkb | dmsimard: yes, though I'm not sure we ever finished the migration to ^ instead of the scripts in puppet-log_processors/files | 15:56 |
| clarkb | iirc the first attempt failed due to daemon management scripts | 15:56 |
| fungi | mtreinish: i apparently missed implementing this part of the netifaces fix for puppet-subunit2sql: https://review.openstack.org/516085 | 15:57 |
| *** slaweq has quit IRC | 15:57 | |
| *** xarses has joined #openstack-infra | 15:59 | |
| *** spectr has quit IRC | 16:02 | |
| *** ijw has joined #openstack-infra | 16:03 | |
| EmilienM | jeblair: good morning, any update on the puppet jobs issue that you were investigating? | 16:04 |
| mnaser | does adding/removing public keys from gerrit sometime to take effect? | 16:04 |
| jeblair | EmilienM: i believe the fix is in production | 16:05 |
| EmilienM | jeblair: perfect, I did recheck | 16:05 |
| fungi | mnaser: it should take effect immediately | 16:06 |
| AJaeger | jeblair: tahnks for your better checking in zuul - check-requirements fails now since it had no run playbook. Fix is currently merging | 16:06 |
| mnaser | fungi: gitreview.user -- oops | 16:06 |
| fungi | indeed, that'd do it | 16:06 |
| AJaeger | team, anybody have time to dig into check-requirements failure? Now the job always succeeds ;( | 16:07 |
| AJaeger | example log http://logs.openstack.org/08/516208/2/check/requirements-check/fb5d824/job-output.json.gz | 16:07 |
| AJaeger | I'm out of ideas ;( | 16:07 |
| *** catintheroof has joined #openstack-infra | 16:07 | |
| *** ramishra has quit IRC | 16:07 | |
| fungi | jeblair: just a heads up in case you haven't seen scrollback or the status log, but ianw restarted the zuul scheduler and executors this morning due to it indefinitely queuing events | 16:08 |
| fungi | may have been related to the gerrit trove instance maintenance | 16:08 |
| jeblair | fungi: ack thx. | 16:08 |
| jeblair | AJaeger: according to the json, the script is being run (though nothing appears in the console log?) | 16:10 |
| openstackgerrit | Merged openstack-infra/zuul feature/zuulv3: Zuul: add file extension to playbook path https://review.openstack.org/516135 | 16:10 |
| openstackgerrit | Merged openstack-infra/zuul feature/zuulv3: Use cached branches on tenant reconfiguration https://review.openstack.org/515971 | 16:10 |
| *** catintheroof has quit IRC | 16:12 | |
| AJaeger | jeblair: it is run but it succeeds - it should fail | 16:13 |
| AJaeger | and locally it fails | 16:13 |
| jeblair | AJaeger: i'm guessing there's no output because it uses the "script" module. we may not have any special handling for that to hook it into zuul_stream. | 16:13 |
| jeblair | AJaeger: have you found the script's output in the json file? | 16:13 |
| AJaeger | jeblair: yes - with debug output http://paste.openstack.org/show/624988/ | 16:14 |
| AJaeger | see "broken-requirement" - and no error | 16:14 |
| AJaeger | locally I get "Requirement set([Requirement(package=u'broken-requirement', location='', specifiers='', markers=u'', comment='', extras=frozenset([]))]) not in openstack/requirements" for tht line | 16:14 |
| AJaeger | but not under Zuul ;( | 16:14 |
| *** dsariel_ has joined #openstack-infra | 16:21 | |
| *** jamesden_ has quit IRC | 16:22 | |
| *** jamesdenton has joined #openstack-infra | 16:23 | |
| jeblair | AJaeger: i think it probably relates to this: https://git.openstack.org/cgit/openstack/requirements/tree/playbooks/files/project-requirements-change.py#n187 | 16:24 |
| jeblair | AJaeger: the 'master' branch of that repo contains the proposed future state of master. | 16:25 |
| *** dsariel__ has joined #openstack-infra | 16:25 | |
| jeblair | AJaeger: i still don't really understand everything that script is doing, but based on the comments at least, i suspect that part may no longer be functioning as intended. | 16:26 |
| *** slaweq has joined #openstack-infra | 16:27 | |
| AJaeger | jeblair: ;/ | 16:27 |
| AJaeger | is mordred around? I think he did that change | 16:28 |
| *** dsariel_ has quit IRC | 16:28 | |
| jeblair | AJaeger: so we're probably hitting the "unchanged" case (the first one) | 16:29 |
| *** sflanigan has quit IRC | 16:29 | |
| *** sflanigan has joined #openstack-infra | 16:29 | |
| *** sflanigan has joined #openstack-infra | 16:29 | |
| jeblair | AJaeger: i think that may have been the behavior for a long time | 16:29 |
| jeblair | AJaeger: i think the behavior we wanted from the script is "if you don't change a req, it's fine. but if you do change it, it must match the current value" | 16:30 |
| AJaeger | jeblair: yes, it gets the Unchanged one | 16:30 |
| jeblair | AJaeger: that way you weren't forced to upgrade req B just because you were trying to upgrade req A | 16:30 |
| *** dsariel_ has joined #openstack-infra | 16:30 | |
| AJaeger | jeblair: http://paste.openstack.org/show/624989/ is newer log file with more printf debugging ;) | 16:31 |
| jeblair | that seems to confirm it | 16:31 |
| openstackgerrit | Jens Harbott (frickler) proposed openstack-infra/shade master: Fix regression for list_router_interfaces https://review.openstack.org/516247 | 16:31 |
| jeblair | AJaeger: the tricky thing is that zuul doesn't provide the information this job needs. | 16:32 |
| jeblair | we could have the job add a git remote and find out what the current value is, but that's sort of clunky. | 16:33 |
| *** bhavik1 has joined #openstack-infra | 16:33 | |
| clarkb | fungi: looking at the zuulv3 issues etherpad did the launchpad credentials problem in the tag-releases job get sorted out? | 16:34 |
| *** dsariel__ has quit IRC | 16:34 | |
| *** yamahata has quit IRC | 16:34 | |
| fungi | yeah, pretty early on. i didn't realize it was still listed as broken. i'll move it | 16:34 |
| jeblair | we could have zuul do the opposite of what it did before. instead of creating zuul refs to represent the future state, create refs to represent the 'base' state. however, one complexity is, "which base state?" the state before this change, or before the change series, or something else... | 16:34 |
| jeblair | or maybe this script just needs to compare HEAD^1 to HEAD | 16:35 |
| clarkb | would be nice to have everyone do a quick update of their items on the etherpad so we can clear out any that are fixed (I just got to mark one of mine fixed \o/) | 16:35 |
| jeblair | maybe that's enough, because previous changes will have compared their own HEAD^1 to HEAD | 16:35 |
| jeblair | clarkb: ++ i was just doing that too | 16:35 |
| AJaeger | jeblair: would that work with a merge commit? See line 14 of the paste, I run git show there | 16:35 |
| AJaeger | clarkb: good idea. I think I removed mine but will check again later | 16:36 |
| jeblair | AJaeger: ^ or ~ should do the right thing with merges, i'll have to look it up again | 16:36 |
| *** dsariel__ has joined #openstack-infra | 16:36 | |
| fungi | clarkb: oh, for some reason drag-n-drop seems to be working only intermittently for me in that pad. might explain why some of the stuff i thought i dragged into the fixed section is hanging around still | 16:37 |
| jeblair | you can drag and drop? | 16:38 |
| clarkb | I do cut and paste | 16:39 |
| fungi | jeblair: apparently only sometimes | 16:39 |
| clarkb | fungi: the otehr one of yours that I think is fixed is the detached signatures on prerelease | 16:39 |
| fungi | so i'm going to have to remind myself not to rely on it any longer | 16:39 |
| *** dsariel_ has quit IRC | 16:39 | |
| *** vsaienk0 has quit IRC | 16:40 | |
| *** gouthamr has joined #openstack-infra | 16:40 | |
| *** bhavik1 has quit IRC | 16:41 | |
| *** Swami has joined #openstack-infra | 16:41 | |
| *** dsariel__ has quit IRC | 16:42 | |
| AJaeger | clarkb, fungi, could you review https://review.openstack.org/516010 and https://review.openstack.org/516009 - those fix infra/publications publishing. And once those in, please run some dummy commit to see whether it works... | 16:42 |
| *** jpena is now known as jpena|brb | 16:44 | |
| clarkb | looks like gerrit plugins now publish to http://tarballs.openstack.org/gerrit/plugins/javamelody/gerrit/buck-out/gen/plugins/javamelody/ rather than ci/gerrit/$pluginname. I think that is good enough to call that fixed and working (we can just use the new path(s)) | 16:44 |
| jeblair | clarkb: ++ | 16:44 |
| fungi | wfm | 16:45 |
| *** e0ne has quit IRC | 16:46 | |
| clarkb | AJaeger: before we review those, what neesd to be done to get the linter job passing and voting? | 16:46 |
| *** markvoelker_ has joined #openstack-infra | 16:47 | |
| clarkb | thinking it would be good to get that in before we make more job changes | 16:47 |
| jeblair | fungi, AJaeger, clarkb: i think if we "git checkout HEAD^1" we will check out the repo "before the current change is applied". i think that is true regardless of whether the current change was applied as a fast-forward or a merge commit. | 16:47 |
| clarkb | jeblair: what if there are a stack of changes though? | 16:47 |
| clarkb | might need to go further back? | 16:47 |
| AJaeger | clarkb: https://review.openstack.org/516026 It passes ;) | 16:48 |
| jeblair | fungi, AJaeger, clarkb: that would be a slight behavior change from the current requirements script, which actually compares the change under test to some arbitrary point in the past based on when the zuul mergers updated. | 16:48 |
| jeblair | fungi, AJaeger, clarkb: but if we assume that any parent changes already did the same thing with their own single change under test, i think we're still covered | 16:48 |
| jeblair | clarkb: i think that ^ addresses your point... what do you think? | 16:48 |
| AJaeger | jeblair: I pushed https://review.openstack.org/516208 with HEAD^1 fo rtesting | 16:48 |
| *** markvoelker has quit IRC | 16:49 | |
| AJaeger | jeblair: agreed, we should be fine | 16:49 |
| clarkb | jeblair: thats a good point and ya basically the first proposed state woulc check against a good known state | 16:49 |
| fungi | yeah, i think HEAD^1 is probably the actual intent there. it's how we did some of our other ratcheting/comparison jobs in the past anyway | 16:49 |
| AJaeger | clarkb: ofr 516026: We could merge that in the normal linters job as well but I wanted to do that as followup... | 16:49 |
| jeblair | AJaeger: ya i think that's what i'm proposing. i'd probably change that second checkout from 'head' to branch, just so we land back on a clean checkout instead of a detached head. | 16:49 |
| clarkb | and then we can do induction on the rest | 16:49 |
| jeblair | fungi: i like the word ratcheting | 16:49 |
| AJaeger | jeblair: do you want to comment on my test change? I'll wait for test results and then cleanup... | 16:50 |
| jeblair | AJaeger: will do | 16:50 |
| *** slaweq has quit IRC | 16:50 | |
| clarkb | AJaeger: ya tahts fine, I just saw that some of these other job changes have failing linters, want to get that passing before I merge any regressions :) | 16:50 |
| *** slaweq has joined #openstack-infra | 16:50 | |
| AJaeger | clarkb: those all failed since we still had problems at that time but now we're good | 16:51 |
| *** jascott1 has joined #openstack-infra | 16:51 | |
| *** markvoelker_ has quit IRC | 16:51 | |
| *** catintheroof has joined #openstack-infra | 16:52 | |
| jeblair | folks will be happy to know that rackspace has informed us that the planned maintenance was completed sucessfully. | 16:52 |
| fungi | nice | 16:53 |
| openstackgerrit | Doug Hellmann proposed openstack-infra/openstack-zuul-jobs master: do not run the py35 job for deliverable file changes https://review.openstack.org/516394 | 16:54 |
| *** ldnunes has quit IRC | 16:54 | |
| dhellmann | jeblair : the outcome of the earlier discussion about ^^ isn't entirely clear to me, so if there's a different way to achieve that let me know | 16:55 |
| *** tesseract has quit IRC | 16:55 | |
| openstackgerrit | Merged openstack-infra/project-config master: Make ansible-lint voting https://review.openstack.org/516026 | 16:55 |
| *** slaweq has quit IRC | 16:55 | |
| *** catintheroof has quit IRC | 16:56 | |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config master: Fix python-ironicclient branch-tarball https://review.openstack.org/516397 | 16:57 |
| jlvillal | I checked the mailing list and didn't see mention of it in my quick search. Is Jenkins now running on stable/ocata instead of ZuuL? I noticed this patch of mine: https://review.openstack.org/#/c/510200/ was voted by Jenkins and not Zuul. | 16:57 |
| openstackgerrit | James E. Blair proposed openstack-infra/project-config master: Remove py35 template from releases https://review.openstack.org/516398 | 16:58 |
| jeblair | dhellmann: that's the alternative ^ | 16:58 |
| dmsimard | clarkb: it looks like there's a sane amount of logging done by https://git.openstack.org/cgit/openstack-infra/puppet-log_processor/tree/files/log-gearman-worker.py -- do we have hints there ? | 16:58 |
| clarkb | jlvillal: there is no jenkins only zuul. Those jobs are from october 10th when we must still have had a jenkins/zuulv2 running | 16:59 |
| jeblair | dhellmann, AJaeger: i *personally* was imagining that for something like this, we would go with the path in 516398, though i agree that 516394 is harmless and we could do that. maybe worth getting some more config-cores to weigh in on it since it's a grey area. | 16:59 |
| jeblair | i cross-linked those 2 changes for easy reference | 17:00 |
| AJaeger | jeblair: I would go with 516398 as well in this specific case. | 17:00 |
| *** markvoelker has joined #openstack-infra | 17:00 | |
| jlvillal | jeblair: Oh duh. Now I realize my real question. Before it was failing Zuul because our job definitions weren't there. Playbooks that is. | 17:00 |
| AJaeger | bbl | 17:01 |
| jlvillal | jeblair: So an error like: ironic-tempest-dsvm-ipa-wholedisk-bios-pxe_ipmitool-tinyipa ironic-tempest-dsvm-ipa-wholedisk-bios-pxe_ipmitool-tinyipa : ERROR No valid playbook found | 17:01 |
| jlvillal | jeblair: Put today, it doesn't say that. Even though we haven't finished our patch to add those. | 17:01 |
| jeblair | jlvillal: link? | 17:01 |
| jlvillal | s/Put today/But today/ | 17:01 |
| jlvillal | jeblair: https://review.openstack.org/#/c/510200/ | 17:01 |
| jlvillal | jeblair: 27-Oct-2017 Zuul job has an error | 17:01 |
| jlvillal | jeblair: Today's Zuul job does not | 17:02 |
| dhellmann | jeblair : are the lists of irrelevant-files additive? should we extend the list in 516398? | 17:02 |
| jlvillal | jeblair: We haven't yet landed: https://review.openstack.org/#/c/513697/ | 17:02 |
| *** tosky has quit IRC | 17:04 | |
| jlvillal | jeblair: Could it be because of: http://lists.openstack.org/pipermail/openstack-dev/2017-October/124064.html ? | 17:05 |
| *** markvoelker has quit IRC | 17:05 | |
| jeblair | dhellmann: good point, it is not. so if you want that whole list, you would need to copy it. that does swing the pendulum a bit toward your solution, i think. | 17:05 |
| *** yamahata has joined #openstack-infra | 17:07 | |
| *** ldnunes has joined #openstack-infra | 17:07 | |
| jeblair | dhellmann: wait... it may not be necessary, let me check something :) | 17:08 |
| clarkb | iirc we said we'd maintain shared sets of irrelvant files for this reason | 17:08 |
| clarkb | and that would be the compromise | 17:08 |
| clarkb | (so I am fine with dhellmann's chnage) | 17:08 |
| *** camunoz has joined #openstack-infra | 17:10 | |
| *** Apoorva has quit IRC | 17:10 | |
| jeblair | okay, so in order for a job to run, it has to match not only a variant on the project pipeline, but at least one global definition/variant of the job. since the only definition of openstack-tox-py35 contains those irrelevant-files matchers, it should do what we want. | 17:11 |
| clarkb | jeblair: it being your change or it being dhellmann's or both? | 17:12 |
| jeblair | clarkb: well, both. but i was responding to myself about my own change. | 17:12 |
| jeblair | (i'm writing out the full matrix of possibilities real quick to triple check) | 17:13 |
| *** cshastri has joined #openstack-infra | 17:14 | |
| jeblair | okay, i'm wrong -- if a change modified files in both deliverables/ and docs/, with my proposed change it would run (because it would match the project-pipeline variant on the doc/ file, and it would match the job definition on the deliverabels/ file) | 17:16 |
| *** hashar is now known as hasharAway | 17:16 | |
| jeblair | (my change only really handles a case where a change touches deliverables/ but nothing else) | 17:16 |
| jeblair | so yeah, to be equivalent, we'd have to copy everything over. | 17:17 |
| *** harlowja has joined #openstack-infra | 17:19 | |
| pabelanger | looks like ze02 is upwards of 40.00 load | 17:20 |
| pabelanger | checking why | 17:20 |
| jeblair | pabelanger: i think ianw posted a paste of a log entry that probably explains it. a bug in zuul. check the scrollback. | 17:20 |
| pabelanger | looking | 17:20 |
| jeblair | jlvillal: almost certainly related | 17:20 |
| EmilienM | pabelanger: would it be possible to re-enqueue 508870 in the gate please? we had to revert OVS 2.8 in RDO & and need 508870 for the 51 days promotion... we think it's our last blocker now | 17:20 |
| jlvillal | jeblair: Okay. Thanks. | 17:21 |
| pabelanger | AttributeError: 'ExecutorExecuteWorker' object has no attribute 'unregisterFunction' | 17:21 |
| jeblair | pabelanger: that's the one | 17:21 |
| pabelanger | yah, I can confirm that | 17:21 |
| *** jpena|brb is now known as jpena | 17:21 | |
| pabelanger | kk, let me see if I can patch it | 17:21 |
| jeblair | jlvillal: is the current behavior good? | 17:21 |
| jlvillal | jeblair: Not so much. Since it votes +1 based on just voting for unit tests and pep8 jobs. | 17:22 |
| *** trown is now known as trown|lunch | 17:22 | |
| *** prometheanfire has quit IRC | 17:22 | |
| jlvillal | jeblair: None of the devstack jobs get run, it appears | 17:22 |
| jeblair | jlvillal: okay, let me dig a little deeper | 17:22 |
| jlvillal | jeblair: Thanks | 17:23 |
| *** tmorin has quit IRC | 17:23 | |
| *** markvoelker has joined #openstack-infra | 17:23 | |
| *** hasharAway has quit IRC | 17:23 | |
| *** camunoz has quit IRC | 17:24 | |
| jeblair | jlvillal: yeah, https://review.openstack.org/513697 is what you need. sorry about the change. the good news though is that at least now it will be running the correct jobs on ocata, not some unpredictable mismatch of master + ocata jobs. | 17:24 |
| *** dhinesh has joined #openstack-infra | 17:24 | |
| jeblair | jlvillal: you should be able to rebase or depends-on that to get other changes moving without waiting for it to land | 17:24 |
| jlvillal | jeblair: Thanks. | 17:25 |
| jlvillal | vdrok: ^^^^ | 17:25 |
| *** rlandy is now known as rlandy|brb | 17:26 | |
| openstackgerrit | Paul Belanger proposed openstack-infra/zuul feature/zuulv3: Fix syntax with gear unRegisterFunction() https://review.openstack.org/516403 | 17:26 |
| *** catintheroof has joined #openstack-infra | 17:27 | |
| *** jcoufal has joined #openstack-infra | 17:27 | |
| *** markvoelker has quit IRC | 17:28 | |
| *** hashar has joined #openstack-infra | 17:28 | |
| clarkb | jeblair: are you ok with dhellmann's change then to avoid duplication or do you think we shoudl duplicate in this case? | 17:29 |
| *** jcoufal_ has quit IRC | 17:29 | |
| *** markvoelker has joined #openstack-infra | 17:30 | |
| *** rbrndt has quit IRC | 17:30 | |
| openstackgerrit | Ronelle Landy proposed openstack-infra/tripleo-ci master: Use rdoproject-based settings for containers and dlrn_hash for images https://review.openstack.org/514668 | 17:30 |
| jeblair | clarkb: i'm okay with dhellmann's change. i voted +2 on it and -1 on my own. | 17:30 |
| *** rbrndt has joined #openstack-infra | 17:30 | |
| *** rbrndt has quit IRC | 17:30 | |
| *** nikhil has joined #openstack-infra | 17:31 | |
| clarkb | ok /me goes to review it then | 17:31 |
| clarkb | oh AJaeger wins | 17:31 |
| *** Apoorva has joined #openstack-infra | 17:32 | |
| *** Apoorva has quit IRC | 17:32 | |
| *** catintheroof has quit IRC | 17:32 | |
| *** Apoorva has joined #openstack-infra | 17:32 | |
| *** mat128 has quit IRC | 17:32 | |
| *** rbrndt has joined #openstack-infra | 17:32 | |
| *** markvoelker has quit IRC | 17:34 | |
| dmsimard | infra-root: It's a bit hard for me to troubleshoot the elastic-recheck issue without being able to poke at the machine running log processor, I've documented breadcrumbs I know about in the zuulv3-issues pad. | 17:34 |
| *** slaweq has joined #openstack-infra | 17:34 | |
| *** mat128 has joined #openstack-infra | 17:35 | |
| clarkb | dmsimard: you should be able to run it all locally fwiw | 17:35 |
| dmsimard | clarkb: including connecting to the geard server ? | 17:35 |
| clarkb | elastic rechecks relies completely on unauthenticated read only access to elasticsearch | 17:35 |
| clarkb | dmsimard: we may be getting streams crossed here | 17:36 |
| dmsimard | clarkb: I have a feeling the issue could be around here https://git.openstack.org/cgit/openstack-infra/puppet-log_processor/tree/files/log-gearman-worker.py#n243 | 17:36 |
| pabelanger | IIRC, I was able to get it running locally before we did the great upgrade from precise to trusty | 17:36 |
| dhellmann | jeblair, clarkb, AJaeger : thanks for puzzling through that. | 17:36 |
| *** sflanigan has quit IRC | 17:37 | |
| *** sflanigan has joined #openstack-infra | 17:37 | |
| *** sflanigan has joined #openstack-infra | 17:37 | |
| *** catintheroof has joined #openstack-infra | 17:37 | |
| jeblair | dhellmann: thanks for your patience :) | 17:37 |
| clarkb | dmsimard: the bug is in project-config/roles/submit-log-processor-jobs/library/submit_log_processor_jobs.py | 17:38 |
| clarkb | dmsimard: we don't remove the .gz suffix from the file paths when we submit them | 17:38 |
| clarkb | dmsimard: line 89 output['source_url'] = output['event']['fields']['log_url'] should remoev the .gz if present I think | 17:39 |
| *** slaweq has quit IRC | 17:39 | |
| dmsimard | clarkb: but even then, I mean... it looks like log-gearman-worker was written to handle gzipped content ? | 17:39 |
| jlvillal | jeblair: vdrok: So we just approved: https://review.openstack.org/#/c/513697/ and I see it in the gate. But it doesn't have any devstack jobs as seen in the Zuul v3 status page. | 17:39 |
| jlvillal | jeblair: vdrok: So that seems...worrisome... | 17:40 |
| dmsimard | clarkb: we can try removing ".gz" if there's one, it doesn't cost us much, I guess | 17:40 |
| jeblair | jlvillal: looking | 17:40 |
| clarkb | dmsimard: yes it is, apache serves /foo and /foo.gz as the same thing. And if you tell apache that you can handle gzip'd content you get gzipped content regardless of path | 17:40 |
| vdrok | jeblair: also, small question regarding the recent email, do we need to backport the legacy-jobs.yaml in addition to project.yaml and playbooks? | 17:40 |
| clarkb | dmsimard: so the "canonical" file name in our case (at least based on all the queries we've written) is to not include the gz (and we didn't include it in the old log jenkins workers iirc) | 17:41 |
| vdrok | meaning this one https://github.com/openstack/ironic-lib/blob/master/zuul.d/legacy-ironic-lib-jobs.yaml | 17:41 |
| AJaeger | jeblair: the check-requirement change fails now! Now one more test... | 17:42 |
| *** catintheroof has quit IRC | 17:42 | |
| dmsimard | clarkb: makes sense... the reason why I wondered if the issue resided elsewhere is because it worked before | 17:42 |
| *** ldnunes has quit IRC | 17:42 | |
| dmsimard | clarkb: regardless of elastic-recheck, logstash.o.o doesn't contain much log entries at all matching filenames job-output.txt/job-output.txt.gz | 17:42 |
| dmsimard | oh, hang on | 17:42 |
| dmsimard | I'm confusing elastic-recheck and submitting gear jobs | 17:43 |
| *** ldnunes has joined #openstack-infra | 17:43 | |
| jeblair | jlvillal, vdrok: yes, that needs to be backported as well. that's why those aren't showing up. sorry i missed that was absent from 513697 | 17:43 |
| clarkb | dmsimard: it worked before beacuse system-config/modules/openstack_project/files/logstash/jenkins-log-client.yaml did not specify the .gz and did not do file matching uses regexes like current system does | 17:43 |
| vdrok | jeblair: no worries, thank you. will add those as well | 17:43 |
| *** sflanigan has quit IRC | 17:43 | |
| clarkb | dmsimard: so all of the jobs we submitted had no .gz suffix. So all of the e-r queries don't query for an e-r suffix. The new role in ansible does a file pattern match against the actual files then submits them as they are on disk (this has the .gz) and so no all of the filenames in elasticsearch get indexed with the .gz | 17:44 |
| *** cshastri has quit IRC | 17:44 | |
| *** sflanigan has joined #openstack-infra | 17:44 | |
| clarkb | either path should work with apache so its mostly a matter of just getting the indexed "filename" and "tags" attributes to line up the way we ant | 17:44 |
| jeblair | vdrok: your link was for the ironic-lib repo, but i believe there's a similar file in ironic which is what jlvillal was asking about | 17:44 |
| clarkb | we could decide to just go to .gz moving forward or go back to dropping it entirely and be backward compatible | 17:45 |
| vdrok | yeah, exactly | 17:45 |
| jeblair | vdrok: i assume the same thing needs to happen in both repos. but i just wanted to be clear. | 17:45 |
| *** slaweq has joined #openstack-infra | 17:45 | |
| jlvillal | jeblair: Understood. Thanks. | 17:46 |
| andreykurilin | hi! I'm experimenting with zuul_v3 (trying to do simple things), but I'm getting weird error while parsing .zuul.yaml. Can anyone help me? | 17:46 |
| clarkb | dmsimard: basically we aren't putting the data into a the database the same way that we were so the existing queries don't match that data. We can either update the queries or update the data inserts | 17:47 |
| *** Goneri has joined #openstack-infra | 17:47 | |
| dmsimard | clarkb: yeah, I was confused between submitting, processing and e-r | 17:48 |
| jeblair | andreykurilin: of course! do you have a link to the change? | 17:49 |
| andreykurilin | jeblair: sure, thanks. https://review.openstack.org/#/c/508839/5/zuul.d/zuul.yaml | 17:50 |
| dmsimard | clarkb: still a bit confused... pretend we're submitting with the .gz suffix right now. Why isn't there more results for filename:"job-output.txt.gz" ? http://logstash.openstack.org/#dashboard/file/logstash.json?query=filename%3A%5C%22job-output.txt.gz%5C%22 | 17:50 |
| clarkb | dmsimard: to update the inserts we would drop the .gz suffix in the log processor submit job role. To update the queries we'd update elastic recheck's queries to all include .gz for filename and url and tags now | 17:50 |
| andreykurilin | jeblair: an error says "Configuration item has more than one key." | 17:50 |
| *** dprince has quit IRC | 17:50 | |
| andreykurilin | jeblair: I checked the number of keys after loading file via yaml.safe_load and eveythink looks right | 17:51 |
| jeblair | andreykurilin: ah, the playbooks can't go in zuul.d. you can put them anywhere in the repo except there; that dir can only have zuul config files. we usually put them in the top level (just playbooks/) | 17:52 |
| clarkb | dmsimard: there are we are just barely under 7 days behind | 17:52 |
| clarkb | dmsimard: so if you stretch the time frame out to 7 days you'll see a ton of hits and the received at field says they are recent | 17:53 |
| dmsimard | clarkb: oh no wonder I'm confused.. is it going to catch up at this rate ? | 17:53 |
| andreykurilin | jeblair: ok, got it. will try. thanks:) | 17:53 |
| jeblair | clarkb: just in time to delete them? | 17:53 |
| clarkb | its possible the actual indexing is completely fubar from the change in console log formats | 17:53 |
| *** camunoz has joined #openstack-infra | 17:53 | |
| clarkb | ya we might want to consider restarting the logstash server and only roll forward at this point | 17:53 |
| jeblair | or, i guess, we still keep 10 days or whatever, it's just from day -17 to -7 ? | 17:53 |
| clarkb | would probably be easier to debug performance issues that way too | 17:53 |
| clarkb | jeblair: no its day -10 to now regardless of how far behind we are | 17:54 |
| jeblair | oh ok | 17:54 |
| clarkb | my guess is that at least some of this is related to the format change of the console logs | 17:54 |
| clarkb | we may need to update the logstash rules for that and give it a new type we can index more quikcly | 17:54 |
| andreykurilin | jeblair: does it mean that I can split zuul.yaml by several files (job descriptions and which jobs the project check should launch) and put them in zuul.d and everything will work? | 17:55 |
| jeblair | andreykurilin: yes. keep in mind in some cases, job ordering can matter, so you may want to keep all the jobs in one file, or name the files so they lexigraphically sort in the right order. | 17:56 |
| andreykurilin | jeblair: thanks! | 17:56 |
| jeblair | andreykurilin: https://docs.openstack.org/infra/zuul/feature/zuulv3/user/config.html#configuration-loading and the section after have more details | 17:57 |
| *** Guest548 has quit IRC | 17:57 | |
| andreykurilin | great, thx | 17:58 |
| dmsimard | clarkb: oh, it's probably failing to grok everything I guess | 17:58 |
| clarkb | dmsimard: I do notice a lot of message:"%{logmessage}" 's which implies something is breaking ya | 17:59 |
| clarkb | though that didn't hit a grok parse failure but I think it effectively is | 17:59 |
| openstackgerrit | Merged openstack-infra/openstack-zuul-jobs master: do not run the py35 job for deliverable file changes https://review.openstack.org/516394 | 17:59 |
| *** ijw has quit IRC | 18:00 | |
| clarkb | dmsimard: the way I test logsatsh with problems like this is I run a local logstash daemon of whatever version we are running (we can look that up). Then replace our tcp input and elasticsearch output with file input from some local file and stdout output | 18:00 |
| *** ijw has joined #openstack-infra | 18:01 | |
| clarkb | but I'm guessing formatting and whitespace handling etc is just different enough that we may be having some difficulty. It is also possible that the volume is significantly higher for some reason | 18:01 |
| clarkb | like the regex method is adding a lot more files than we used to have unexpectedly | 18:01 |
| dmsimard | clarkb: yeah last time I tried to test that I ended up with a logstash that was way too recent :) | 18:01 |
| dmsimard | I'll probably just try puppet-openstackci next time | 18:01 |
| openstackgerrit | Paul Belanger proposed openstack-infra/project-config master: Remove legacy jobs for ansible-role-zuul https://review.openstack.org/516412 | 18:01 |
| *** trown|lunch is now known as trown | 18:02 | |
| jeblair | if you're curious: 51/78 zuulv3-paths changes outstanding (27 merged or abandoned so far) | 18:02 |
| jeblair | and about half of those have at least a +2 | 18:02 |
| clarkb | the number of hours we are behind is definitely increasing according to e-r | 18:03 |
| clarkb | we also lost 5 of our indexers and gave them to zuul for use as mergers? or executros (maybe both) | 18:03 |
| pabelanger | clarkb: we should be able to bring them back online now that we deleted zuul-launchers | 18:04 |
| pabelanger | atleast 6 of them | 18:04 |
| clarkb | logstash-2017.10.23 (the index for that day) has 1.35 billion records | 18:05 |
| clarkb | which is high compared to where we were before iirc | 18:05 |
| *** markvoelker has joined #openstack-infra | 18:05 | |
| *** nicolasbock has quit IRC | 18:05 | |
| clarkb | so I think the biggest culprit is likely volume | 18:05 |
| clarkb | we are indexing about 2-2.5k requests per second which seems low too. Peak is closer to 10k iirc | 18:06 |
| clarkb | so could be a combination of volume, fewer indexers, and slower logstash processing. I'd probably start by tackling one at a time. Maybe add back in the 5 indexers | 18:07 |
| clarkb | then look at volume then look at logstash performance? | 18:07 |
| *** slaweq has quit IRC | 18:08 | |
| *** diablo_rojo has joined #openstack-infra | 18:09 | |
| *** markvoelker has quit IRC | 18:09 | |
| pabelanger | sure, shouldn't be difficult to bring them back online | 18:09 |
| *** rlandy|brb is now known as rlandy | 18:09 | |
| pabelanger | see how things preform | 18:10 |
| *** markvoelker has joined #openstack-infra | 18:10 | |
| *** slaweq has joined #openstack-infra | 18:13 | |
| *** jcoufal_ has joined #openstack-infra | 18:13 | |
| jeblair | AJaeger: pointed out an error in 516208 | 18:14 |
| *** jcoufal has quit IRC | 18:14 | |
| jeblair | clarkb, dmsimard: http://paste.openstack.org/show/624996/ is still a thing | 18:15 |
| clarkb | jeblair: ok good to know though I don't think that is why we are behind based on the 1.35 billion records for the 23rd | 18:15 |
| dmsimard | jeblair: those are from the executor right ? because they're not in job-output.txt/.json | 18:16 |
| jeblair | clarkb: right. if anything, that's causing the backlog to be less worse | 18:16 |
| jeblair | dmsimard: it's from the live log stream | 18:16 |
| dmsimard | jeblair: ah. | 18:16 |
| jeblair | that error still has me stumped. like, my next step would be to try to get a packet capture. | 18:18 |
| *** slaweq has quit IRC | 18:18 | |
| *** jpena is now known as jpena|off | 18:18 | |
| *** markvoelker has quit IRC | 18:19 | |
| *** markvoelker has joined #openstack-infra | 18:19 | |
| *** rkukura has joined #openstack-infra | 18:19 | |
| jeblair | the ansible module for that is supposed to wait for a connection before attempting to submit the job | 18:19 |
| *** markvoelker_ has joined #openstack-infra | 18:20 | |
| jeblair | ohh.... | 18:21 |
| jeblair | the job queue is kind of big | 18:21 |
| jeblair | there's a 30 second timeout on getting a response to submitting a job | 18:21 |
| jeblair | and that error shows up 30s after the task starts | 18:21 |
| clarkb | this may be an argument for cutting our losses and just starting fresh | 18:21 |
| AJaeger | jeblair: ooh - thanks | 18:22 |
| jeblair | so it may be that we're connecting, but the logstash geard takes more than 30s to respond to submit job because of the 600k backlog. | 18:22 |
| clarkb | could also be part of the slowness if workers are getting jobs slowly | 18:22 |
| *** markvoelker has quit IRC | 18:23 | |
| clarkb | and then base performance on a known good start state | 18:23 |
| *** pblaho has quit IRC | 18:26 | |
| *** jcoufal has joined #openstack-infra | 18:26 | |
| *** e0ne has joined #openstack-infra | 18:27 | |
| dhinesh | is there an nodepool element for java installations? | 18:27 |
| clarkb | dhinesh: I'm not sure, you can look in diskimage-builder/elements if not it is easy to add one yourself if you just need to install packages | 18:28 |
| clarkb | dhinesh: you can see the infra-package-needs element as an example | 18:28 |
| *** jcoufal_ has quit IRC | 18:29 | |
| clarkb | I'm going to grab an early lunch. If the e-r graph shows we are even further behind my vote is we just go ahead and restart the logstash server and see if that helps stabilize things | 18:29 |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/openstack-zuul-jobs master: Cleanup ironic-inspector legacy jobs https://review.openstack.org/515738 | 18:29 |
| pabelanger | clarkb: I can stand up a few more workers if we want to do that | 18:30 |
| clarkb | *restart the gearman server on logstash.o.o | 18:30 |
| clarkb | pabelanger: I think that would be good too | 18:30 |
| pabelanger | kk | 18:30 |
| pabelanger | I'll get some coffee and start | 18:30 |
| clarkb | tyty | 18:30 |
| dhinesh | clarkb: ok thanks | 18:30 |
| jeblair | clarkb, dmsimard: i think geard is slow because of stats reporting. | 18:30 |
| jeblair | the _updateStats method is pretty inefficient. everything else about adding or removing jobs should be fast, but that one is slow. it's likely causing jobs to be removed more slowly than they otherwise should be. | 18:31 |
| *** markvoelker has joined #openstack-infra | 18:32 | |
| *** jcoufal_ has joined #openstack-infra | 18:32 | |
| jeblair | fortunately, it can be improved fairly easily. | 18:33 |
| *** markvoelker_ has quit IRC | 18:33 | |
| *** jcoufal has quit IRC | 18:34 | |
| *** rkukura has quit IRC | 18:36 | |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config master: Remove zuul/mapping and job https://review.openstack.org/516029 | 18:36 |
| AJaeger | jeblair, mordred , can we remove this now? ^ | 18:36 |
| *** dprince has joined #openstack-infra | 18:37 | |
| *** markvoelker has quit IRC | 18:37 | |
| openstackgerrit | Merged openstack-infra/project-config master: Remove legacy ironic-inspector jobs https://review.openstack.org/515737 | 18:37 |
| *** rkukura has joined #openstack-infra | 18:37 | |
| dmsimard | jeblair: happy you feel that way, I'm totally in uncharted territory here :) | 18:37 |
| *** dtantsur is now known as dtantsur|afk | 18:40 | |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/zuul feature/zuulv3: Remove zuul-migrate job https://review.openstack.org/516028 | 18:40 |
| AJaeger | jeblair: and this one can go as well, can't it? ^ | 18:40 |
| *** felipemonteiro_ has quit IRC | 18:41 | |
| *** ccamacho has quit IRC | 18:41 | |
| *** markvoelker has joined #openstack-infra | 18:41 | |
| *** catintheroof has joined #openstack-infra | 18:43 | |
| *** mikal has quit IRC | 18:43 | |
| *** felipemonteiro has joined #openstack-infra | 18:44 | |
| jeblair | clarkb, pabelanger: how do you feel about dropping the "workers" line from the gearman queue graphs? | 18:44 |
| jeblair | it's not too hard to keep, but if it's not useful, it'd be slightly more efficient to drop it | 18:45 |
| *** rwsu has quit IRC | 18:45 | |
| pabelanger | jeblair: I don't use it myself | 18:45 |
| *** jcoufal has joined #openstack-infra | 18:45 | |
| *** felipemonteiro_ has joined #openstack-infra | 18:45 | |
| *** jcoufal_ has quit IRC | 18:45 | |
| openstackgerrit | James E. Blair proposed openstack-infra/gear master: server: make stats more efficient https://review.openstack.org/516422 | 18:47 |
| jeblair | pabelanger, dmsimard, clarkb: ^ happy to put workers back in if folks want it | 18:47 |
| *** catintheroof has quit IRC | 18:48 | |
| clarkb | ya we can probably drop.it | 18:49 |
| *** felipemonteiro has quit IRC | 18:49 | |
| *** catintheroof has joined #openstack-infra | 18:49 | |
| *** salv-orlando has joined #openstack-infra | 18:52 | |
| *** ijw has quit IRC | 18:54 | |
| *** mikal has joined #openstack-infra | 18:55 | |
| *** salv-orl_ has quit IRC | 18:55 | |
| *** jcoufal_ has joined #openstack-infra | 18:57 | |
| AJaeger | clarkb, do you want to review https://review.openstack.org/516010 and https://review.openstack.org/516009 now since the linters jobs are voting? those fix infra/publications publishing. And once those in, please run some dummy commit to see whether it works... | 18:58 |
| *** baoli has quit IRC | 18:59 | |
| *** jcoufal has quit IRC | 18:59 | |
| *** andreww has quit IRC | 19:00 | |
| *** dprince has quit IRC | 19:01 | |
| *** dprince has joined #openstack-infra | 19:01 | |
| *** rbrndt has quit IRC | 19:01 | |
| *** ldnunes has quit IRC | 19:04 | |
| openstackgerrit | Paul Belanger proposed openstack-infra/system-config master: mqtt-root-CA.pem.crt requires ::logstash to be setup https://review.openstack.org/516426 | 19:06 |
| *** ijw has joined #openstack-infra | 19:06 | |
| pabelanger | clarkb: ^logstash-worker launch issue | 19:08 |
| *** baoli has joined #openstack-infra | 19:09 | |
| *** diablo_rojo has quit IRC | 19:12 | |
| *** shardy has quit IRC | 19:14 | |
| openstackgerrit | Sam Yaple proposed openstack-infra/project-config master: Publish requirements loci images to DockerHub https://review.openstack.org/512941 | 19:16 |
| *** yamamoto has quit IRC | 19:16 | |
| *** rkukura has quit IRC | 19:17 | |
| *** rkukura has joined #openstack-infra | 19:19 | |
| *** diablo_rojo has joined #openstack-infra | 19:20 | |
| *** Goneri has quit IRC | 19:24 | |
| *** markvoelker has quit IRC | 19:34 | |
| *** markvoelker has joined #openstack-infra | 19:34 | |
| *** rwsu has joined #openstack-infra | 19:35 | |
| *** mat128 has quit IRC | 19:35 | |
| *** mat128 has joined #openstack-infra | 19:36 | |
| *** ijw has quit IRC | 19:40 | |
| *** ociuhandu has quit IRC | 19:40 | |
| clarkb | ok back | 19:40 |
| clarkb | pabelanger: I've approved your fix | 19:41 |
| clarkb | jeblair: gear change lgtm | 19:42 |
| clarkb | I figure we can get that in and released then restart the logstash.o.o geard to pick it up? | 19:43 |
| clarkb | picking up the infra update session again, does anyone have a native zuulv3 job that would be a good example for that atlk? maybe the devstack job? | 19:45 |
| fungi | i suppose that also raises the question as to whether you want to showcase how flexibly it can define complex jobs, or how simple jobs are under v3 | 19:48 |
| fungi | the native v3 devstack jobs are a lot more understandable than devstack-gate, but if the audience is people who haven't already tried reading through the d-g scripts then the v3 devstack job is more likely to frighten them ;) | 19:49 |
| clarkb | ha | 19:49 |
| clarkb | This IS simpler honest | 19:49 |
| clarkb | I think I may put the devstack job in to have something but can always swap out with something else later | 19:50 |
| clarkb | fungi: do you want to be second reviewer on https://review.openstack.org/#/c/516422/ maybe? | 19:50 |
| clarkb | *second +2 reviewer | 19:50 |
| fungi | a smaller-scope job may be easier to fit entire playbooks on the screen, and take less explaining (at the loss of demonstrating some more advanced options of course) | 19:51 |
| *** rkukura has quit IRC | 19:51 | |
| fungi | so really depends on what direction you want to take it | 19:51 |
| ianw | pabelanger: so the executor should have stopped itself when the load gets to high, but actually crashed itself? | 19:53 |
| *** rwsu has quit IRC | 19:55 | |
| *** jcoufal has joined #openstack-infra | 19:55 | |
| clarkb | fungi: pabelanger ianw https://review.openstack.org/515950 updated with a heavily modified devstack job definition | 19:57 |
| clarkb | hopefully illustrates things without being too large to fit on a screen | 19:57 |
| *** jcoufal_ has quit IRC | 19:58 | |
| *** rbrndt has joined #openstack-infra | 20:00 | |
| *** rwsu has joined #openstack-infra | 20:02 | |
| *** amoralej is now known as amoralej|off | 20:02 | |
| *** baoli has quit IRC | 20:04 | |
| clarkb | jeblair: are we submitting logstash jobs on aborted zuul jobs? | 20:04 |
| clarkb | jeblair: that may account for extra volume if so (we had previously skipped them) | 20:05 |
| *** rkukura has joined #openstack-infra | 20:05 | |
| clarkb | dmsimard: ^ | 20:05 |
| *** bobh has joined #openstack-infra | 20:06 | |
| mwhahaha | so if a change is stuck in the check queue, is there something that can be done to kick it out? http://zuulv3.openstack.org/static/stream.html?uuid=a15efdf243e34ee6bd5e97b241b7065a&logfile=console.log | 20:07 |
| mwhahaha | looks like it timed out but hasn't updated in some time | 20:07 |
| mwhahaha | change 512466,5 | 20:07 |
| *** mat128 has quit IRC | 20:07 | |
| clarkb | a new patchset is the big hammer reset button | 20:07 |
| *** dimak has quit IRC | 20:08 | |
| *** lihi has quit IRC | 20:08 | |
| *** leyal has quit IRC | 20:08 | |
| fungi | abandoning and restoring may also work | 20:08 |
| clarkb | depending on why it is stuck we may have to restart some bit of zuul to get it to notice new state | 20:08 |
| AJaeger | jeblair: the reuqirements change looks fine, I tested with success and failing now, added links tohttps://review.openstack.org/#/c/516208 - thanks for help! | 20:08 |
| *** oanson has quit IRC | 20:08 | |
| *** snapiri has quit IRC | 20:08 | |
| mwhahaha | well it seem that zuul is stuck so I don't want to touch it unless i have to | 20:08 |
| mwhahaha | also it's a cherrypick w/o anything to rebase :D So i guess i could force it but it seems that there might be a few of these | 20:09 |
| *** jcoufal_ has joined #openstack-infra | 20:09 | |
| mwhahaha | 507796,7 is also stuck in a similar fashion | 20:09 |
| fungi | mwhahaha: check the stream there... "RUN END RESULT_TIMED_OUT" | 20:09 |
| * mwhahaha is aware | 20:10 | |
| mwhahaha | but it seems to be stuck on copying files from node | 20:10 |
| fungi | ahh, i see, it's been on the copy files task, yes | 20:10 |
| dmsimard | clarkb: depends what you mean by aborted, if the jobs are killed without running POST-RUNs it won't be submitted I guess ? | 20:10 |
| fungi | mwhahaha: i wonder how long it will take to time out that task | 20:10 |
| jeblair | clarkb: we should not be running any post-playbooks on aborted jobs, so if we see that, it's a bug | 20:10 |
| *** jcoufal has quit IRC | 20:10 | |
| jeblair | dmsimard: ^ | 20:10 |
| mwhahaha | fungi: yea i wanted to raise the issue, i guess i'll just wait | 20:11 |
| dmsimard | jeblair: I suspected as much | 20:11 |
| fungi | jeblair: does the timeout used for the run task also get used for all other tasks, or are there separate timeouts for those? | 20:11 |
| clarkb | gotcha so an aborted job is properly stopped before it can get that far eg no logs at all? | 20:11 |
| jeblair | fungi: same timeout. that's a mis-feature we should correct, but we haven't discussed how yet. | 20:11 |
| jeblair | (because obviously it's way too long. however, it will *eventually* hit) | 20:12 |
| fungi | i was watching one the other day where the node had apparently gone missing mid-job, and all the post tasks then waited 30 minutes each to time out before the job could finally 5report | 20:12 |
| jeblair | clarkb: yes, including even if we abort in an early post-playbook, we won't run later ones. | 20:12 |
| mwhahaha | these failures seem to indicate the vm went missing | 20:12 |
| fungi | mwhahaha: okay, so my guess is something happened to that node and the post playbooks will each wait for up to the job timeout (some 3 hours from the looks of it?) before moving on to the next | 20:12 |
| jeblair | fungi: that also feels like something wrong with ansible; we shouldn't be waiting 30m for ansible to tell us it can't connect. | 20:13 |
| mwhahaha | sooooo i should probably force the issue i guess? | 20:13 |
| mwhahaha | abandon/restore | 20:13 |
| fungi | jeblair: yeah, not sure how ansible is failing to notice that it can't run a post task | 20:13 |
| mwhahaha | there's some nova patch with the same problem | 20:14 |
| mwhahaha | i'll abandon/restore the tripleo patches stuck with this | 20:14 |
| *** slaweq has joined #openstack-infra | 20:14 | |
| jeblair | at some point we should look more closely at one of these and see what we could improve | 20:14 |
| *** baoli has joined #openstack-infra | 20:15 | |
| fungi | mwhahaha: that seems to have done it | 20:15 |
| fungi | "POST-RUN END RESULT_ABORTED" | 20:15 |
| fungi | i'm guessing that corresponds to your abandoning | 20:16 |
| mwhahaha | yea | 20:16 |
| clarkb | jeblair: thank you for confirming | 20:16 |
| mwhahaha | i thought there used to be a shorter timeout on the post log stuff | 20:16 |
| * mwhahaha shrugs | 20:16 | |
| mwhahaha | i'll let you folks figure it out :D | 20:16 |
| *** salv-orlando has quit IRC | 20:17 | |
| fungi | mwhahaha: yeah, jeblair noted above that it's a misfeature needing correcting when someone gets time | 20:17 |
| *** yamamoto has joined #openstack-infra | 20:17 | |
| *** salv-orlando has joined #openstack-infra | 20:17 | |
| *** bobh has quit IRC | 20:18 | |
| *** eharney has quit IRC | 20:18 | |
| jeblair | for starters, do we make it a system-wide setting? like "each pre/post playbook gets X minutes to run"? or do we say the the entire pre phase gets X minutes, and the entire post phase Y minutes? do we let jobs override it? | 20:19 |
| mwhahaha | there was always the 3hour upper limit for all things i thought | 20:19 |
| mwhahaha | is that gone? | 20:19 |
| mwhahaha | in the past when this failed, it always reported back at about 3hours 5mins or something | 20:20 |
| jeblair | mwhahaha: i believe that was just convention | 20:20 |
| *** kgiusti has left #openstack-infra | 20:20 | |
| jeblair | mwhahaha: that was likely the job timeout | 20:20 |
| mwhahaha | no | 20:20 |
| mwhahaha | job timeout was 2:45 | 20:20 |
| mwhahaha | zuul timeout was 3 | 20:20 |
| mwhahaha | at least that's how it was showing up when it occured | 20:21 |
| *** salv-orlando has quit IRC | 20:21 | |
| mwhahaha | our limit was 2:30 max job time and if we ran into this type of situation where the vm went away it would end around 3 hours | 20:21 |
| jeblair | mwhahaha: i don't know what the math was, and it's probably not worth looking at now, but we didn't have a hard-coded limit. some jobs implemented their own timeouts inside of the job timeout though. | 20:21 |
| *** bobh has joined #openstack-infra | 20:22 | |
| fungi | timeouts under v2 were at job-level granularity owing to inheriting from jenkins design | 20:22 |
| fungi | devstack-gate (and some similar job frameworks) implemented a sort of "soft" timeout where they estimated their setup.teardown activities and then killed the main job payload leaving time to copy logs and what have you | 20:22 |
| jeblair | mwhahaha: oh, if the node went away, maybe there was a 30m watchdog for post, and a 5 minute grace period? | 20:22 |
| mwhahaha | yea | 20:23 |
| mwhahaha | that seems to be about right | 20:23 |
| mwhahaha | either way it seems that you would want an upper limit from the zuul operator standpoint to say no job can run longer than X | 20:23 |
| fungi | though i tend to agree the real issue here is that ansible doesn't seem to indicate that it's having trouble getting a response from the node | 20:24 |
| fungi | and we should likely have something global to timeout fairly quickly on connectivity/response issues | 20:24 |
| fungi | independent of job-level or playbook-level or task-level timeouts | 20:25 |
| mwhahaha | that would be ideal but short of adding additional functionality to constantly monitor status of nodes, being able to says the entire process can't exceed X would be a start | 20:25 |
| jeblair | mwhahaha: we actually do have that now -- there's a max job timeout. so once we have pre/post timeouts implemented, that will be more sane. | 20:25 |
| *** e0ne has quit IRC | 20:25 | |
| mwhahaha | what's the current max job timeout right now? | 20:26 |
| jeblair | mwhahaha: equal to the highest value used under zuul v2 :) so probably 3h and change? | 20:26 |
| *** larainema has quit IRC | 20:26 | |
| mwhahaha | but pre/post aren't captured in that value? | 20:26 |
| *** yamamoto has quit IRC | 20:26 | |
| jeblair | mwhahaha: they use the same value. that's the misfeature. | 20:26 |
| mwhahaha | ah so it's 3+3+3 | 20:27 |
| jeblair | yep. or even more. maybe 3+3+3+3 | 20:27 |
| jeblair | 3*(n_playbooks) | 20:27 |
| mwhahaha | :o | 20:27 |
| fungi | granted, a timeout on your run playbook means only post playbooks set to run on failure count toward that | 20:28 |
| clarkb | anyone else willing to review https://review.openstack.org/#/c/516422/ so that we can get a release out and restart gearman on logstash.o.o with that in place? | 20:28 |
| fungi | clarkb: yeah, i have it pulled up from before when you asked. just about done looking at it | 20:29 |
| fungi | approved, in it goes | 20:30 |
| clarkb | AJaeger: are you still around? I have a qusetion about the publications publishing | 20:30 |
| clarkb | AJaeger: specifically https://review.openstack.org/#/c/516010/4/playbooks/publications/run.yaml won't that just copy all the content in the root of the repo ? | 20:30 |
| clarkb | AJaeger: but then in https://review.openstack.org/#/c/516010/4/playbooks/publish/publications.yaml we only copy sphinx output? | 20:31 |
| AJaeger | clarkb: yes, again ;) | 20:32 |
| clarkb | AJaeger: so why do the first copy if we only publish the sphinx output? | 20:32 |
| AJaeger | clarkb: so, run.yaml copies everything over - like it was done with jenkins/jobs | 20:32 |
| AJaeger | clarkb: we do not run sphinx at all. We just use the sphinx module to copy arbitrary data | 20:33 |
| clarkb | I see | 20:33 |
| clarkb | and sphinx will look for _out? | 20:33 |
| AJaeger | clarkb: similar to http://git.openstack.org/cgit/openstack-infra/project-config/tree/playbooks/publish/infra-index.yaml | 20:34 |
| clarkb | AJaeger: ok I think there may be one little bug in the script though I'll leave a comment shortly | 20:34 |
| *** jcoufal_ has quit IRC | 20:34 | |
| *** markvoelker has quit IRC | 20:35 | |
| pabelanger | ianw: yah, it should stop accepting more jobs | 20:35 |
| clarkb | AJaeger: comment posted | 20:35 |
| AJaeger | clarkb: indeed, found one as well - thanks | 20:35 |
| pabelanger | sorry, back from late lunch, returning to launch logstash-workers | 20:37 |
| AJaeger | clarkb: that bug was already in JJB and http://git.openstack.org/cgit/openstack-infra/openstack-zuul-jobs/tree/playbooks/legacy/infra-publications-publish/run.yaml#n75 - good find | 20:38 |
| clarkb | AJaeger: its not set -e so it probably just "worked" | 20:39 |
| openstackgerrit | Merged openstack-infra/system-config master: mqtt-root-CA.pem.crt requires ::logstash to be setup https://review.openstack.org/516426 | 20:39 |
| AJaeger | ;) | 20:39 |
| clarkb | pabelanger: ^ fyi | 20:39 |
| *** bobh has quit IRC | 20:39 | |
| *** ijw has joined #openstack-infra | 20:40 | |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config master: Fix openstack-infra publishing https://review.openstack.org/516010 | 20:42 |
| AJaeger | clarkb: fixed the two problems ^ | 20:42 |
| * AJaeger waves good night | 20:42 | |
| pabelanger | clarkb: danke | 20:42 |
| openstackgerrit | Merged openstack-infra/gear master: server: make stats more efficient https://review.openstack.org/516422 | 20:44 |
| ianw | clarkb: not sure if comments on 515950 are helpful or not :) just some general thoughts | 20:45 |
| *** markvoelker has joined #openstack-infra | 20:45 | |
| *** ijw has quit IRC | 20:46 | |
| *** ijw has joined #openstack-infra | 20:46 | |
| clarkb | ianw: ya I like the idea of demoing the old to new maybe drop my devstack example and replace it with your dib migrated examples? | 20:46 |
| clarkb | we can illustrate the migration and native jobs that way | 20:47 |
| clarkb | and ya I think we can frame volunteering as it is different now and easier to dig in as projects manage their own jobs now | 20:48 |
| *** markvoelker has quit IRC | 20:49 | |
| *** markvoelker has joined #openstack-infra | 20:49 | |
| fungi | i too have added some comments | 20:49 |
| jeblair | clarkb, ianw: i think the import-into-repos thing is easy for folks to get from the docs. i think showing what a v3-native job looks like is the better thing to put in a presentation. | 20:49 |
| clarkb | jeblair: so more like what we haev with devstack example in current ps? | 20:50 |
| fungi | ianw: it's possible, like you mentioned on the other topic in there, that referencing is better than too much detailed rehashing. the sales pitch is already up at https://governance.openstack.org/tc/reference/top-5-help-wanted.html#community-infrastructure-sysadmins | 20:50 |
| jeblair | whether that's the devstack example, or a different one, that's what i'd focus on. we can still highlight that it's in-repo, but highlighting the new functionality is a good topic for this presentation, whereas the in-repoification is better in documents (or a dedicated presentation like pabelangers) | 20:51 |
| clarkb | ya I'll update to a link to pabelangers talk which should get people video | 20:51 |
| ianw | i want to fiddle more with dib logs etc and get it all right. it might be a good example of a non-devstack job ... as much of the devstack jobs should be templated out | 20:55 |
| clarkb | let me update based on current feedback except for the example job stuff then we can iterate on that bit | 20:55 |
| *** jtomasek_ has quit IRC | 20:58 | |
| dmsimard | infra-root: FYI ara 0.14.5 is out, it's the version that is tested to work with 2.4.1, contains the wsgi sqlite middleware and also contains the firefox permalink fix. | 20:59 |
| dmsimard | I suppose it requires an update on the executors to be effective, feel free to do that whenever you'd like, it's backwards compatible with 2.3 and 2.2 | 21:00 |
| jeblair | dmsimard: i assume it's going to happen automatically within 30m or so :) | 21:02 |
| dmsimard | magic | 21:02 |
| *** xingchao has joined #openstack-infra | 21:02 | |
| *** markvoelker_ has joined #openstack-infra | 21:02 | |
| dmsimard | I'll keep an eye out for issues just in case, btw the version of ara is at the top right of the reports. | 21:02 |
| *** jascott1 has quit IRC | 21:02 | |
| *** jascott1 has joined #openstack-infra | 21:03 | |
| jeblair | dmsimard: it will *probably* not actually require any restarts to take effect either. | 21:03 |
| *** markvoelker has quit IRC | 21:04 | |
| *** jascott1 has quit IRC | 21:04 | |
| *** jascott1 has joined #openstack-infra | 21:04 | |
| dmsimard | jeblair: I don't think it would, it'd just be loaded for the next ansible-playbook run | 21:05 |
| dmsimard | although I wonder if there's some race that could be triggered if it's updated in the middle of some runs | 21:05 |
| *** akscram1 has quit IRC | 21:06 | |
| *** jascott1 has quit IRC | 21:06 | |
| *** akscram1 has joined #openstack-infra | 21:07 | |
| *** trown is now known as trown|outtypewww | 21:07 | |
| clarkb | ianw: fungi ok new patchset pushed, fungi can you take a look and see if that addresses your comments? | 21:09 |
| *** smatzek has quit IRC | 21:09 | |
| fungi | sure | 21:09 |
| *** smatzek has joined #openstack-infra | 21:10 | |
| dmsimard | fungi, ianw, clarkb: what I feel is missing from the "infrastructure sysadmin" bit (and I probably mentioned this before) is how to get onboarded/ramped up. For reviews and contributing to projects it's relatively straightforward and I guess you could say infra-root are expected to be core on openstack-infra projects so it's a logical step... | 21:11 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul feature/zuulv3: Add support for override-checkout, deprecate override-branch https://review.openstack.org/516451 | 21:11 |
| *** slaweq has quit IRC | 21:11 | |
| *** esberglu has quit IRC | 21:11 | |
| dmsimard | But then what ? Is it particularly system-config/puppet-openstackci for example ? Or just a matter of gaining more experience on different infra projects ? It's not clear | 21:11 |
| clarkb | fungi: it actually does work in 80x24 fwiw, its just the devstack example gets cut off a little | 21:11 |
| jeblair | i don't remember who needed the override-checkout thing... it didn't make it into the etherpad (until i just put it there) | 21:12 |
| *** slaweq has joined #openstack-infra | 21:12 | |
| fungi | dmsimard: have you had a look at the separate onboarding deck clarkb is prepping for next week? | 21:12 |
| *** esberglu has joined #openstack-infra | 21:12 | |
| dmsimard | fungi: I guess not :/ | 21:12 |
| clarkb | jeblair: it was a neutron plugin iirc | 21:12 |
| fungi | dmsimard: clarkb: maybe the help wanted topic there should call out our onboarding slot on the schedule too? | 21:12 |
| jeblair | dmsimard: yes, an update in the middle of a run could definitely error out. but the window is pretty small. | 21:13 |
| fungi | assuming the project update is on the schedule earlier than the onboarding slot anyway | 21:13 |
| clarkb | fungi: ya though I don't think onboarding is recorded | 21:13 |
| clarkb | fungi: its not the order is in reverse | 21:13 |
| clarkb | dmsimard: https://review.openstack.org/#/c/515853/ fwiw | 21:13 |
| fungi | clarkb: oh well, that's a missed opportunity | 21:13 |
| jeblair | help wanted: recording onboarding session | 21:13 |
| fungi | hah | 21:13 |
| fungi | nice one | 21:13 |
| clarkb | fungi: dmsimard but maybe that deck need a "you want root" slide | 21:13 |
| *** xingchao has quit IRC | 21:14 | |
| dmsimard | I'll have to watch the youtube recording :) | 21:14 |
| fungi | in boston, recording for the onboarding sessions was called out as a nice-to-have for future iterations | 21:14 |
| *** smatzek has quit IRC | 21:14 | |
| *** dhinesh has quit IRC | 21:15 | |
| *** pcaruana has quit IRC | 21:15 | |
| dmsimard | that reminds me | 21:16 |
| *** slaweq has quit IRC | 21:16 | |
| dmsimard | I made this little javascript bookmarklet that redirects me from github.com to git.openstack.org :D | 21:16 |
| clarkb | dmsimard: as far as answering your question (and I'll try to make a slide out of this) I think you are right, step 0 is reviews, step 1 is core on projects. Then its a matter o growing/generating/creating trust based on those reviews because there tends to be straightforward undo type buttons for core review actions but not for root actions necessarily | 21:16 |
| *** esberglu has quit IRC | 21:16 | |
| *** slaweq has joined #openstack-infra | 21:16 | |
| *** slaweq has quit IRC | 21:16 | |
| clarkb | basically looking for judgement from things like reviews and stuff so that we don't need to worry about lack of undo button once handing that out | 21:17 |
| clarkb | that said we've all had our derps so I don't think we expect perfection :) | 21:17 |
| *** salv-orlando has joined #openstack-infra | 21:18 | |
| *** mrunge has quit IRC | 21:18 | |
| dmsimard | clarkb: Basically, some (documented) pointers on what it is when we mention that "infra needs help". When we discussed this internally a few weeks back, some people asked what we needed help with and there was no clear answer. | 21:20 |
| *** dave-mccowan has quit IRC | 21:20 | |
| dmsimard | Some folks aren't much of a "self starter", they'd be willing to help but need to be given a direction. Others, you just throw them and they'll figure out something to help with. | 21:21 |
| *** mrunge has joined #openstack-infra | 21:21 | |
| clarkb | thats fair, though that is also different question than the earlier one. We can be better about writing up a low hanging fruit list | 21:21 |
| dmsimard | I guess we could be doing a better job of tracking the things that we're interested in doing (backlog? storyboard? etc.) | 21:21 |
| clarkb | ya I think its a combo of low hanging fruit (here is good places to get feet wet) and backlog of more important work that people can jump on as they get more involved | 21:22 |
| dmsimard | clarkb: yes, it's a different question but tangentially related to contributing to infra :) | 21:22 |
| *** salv-orlando has quit IRC | 21:22 | |
| clarkb | Today the best place to start is probably with our specs list and particularly priority efforts as far as backloggy stuff goes | 21:22 |
| jeblair | what we need is a self-starter who's motivated to help others find what to work on :) | 21:23 |
| dmsimard | ++ haha | 21:23 |
| clarkb | http://specs.openstack.org/openstack-infra/infra-specs/ ist not going to give you small time chunked tasks, but does give flavor to "this is the work we are doing and what is important" | 21:23 |
| *** esberglu has joined #openstack-infra | 21:23 | |
| jeblair | i mean, i added a smiley, and it's borderline tautological, but in all honesty, maybe that's something we should highlight as being very valuable :) | 21:23 |
| clarkb | we do have http://specs.openstack.org/openstack-infra/infra-specs/#help-wanted | 21:24 |
| clarkb | I'll add ^ to the onboarding presentation as well | 21:24 |
| clarkb | and I guess work to updating that and making sure it reflects current wants/needs | 21:24 |
| dmsimard | jeblair: there's definitely value in someone who's able to organize/feed/unblock work in general | 21:25 |
| SamYaple | can i get a +3 on this ceph stretch mirror? https://review.openstack.org/#/c/513591/ | 21:25 |
| dmsimard | that person is not me | 21:25 |
| * dmsimard is horrible at organizing his work | 21:25 | |
| clarkb | fungi: you are assignee on http://specs.openstack.org/openstack-infra/infra-specs/specs/ptgbot.html is that implemented now? I think it is | 21:25 |
| *** jascott1 has joined #openstack-infra | 21:26 | |
| fungi | clarkb: oh, yep. i'll propose a change to move it to implemented. thanks for the reminder! | 21:27 |
| openstackgerrit | John L. Villalovos proposed openstack-infra/project-config master: ironic: Remove publishing to pypi https://review.openstack.org/516453 | 21:28 |
| openstackgerrit | Jeremy Stanley proposed openstack-infra/infra-specs master: Mark the ptgbot spec implemented https://review.openstack.org/516454 | 21:30 |
| fungi | clarkb: ^ | 21:30 |
| *** esberglu has quit IRC | 21:31 | |
| *** esberglu has joined #openstack-infra | 21:32 | |
| *** esberglu has quit IRC | 21:32 | |
| openstackgerrit | John L. Villalovos proposed openstack-infra/project-config master: ironic: Remove publish-to-pypi add release-openstack-server https://review.openstack.org/516453 | 21:32 |
| *** esberglu has joined #openstack-infra | 21:32 | |
| pabelanger | hmm, iptables is failing to restart on logstash-workers | 21:34 |
| clarkb | pabelanger: are you using xenial? | 21:35 |
| pabelanger | nope, trusty | 21:35 |
| pabelanger | I think it might be something with ipv6 | 21:35 |
| dmsimard | it's iptables-persistent, right ? xenial is netfilter-persistent | 21:35 |
| clarkb | dmsimard: ya | 21:35 |
| clarkb | pabelanger: also it relies on dns iirc | 21:35 |
| pabelanger | yah, running with --keep this time | 21:35 |
| *** smatzek has joined #openstack-infra | 21:36 | |
| *** jcoufal has joined #openstack-infra | 21:36 | |
| clarkb | so if we deleted the dns records for 16-20 and didn't update puppet it may be failing on those lookups | 21:36 |
| *** thorst has quit IRC | 21:38 | |
| pabelanger | k, let me check | 21:38 |
| pabelanger | clarkb: no DNS, so we did delete them | 21:39 |
| clarkb | dmsimard: https://review.openstack.org/516455 starting with that | 21:40 |
| pabelanger | know more in a few minutes | 21:40 |
| clarkb | pabelanger: you can likely add dns records with short TTLs that point to localhost or something, then update them after you have instances booted? | 21:40 |
| *** smatzek has quit IRC | 21:40 | |
| dmsimard | clarkb: root access is granular or it's an all of nothing deal ? | 21:41 |
| clarkb | dmsimard: its basically all or nothing | 21:41 |
| clarkb | dmsimard: thats the crux of the problem | 21:41 |
| dmsimard | oh, yeah, that's a bummer. I thought there was some amount of granularity so you could start little by little to give access to certain things to certain people | 21:41 |
| dmsimard | helps with the trust building exercise I suppose | 21:42 |
| dmsimard | afk food, I'm starting to digest myself | 21:42 |
| clarkb | dmsimard: we can give out root on a host by host basis, but we've generally not done that as a path to root thing | 21:42 |
| SamYaple | you can *totally* trust me | 21:42 |
| SamYaple | im trustable | 21:42 |
| clarkb | instead its been for individuals that aren't interested in having complete system authority and just want to help on a small subset | 21:42 |
| dmsimard | SamYaple: you need a business card with your public key on it | 21:42 |
| *** baoli has quit IRC | 21:42 | |
| *** wolverineav has quit IRC | 21:42 | |
| pabelanger | yah, standing up the infra locally is likey a good path forward. See how everything works, almost everything should be able to run local | 21:43 |
| pabelanger | eg: trusty->xenial migration will likely be a good process to infra-root too. We'll likely need to do that in the next 6 months | 21:44 |
| *** rwsu has quit IRC | 21:44 | |
| pabelanger | eg: making sure puppet manifests work locally on xenial, then fixing issues | 21:44 |
| *** rodrigods has quit IRC | 21:45 | |
| *** rodrigods has joined #openstack-infra | 21:45 | |
| SamYaple | dmsimard: so i wont say which company, but when i put in my public ssh key with the comment 'null@null' i break some profiles.... (*cough*rackspace*cough*) | 21:45 |
| clarkb | pabelanger: we should definitely add that to the help wanted specs list | 21:45 |
| clarkb | pabelanger: which probably means writing a spec for that | 21:45 |
| SamYaple | listen properly sanatize everything. including ssh keys people | 21:45 |
| clarkb | (because you are right that would be a good place to get familiar with everything) | 21:45 |
| pabelanger | clarkb: yah | 21:45 |
| *** rodrigods has quit IRC | 21:45 | |
| *** rodrigods has joined #openstack-infra | 21:45 | |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/os-testr master: Updated from global requirements https://review.openstack.org/503645 | 21:47 |
| *** Sukhdev has joined #openstack-infra | 21:48 | |
| fungi | SamYaple: sounds like they also had the incorrect notion that the comment portion of an authorized keys line was anything other than completely arbitrary data | 21:48 |
| *** aeng has joined #openstack-infra | 21:49 | |
| fungi | inappropriate to assume it contains a @ character even | 21:49 |
| SamYaple | indeed. tis exactly what happened | 21:49 |
| SamYaple | yea the web form required teh @ | 21:49 |
| SamYaple | which is funny | 21:49 |
| fungi | can contain whitespace characters too | 21:49 |
| fungi | same for the pubkey format | 21:49 |
| SamYaple | just like email addresses can contain comments (no seriuosly, that RFC is insane) | 21:49 |
| *** boden has quit IRC | 21:50 | |
| SamYaple | but most places only accept a small subset of valid addresses | 21:50 |
| fungi | ssh-rsa AAAAB3...QoQ== this is my workstation's public key | 21:51 |
| fungi | totally valid | 21:51 |
| *** rcernin has joined #openstack-infra | 21:51 | |
| fungi | sticking an e-mail address in there is purely convention | 21:52 |
| fungi | also leaving off the comment field entirely works fine | 21:52 |
| SamYaple | huh. i thought it was user@host, not email address | 21:52 |
| SamYaple | that ssh-key generated i mean | 21:52 |
| fungi | formatted as an e-mail address (i.e. using an @ between the username and hostname) | 21:53 |
| SamYaple | i thought email addys required FQDN, and FQDN required a TLD | 21:54 |
| SamYaple | but i *really* dont want to think about this more :P | 21:54 |
| fungi | yeah, for openssh behavior it'll use the short hostname unless you have set your hostname on that system to be an fqdn (which for most of my systems it is) | 21:55 |
| SamYaple | hostname as fqdn breaks lots of things | 21:56 |
| clarkb | drives me nuts because foo. is a perfectly valid fqdn in many places (like my home network used to be) | 21:57 |
| *** thorst has joined #openstack-infra | 21:58 | |
| SamYaple | oh yea lets complain about things violating RFCs | 21:59 |
| SamYaple | goody! | 21:59 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul feature/zuulv3: Add support for override-checkout, deprecate override-branch https://review.openstack.org/516451 | 22:00 |
| jeblair | it's zuul meeting time in #openstack-meeting-alt | 22:00 |
| *** wolverineav has joined #openstack-infra | 22:01 | |
| fungi | ooh! | 22:01 |
| *** thorst has quit IRC | 22:02 | |
| *** dprince has quit IRC | 22:02 | |
| *** wolverineav has quit IRC | 22:05 | |
| *** wolverineav has joined #openstack-infra | 22:05 | |
| pabelanger | ping: unknown host subunit-worker01.openstack.org | 22:08 |
| pabelanger | that is our firewall issue | 22:08 |
| pabelanger | guess we deleted that host? | 22:08 |
| pabelanger | clarkb: ^ | 22:08 |
| clarkb | ya I think we did, guessing the dns record wasn't removed? | 22:09 |
| clarkb | or do we explicitly ping it? | 22:09 |
| *** leyal has joined #openstack-infra | 22:09 | |
| *** oanson has joined #openstack-infra | 22:09 | |
| pabelanger | no, I manually pinged | 22:09 |
| *** dimak has joined #openstack-infra | 22:09 | |
| pabelanger | I'll check and see what is going on with it | 22:09 |
| clarkb | well it was a preceise host | 22:10 |
| *** lihi has joined #openstack-infra | 22:10 | |
| clarkb | so we deleted it, guessing its just dns that needs updating (and was missed on deletion whcih may be my bad, I can't remember if I deleted that host or not) | 22:10 |
| *** wolverineav has quit IRC | 22:10 | |
| *** hashar has quit IRC | 22:11 | |
| fungi | yeah, i deleted it from dns | 22:11 |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/os-testr master: Updated from global requirements https://review.openstack.org/503645 | 22:11 |
| *** jcoufal has quit IRC | 22:11 | |
| clarkb | oh in that case we probably have to update system-config to remove it from the geraman hosts list | 22:11 |
| fungi | waiting on 516085 to get reviewed/approved so i can try another rebuild | 22:11 |
| *** jcoufal has joined #openstack-infra | 22:11 | |
| fungi | but yeah, we can delete it from puppet in the interim... just remove from host lists in manifests/site.pp in system-config? | 22:12 |
| clarkb | oh let me review that | 22:12 |
| *** snapiri has joined #openstack-infra | 22:12 | |
| *** prometheanfire has joined #openstack-infra | 22:13 | |
| fungi | i have reason to believe the rebuild will work once that merges | 22:13 |
| clarkb | I've approved it, nice and simple | 22:13 |
| fungi | thanks clarkb! i'll fire up another replacement in a few minutes once that merges, and if i get it into dns then we can avoid temporarily removing and readding it in system-config i guess | 22:14 |
| fungi | pabelanger: you okay with that plan? | 22:14 |
| openstackgerrit | Paul Belanger proposed openstack-infra/system-config master: Remove subunit-worker01.openstack.org https://review.openstack.org/516460 | 22:14 |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/os-testr master: Updated from global requirements https://review.openstack.org/503645 | 22:15 |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/os-testr master: Updated from global requirements https://review.openstack.org/503645 | 22:15 |
| pabelanger | fungi: ah, so 516460 isn't needed then | 22:15 |
| *** marst has quit IRC | 22:15 | |
| fungi | pabelanger: well, don't abandon it yet, and if this doesn't totally fix the replacement build i'm attempting then we can go ahead with the removal while i iterate further | 22:16 |
| fungi | i'd rather not hold you up | 22:16 |
| pabelanger | sure | 22:16 |
| openstackgerrit | Kendall Nelson proposed openstack-infra/project-config master: Setup Contributor Guide in Storyboard https://review.openstack.org/516462 | 22:17 |
| *** salv-orlando has joined #openstack-infra | 22:19 | |
| prometheanfire | so... does openstack-infra run the mail list? | 22:19 |
| *** jcoufal_ has joined #openstack-infra | 22:19 | |
| prometheanfire | if so, may want to fix dmarc | 22:19 |
| prometheanfire | https://dmarc.org/wiki/FAQ#I_operate_a_mailing_list_and_I_want_to_interoperate_with_DMARC.2C_what_should_I_do.3F :D | 22:20 |
| clarkb | #status log gear 0.11.0 tagged wtih statsd performance improvements | 22:21 |
| *** jcoufal has quit IRC | 22:21 | |
| openstackstatus | clarkb: finished logging | 22:21 |
| clarkb | prometheanfire: I'm sure jeblair could write a novel on this but there isn't really a good way to fix dmarc | 22:21 |
| clarkb | prometheanfire: we looked into it extensively when yahoo decided to break mailing lists and tl;dr was there are no good options | 22:22 |
| fungi | prometheanfire: we're happy if you want to help us lobby for getting rid of dmarc though | 22:22 |
| prometheanfire | lol, I have dmarc as strict on my server | 22:22 |
| clarkb | prometheanfire: so you break mailing lists too L:P | 22:22 |
| fungi | right now the recommendation is to not post from or subscribe to our mailing lists with addresses at dmarc-enforcing domains | 22:22 |
| clarkb | er :P | 22:22 |
| prometheanfire | I think I sent as the wrong user... | 22:22 |
| *** salv-orlando has quit IRC | 22:23 | |
| prometheanfire | gentoo.org doesn't have dmarc and that's my ml user for openstack | 22:23 |
| *** felipemonteiro_ has quit IRC | 22:23 | |
| openstackgerrit | Merged openstack-infra/project-config master: Remove legacy jobs for ansible-role-zuul https://review.openstack.org/516412 | 22:23 |
| fungi | if we can upgrade the listserv to a newer ubuntu release, the version of mailman it has includes some dmarc mitigation options we might be able to try | 22:23 |
| jeblair | basically, people who have their dmarc configs are saying, in essence, this domain is not used for mailing lists | 22:24 |
| jeblair | that's a fine thing to say | 22:24 |
| prometheanfire | fungi: ya, that'd be nice | 22:24 |
| jeblair | i personally think it's weird to do that for some domains | 22:24 |
| jeblair | but basically, dmarc was meant for paypal to say "paypal.com only sends out receipt spam. it's not meant for mailing lists." | 22:25 |
| *** slaweq has joined #openstack-infra | 22:25 | |
| *** salv-orlando has joined #openstack-infra | 22:26 | |
| *** jcoufal_ has quit IRC | 22:26 | |
| *** jcoufal has joined #openstack-infra | 22:28 | |
| openstackgerrit | Ian Wienand proposed openstack-infra/zuul feature/zuulv3: Check start time for wait_time key https://review.openstack.org/516465 | 22:29 |
| jeblair | #topic v3 roadmap | 22:30 |
| jeblair | gah | 22:31 |
| * fungi sweeps that one over to the other buffer | 22:32 | |
| openstackgerrit | Merged openstack-infra/puppet-subunit2sql master: Make subunit2sql install depend on netifaces https://review.openstack.org/516085 | 22:34 |
| fungi | pabelanger: ^ it merged, so trying again now | 22:37 |
| pabelanger | fungi: ack | 22:37 |
| fungi | i'll let you know once it's in dns or when i approve your system-config change | 22:37 |
| pabelanger | great | 22:37 |
| openstackgerrit | Merged openstack-infra/project-config master: Add post tarball gate to neutron-tempest-plugin https://review.openstack.org/516188 | 22:38 |
| openstackgerrit | sebastian marcet proposed openstack-infra/openstackid-resources master: API filter server error https://review.openstack.org/516466 | 22:39 |
| openstackgerrit | Merged openstack-infra/openstack-zuul-jobs master: Restrict legacy-tempest-dsvm-nova-wsgi-full to stable/ocata https://review.openstack.org/516334 | 22:42 |
| fungi | clarkb: i'm going to wager that the gear performance fix will also address the massive processing backlog we're seeing with the subunit2sql worker | 22:44 |
| openstackgerrit | Merged openstack-infra/project-config master: Unblock designate-dashboard gate https://review.openstack.org/516223 | 22:46 |
| openstackgerrit | Merged openstack-infra/project-config master: Add post tarball gate to magnum-tempest-plugin https://review.openstack.org/516187 | 22:46 |
| openstackgerrit | Merged openstack-infra/project-config master: Add post tarball gate to zaqar-tempest-plugin https://review.openstack.org/516186 | 22:46 |
| openstackgerrit | Merged openstack-infra/openstackid-resources master: API filter server error https://review.openstack.org/516466 | 22:46 |
| *** catintheroof has quit IRC | 22:48 | |
| *** slaweq has quit IRC | 22:52 | |
| openstackgerrit | Alex Schultz proposed openstack-infra/openstack-zuul-jobs master: Add audit package for beaker jobs https://review.openstack.org/516469 | 22:54 |
| mnaser | mwhahaha: do you know if the audit service automatically starts | 22:56 |
| mwhahaha | good question | 22:56 |
| mwhahaha | probably would be a good idea to ensure it's running let me check | 22:57 |
| *** rbrndt has quit IRC | 22:57 | |
| *** tpsilva has quit IRC | 22:59 | |
| *** aeng has quit IRC | 23:00 | |
| mwhahaha | mnaser: yes, yes it is | 23:00 |
| mwhahaha | mnaser: the install does a systemctl preset auditd.service and the defaults for rhel include enable auditd.service | 23:01 |
| mwhahaha | mnaser: 90-default.preset:enable auditd.service | 23:01 |
| mnaser | ok cools | 23:01 |
| mwhahaha | TIL | 23:01 |
| mnaser | +2 | 23:02 |
| mnaser | also that means we're starting it for no reason in all the other jobs | 23:02 |
| mnaser | so inefficent :-P | 23:02 |
| * mwhahaha re-hides the bodies | 23:02 | |
| mwhahaha | nothing to see here | 23:02 |
| ianw | i have great dejavu about audit and i'm sure we've fixed that before | 23:04 |
| clarkb | in the interest of expediency I am going to update the version of gear on logstash.o.o now | 23:04 |
| clarkb | then restart the geard there | 23:04 |
| clarkb | (rather than waiting for the next puppet run) | 23:04 |
| jeblair | robcresswell: https://review.openstack.org/516451 implements the tag checkout ("override-checkout") we were discussing last week | 23:05 |
| ianw | mnaser / mwhahaha: https://review.openstack.org/#/c/352263/ is what i'm thinking of | 23:05 |
| mnaser | wait so audit is actually installed? | 23:05 |
| mwhahaha | mnaser: rpm-qa.txt should show | 23:05 |
| fungi | pabelanger: i've approved 516460 for now. for some reason even though subunit2sql claims to work with the version of python-netifaces in ubuntu and i can see it getting preinstalled in the puppet apply output, the pip package provider seems to want to do `pip install --upgrade subunit2sql` which results in downloading latest netifaces sdist from pypi and attempting to build it anyway | 23:06 |
| clarkb | actually next puppet run should start at 23:15UTC I think so I'll just wait | 23:06 |
| mnaser | sigh | 23:06 |
| mnaser | it is | 23:06 |
| *** jaypipes has quit IRC | 23:06 | |
| mwhahaha | well where are my logs :( | 23:06 |
| mnaser | so why is -f /var/log/audit/audit.log not evaluating | 23:06 |
| mnaser | maybe it's not started?! | 23:06 |
| mnaser | http://logs.openstack.org/24/505824/2/check/legacy-puppet-beaker-rspec-centos-7/d3b8328/logs/ps.txt.gz -- nope its running | 23:06 |
| mnaser | thanks for pointing that one out ianw | 23:06 |
| mwhahaha | permissions maybe? | 23:07 |
| mnaser | http://logs.openstack.org/24/505824/2/check/legacy-puppet-beaker-rspec-centos-7/d3b8328/job-output.txt.gz#_2017-10-30_22_43_37_618665 | 23:07 |
| mnaser | ohhhhh | 23:07 |
| mnaser | ok lets take this to #puppet-opeonstack | 23:07 |
| clarkb | fungi: ya I expect they are both sad together (the different geard connections) | 23:08 |
| fungi | clarkb: if only i could figure out how to tell puppet/pip not to upgrade subunit2sql's install_requires unless they're too old. but i expect that this is simply a byproducto of attempting to map puppet's limited view of package states to pip's unfortunate limitations for dependency handling | 23:10 |
| clarkb | fungi: maybe use an exec and pass an appropriate incantation to pip? maybe with a pin? | 23:11 |
| clarkb | basically DIY | 23:11 |
| fungi | we'd need to switch to installing subunit2sql from source | 23:12 |
| fungi | a la continuous deployment | 23:12 |
| clarkb | ok gear is updated on logstash.o.o I'm restarting the daemon now | 23:12 |
| *** xarses has quit IRC | 23:14 | |
| clarkb | that is done, but ran into a hiccup, fix in a moment | 23:15 |
| *** sdague has quit IRC | 23:16 | |
| *** yamamoto has joined #openstack-infra | 23:17 | |
| openstackgerrit | Clark Boylan proposed openstack-infra/system-config master: Remove zl's from jenkins-logstash-client config https://review.openstack.org/516473 | 23:17 |
| clarkb | infra-root ^ that is sort of the halfway workaround I used locally | 23:17 |
| clarkb | long term (like after summit) we should switch it to just run a standalone geard | 23:17 |
| *** aeng has joined #openstack-infra | 23:17 | |
| openstackgerrit | Merged openstack-infra/shade master: Zuul: add file extension to playbook path https://review.openstack.org/516145 | 23:19 |
| *** vhosakot has joined #openstack-infra | 23:19 | |
| clarkb | #status log geard (really jenkins-log-client) restarted on logstash.o.o to pick up gear 0.11.0 performance improvements. https://review.openstack.org/516473 needed to workaround zuul transition there. | 23:19 |
| openstackstatus | clarkb: finished logging | 23:19 |
| *** jcoufal_ has joined #openstack-infra | 23:23 | |
| *** hongbin has quit IRC | 23:23 | |
| *** jcoufal has quit IRC | 23:24 | |
| *** tosky has joined #openstack-infra | 23:25 | |
| *** baoli has joined #openstack-infra | 23:25 | |
| *** dave-mccowan has joined #openstack-infra | 23:28 | |
| *** LindaWang has quit IRC | 23:28 | |
| pabelanger | fungi: ack, thanks | 23:29 |
| *** LindaWang has joined #openstack-infra | 23:29 | |
| clarkb | fungi: so netifaces needed by setuptools conflicts with that needed by subunit? | 23:30 |
| *** Swami has quit IRC | 23:30 | |
| openstackgerrit | Merged openstack-infra/system-config master: Remove subunit-worker01.openstack.org https://review.openstack.org/516460 | 23:30 |
| clarkb | dmsimard: kibana results looking much better now | 23:31 |
| clarkb | we'll have a hole in the data where we didn't index but considering we only keep 10 days anyways that isn't a huge deal | 23:31 |
| clarkb | just roll forward | 23:31 |
| fungi | clarkb: nope. pip is just using --upgrade-strategy=eager (default) instead of --upgrade-strategy=only-if-needed | 23:31 |
| *** lbragstad has quit IRC | 23:32 | |
| fungi | so even though the python-netifaces version in xenial is sufficient to meet the version range oslo.utils requires, pip attempts to upgrade to the most recent version pypi has when upgrading subunit2sql | 23:32 |
| clarkb | fungi: does subunit2sql need a cap on it? | 23:32 |
| fungi | i don't see how that would help | 23:33 |
| clarkb | it should stop upgrading to latest and just to the version that satisfies the cap? | 23:33 |
| fungi | pip install --upgrade wants to install the latest version of all your install_requires | 23:33 |
| clarkb | but won't if you have a cap aiui | 23:33 |
| fungi | ahh, i see, you mean should we ensure=x.y.x for subunit2sql instead of ensure=latest | 23:33 |
| *** gyee has quit IRC | 23:33 | |
| clarkb | or <x.y.z | 23:34 |
| clarkb | but I may be misunderstanding why this is problematic | 23:34 |
| fungi | we want to avoid the pip package provider from upgrading subunit2sql's (transitive) dependencies because several of them include c extensions and so are getting preinstalled with distro packages | 23:34 |
| fungi | pymysql, psycopg2 and netifaces to be exact | 23:35 |
| fungi | er, not pymysql but mysql-python i guess | 23:35 |
| clarkb | we don't need mysql-python if using pymysql so that might be something we can cleanup if we isntall both | 23:36 |
| fungi | these come into play because subunit2sql requires oslo.db which in turn requires oslo.utils | 23:36 |
| clarkb | but got it | 23:36 |
| fungi | and anyway, the current pain is netifaces | 23:36 |
| fungi | because it's only distributed on pypi as an sdist and needs local compiler and python headers to build | 23:37 |
| clarkb | but ya I think you can do something like make a pip install exec that does pip install netifaces==distro-version subunit2sql | 23:37 |
| fungi | ahh, fair enough that would likely work around it | 23:37 |
| fungi | though the trick is working out when to trigger that pip install so it doesn't happen at every puppet apply | 23:37 |
| *** baoli has quit IRC | 23:38 | |
| fungi | at least for continuous deployment we can trigger it exclusively when the git repo updates | 23:38 |
| *** thorst has joined #openstack-infra | 23:39 | |
| clarkb | you could do a pip search unless/onlyif | 23:41 |
| clarkb | that might be complicated though | 23:41 |
| *** jcoufal has joined #openstack-infra | 23:43 | |
| *** ihrachys has quit IRC | 23:44 | |
| *** jcoufal_ has quit IRC | 23:46 | |
| *** bobh has joined #openstack-infra | 23:47 | |
| *** jcoufal_ has joined #openstack-infra | 23:47 | |
| *** jcoufal has quit IRC | 23:50 | |
| *** esberglu_ has joined #openstack-infra | 23:51 | |
| *** aeng has quit IRC | 23:51 | |
| *** tosky has quit IRC | 23:52 | |
| *** aeng has joined #openstack-infra | 23:52 | |
| *** stakeda has joined #openstack-infra | 23:53 | |
| *** ijw has quit IRC | 23:53 | |
| *** ijw has joined #openstack-infra | 23:54 | |
| *** esberglu has quit IRC | 23:54 | |
| *** jcoufal_ has quit IRC | 23:55 | |
| fungi | i don't see any way to tell puppet's pip package provider to switch to doing --upgrade-strategy=only-if-needed | 23:58 |
| fungi | which would be the ideal solution for this | 23:58 |
| clarkb | ya I think the provider is fairly hard coded in its execution of pip | 23:58 |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!