| *** ijw has joined #openstack-infra | 00:05 | |
| *** dhinesh has joined #openstack-infra | 00:12 | |
| *** ijw has quit IRC | 00:13 | |
| *** longkb has joined #openstack-infra | 00:16 | |
| *** sthussey has quit IRC | 00:17 | |
| *** gyee has quit IRC | 00:21 | |
| *** graphene has quit IRC | 00:31 | |
| *** threestrands has joined #openstack-infra | 00:32 | |
| *** threestrands has quit IRC | 00:32 | |
| *** threestrands has joined #openstack-infra | 00:32 | |
| *** spsurya has quit IRC | 00:35 | |
| openstackgerrit | Merged openstack-infra/project-config master: Revert "Revert "Revert "Disable inap-mtl01 provider""" https://review.openstack.org/600577 | 00:42 |
|---|---|---|
| clarkb | I'll keep an eye on ^ | 00:42 |
| clarkb | anyone will to review 600575 to speed up puppet runs, it passes tests | 00:43 |
| *** spartakos has quit IRC | 00:45 | |
| *** dave-mccowan has joined #openstack-infra | 00:51 | |
| *** hamzy has joined #openstack-infra | 00:55 | |
| *** hongbin_ has joined #openstack-infra | 00:55 | |
| *** akhila has quit IRC | 01:01 | |
| *** Emine has quit IRC | 01:11 | |
| *** Emine has joined #openstack-infra | 01:11 | |
| ianw | clarkb: yeah, sorry ... think it looks good, just getting my head around globs | 01:17 |
| openstackgerrit | Kendall Nelson proposed openstack-infra/storyboard-webclient master: Show Email Addresses when Searching https://review.openstack.org/589713 | 01:18 |
| clarkb | once the current remote puppet afs finishes we'll run the else playbook and should update nl03 with inap max servers | 01:19 |
| *** markvoelker has quit IRC | 01:19 | |
| clarkb | have logstash query ready to go as well `node_provider:"inap-mtl01" AND filename:"job-output.txt" AND message:"gzip console log and json output"` | 01:19 |
| *** markvoelker has joined #openstack-infra | 01:20 | |
| clarkb | the message filter there is just to get single return per job to quickly check success vs failure | 01:20 |
| *** bobh has joined #openstack-infra | 01:22 | |
| *** markvoelker has quit IRC | 01:24 | |
| zxiiro | can I use shade to share an image with another tenant? or do I need to dig into the sdk for that | 01:24 |
| clarkb | zxiiro: openstackcloud.update_image_properties is probably what you want in shade | 01:27 |
| clarkb | it will do the json diff patching for you, you just supply the new values that you want iirc | 01:27 |
| *** camunoz has quit IRC | 01:28 | |
| zxiiro | cool will dig into that | 01:28 |
| *** ijw has joined #openstack-infra | 01:35 | |
| *** bobh has quit IRC | 01:37 | |
| clarkb | nl03's nodepool.yaml has reenabled inap | 01:39 |
| *** markvoelker has joined #openstack-infra | 01:41 | |
| clarkb | nodes are coming online and getting used, I'll watch logstash for any abnormalities | 01:43 |
| clarkb | I wonder if this is the first time we've had our periodic jobs wrap around | 01:46 |
| clarkb | or are in danger of that at least | 01:46 |
| clarkb | an openstack-tox-docs job has succeeded in inap-mtl01 | 01:49 |
| clarkb | that is a good first sign | 01:49 |
| clarkb | thats neat you can see gate resets in the nodepool utiliation graphs | 01:54 |
| *** ijw has quit IRC | 01:55 | |
| clarkb | reset did fail on inap but appears to be valid pylint fail in neutron | 01:55 |
| clarkb | http://logs.openstack.org/47/583847/2/gate/openstack-tox-pep8/f5daa43/job-output.txt.gz#_2018-09-07_01_52_01_804084 | 01:55 |
| clarkb | there are a lot of neutron changes too so we should expect a bunch of resets I bet | 01:56 |
| *** ijw_ has joined #openstack-infra | 01:58 | |
| clarkb | that is the only inap failure indexed so far | 02:00 |
| clarkb | 20 successes | 02:01 |
| *** dhinesh has quit IRC | 02:01 | |
| *** ijw_ has quit IRC | 02:03 | |
| *** apetrich_ has joined #openstack-infra | 02:04 | |
| clarkb | ianw: ^ anything else you'd like checked before I call this good? I've also pinged the neutron channel about their gate resets | 02:04 |
| ianw | clarkb: nope, i'll keep an eye. the only thing i think was wrong was the mirror "timeouts" (i was a little surprised when that turned out to be an infinite timeout with the server gone ;) | 02:05 |
| *** apetrich has quit IRC | 02:07 | |
| *** diablo_rojo has quit IRC | 02:08 | |
| clarkb | alright then, I'm finding dinner | 02:11 |
| *** Bhujay has joined #openstack-infra | 02:12 | |
| *** longkb has quit IRC | 02:16 | |
| *** longkb has joined #openstack-infra | 02:17 | |
| *** ramishra has joined #openstack-infra | 02:22 | |
| *** r-daneel has quit IRC | 02:23 | |
| *** jamesmcarthur has joined #openstack-infra | 02:34 | |
| *** bhavikdbavishi has joined #openstack-infra | 02:36 | |
| *** psachin has joined #openstack-infra | 02:48 | |
| *** Bhujay has quit IRC | 02:56 | |
| *** jamesmcarthur has quit IRC | 02:59 | |
| *** ijw has joined #openstack-infra | 03:01 | |
| *** ijw has quit IRC | 03:05 | |
| openstackgerrit | Merged openstack-infra/storyboard-webclient master: Expose ordering criteria in search UI https://review.openstack.org/577870 | 03:07 |
| *** r-daneel has joined #openstack-infra | 03:13 | |
| *** jamesmcarthur has joined #openstack-infra | 03:21 | |
| *** dave-mccowan has quit IRC | 03:25 | |
| *** eernst has quit IRC | 03:27 | |
| *** eernst has joined #openstack-infra | 03:30 | |
| *** annp has joined #openstack-infra | 03:32 | |
| *** eernst has quit IRC | 03:32 | |
| *** eernst has joined #openstack-infra | 03:34 | |
| *** nicolasbock has quit IRC | 03:36 | |
| *** eernst has quit IRC | 03:39 | |
| *** jamesmcarthur has quit IRC | 03:39 | |
| *** jamesmcarthur has joined #openstack-infra | 03:43 | |
| *** r-daneel has quit IRC | 03:47 | |
| *** r-daneel has joined #openstack-infra | 03:47 | |
| *** toabctl has quit IRC | 03:48 | |
| *** toabctl has joined #openstack-infra | 03:57 | |
| *** udesale has joined #openstack-infra | 04:05 | |
| *** spsurya has joined #openstack-infra | 04:10 | |
| *** pgadiya has joined #openstack-infra | 04:13 | |
| *** pgadiya has quit IRC | 04:13 | |
| *** rlandy has quit IRC | 04:16 | |
| *** jamesmcarthur has quit IRC | 04:16 | |
| *** jamesmcarthur has joined #openstack-infra | 04:17 | |
| *** Bhujay has joined #openstack-infra | 04:17 | |
| *** Bhujay has quit IRC | 04:18 | |
| *** Bhujay has joined #openstack-infra | 04:19 | |
| *** hongbin__ has joined #openstack-infra | 04:22 | |
| *** hongbin_ has quit IRC | 04:25 | |
| *** spartakos has joined #openstack-infra | 04:29 | |
| *** dhajare has joined #openstack-infra | 04:31 | |
| *** psachin has quit IRC | 04:34 | |
| AJaeger | config-core, https://review.openstack.org/#/c/597598/ https://review.openstack.org/597590 and https://review.openstack.org/597582 are repos that have migrated jobs now as part of python3-first, please +2A | 04:36 |
| *** psachin has joined #openstack-infra | 04:36 | |
| AJaeger | clarkb: no, this is not the first time with periodic jobs wrap around - but it's rare | 04:36 |
| *** jamesmcarthur has quit IRC | 04:38 | |
| AJaeger | config-core, please also review https://review.openstack.org/598487 https://review.openstack.org/600457 and https://review.openstack.org/600457 | 04:41 |
| *** jamesmcarthur has joined #openstack-infra | 04:42 | |
| *** dhinesh has joined #openstack-infra | 04:44 | |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config master: Retire developement-proposals (Step 2) https://review.openstack.org/600644 | 04:45 |
| *** dhinesh_ has joined #openstack-infra | 04:46 | |
| *** dhinesh has quit IRC | 04:49 | |
| *** psachin has quit IRC | 04:51 | |
| *** psachin has joined #openstack-infra | 04:54 | |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config master: Retire developement-proposals (Step 2) https://review.openstack.org/600644 | 04:56 |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config master: Retire developement-proposals (Step 4) https://review.openstack.org/600648 | 04:56 |
| *** rcernin has quit IRC | 04:57 | |
| *** spartakos has quit IRC | 04:57 | |
| *** kjackal has joined #openstack-infra | 05:06 | |
| *** kjackal_ has joined #openstack-infra | 05:06 | |
| *** psachin` has joined #openstack-infra | 05:07 | |
| AJaeger | timburke, corvus, the test results are back - http://logs.openstack.org/40/600540/1/check/swift-dsvm-functional/0a61993/job-output.txt.gz shows that while we initally install for devstack python-swiftclient from git, the tox role uses the package. The other job is fine. timburke, so we can move forward with your change! | 05:08 |
| *** psachin has quit IRC | 05:09 | |
| *** masber has quit IRC | 05:12 | |
| *** spartakos has joined #openstack-infra | 05:12 | |
| *** jamesmcarthur has quit IRC | 05:12 | |
| *** jamesmcarthur has joined #openstack-infra | 05:16 | |
| *** akhila has joined #openstack-infra | 05:21 | |
| *** ssbarnea|bkp has quit IRC | 05:22 | |
| *** e0ne has joined #openstack-infra | 05:23 | |
| *** hongbin__ has quit IRC | 05:23 | |
| *** e0ne has quit IRC | 05:24 | |
| *** e0ne has joined #openstack-infra | 05:28 | |
| *** e0ne has quit IRC | 05:29 | |
| *** rcernin has joined #openstack-infra | 05:30 | |
| *** akhila has quit IRC | 05:41 | |
| *** e0ne has joined #openstack-infra | 05:43 | |
| *** r-daneel has quit IRC | 05:44 | |
| *** quiquell has joined #openstack-infra | 05:45 | |
| *** jamesmcarthur has quit IRC | 05:46 | |
| *** e0ne has quit IRC | 05:47 | |
| *** e0ne has joined #openstack-infra | 05:48 | |
| *** jamesmcarthur has joined #openstack-infra | 05:50 | |
| *** psachin` has quit IRC | 05:50 | |
| *** e0ne has quit IRC | 05:50 | |
| *** e0ne has joined #openstack-infra | 05:51 | |
| *** e0ne has quit IRC | 05:57 | |
| *** ijw has joined #openstack-infra | 06:00 | |
| *** ijw has quit IRC | 06:04 | |
| *** ijw has joined #openstack-infra | 06:05 | |
| *** ijw_ has joined #openstack-infra | 06:06 | |
| openstackgerrit | OpenStack Proposal Bot proposed openstack-infra/project-config master: Normalize projects.yaml https://review.openstack.org/600658 | 06:09 |
| *** ijw has quit IRC | 06:10 | |
| *** cshastri has joined #openstack-infra | 06:10 | |
| *** pcaruana has joined #openstack-infra | 06:13 | |
| *** dhinesh_ has quit IRC | 06:14 | |
| *** jamesmcarthur has quit IRC | 06:15 | |
| *** jamesmcarthur has joined #openstack-infra | 06:16 | |
| *** e0ne has joined #openstack-infra | 06:16 | |
| *** e0ne has quit IRC | 06:18 | |
| openstackgerrit | Tobias Henkel proposed openstack-infra/nodepool master: Fix label name in reported label stats https://review.openstack.org/600659 | 06:22 |
| *** bhavikdbavishi has quit IRC | 06:23 | |
| *** bhavikdbavishi has joined #openstack-infra | 06:24 | |
| openstackgerrit | Merged openstack-infra/project-config master: remove job settings for storlets repositories https://review.openstack.org/597598 | 06:26 |
| *** holser_ has joined #openstack-infra | 06:29 | |
| openstackgerrit | Merged openstack-infra/project-config master: remove job settings for kolla repositories https://review.openstack.org/597590 | 06:30 |
| openstackgerrit | Merged openstack-infra/project-config master: remove job settings for ec2-api repositories https://review.openstack.org/597582 | 06:30 |
| gmann | clarkb: corvus ianw frickler last patch to finish Rocky testing setup work. It is passing now- https://review.openstack.org/#/c/591594/ | 06:30 |
| *** quiquell is now known as quique|rover|bbl | 06:36 | |
| *** hashar has joined #openstack-infra | 06:42 | |
| *** aojea has joined #openstack-infra | 06:43 | |
| *** jamesmcarthur has quit IRC | 06:51 | |
| *** ginopc has joined #openstack-infra | 06:57 | |
| *** jamesmcarthur has joined #openstack-infra | 06:58 | |
| *** jtomasek has joined #openstack-infra | 07:00 | |
| AJaeger | frickler: could you put https://review.openstack.org/598487 and https://review.openstack.org/600644 and https://review.openstack.org/#/c/600476/ on your review queue, please? | 07:03 |
| AJaeger | ianw: thanks for reviews! | 07:03 |
| *** quique|rover|bbl has quit IRC | 07:06 | |
| openstackgerrit | Merged openstack-infra/openstack-zuul-jobs master: Use tox-docs for specs repositories https://review.openstack.org/600457 | 07:06 |
| *** dhajare has quit IRC | 07:06 | |
| *** quiquell has joined #openstack-infra | 07:06 | |
| *** quiquell has quit IRC | 07:07 | |
| *** quiquell|rover has joined #openstack-infra | 07:07 | |
| *** quiquell|rover has quit IRC | 07:07 | |
| *** quiquell|rover has joined #openstack-infra | 07:08 | |
| *** quiquell|rover has quit IRC | 07:11 | |
| AJaeger | dirk: could you +A https://review.openstack.org/#/c/598973/ , please? | 07:11 |
| *** florianf has joined #openstack-infra | 07:12 | |
| *** quiquell has joined #openstack-infra | 07:12 | |
| *** quiquell is now known as quiquell|rover | 07:12 | |
| dirk | AJaeger: done | 07:12 |
| AJaeger | thanks, dirk | 07:13 |
| openstackgerrit | Merged openstack-infra/project-config master: Normalize projects.yaml https://review.openstack.org/600658 | 07:13 |
| openstackgerrit | Merged openstack-infra/project-config master: Remove references to pypi mirroring from dashboard https://review.openstack.org/600364 | 07:14 |
| openstackgerrit | Merged openstack-infra/project-config master: Use tox-docs for specs publishing https://review.openstack.org/600458 | 07:16 |
| *** ccamacho has joined #openstack-infra | 07:17 | |
| *** dhajare has joined #openstack-infra | 07:20 | |
| *** tesseract has joined #openstack-infra | 07:27 | |
| evrardjp | morning | 07:30 |
| *** pguimaraes has quit IRC | 07:30 | |
| *** jamesmcarthur has quit IRC | 07:33 | |
| *** dhajare has quit IRC | 07:33 | |
| * SotK notices that the publish-openstack-javascript-content job appears to have become broken: http://zuul.openstack.org/builds.html?job_name=publish-openstack-javascript-content | 07:34 | |
| *** dhajare has joined #openstack-infra | 07:34 | |
| *** dhajare has quit IRC | 07:35 | |
| openstackgerrit | Thierry Carrez proposed openstack-infra/puppet-ptgbot master: Make the FC SIG freed slots actually reservable https://review.openstack.org/600665 | 07:35 |
| *** dhajare has joined #openstack-infra | 07:35 | |
| *** pblaho has joined #openstack-infra | 07:35 | |
| AJaeger | SotK: http://logs.openstack.org/67/6776d05c12596c4dbd2b4c737b1cf6bb23884ef2/post/publish-openstack-javascript-content/2ebe59d/job-output.txt.gz#_2018-09-07_03_33_23_588274 is the problem... | 07:37 |
| AJaeger | SotK: let's ask fungi, he looked into some of this already earlier this week (or last one) (gpg temp signing key) | 07:38 |
| openstackgerrit | Tobias Henkel proposed openstack-infra/zuul master: WIP: Match tag items against containing branches https://review.openstack.org/578557 | 07:38 |
| openstackgerrit | Thierry Carrez proposed openstack/ptgbot master: Update unbook documentation https://review.openstack.org/600520 | 07:39 |
| *** pguimaraes has joined #openstack-infra | 07:43 | |
| *** dtantsur|afk is now known as dtantsur | 07:46 | |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/openstack-zuul-jobs master: Remove legacy-cloudkitty-dsvm-install https://review.openstack.org/600668 | 07:48 |
| *** jpena|off is now known as jpena | 07:51 | |
| *** alexchadin has joined #openstack-infra | 07:52 | |
| *** ramishra has quit IRC | 07:58 | |
| *** tosky has joined #openstack-infra | 07:59 | |
| *** jpich has joined #openstack-infra | 08:01 | |
| *** shardy has joined #openstack-infra | 08:08 | |
| *** slaweq_ is now known as slaweq | 08:10 | |
| *** ssbarnea|bkp has joined #openstack-infra | 08:11 | |
| *** ramishra has joined #openstack-infra | 08:14 | |
| *** electrofelix has joined #openstack-infra | 08:14 | |
| *** ethfci has joined #openstack-infra | 08:16 | |
| *** spartakos has quit IRC | 08:18 | |
| hwoarang | i notice that some opensuse jobs are failing with RETRY_LIMIT such as https://review.openstack.org/600272 but i can't figure out what exactly it is retrying for | 08:23 |
| hwoarang | any clues? | 08:23 |
| AJaeger | hwoarang: that's an indication that the pre.yaml failed | 08:25 |
| AJaeger | hwoarang: http://logs.openstack.org/72/600272/1/check/openstack-ansible-functional-opensuse-423/b23c425/ara-report/ | 08:25 |
| AJaeger | See the red x in front of pre.yaml? Check which one failed... | 08:25 |
| hwoarang | oh good point | 08:25 |
| hwoarang | oh mirrors again :( | 08:26 |
| AJaeger | yeah ;( | 08:26 |
| hwoarang | ok infra mirrors are out of sync again | 08:28 |
| hwoarang | *sigh* | 08:28 |
| * hwoarang will wait until the next sync before he investigates more | 08:28 | |
| *** olivierb has joined #openstack-infra | 08:29 | |
| *** e0ne has joined #openstack-infra | 08:32 | |
| *** threestrands has quit IRC | 08:33 | |
| openstackgerrit | Merged openstack-infra/project-config master: Add development-proposals to spec index https://review.openstack.org/598487 | 08:33 |
| openstackgerrit | Merged openstack-infra/project-config master: Remove system-config project config https://review.openstack.org/600476 | 08:33 |
| openstackgerrit | Merged openstack-infra/project-config master: Retire developement-proposals (Step 2) https://review.openstack.org/600644 | 08:33 |
| *** dhajare has quit IRC | 08:33 | |
| hwoarang | even mirrors.kernel.org (the source of infra mirrors) seems to be lagging by a day compared to downloads.opensuse.org :/ | 08:34 |
| *** derekh has joined #openstack-infra | 08:40 | |
| *** ssbarnea|bkp has quit IRC | 08:43 | |
| *** psachin has joined #openstack-infra | 08:46 | |
| *** ssbarnea|bkp has joined #openstack-infra | 08:54 | |
| *** dhajare has joined #openstack-infra | 08:54 | |
| *** rcernin has quit IRC | 09:16 | |
| *** mriedem has joined #openstack-infra | 09:23 | |
| *** nicolasbock has joined #openstack-infra | 09:39 | |
| *** nicolasbock has quit IRC | 09:40 | |
| *** pblaho has quit IRC | 09:45 | |
| *** e0ne has quit IRC | 09:47 | |
| *** ssbarnea1 is now known as ssbarnea | 09:53 | |
| hwoarang | dirk fyi^ | 09:54 |
| *** jtomasek has quit IRC | 09:55 | |
| *** gfidente has joined #openstack-infra | 10:07 | |
| hwoarang | infra-root could you please check again if there is something wrong with the opensuse-mirror script? opensuse mirrors seem to not have been synced since yesterday afternoon | 10:11 |
| *** roman_g has joined #openstack-infra | 10:12 | |
| *** jd_ has quit IRC | 10:25 | |
| openstackgerrit | Markos Chandras (hwoarang) proposed openstack-infra/system-config master: modules: mirror: opensuse: Ensure that old distros are wiped from mirrors https://review.openstack.org/600701 | 10:25 |
| openstackgerrit | Markos Chandras (hwoarang) proposed openstack-infra/system-config master: modules: mirror: opensuse: Ensure that old distros are wiped from mirrors https://review.openstack.org/600701 | 10:27 |
| openstackgerrit | Tobias Henkel proposed openstack-infra/zuul master: WIP: Match tag items against containing branches https://review.openstack.org/578557 | 10:27 |
| *** jd_ has joined #openstack-infra | 10:28 | |
| *** alexchadin has quit IRC | 10:32 | |
| *** alexchadin has joined #openstack-infra | 10:33 | |
| *** alexchadin has quit IRC | 10:33 | |
| *** alexchadin has joined #openstack-infra | 10:34 | |
| *** alexchadin has quit IRC | 10:34 | |
| *** alexchadin has joined #openstack-infra | 10:35 | |
| *** alexchadin has quit IRC | 10:35 | |
| *** alexchadin has joined #openstack-infra | 10:35 | |
| *** alexchadin has quit IRC | 10:36 | |
| *** bhavikdbavishi has quit IRC | 10:38 | |
| *** dave-mccowan has joined #openstack-infra | 10:51 | |
| *** ramishra_ has joined #openstack-infra | 10:51 | |
| *** jtomasek has joined #openstack-infra | 10:53 | |
| *** ramishra has quit IRC | 10:53 | |
| *** pbourke has quit IRC | 11:00 | |
| *** jpena is now known as jpena|lunch | 11:01 | |
| *** pbourke has joined #openstack-infra | 11:02 | |
| *** annp has quit IRC | 11:13 | |
| *** udesale has quit IRC | 11:16 | |
| *** jamesdenton has joined #openstack-infra | 11:20 | |
| *** EmilienM is now known as EvilienM | 11:25 | |
| *** takamatsu has joined #openstack-infra | 11:27 | |
| *** nicolasbock has joined #openstack-infra | 11:29 | |
| *** eernst has joined #openstack-infra | 11:32 | |
| *** eernst has quit IRC | 11:37 | |
| *** r-daneel has joined #openstack-infra | 11:39 | |
| *** ssbarnea|bkp has quit IRC | 11:43 | |
| *** jamesmcarthur has joined #openstack-infra | 11:48 | |
| *** jpena|lunch is now known as jpena | 11:54 | |
| *** jamesmcarthur has quit IRC | 11:54 | |
| *** andreaf has joined #openstack-infra | 12:00 | |
| pabelanger | rsync: failed to connect to rsync.opensuse.org (62.146.92.202): Connection timed out (110) | 12:01 |
| pabelanger | hwoarang: ^ | 12:01 |
| pabelanger | that's about all I see right now | 12:01 |
| *** alexchadin has joined #openstack-infra | 12:05 | |
| odyssey4me | pabelanger: I think https://review.openstack.org/#/c/600701/2 might be to address the issue? | 12:07 |
| odyssey4me | hwoarang: ^ ? | 12:07 |
| *** alexchadin has quit IRC | 12:09 | |
| *** alexchadin has joined #openstack-infra | 12:11 | |
| *** elod has quit IRC | 12:13 | |
| *** elod has joined #openstack-infra | 12:13 | |
| *** efried is now known as fried_rice | 12:14 | |
| *** trown|outtypewww is now known as trown | 12:15 | |
| *** eharney has quit IRC | 12:17 | |
| *** jtomasek has quit IRC | 12:20 | |
| *** elod has quit IRC | 12:25 | |
| *** kjackal has quit IRC | 12:25 | |
| *** kjackal_ has quit IRC | 12:25 | |
| *** sthussey has joined #openstack-infra | 12:29 | |
| *** camunoz has joined #openstack-infra | 12:29 | |
| *** psachin has quit IRC | 12:31 | |
| *** longkb has quit IRC | 12:40 | |
| *** rlandy has joined #openstack-infra | 12:40 | |
| *** kjackal has joined #openstack-infra | 12:44 | |
| *** hashar is now known as hasharAway | 12:44 | |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config master: remove job settings for watcher repositories https://review.openstack.org/593340 | 12:49 |
| *** e0ne has joined #openstack-infra | 12:49 | |
| AJaeger | config-core, next python3-first migration to review, please - https://review.openstack.org/593340 | 12:52 |
| *** jamesmcarthur has joined #openstack-infra | 12:54 | |
| *** tmorin has joined #openstack-infra | 12:57 | |
| tmorin | hi infra-root: I have https://review.openstack.org/599321 which is W+1 but not pushed to gate queue by zuul | 12:59 |
| *** ssbarnea|bkp has joined #openstack-infra | 12:59 | |
| *** ssbarnea|bkp2 has joined #openstack-infra | 12:59 | |
| tmorin | infra-root: I suspect this is because zuul believes that it's Depends-On is not merged | 12:59 |
| *** ssbarnea|bkp2 has quit IRC | 12:59 | |
| *** ssbarnea|bkp has joined #openstack-infra | 13:00 | |
| tmorin | infra-root: but the Depends-On gerrit change *has merged* in this branch | 13:00 |
| tmorin | but this Depends-On still ain't merged in other branches: is this what is confusing zuul ? | 13:00 |
| tosky | not an infra core, but maybe that's one of the reasons for using the new syntax of Depends-On | 13:01 |
| AJaeger | tmorin: was the depends-on merged when the W+1 given | 13:01 |
| tmorin | AJaeger: yes | 13:01 |
| AJaeger | tmorin: and yes, if you use "Ixxx", it waits for *all* changes to merge. | 13:01 |
| tmorin | AJaeger: ah ok... I thought it was doing that branch per branch | 13:02 |
| *** bobh has joined #openstack-infra | 13:02 | |
| AJaeger | tmorin: either use new syntax Depends-On: URL for a single change - or remove and reapprove ;) | 13:02 |
| tmorin | AJaeger: wasn't that the behavior, at least some time ago ? | 13:02 |
| AJaeger | tmorin: no, never | 13:02 |
| tmorin | AJaeger: ok... | 13:02 |
| fungi | tmorin: the behavior with change-id type depends-on has always been that you're depending on every change in any project and any branch with that id | 13:03 |
| openstackgerrit | Markus Hosch proposed openstack-infra/zuul master: Reload tenant in case of new project branches https://review.openstack.org/600088 | 13:03 |
| AJaeger | fungi, could I trouble you to review 593340 , please? | 13:03 |
| tmorin | fungi: so it means that when backporting a fix with a Depends-On, the Depends-On very often has to be adjusted to the right change URL | 13:04 |
| *** ansmith has joined #openstack-infra | 13:04 | |
| fungi | tmorin: sure, that could be seen as a feature i suppose since more people wanted to be able to depend on changes for specific branches rather than on all changes with the same id | 13:05 |
| AJaeger | ah, frickler did it - thanks. fungi, no need for that one... | 13:05 |
| fungi | cool, i was wondering why it needed looking at as it was already approved ;) | 13:06 |
| AJaeger | config-core, some more repos are ready for python3-first: https://review.openstack.org/598974 https://review.openstack.org/597597 https://review.openstack.org/597574 | 13:06 |
| fungi | SotK: AJaeger: i'll see if we somehow regressed the js jobs by adding the artifact signing role | 13:06 |
| AJaeger | thanks, fungi | 13:06 |
| SotK | fungi: thanks | 13:06 |
| tmorin | fungi: I can leave with that, but if the behavior was "merge if the change with this I... in the same branch has merged" , that would possibly make this more confortable | 13:07 |
| *** yamamoto has joined #openstack-infra | 13:08 | |
| fungi | tmorin: well, we've never had that. also zuul v3 adds the ability for us to do cross-project dependencies between gerrit changes and github pull requests. the latter need a url-specific dependency identifier anyway which is why using gerrit ids for depends-on is deprecated | 13:09 |
| *** agopi has quit IRC | 13:10 | |
| fungi | AJaeger: SotK: looking at an example failure, it looks like we failed to declare the gpg_key variable the sign-artifacts role expects | 13:11 |
| fungi | but also, this is a job which runs on non-release events so i don't think we should use that role there | 13:11 |
| openstackgerrit | Merged openstack-infra/project-config master: remove job settings for watcher repositories https://review.openstack.org/593340 | 13:11 |
| *** elod has joined #openstack-infra | 13:11 | |
| fungi | i wonder if we're reusing the same job for both branch tip tarballs and tagged release tarballs when it comes to javascript projects? | 13:11 |
| *** aojea has quit IRC | 13:12 | |
| fungi | http://logs.openstack.org/67/6776d05c12596c4dbd2b4c737b1cf6bb23884ef2/post/publish-openstack-javascript-content/2ebe59d/ara-report/result/1b961c7f-6c10-4324-85ee-687cae71560e/ | 13:12 |
| tmorin | fungi: really ? I would have thought the pattern to be not uncommon: fix X in project A allowing a fix in project B: merge X in master, stable/x, stable/y -- in parallel, merge the fix in B master, stable/x, stable/y ... | 13:12 |
| *** myoung has joined #openstack-infra | 13:13 | |
| fungi | tmorin: it might be a common case, but it's not a feature we've had in the past | 13:13 |
| tmorin | fungi, AJaeger: ok... fair enough.. thanks for your explanations! | 13:13 |
| AJaeger | fungi, we have publish-openstack-javascript-tarball as post job | 13:14 |
| fungi | tmorin: more often it seems like we ran into problems where a fix needed to merge to be backported to different branches to solve upgrade failures and so changes needed to depends-on fixes in other branches | 13:15 |
| AJaeger | fungi, indeed we use that content job in both post and release pipelines - while the nodejs4-publish-to-npm template uses different jobs | 13:15 |
| fungi | i.e. backport to stable/queens needed a backport merged on stable/pike first and then the stable/rocky change depended on merging a fix to stable/queens... | 13:16 |
| fungi | AJaeger: nodejs4-publish-to-npm doesn't take care of our releases published to tarballs.o.o i guess, which are the ones we actually want merged | 13:17 |
| fungi | i guess we need different versions of this job so that ones triggered from tags get signed and those triggered from ref updates don't | 13:17 |
| AJaeger | fungi, those are different - just wanted to point out that I think you're on the right track that we run same job in both post and release | 13:18 |
| *** tmorin has left #openstack-infra | 13:19 | |
| *** eharney has joined #openstack-infra | 13:23 | |
| *** agopi has joined #openstack-infra | 13:30 | |
| *** jistr is now known as jistr|call | 13:32 | |
| *** kjackal has quit IRC | 13:35 | |
| *** rh-jelabarre has joined #openstack-infra | 13:36 | |
| *** dhajare has quit IRC | 13:38 | |
| *** jtomasek has joined #openstack-infra | 13:42 | |
| *** kjackal has joined #openstack-infra | 13:44 | |
| AJaeger | dhellmann: I think OpenStackClient is also ready - want to un-WIP https://review.openstack.org/#/c/597577 ? | 13:53 |
| dhellmann | AJaeger : you're right; done | 13:54 |
| AJaeger | config-core, some more repos are ready for python3-first: https://review.openstack.org/598974 https://review.openstack.org/597597 https://review.openstack.org/597574 https://review.openstack.org/#/c/597577 - please review | 13:55 |
| AJaeger | dhellmann: that means roughly half of teams are done? | 13:56 |
| dhellmann | 28/66 | 13:56 |
| dhellmann | 42% | 13:56 |
| dhellmann | only 7 teams haven't started yet | 13:57 |
| AJaeger | plus those 4 - makes 32/66 -> 49 per cent... | 13:57 |
| dhellmann | we have 406 open patches to import settings | 13:57 |
| dhellmann | yeah, I don't count them done until those cleanup patches merge :-) | 13:57 |
| AJaeger | fair | 13:57 |
| AJaeger | 406 ;( | 13:58 |
| dhellmann | done-ish :-) | 13:58 |
| * AJaeger tried pushing some patches through the last days... | 13:58 | |
| dhellmann | we have several teams with 10s of open patches | 13:58 |
| dhellmann | only 2 of the teams that haven't started have more than 10 repos | 13:58 |
| dhellmann | yeah, you've been a review machine | 13:58 |
| dhellmann | we have a pretty high recheck rate on these, though | 13:59 |
| dhellmann | the stable branches are in bad shape | 13:59 |
| dhellmann | it's even hard to land anything in master for a few projects | 13:59 |
| AJaeger | dhellmann: yeah, indeed, stable is in bad shape for many branches and repos | 13:59 |
| AJaeger | dhellmann: we should ask a couple of projects to retire repos ;) | 14:00 |
| *** TheJulia is now known as needssleep | 14:00 | |
| dhellmann | I need to pull some stats about which jobs caused rechecks | 14:00 |
| dhellmann | charms hasn't started and they have 80 :-) | 14:00 |
| * dhellmann has to step out for pre-travel errands | 14:00 | |
| AJaeger | fun... | 14:00 |
| *** eernst has joined #openstack-infra | 14:04 | |
| ssbarnea | i see lots of timeout failures on post collect which is apparently 30mins. Can we make this 35-40min as I bet this would avoid 9/10 timeouts happening there, as I seen it happening very close to the last tasks. | 14:05 |
| openstackgerrit | Merged openstack-infra/nodepool master: Add metric for image build result https://review.openstack.org/590412 | 14:06 |
| *** quiquell|rover has quit IRC | 14:08 | |
| *** alexchadin has quit IRC | 14:10 | |
| *** kjackal has quit IRC | 14:13 | |
| *** kjackal has joined #openstack-infra | 14:16 | |
| *** cshastri has quit IRC | 14:16 | |
| *** lbragstad is now known as elbragstad | 14:18 | |
| mnaser | AJaeger, dhellmann: pushed those out | 14:21 |
| *** holser_ has quit IRC | 14:22 | |
| fungi | ssbarnea: which jobs? | 14:22 |
| *** holser__ has joined #openstack-infra | 14:22 | |
| ssbarnea|bkp | fungi the magic http://logs.openstack.org/84/583984/4/check/tripleo-ci-centos-7-containerized-undercloud-upgrades/d89eaff/job-output.txt.gz#_2018-09-07_13_28_44_198036 | 14:23 |
| Shrews | ssbarnea|bkp: you should be able to change that by changing the value for https://zuul-ci.org/docs/zuul/user/config.html#attr-job.post-timeout | 14:24 |
| hwoarang | pabelanger: ok thank you. it seems that the kernel.org mirror is outdated | 14:25 |
| AJaeger | thanks, mnaser | 14:25 |
| *** eernst has quit IRC | 14:25 | |
| ssbarnea|bkp | Shrews thanks, i am trying to see if this works. I seen timeouts too often. | 14:26 |
| *** eernst has joined #openstack-infra | 14:26 | |
| AJaeger | config-core, a trivial job removal for review, please - https://review.openstack.org/600668 | 14:26 |
| openstackgerrit | Merged openstack-infra/project-config master: remove job settings for Packaging-rpm repositories https://review.openstack.org/598974 | 14:29 |
| openstackgerrit | Merged openstack-infra/nodepool master: Improve static provider to check non-ssh hosts https://review.openstack.org/600079 | 14:32 |
| *** pcaruana has quit IRC | 14:33 | |
| openstackgerrit | Merged openstack-infra/project-config master: remove job settings for qinling repositories https://review.openstack.org/597597 | 14:35 |
| openstackgerrit | Merged openstack-infra/project-config master: remove job settings for OpenStack-Helm repositories https://review.openstack.org/597574 | 14:35 |
| openstackgerrit | Merged openstack-infra/project-config master: remove job settings for OpenStackClient repositories https://review.openstack.org/597577 | 14:35 |
| openstackgerrit | Merged openstack-infra/nodepool master: Fix missing node state refresh in static driver https://review.openstack.org/600080 | 14:35 |
| openstackgerrit | Merged openstack-infra/nodepool master: Fix use of wrong label static test fixture https://review.openstack.org/600081 | 14:35 |
| dhellmann | mnaser : thanks! | 14:41 |
| corvus | ssbarnea|bkp: what is that playbook doing that takes 30+ minutes? | 14:42 |
| corvus | ssbarnea|bkp: collect logs, i see -- but why does it take 30+ minutes to do that? | 14:43 |
| *** anteaya has joined #openstack-infra | 14:49 | |
| *** spartakos has joined #openstack-infra | 14:52 | |
| ssbarnea | corvus: this is something i plan to investigate and address (or at least ping those that can fix it). mainky upgrades to generate two set of tempest results which are converted to xml. also ARA xml generation is taking more as there are more tasks to run. | 14:54 |
| *** apetrich_ has quit IRC | 14:58 | |
| corvus | ssbarnea: do you mean you run tempest in the post playbook? | 14:58 |
| ssbarnea | no way, that should be part of the tempest. I will check and ping you. I am on PTO all next week so now I am trying to get wip work sorted. | 15:00 |
| *** holser__ has quit IRC | 15:03 | |
| *** jistr|call is now known as jistr | 15:03 | |
| *** holser_ has joined #openstack-infra | 15:04 | |
| *** d0ugal has quit IRC | 15:04 | |
| *** gyee has joined #openstack-infra | 15:06 | |
| fungi | ssbarnea: is it just the tripleo-ci-centos-7-containerized-undercloud-upgrades job which is timing out this way, or others too? | 15:06 |
| *** ginopc has quit IRC | 15:07 | |
| openstackgerrit | Doug Hellmann proposed openstack-infra/project-config master: remove job settings for heat repositories https://review.openstack.org/597289 | 15:08 |
| clarkb | corvus: ianw +2'd https://review.openstack.org/#/c/600575/ (afs ansible node selection fix) if you want to take another look and approve? I don't think we need to add yet another group | 15:09 |
| fungi | ssbarnea: another build which didn't time out for that same patchset seems to contain 53mib of data in 4392 files under the logs subdir... does that seem like a lot compared to your other jobs? | 15:09 |
| clarkb | https://github.com/openstack/tripleo-quickstart-extras/blob/master/roles/collect-logs/tasks/collect.yml it is a rather extensive set of things being collected. I'm guessing the -containers jobs are slower as they grab all the container logs | 15:13 |
| clarkb | fungi: ssbarnea probably need insight into how long each of those individual tasks within the play are taking | 15:14 |
| *** jtomasek has quit IRC | 15:17 | |
| ssbarnea|bkp | fungi not really, sounds like average to me. this reminded me of something i seen in the last two weeks, the task that was building the list of files was timing out, it was looking as stuck trying to create the log size stats. | 15:17 |
| fungi | possible we're more constrained for bandwidth/throughput collecting files from nodes in some providers than others? | 15:18 |
| ssbarnea|bkp | the problem is that in this case we don't have the http://logs.openstack.org/84/583984/4/check/tripleo-ci-centos-7-containerized-undercloud-upgrades/5ac6e52/logs/quickstart_collect_logs.log | 15:18 |
| ssbarnea|bkp | this was the success run, with 10min only, so there is a chance that the timeout is caused by some kind of stuck task. | 15:19 |
| *** e0ne has quit IRC | 15:21 | |
| fungi | yeah, ara reports the elapsed time on the post-logs.yaml playbook in that run as a whipping 6 seconds | 15:21 |
| fungi | s/whipping/whopping/ | 15:21 |
| *** shardy has quit IRC | 15:21 | |
| corvus | clarkb: yes, sorry i thought i got that yesterday. +3 | 15:21 |
| *** akhila has joined #openstack-infra | 15:22 | |
| corvus | fungi: https://review.openstack.org/600558 could use a look from you when you have a sec | 15:23 |
| fungi | ssbarnea|bkp: oh, yeah so even on that 5ac6e52 build we're missing details on the fnial post-logs.yaml task | 15:23 |
| *** akhila has quit IRC | 15:23 | |
| *** d0ugal has joined #openstack-infra | 15:25 | |
| clarkb | looks like we cleared the backlog overnight | 15:28 |
| clarkb | appears neutron may have addressed the pep8 job failures too that were reseting the gate often | 15:29 |
| *** fried_rice has quit IRC | 15:29 | |
| *** ramishra_ has quit IRC | 15:30 | |
| openstackgerrit | Merged openstack-infra/openstack-zuul-jobs master: Remove legacy-cloudkitty-dsvm-install https://review.openstack.org/600668 | 15:30 |
| clarkb | fungi: https://review.openstack.org/#/c/600557/ is likely also of interest | 15:31 |
| clarkb | should fix cloud launcher on bridge.o.o, looks like this particular issue was affecting cloud launcher on puppetmster too | 15:32 |
| *** efried has joined #openstack-infra | 15:32 | |
| dmsimard | fungi: the post playbook in ara is sort of flawed because the report is saved before the playbook actually ends | 15:33 |
| fungi | right, that last task is the one which collects it | 15:33 |
| fungi | i don't know that i'd call it "flawed" just a known caveat | 15:33 |
| dmsimard | Yup, just wanted to make sure that was understood | 15:33 |
| *** Dobroslaw has quit IRC | 15:33 | |
| dmsimard | Sure | 15:33 |
| clarkb | this is why the old jenkins scp plugin console log copy had a hack of spawning a thread that would continue running after the job completed to finish copying the console log in its entireety | 15:35 |
| clarkb | what this did was move the hard to debug point from within the job context to the jenkins master | 15:35 |
| clarkb | more use friendly but ugh pain to debug | 15:35 |
| hwoarang | infra-root: could you confirm with me that mirror-update.openstack.org/23.253.92.251 is the host which runs the various distro mirror scripts? | 15:37 |
| fungi | hwoarang: for the moment it is. also 2001:4800:7818:103:be76:4eff:fe04:5f3a | 15:37 |
| fungi | hwoarang: it can change over time though, we don't guarantee a stable source address there | 15:38 |
| pabelanger | 104.130.141.75 is mirror-update for me | 15:38 |
| fungi | er, right | 15:38 |
| fungi | mirror-update.openstack.org is an alias for mirror-update01.openstack.org | 15:38 |
| fungi | and mirror-update01.openstack.org is 104.130.141.75 and 2001:4800:7818:103:be76:4eff:fe04:5f3a | 15:38 |
| hwoarang | oh it changed because on stage.opensuse.org we had 23.253.92.251 on whitelist | 15:38 |
| hwoarang | ok so a dynamic ip will not work in that case. need to find another way then | 15:39 |
| fungi | hwoarang: aha, yes whitelisting the address for our mirror updater won't really be a good solution | 15:39 |
| clarkb | corvus: https://review.openstack.org/#/c/600562/1 does make the logging much better, but ianw points out we continue to leak that info to the main console log. I guess ansible writes to the logfile and to stdout by default? I +2'd because I think it is an incremental improvement as is | 15:39 |
| fungi | hwoarang: it's not so much that it's a dynamic ip address, more that our infrastructure is dynamic and we upgrade servers by rebuilding and replacing them | 15:40 |
| fungi | so servers are getting replaced with other servers which have different ip addresses | 15:40 |
| hwoarang | hmm i see | 15:40 |
| fungi | we keep a static cname resource record in dns pointing to whatever the current server is for that service | 15:41 |
| fungi | and update the cname whenever the service moves to a different/new server | 15:41 |
| hwoarang | ok so the mirror-update.openstack.org hostname is static | 15:42 |
| hwoarang | ok i will check if we can do whitelisting based on that | 15:42 |
| openstackgerrit | Merged openstack-infra/system-config master: Serialize puppet on afs servers properly https://review.openstack.org/600575 | 15:42 |
| fungi | static for the foreseeable future, though i could imagine replacing it with a ci job at some point so we no longer relied on a persistent server to do that work at all | 15:43 |
| *** ssbarnea|bkp has quit IRC | 15:43 | |
| hwoarang | i see | 15:43 |
| *** camunoz has quit IRC | 15:43 | |
| fungi | and i think we don't want our mirror servers to be in any sort of privileged position for the data we mirror. it's something that has to be kept up/communicated over time if we end up trying to maintain such a relationship, and also means additional hurdles for anyone else who wants to deploy a similar solution | 15:44 |
| corvus | clarkb: yeah, let's push it in so it's there when we need to debug apply test | 15:44 |
| clarkb | ansible run just started 15 minutes ago so about an hour and a half until we see 600575 tak effect | 15:46 |
| corvus | clarkb: oh? what did our full runtime end up being after switching back to -f20? | 15:47 |
| clarkb | corvus: it lowered it by a few minutes but because we trigger on a 15 minute cron period the effective rotation was the same iirc | 15:47 |
| clarkb | corvus: I think the -f 1 with ~30 nodes is dominating the runtime as that cost is about half an hour | 15:48 |
| corvus | clarkb: hrm, last 6 runs look to be 75 mins | 15:48 |
| corvus | clarkb: ok. let's let that land, then do -f50 | 15:48 |
| pabelanger | corvus: clarkb: fungi: mordred: With ptg next week, do we have any thoughts are ara-reports for bridge.o.o? EG: enable apache2 on bridge.o.o to host them, or some other process with new server? | 15:52 |
| clarkb | pabelanger: I think we should have reporting like that, we just need to make sure we aren't leaking sensitive info first. Apparently the sensitive info we are logging is a bug and mordred intended on fixing that so this should be doabl | 15:53 |
| corvus | pabelanger: good question! we also need a database. | 15:53 |
| clarkb | we probably want a bigger server if we are going to run a webserver on it too | 15:53 |
| corvus | clarkb: i don't think the openstacksdk stuff will go into ara. that's python logging module stuff; shouldn't touch the ara callback. | 15:53 |
| *** yamamoto has quit IRC | 15:53 | |
| corvus | but we may need to think about setting no_log in places | 15:54 |
| *** yamamoto has joined #openstack-infra | 15:54 | |
| *** dtantsur is now known as dtantsur|afk | 15:54 | |
| clarkb | ya I think ara will dump host and group vars into output in places | 15:54 |
| * corvus looks at cloud-launcher | 15:54 | |
| corvus | whatever we do -- we should set it up firewalled off to start so we can inspect it manually | 15:55 |
| pabelanger | clarkb: re security, I figure once we decided which server ara runs on, we can first keep it firewalled off to http, then infra-root and do some audit of playbook runs (kinda how we first disabled jenkins UI). Then confirm things good and open to public | 15:55 |
| pabelanger | what corvus said | 15:55 |
| clarkb | corvus: ++ | 15:55 |
| corvus | pabelanger: good idea :) | 15:55 |
| clarkb | and pabelanger too | 15:55 |
| pabelanger | would we want local DB on server or trove? | 15:56 |
| corvus | i guess trove is still what we're doing | 15:56 |
| openstackgerrit | Merged openstack-infra/system-config master: Use git-servers group in remote_puppet_else https://review.openstack.org/600576 | 15:56 |
| openstackgerrit | Merged openstack-infra/system-config master: Up profiled tasks limit https://review.openstack.org/600256 | 15:56 |
| openstackgerrit | Merged openstack-infra/system-config master: Update launch README for bridge https://review.openstack.org/600558 | 15:56 |
| corvus | i think the main question is -- how to install and run ara? do we want to make it one of the first 'run in container' services? | 15:56 |
| clarkb | for something like this I'm not sure it matters too much, it isn't state we need to persist carefully | 15:57 |
| clarkb | re db | 15:57 |
| clarkb | as for install and run ara, if it is going on the bridge I think that it would be ansibled | 15:57 |
| corvus | yeah, but trove is still pretty easy | 15:57 |
| corvus | clarkb: right, but installed globally? in a venv? seems like container would be better? | 15:57 |
| *** Bhujay has quit IRC | 15:58 | |
| *** yamamoto has quit IRC | 15:59 | |
| openstackgerrit | Markos Chandras (hwoarang) proposed openstack-infra/system-config master: Revert "Use Kernel.org mirror for OpenSUSE" https://review.openstack.org/600805 | 16:00 |
| hwoarang | infra-root: can you help me get ^^ in to resolve the CI opensuse failures for the time being? :/ TIA | 16:00 |
| hwoarang | dirk fyi^ | 16:00 |
| fungi | gonna grab lunch and run pre-trip errands but should be back soonish | 16:00 |
| clarkb | corvus: maybe? I don't actually know what ara looks like as a service | 16:00 |
| clarkb | I've only used it as generate a thing or weird apache middleware | 16:00 |
| *** jamesmcarthur has quit IRC | 16:06 | |
| openstackgerrit | Merged openstack-infra/system-config master: Write ansible log to fileout in apply test https://review.openstack.org/600562 | 16:06 |
| *** jamesmcarthur has joined #openstack-infra | 16:06 | |
| corvus | backlog is okay today (everyone is packing?) so how about i restart zuul in a few minutes? | 16:09 |
| clarkb | wfm | 16:09 |
| pabelanger | 312804 | 16:09 |
| pabelanger | oops | 16:10 |
| *** jamesmcarthur has quit IRC | 16:10 | |
| openstackgerrit | Merged openstack-infra/system-config master: Don't set security groups in openstackci-packethost https://review.openstack.org/600557 | 16:10 |
| pabelanger | clarkb: corvus: fungi: I think we can do virtualenv first for ara, and if that goes well move into container later? | 16:11 |
| pabelanger | but yah, ara on bridge.o.o, which then stores into trovedb | 16:11 |
| corvus | pabelanger: hrm. i wonder why we don't just do container? isn't that what we decided in the spec? | 16:11 |
| pabelanger | we then should be able to stand up ara.openstack.org, another minimal server | 16:11 |
| pabelanger | corvus: I mean, i guess we could. I'm not sure yet how that looks from ansible pov, I guess docker_image task should be okay | 16:12 |
| corvus | pabelanger: seems like a great thing to work out at the ptg :) | 16:12 |
| pabelanger | indeed | 16:12 |
| pabelanger | and pbrx jobs for ara also | 16:12 |
| corvus | dmsimard: is there an existing published ara server container image? | 16:13 |
| *** holser_ has quit IRC | 16:14 | |
| *** felipemonteiro has joined #openstack-infra | 16:14 | |
| *** felipemonteiro_ has joined #openstack-infra | 16:16 | |
| corvus | do we think the zuul_restart playbook is working now? i think the issues were the init scripts which got fixed? | 16:16 |
| corvus | i'm going to give it a shot :) | 16:17 |
| *** felipemonteiro has quit IRC | 16:19 | |
| corvus | the web.pid file was not removed | 16:19 |
| *** mgagne has left #openstack-infra | 16:20 | |
| *** mgagne has joined #openstack-infra | 16:20 | |
| corvus | we're generating ssh keys for projects | 16:21 |
| *** mriedem is now known as mriedem_bbq | 16:22 | |
| pabelanger | cool! | 16:23 |
| *** ssbarnea|bkp has joined #openstack-infra | 16:23 | |
| corvus | that's done; loading branches now | 16:24 |
| *** bnemec is now known as beekneemech | 16:24 | |
| *** olivierb has quit IRC | 16:24 | |
| corvus | cat jobs | 16:25 |
| corvus | the rest of the restart playbook seems to be going well; most of the ze's have stopped. it's just waiting for the last few to restart them | 16:25 |
| corvus | mergers are already up | 16:25 |
| clarkb | what was the fix for the init scripts, I don't remember that going in. I'm guessing waiting for service to actually stop and clean up pids on its own? | 16:26 |
| corvus | clarkb: yep | 16:26 |
| corvus | but it looks like zuul-web didn't delete its own pidfile for some reason | 16:26 |
| *** dhinesh has joined #openstack-infra | 16:26 | |
| corvus | the restart playbook is pretty usable in the current state though. you just have to delete web.pid manually once it stops. that's the only hiccup so far. | 16:27 |
| corvus | and the executors have started again | 16:27 |
| clarkb | and then start zuul-web manually? | 16:27 |
| *** spartakos has quit IRC | 16:27 | |
| corvus | clarkb: no, playbook will do that | 16:27 |
| corvus | http://paste.openstack.org/show/729697/ | 16:27 |
| *** e0ne has joined #openstack-infra | 16:28 | |
| corvus | clarkb: it just gets stuck at "wait to stop zuul web", then you go delete web.pid, and then it resumes | 16:28 |
| clarkb | gotcha | 16:28 |
| corvus | it's back up; re-enqueing now | 16:28 |
| corvus | we need to check on the logs for a job to make sure we can still upload them correctly (and haven't broken secrets somehow) | 16:29 |
| *** dhinesh has quit IRC | 16:30 | |
| *** florianf has quit IRC | 16:30 | |
| *** jamesmcarthur has joined #openstack-infra | 16:32 | |
| clarkb | corvus: http://logs.openstack.org/27/596427/4/check/tox-py35/1e8a9e6/ other than a suspicuously short url that lgtm | 16:32 |
| corvus | \o/ | 16:32 |
| corvus | enqueue is finished | 16:33 |
| *** jpich has quit IRC | 16:34 | |
| *** yamamoto has joined #openstack-infra | 16:35 | |
| corvus | so i think we're ready to try some CD stuff next week :) | 16:36 |
| corvus | http://zuul.openstack.org/api/project-ssh-key/openstack-infra/system-config.pub | 16:36 |
| corvus | well, as soon as https://review.openstack.org/599073 lands | 16:37 |
| *** e0ne has quit IRC | 16:42 | |
| *** hasharAway is now known as hasharDinner | 16:43 | |
| clarkb | run with the afs fix in it has just started | 16:46 |
| dirk | hwoarang: thanks. I'm a bit concerned about gwdg.de again, see inline comment | 16:46 |
| dirk | hwoarang: we had issues with tumbleweed, which is pretty large, and gwdg.de is pretty slow in the US | 16:46 |
| dirk | it took more than the AFS timeout to sync which tripped something over | 16:47 |
| *** felipemonteiro_ has quit IRC | 16:53 | |
| hwoarang | dirk: but tw is still using rsync.o.o so it should be ok right? | 16:54 |
| hwoarang | dirk: maybe we can try the leaseweb mirrors in the US? i have 0 metrics on which mirrors are stable and which are not so it's a shot in the dark | 16:56 |
| openstackgerrit | Markos Chandras (hwoarang) proposed openstack-infra/system-config master: Revert "Use Kernel.org mirror for OpenSUSE" https://review.openstack.org/600805 | 16:58 |
| hwoarang | ^^ with the leaseweb mirror | 16:59 |
| *** myoung is now known as myoung|dr | 16:59 | |
| *** Swami has joined #openstack-infra | 17:01 | |
| *** derekh has quit IRC | 17:01 | |
| *** jpena is now known as jpena|off | 17:01 | |
| *** e0ne has joined #openstack-infra | 17:02 | |
| *** psachin has joined #openstack-infra | 17:02 | |
| evrardjp | thanks hwoarang | 17:09 |
| *** trown is now known as trown|lunch | 17:09 | |
| *** e0ne has quit IRC | 17:11 | |
| clarkb | doing more ansible performance debugging and significant amount of time seems to be spent generating inventory from our inventory sources. `sudo ansible logstash-worker01.openstack.org -vvv -m ping` illustrates this | 17:16 |
| clarkb | this cost is probably relatively low for our puppet_run_else playbook but high for others that execute against a small number of nodes | 17:17 |
| *** tesseract has quit IRC | 17:20 | |
| corvus | clarkb: there's a cache, but i'm not sure how it works or what the timeout situation is | 17:21 |
| *** caphrim007 has joined #openstack-infra | 17:22 | |
| dmsimard | corvus: I know that harlowja and SpamapS run ara in a container but I haven't set up anything "official" | 17:23 |
| dmsimard | There's two ansible roles that I know of, https://github.com/openstack/ansible-role-ara (that I'm trying to make a thing eventually) as well as https://github.com/robertdebock/ansible-role-ara | 17:23 |
| corvus | dmsimard: interested in publishing containers automatically on every commit to dockerhub? | 17:23 |
| corvus | s/containers/images/ | 17:24 |
| clarkb | corvus: oddly strace seems to imply that reading the cache for all nodes is at least part of the slowness | 17:24 |
| *** jtomasek has joined #openstack-infra | 17:24 | |
| corvus | Shrews: ^ both of these conversations may interest you | 17:24 |
| *** ssbarnea|bkp2 has joined #openstack-infra | 17:25 | |
| dmsimard | corvus: I don't have a personal use case for that yet but sure | 17:26 |
| *** ssbarnea|bkp has quit IRC | 17:26 | |
| corvus | dmsimard: so ara (short for ansible-role-ara) installs ara in a venv, yeah? | 17:26 |
| clarkb | it appears to read all the contents of /var/cache/ansible/facts to run against a single node. I'm guessing it does this as part of determining group membership to determine where to run the module/play | 17:26 |
| *** ijw_ has quit IRC | 17:27 | |
| *** jmorgan1 has quit IRC | 17:28 | |
| dmsimard | corvus: the default behavior of the openstack ara role is to install the latest version from pypi in a virtualenv and have the built-in webserver run as a systemd service, yes. | 17:28 |
| dmsimard | corvus: ideally I want to get to a point where the role can be used to test ara itself, I've been experimenting on and off when time allows: https://review.openstack.org/#/c/557933/ | 17:28 |
| openstackgerrit | Merged openstack-infra/zuul-jobs master: add-build-sshkey: Remove only the master key https://review.openstack.org/599073 | 17:29 |
| corvus | dmsimard: thanks -- we might use that to set up our system-config ara, or we might go the container route, and if you don't object, throw some patches at the ara repo to do those image builds | 17:29 |
| dmsimard | FWIW I'm going to take a first good look at the usability of the ara 1.0 backend, API and callback next week with tristanC | 17:29 |
| dmsimard | I landed what I think are the last big schema changes | 17:30 |
| dmsimard | let me show you something | 17:31 |
| corvus | can plays have names? | 17:33 |
| clarkb | corvus: Shrews reading the docs on fact_caching config option implies it is host specific (because well facts are host specific) so I don't think ansible should be reading all of the fact caches when running against a single node. It should load that node's facts from cache only | 17:33 |
| clarkb | I don't know how much a real world timesaving this would be with our large puppet_run_else playbook but seems like a bug in ansible? maybe someone with a bit more knowledge of how ansible is supposed to work on startup can check | 17:34 |
| clarkb | corvus: we also seem to rarely use as many forks as we have allowed for | 17:35 |
| corvus | clarkb: seems like the big win on forks would be puppet_else, yeah? | 17:35 |
| clarkb | but that may just be difference between human poll rate and cpu execution rate :) | 17:35 |
| clarkb | corvus: ya | 17:35 |
| clarkb | I've watched else do 1-19 forks just doing a poll via ps | 17:36 |
| clarkb | corvus: also base since that runs against everything | 17:36 |
| *** psachin has quit IRC | 17:36 | |
| *** jmorgan1 has joined #openstack-infra | 17:37 | |
| *** spartakos has joined #openstack-infra | 17:39 | |
| *** diablo_rojo has joined #openstack-infra | 17:40 | |
| clarkb | for those of you in channel attending the PTG I just sent email to the list making Monday evening beer garden team dinner official. Suggest we meet at 6pm in the PTG hotel lobby to group up into taxi/uber/lyft travel | 17:45 |
| *** roman_g has quit IRC | 17:45 | |
| clarkb | dhellmann: diablo_rojo smcginnis ttx infra-root config-core ^ | 17:45 |
| diablo_rojo | clarkb, cool :) Thanks! | 17:46 |
| dhellmann | clarkb : sounds good | 17:47 |
| dmsimard | clarkb: http://ovh.dmsimard.com:8000/api/v1/ | 17:48 |
| dmsimard | er, that was meant for corvus | 17:49 |
| dmsimard | ah, found a bug :D | 17:49 |
| dmsimard | sec | 17:50 |
| clarkb | corvus: ianw afs fix got our period down to one hour, was an hour 15-30 minutes prior | 17:50 |
| clarkb | not bad, but plenty more to improve upon I think. corvus is there a -f 50 change yet? | 17:51 |
| corvus | clarkb: not yet, about to write it | 17:51 |
| *** mriedem_bbq is now known as mriedem | 17:52 | |
| openstackgerrit | James E. Blair proposed openstack-infra/system-config master: Name plays in playbooks https://review.openstack.org/600827 | 17:52 |
| openstackgerrit | James E. Blair proposed openstack-infra/system-config master: Increase forks to 50 https://review.openstack.org/600828 | 17:53 |
| corvus | clarkb: ^ | 17:54 |
| clarkb | infra-root ^ any chance you can review 600828 real quick? We are making progress on improving ansible performance pre PTG which will be helpful as we try to do things at the PTG | 17:55 |
| *** electrofelix has quit IRC | 17:55 | |
| clarkb | corvus: reading 600827 I think we may want to set strategy: free on more of the base plays | 17:56 |
| clarkb | they are largely indepednent and we don't need to do things in lock step | 17:56 |
| *** electrofelix has joined #openstack-infra | 17:56 | |
| corvus | clarkb: yeah, i think you're right | 17:56 |
| clarkb | that should get us better thread utilization | 17:57 |
| clarkb | I'll write that change | 17:59 |
| *** electrofelix has quit IRC | 17:59 | |
| dmsimard | corvus: okay, sorry -- here you go: http://ovh.dmsimard.com:8000/api/v1/ | 17:59 |
| openstackgerrit | James E. Blair proposed openstack-infra/system-config master: Move the !ci-backup play next to the rest of the servers https://review.openstack.org/600833 | 17:59 |
| dmsimard | sent a patch to add support for names :) https://review.openstack.org/#/c/600834/ | 18:01 |
| openstackgerrit | Clark Boylan proposed openstack-infra/system-config master: Run base playbook with strategy free https://review.openstack.org/600835 | 18:02 |
| *** ijw has joined #openstack-infra | 18:02 | |
| corvus | dmsimard: how does a playbook name work? | 18:03 |
| corvus | dmsimard: and hey, look, it's an ara 1.0! :) | 18:03 |
| corvus | dmsimard: i mean, where would you put the name for a playbook? | 18:04 |
| corvus | (since a playbook is strictly a list of plays aiui) | 18:04 |
| openstackgerrit | James E. Blair proposed openstack-infra/system-config master: Remove !ci-backup play https://review.openstack.org/600837 | 18:04 |
| clarkb | corvus: the filename maybe? thought you can use --- to have multiple documents in a single file | 18:05 |
| dmsimard | corvus: What you're looking at is the backend and the API -- I haven't settled on the best approach to give playbooks labels and names from a client perspective yet. | 18:05 |
| dmsimard | corvus: It could be a simple ansible module or something else, what would be a convenient way for you to do it ? | 18:06 |
| dmsimard | From an API perspective, they can be done as POST or PATCH so it doesn't need to be at runtime | 18:07 |
| *** bharath_t has joined #openstack-infra | 18:07 | |
| *** eharney has quit IRC | 18:07 | |
| corvus | dmsimard: i'd love it if that were an ansible thing, like if ansible allowed you to name playbooks and that was the ara interface. if i were to suggest an implementation, i'd suggest a sort of "meta" play that you add to a playbook to name it (a la zuul's pragma config item). or maybe a special comment syntax. | 18:08 |
| *** trown|lunch is now known as trown | 18:09 | |
| corvus | i think it would feel weird to have special ara modules in playbooks. i like that ara's heisenburg factor is low. :) | 18:09 |
| dmsimard | something I was thinking about was to use ansible vars | 18:10 |
| dmsimard | i.e, ansible-playbook -i foo -e "playbook_name=foo playbook_label=bar" | 18:11 |
| corvus | dmsimard: oh, for the zuul use case i think that would be great | 18:11 |
| *** dhinesh has joined #openstack-infra | 18:12 | |
| dmsimard | I'm pretty excited about the use of labels FWIW, it's what I ended up going for to categorize playbooks. It's very generic so it could be used to identify phases (such as pre/run/post), or results (success/failure), etc. | 18:12 |
| *** anteaya has quit IRC | 18:13 | |
| dmsimard | oh, a new addition is the notion of host aliases -- to track a single host across multiple playbook runs | 18:13 |
| *** anteaya has joined #openstack-infra | 18:14 | |
| dmsimard | i.e, click on a host to see what playbooks it ran and what were the results | 18:14 |
| *** jamesmcarthur has quit IRC | 18:16 | |
| SpamapS | dmsimard: coming to Denver? I'd love to get some ARA hacking time in with you. | 18:17 |
| SpamapS | Including possibly contributing our containerized ara stuff somewhere. | 18:17 |
| *** mshalamov has quit IRC | 18:17 | |
| dmsimard | SpamapS: I'm not, I'll be in Ausstin next month though | 18:18 |
| dmsimard | Austin* | 18:18 |
| SpamapS | Oh we can do it there then. | 18:18 |
| SpamapS | I'll be at AnsibleFest too | 18:18 |
| SpamapS | labels++ btw | 18:18 |
| bharath_t | clarkb, do we need to create project manually in launchpad.net for bugs/blueprints? | 18:19 |
| *** ijw has quit IRC | 18:19 | |
| *** _Cyclone_ has quit IRC | 18:19 | |
| clarkb | bharath_t: I think your projects specifies they would use storyboard in the yaml file. Storyboard projects will have been auto created for you. The project creators doc should have all this info for you too | 18:20 |
| *** gfidente has quit IRC | 18:21 | |
| *** _Cyclone_ has joined #openstack-infra | 18:23 | |
| bharath_t | clarkb, okay got it ... I misunderstood | 18:24 |
| dhinesh | is there any latest documentation\guide which would help in building a third-party CI from scratch? | 18:26 |
| clarkb | dhinesh: https://docs.openstack.org/infra/system-config/third_party.html | 18:26 |
| openstackgerrit | Goutham Pacha Ravi proposed openstack-infra/project-config master: remove job settings for manila repositories https://review.openstack.org/593898 | 18:26 |
| fungi | dhinesh: does the current one not work? | 18:26 |
| *** jamesmcarthur has joined #openstack-infra | 18:29 | |
| dhinesh | fungi: unfortunately the working version was compromised unintentionally by someone when I was on break | 18:32 |
| dhinesh | fungi: I am using the original https://docs.openstack.org/infra/openstackci/third_party_ci.html to bring it up. i am using older versions of nodepool like 0.4.0 and zuul 2.5.1 and they seem to be be having issues, when i try to push images built images using nodepool-builder | 18:34 |
| fungi | dhinesh: ahh, that's what i was wondering. we've of course moved on to newer zuul/nodepool without jenkins which means job rewrites and all that, so had hoped the openstackci third-party ci setups would continue to use the older versions of stuff | 18:35 |
| *** diablo_rojo has quit IRC | 18:35 | |
| *** diablo_rojo has joined #openstack-infra | 18:35 | |
| *** bobh has quit IRC | 18:36 | |
| *** bobh has joined #openstack-infra | 18:36 | |
| openstackgerrit | Merged openstack-infra/devstack-gate master: Update grenade settings for stable/rocky https://review.openstack.org/591594 | 18:39 |
| dhinesh | fungi: yeah that is what i was hoping as well, to build the way i had originally setup, but am running into errors with python modules | 18:41 |
| smcginnis | Looks like we may have an oddity with the docs.openstack.org site and the publish-openstack-tox-docs-direct job. | 18:41 |
| smcginnis | It's getting a redirect to /latest/ when it shouldn't. | 18:41 |
| smcginnis | So using https://docs.openstack.org/operations-guide/index.html works. | 18:41 |
| smcginnis | Using https://docs.openstack.org/operations-guide/ redirects and 404's. | 18:42 |
| fungi | dhinesh: can you link those "errors with python modules" from details on http://paste.openstack.org/ ? | 18:42 |
| smcginnis | Is that an openstackdocstheme thing that does the redirect or something with our site config? | 18:42 |
| fungi | smcginnis: i think the redirects to latest/ are in a .htaccess file in the openstack-manuals repo | 18:43 |
| smcginnis | fungi: This isn't in -manuals. | 18:43 |
| fungi | is the operations guide intended to be unversioned or release-specific? | 18:43 |
| smcginnis | Unversioned. | 18:43 |
| smcginnis | It uses publish-openstack-tox-docs-direct, which appears to be right. | 18:43 |
| dhinesh | fungi: sure http://paste.openstack.org/show/729706/ | 18:44 |
| smcginnis | As the redirected docs are not there as expected. | 18:44 |
| smcginnis | So really just need to understand where the redirect is coming from. | 18:44 |
| fungi | smcginnis: and you're sure it's not https://git.openstack.org/cgit/openstack/openstack-manuals/tree/www/.htaccess | 18:45 |
| openstackgerrit | sebastian marcet proposed openstack-infra/openstackid-resources master: Added new endpoints for Track Tag Groups https://review.openstack.org/600845 | 18:45 |
| smcginnis | fungi: I don't see anything similar in this repo - http://git.openstack.org/cgit/openstack/operations-guide/tree/ | 18:46 |
| fungi | smcginnis: what i'm trying to say is the openstack-manuals www/.htaccess file is deployed to the root of https://docs.openstack.org/ | 18:47 |
| smcginnis | It had been working. | 18:47 |
| fungi | when did it first break? | 18:47 |
| smcginnis | Not sure really. It had been fine and we've gotten a few updates. | 18:47 |
| smcginnis | Then today I was prep'ing an etherpad for the ops meetup to talk about it and realized it isn't working anymore. | 18:48 |
| corvus | https://docs.openstack.org/.htaccess | 18:48 |
| fungi | huh, interesting that apache doesn't refuse to serve that file | 18:48 |
| fungi | i thought normally it wouldn't serve dotfiles | 18:48 |
| smcginnis | Sure enough, operations guide is in the list of redirectmatch 301 | 18:50 |
| smcginnis | So it's now pulled into here: https://git.openstack.org/cgit/openstack/openstack-manuals/tree/www/.htaccess#n198 | 18:50 |
| smcginnis | Where is REGULAR_REPOS set? | 18:51 |
| fungi | http://codesearch.openstack.org/?q=REGULAR_REPOS | 18:51 |
| smcginnis | Yeah, looking at those. | 18:52 |
| fungi | seems tools/www-generator.py sets it | 18:52 |
| openstackgerrit | James E. Blair proposed openstack-infra/project-config master: Add cherrypy to third-party ci https://review.openstack.org/600846 | 18:52 |
| fungi | smcginnis: hah, i wonder if the readdition to governance is what broke it ;) | 18:53 |
| smcginnis | Oh, that could be. I don't think I've looked at it since that patch was still pending out there. | 18:53 |
| fungi | seems there's a loop over the governance sigs list in the _get_official_repos() function | 18:54 |
| fungi | i wonder if it's no longer parsing that file successfully | 18:54 |
| *** ijw has joined #openstack-infra | 18:54 | |
| fungi | the schema for it did change with the addition of the security-sig repos | 18:54 |
| smcginnis | I don't see any kind of exclusion list there. | 18:54 |
| *** efried is now known as fried_rice | 18:56 | |
| fungi | no, _get_official_repos() does also seem to add the _GOVERNANCE_SIGS_URL repos to the regular_repos list after all | 18:56 |
| fungi | dhellmann: ^ if you're still around, you had some hand in designing that docs transition... do you recall if there was an escape hatch built in for repos publishing documentation which didn't want a redirect to latest/ ? | 18:57 |
| smcginnis | operations-guide appears to be the only one using that publish-direct job | 18:57 |
| *** r-daneel has quit IRC | 18:58 | |
| dhellmann | fungi : I don't know if that case came up specifically | 18:59 |
| smcginnis | Looks like it would be easy enough to add an exlusion check in the www-generator. But where should it go instead? | 18:59 |
| smcginnis | Err, I suppose nowhere, since it shouldn't redirect at all? | 19:00 |
| dhellmann | yeah | 19:00 |
| smcginnis | OK, I'll add an exclusion list of one for now. | 19:01 |
| dhellmann | maybe this repo shouldn't be in the REGULAR_REPOS list | 19:02 |
| smcginnis | Exactly. | 19:02 |
| fungi | right, i think smcginnis is looking at how to exclude it from REGULAR_REPOS | 19:03 |
| dhellmann | ok | 19:03 |
| fungi | right now that list is autogenerated by https://git.openstack.org/cgit/openstack/openstack-manuals/tree/tools/www-generator.py#n364 | 19:03 |
| dhellmann | I wonder what other redirects that's going to disable | 19:03 |
| fungi | and just includes everything official | 19:03 |
| dhellmann | oh, that seems to be the only place REGULAR_REPOS is used | 19:03 |
| clarkb | corvus: any idea if cherrypy is interested in that? (or are you planning to not report for now (I think that requires a new pipeline though)) | 19:03 |
| smcginnis | If this repo clone ever finishes, you can check if I did it right. ;) | 19:04 |
| dhellmann | I'm going to be going offline shortly, but I can look at it over the weekend once I'm in denver | 19:04 |
| smcginnis | Thanks dhellmann | 19:05 |
| smcginnis | I'm presenting it Tuesday morning, so I hope we have plenty of time to clear things up by then. | 19:05 |
| dhellmann | yeah, we should be able to fix it by then | 19:05 |
| corvus | clarkb: not planning on reporting; just want to use it in a required-projects line | 19:05 |
| dhellmann | plan to talk slowly ;-) | 19:05 |
| corvus | (at some point, i'll fix zuul so you don't need to do that to add something to required-projects...) | 19:06 |
| smcginnis | dhellmann: ;) | 19:06 |
| *** jamesmcarthur has quit IRC | 19:06 | |
| *** eernst has quit IRC | 19:07 | |
| smcginnis | dhellmann: Oh, I can just add it to http://git.openstack.org/cgit/openstack/openstack-manuals/tree/tools/www-generator.py#n349 I think? | 19:08 |
| openstackgerrit | Merged openstack-infra/system-config master: Name plays in playbooks https://review.openstack.org/600827 | 19:11 |
| openstackgerrit | Jeremy Stanley proposed openstack-infra/project-config master: Revert "add signing steps to javascript publishing job" https://review.openstack.org/600850 | 19:13 |
| openstackgerrit | Jeremy Stanley proposed openstack-infra/project-config master: Revert "add signing steps to javascript publishing job" https://review.openstack.org/600850 | 19:14 |
| openstackgerrit | Merged openstack-infra/system-config master: Increase forks to 50 https://review.openstack.org/600828 | 19:14 |
| *** jamesmcarthur has joined #openstack-infra | 19:15 | |
| *** anteaya has quit IRC | 19:16 | |
| *** graphene has joined #openstack-infra | 19:18 | |
| *** graphene has quit IRC | 19:23 | |
| *** graphene has joined #openstack-infra | 19:25 | |
| *** beekneemech has quit IRC | 19:29 | |
| *** r-daneel has joined #openstack-infra | 19:30 | |
| zxiiro | anyone here know how to deal with https://docs.openstack.org/bandit/latest/blacklists/blacklist_calls.html#b310-urllib-urlopen | 19:30 |
| zxiiro | It says to audit for URL schemes, and I do a re.match() to check that it's only what I want but I'm not sure how to satisfy whatever the checker is checking for. | 19:30 |
| clarkb | zxiiro: I'm not sure you may have to flag the line with # noqa or similar? | 19:31 |
| clarkb | (and assert you've preventing unexpected local opens) | 19:31 |
| zxiiro | yeah I figure that'd be the easiest solution but was curious if there was a more correct way to assert that I've done my homework. | 19:32 |
| *** EvilienM is now known as EmilienM | 19:33 | |
| fungi | zxiiro: is the code hitting that public? | 19:34 |
| clarkb | fungi: care to review https://review.openstack.org/#/c/600835/ ? I think that may help speed up ansible runtime | 19:35 |
| zxiiro | fungi: yep, take a look at this https://gerrit.linuxfoundation.org/infra/#/c/12498/4/lftools/openstack/image.py | 19:35 |
| *** r-daneel_ has joined #openstack-infra | 19:35 | |
| zxiiro | I added "# nosec" to the line in question to make bandit happy but I'm not sure if that's the right approach | 19:36 |
| *** jbadiapa has quit IRC | 19:36 | |
| *** r-daneel has quit IRC | 19:37 | |
| *** r-daneel_ is now known as r-daneel | 19:37 | |
| corvus | clarkb: looks like the run after the current one will be -f50 | 19:38 |
| clarkb | corvus: yup | 19:38 |
| fungi | zxiiro: i'll see if i have any ideas, though the bandit authors are officially hanging out in ##python-code-quality these days | 19:38 |
| *** ijw has quit IRC | 19:39 | |
| zxiiro | fungi: sure no worries I can ask them too. Thanks for the direction. | 19:39 |
| clarkb | fwiw my concern there would be more worrying about verifying the downloaded image hash prior to uploading to the cloud more so than if someone wanted to upload a file using ftp or file:// | 19:39 |
| clarkb | but I also don't have quite the security paranoia of others | 19:39 |
| zxiiro | clarkb: that's a good idea i should check the checksum at least heh | 19:40 |
| fungi | yeah, i have a feeling what's tripping that is passing a variable straight into the urlretrieve parameters | 19:41 |
| fungi | but without digging into the bandit source hard to say what it's actually expecting instead | 19:41 |
| fungi | i agree it could do with some improved documentation on that check | 19:41 |
| fungi | also, to clarkb's point, not making some cursory checks of the image file does seem risky | 19:43 |
| *** e0ne has joined #openstack-infra | 19:45 | |
| *** spartakos has quit IRC | 19:49 | |
| *** hasharDinner is now known as hashar | 19:55 | |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config master: Move coverage jobs to voting check job https://review.openstack.org/432836 | 19:55 |
| AJaeger | ianw: updated description ^ | 19:56 |
| AJaeger | clarkb, fungi, I would value your input on 432836 overall - whether general direction is fine. | 19:56 |
| clarkb | I've found coverage in check as being valuable for inspecting test results. I do wonder about applying it globally, but projects can always remove it after if they don't care for it | 19:58 |
| *** bharath_t has quit IRC | 19:59 | |
| fungi | clarkb: any idea why system-config-run-base is failing on 600835? | 19:59 |
| corvus | i was just starting to look at that myself | 20:00 |
| fungi | AJaeger: one counterpoint to the argument you make in 432836 is that the linked discussion from 2016 predates the zuul v3 builds dashboard | 20:00 |
| AJaeger | clarkb: I found quite a few broken cover jobs - that never worked in post ;( | 20:00 |
| AJaeger | fungi: yeah, it does. Still, nobody checks if I look at post jobs that fail completely ;( | 20:01 |
| fungi | i don't think finding the latest post pipeline coverage runs is as hard as it was back then, but i do expect that people still aren't looking at them much | 20:01 |
| corvus | fungi, clarkb: http://logs.openstack.org/35/600835/1/check/system-config-run-base/680f906/job-output.txt.gz#_2018-09-07_19_44_47_600584 is that the failure? | 20:01 |
| clarkb | fungi: http://logs.openstack.org/35/600835/1/check/system-config-run-base/680f906/job-output.txt.gz#_2018-09-07_19_44_47_600584 | 20:01 |
| fungi | you both pasted the same url, so i'm guessing that's it ;) | 20:01 |
| clarkb | apt must've failed or maybe it tried using the wrong package manager again? | 20:03 |
| clarkb | (I don't actually know how we fixed that() | 20:03 |
| fungi | https://packages.ubuntu.com/bionic/logrotate | 20:03 |
| fungi | indeed | 20:03 |
| fungi | oh, wait, centos7 | 20:04 |
| fungi | what? | 20:05 |
| fungi | i guess that next line is output from the centos7 node in the job (indirectly) and not the bridge node? | 20:05 |
| clarkb | bionic and centos7 are multiple nodes running the same task, bionic failed but centos7 succeeded I think | 20:05 |
| fungi | ahh, yeah, my eyeballs were skipping past the [bionic] tag at the beginning of that line | 20:06 |
| corvus | i will say that free makes things a bit harder to follow | 20:08 |
| corvus | (but ara will help with that) | 20:08 |
| clarkb | we can compare the base playbook runtime too in production before and after to decide if it is worthwhile | 20:09 |
| corvus | i'm trying to figure out whether/when the "Update apt cache" handler ran on bionic | 20:09 |
| fungi | dhinesh: any chance you can confirm what version of openstackclient you have installed there? that AttributeError exception is raised by openstack._adapter.request making a call to what looks like probably one of its own class methods... | 20:10 |
| corvus | that not having run (yet) for some reason would cause the error | 20:10 |
| clarkb | corvus: http://logs.openstack.org/35/600835/1/check/system-config-run-base/680f906/job-output.txt.gz#_2018-09-07_19_41_21_178872 it ran there as part of job setup at least | 20:11 |
| corvus | clarkb: yeah, but that's before we overwrite the config | 20:12 |
| *** ijw has joined #openstack-infra | 20:12 | |
| clarkb | corvus: grepping cache_updated I don't see an entry for bionic | 20:13 |
| *** agopi is now known as agopi|off | 20:14 | |
| corvus | clarkb: i agree. i see http://logs.openstack.org/35/600835/1/check/system-config-run-base/680f906/job-output.txt.gz#_2018-09-07_19_44_34_608943 for xenial (after writing the apt config and before installing logrotate). but not for bionic. | 20:15 |
| corvus | clarkb: i wonder if there's some weird interaction between handlers and free | 20:15 |
| clarkb | handlers do seem to have weird interactions. Like that reorder of the unbound play so that handlers run at the right time | 20:16 |
| corvus | https://github.com/ansible/ansible/issues/31504 | 20:16 |
| clarkb | https://github.com/ansible/ansible/commit/5f953d1129925579b7bf6ea2cce1ccf583199801 is the commit we need which I guess isn't in 2.6.2 | 20:17 |
| corvus | confirmed, it's not in 2.6.2 | 20:18 |
| corvus | it's in the 2.7.0 rc | 20:19 |
| fungi | well, at least the good news is that change is what's causing the failure, and our testing kept it from breaking our deployments | 20:19 |
| *** agopi|off has quit IRC | 20:19 | |
| fungi | bad news is we're stuck waiting until ansible 2.7 to make use of that feature i guess? | 20:19 |
| corvus | fungi: yeah. this whole test like production thing is kinda working out :) | 20:20 |
| fungi | all things being equal, i'm fine with this result ;) | 20:20 |
| corvus | does anyone know why users has to be in the base-repos play? | 20:21 |
| clarkb | ya its great to see the testing catch real problems | 20:21 |
| corvus | maybe we can move users to the second play and then just have the first play be linear? | 20:21 |
| fungi | seems like a reasonable enough workaround | 20:22 |
| corvus | (users is long and slow and may benefit from free) | 20:22 |
| clarkb | corvus: do we need linear for other roles like exim and unbound (they install packages too) | 20:22 |
| corvus | i don't see anything in base-repos which suggests the need for users | 20:23 |
| clarkb | I guess we can rely on that happening soon enough after base-repos to be fine most of the time | 20:23 |
| corvus | clarkb: it'd only be for handlers. we do have several roles with handlers. most of them are for restarting services after config file changes. | 20:23 |
| corvus | base-server, snmpd, timezone, iptables, and exim all have handlers | 20:23 |
| *** anteaya has joined #openstack-infra | 20:24 | |
| corvus | hrm. maybe we should avoid free :( | 20:24 |
| clarkb | or handlers | 20:24 |
| clarkb | we can restart services and update package caches explicitly in tasks | 20:24 |
| clarkb | (I don't know why that isn't normal though) | 20:24 |
| corvus | it's hard to do the "restart on config file change" without a handler | 20:24 |
| corvus | i think you'd have to do a bunch of register + when stuff | 20:25 |
| corvus | (and do it for every config file; so you'd end up with "when: config_file.changed or defaults_file.changed" i think.) | 20:26 |
| *** diablo_rojo has quit IRC | 20:26 | |
| corvus | i wonder how hard it would be to install 2.7.0rc1 | 20:27 |
| corvus | we use the ansible pip module to install it | 20:28 |
| clarkb | I want to say ansible doesn't publish rc's to pypi | 20:28 |
| corvus | https://pypi.org/project/ansible/2.7.0rc1/ | 20:28 |
| clarkb | but you can download the sdist off github and install that? it was weird in the past when git submodules were used | 20:28 |
| clarkb | oh neat | 20:28 |
| corvus | https://docs.ansible.com/ansible/2.6/modules/pip_module.html | 20:29 |
| corvus | i don't see anything there about enabling pre-releases... | 20:29 |
| *** ijw has quit IRC | 20:29 | |
| corvus | maybe we just need to do 'version==' ? | 20:29 |
| clarkb | ya pip will install an exact version if you give it one | 20:29 |
| corvus | patch incoming | 20:29 |
| clarkb | that overrides the no rc's by default policy | 20:29 |
| openstackgerrit | James E. Blair proposed openstack-infra/system-config master: Install ansible 2.7.0rc1 on bridge https://review.openstack.org/600861 | 20:31 |
| corvus | that should be rather self-testing | 20:32 |
| *** trown is now known as trown|outtypewww | 20:32 | |
| corvus | -f50 started at 20:00 | 20:33 |
| *** r-daneel has quit IRC | 20:34 | |
| corvus | clarkb: mind if i update your patch to depends-on that one? might get some more useful testing | 20:36 |
| clarkb | go for it | 20:36 |
| openstackgerrit | James E. Blair proposed openstack-infra/system-config master: Run base playbook with strategy free https://review.openstack.org/600835 | 20:36 |
| *** jtomasek has quit IRC | 20:36 | |
| *** ijw has joined #openstack-infra | 20:38 | |
| *** ijw has quit IRC | 20:39 | |
| *** ansmith has quit IRC | 20:42 | |
| dhinesh | fungi: don't think openstackclient is installed on the CI machine, did you mean the version of openstackclient on the private openstack cloud side | 20:46 |
| *** r-daneel has joined #openstack-infra | 20:47 | |
| fungi | dhinesh: sorry, openstacksdk. that traceback you pasted showed shade calling the openstack sdk | 20:47 |
| clarkb | corvus: I think we have to wait for the 2100 run to see the -f50 because the run_all.sh script is what updates system-config on disk | 20:50 |
| clarkb | corvus: the playbooks themselves will use the new version on the first run because it updates then runs plabooks but run_all doesn't reexec itself so has to wait for run + 1 | 20:50 |
| corvus | clarkb: ah right | 20:51 |
| *** e0ne has quit IRC | 20:57 | |
| *** jtomasek has joined #openstack-infra | 20:59 | |
| *** spartakos has joined #openstack-infra | 21:00 | |
| *** jento has quit IRC | 21:01 | |
| *** rh-jelabarre has quit IRC | 21:02 | |
| *** spartakos has quit IRC | 21:02 | |
| *** diablo_rojo has joined #openstack-infra | 21:03 | |
| clarkb | yup see it running with -f 50 now | 21:03 |
| corvus | hrm, we don't run the bridge playbook with -v so i don't see a way to independently confirm that 2.7.0 was installed in the job | 21:05 |
| corvus | oh, i think that's in zuul, we may have it in the zuul ara report then | 21:06 |
| corvus | woot! http://logs.openstack.org/61/600861/1/check/system-config-run-base/6437436/ara-report/result/2711b6f4-2fb7-4cd6-9ee1-a43b7bb4ec6d/ | 21:06 |
| clarkb | and the depends on in 600835 should check that it works for fixing that bug | 21:07 |
| corvus | clarkb, fungi: 2.7.0 appears to work with the base playbook and makes clarkb's change happy. of course, that's very likely a non-deterministic error, so it's hard to confirm success there. but we have at least failed to confirm failure. | 21:07 |
| clarkb | watchign the run_all.sh I am somewhat skeptical that -f 50 is doing what we think it does | 21:08 |
| corvus | i'm game to try it out | 21:08 |
| corvus | clarkb: oh, why? | 21:08 |
| clarkb | we have ~130 nodes which should cause it to be running at 50 forks most of the time but looking at ps its well under 10 most of the time looks like | 21:09 |
| dmsimard | corvus: the ansible version displayed at the top right in ARA is probably misleading | 21:09 |
| dmsimard | Top right = webapp ansible version | 21:09 |
| dmsimard | The real ansible version that was used can be found in the parameters or when clicking on the playbook file | 21:10 |
| corvus | dmsimard: yeah, i'm looking at the bottom of http://logs.openstack.org/61/600861/1/check/system-config-run-base/6437436/ara-report/result/2711b6f4-2fb7-4cd6-9ee1-a43b7bb4ec6d/ which confirms that it installed 2.7.0rc1 on fake bridge. | 21:10 |
| dmsimard | Ah, cool | 21:10 |
| dmsimard | I manage to confuse myself sometimes working on ara, it gets a bit inception-ish | 21:11 |
| corvus | dmsimard: that's zuul's ansible bootstrapping the ansible for a node which is standing in for our bastion host which runs ansible on the control plane | 21:11 |
| dmsimard | Oh boy | 21:12 |
| corvus | clarkb: okay now we're at a place which should be able to do a lot in parallel | 21:15 |
| *** holser_ has joined #openstack-infra | 21:17 | |
| corvus | clarkb: i agree the number is very small. i wonder if these tasks are very quick and ansible can't launch them fast enough? like, would we see better utilization if we had a bunch of slow tasks? | 21:17 |
| clarkb | possibly, we do seem to have better utilization in the run puppet else playbook and puppet is a reltively slow task | 21:18 |
| *** bobh has quit IRC | 21:19 | |
| *** ssbarnea|bkp2 has quit IRC | 21:19 | |
| clarkb | I also wonder if each fork does its own startup process reaing all of the cache data | 21:19 |
| *** vaidy has quit IRC | 21:21 | |
| clarkb | testing manually with `sudo ansible logstash-worker* -m shell -a "sleep 20" -f 10` implies the slow start is a one time cost | 21:21 |
| clarkb | I see two waves of succeses ~20 seconds apart | 21:21 |
| clarkb | (that slow start does make ansible less appealing than an ssh for loop for foreground work though :/) | 21:22 |
| corvus | it's interesting to see the 'ok' lines in the log at a relatively consistent about 4/sec | 21:23 |
| corvus | clarkb: what if it's the inventory? ansible isn't supposed to create more forks than needed for the inventory... our inventory is dynamic | 21:26 |
| clarkb | oh interesting | 21:26 |
| clarkb | ya maybe its only reading N out of the inventory at a time? | 21:27 |
| corvus | i'm digging into code for that now | 21:27 |
| *** mriedem is now known as mriedem_mowtown | 21:28 | |
| corvus | wow that really does take a long time to start up | 21:31 |
| corvus | clarkb: if i run your logstash command with 'sleep 0' it looks more like what we see in production | 21:32 |
| corvus | clarkb: so maybe the idea that it only matters for sufficiently slow tasks is the better theory | 21:32 |
| pabelanger | 517832 | 21:33 |
| pabelanger | gah | 21:33 |
| pabelanger | I hit it everytime I move my laptop | 21:33 |
| clarkb | seems like that would be a bug in ansible right? If you've told it you can run more threads then it should do that and not be hindered by task startup time? | 21:34 |
| clarkb | I don't know enough about ansible itnernals though | 21:35 |
| *** holser_ has quit IRC | 21:35 | |
| *** diablo_rojo has quit IRC | 21:39 | |
| *** tosky has quit IRC | 21:40 | |
| corvus | clarkb: i think the forked worker process only lasts as long as a single task | 21:41 |
| clarkb | if that is the case then python startup is likely to hurt us quite a bit | 21:42 |
| corvus | it looks like there's a lot of stuff that happens for each host-task | 21:43 |
| *** jamesmcarthur has quit IRC | 21:45 | |
| *** jamesmcarthur has joined #openstack-infra | 21:46 | |
| *** anteaya has quit IRC | 21:46 | |
| *** anteaya has joined #openstack-infra | 21:47 | |
| *** eernst has joined #openstack-infra | 21:48 | |
| *** graphene has quit IRC | 21:49 | |
| *** myoung|dr is now known as myoung | 21:49 | |
| corvus | i don't expect free to behave differently in this respect. | 21:49 |
| clarkb | free could potentially reuse forks since it runs to completion for each host, but I have no idea if the implemetnation does that | 21:50 |
| corvus | (of course, it should improve parallelism in general, but it still uses the same task queue system) | 21:50 |
| corvus | clarkb: i think it's basically a fully lineraized task queue. the different strategies just decide what order the tasks run in and where the checkpoints are | 21:51 |
| corvus | but ultimately, each host-task goes onto a queue and assigned to a worker (which will then fork, run that task, and exit) | 21:51 |
| *** agopi|off has joined #openstack-infra | 21:52 | |
| *** bobh has joined #openstack-infra | 21:52 | |
| *** diablo_rojo has joined #openstack-infra | 21:52 | |
| clarkb | that is an unfortuantely slow way to do it with python given startup times | 21:54 |
| corvus | clarkb: you don't incur a python startup time with a fork, though, right? | 21:54 |
| *** jamesmcarthur has quit IRC | 21:54 | |
| clarkb | oh right for some reason I had in my head fork + exec | 21:55 |
| clarkb | if not doing the exec then ya should be fine | 21:55 |
| corvus | yeah, it's multiprocessing.Process under the hood | 21:56 |
| *** spartakos has joined #openstack-infra | 21:56 | |
| *** jtomasek has quit IRC | 21:58 | |
| *** kukacz_ has quit IRC | 22:02 | |
| *** kukacz_ has joined #openstack-infra | 22:04 | |
| corvus | i'm looking at the openstacksdk docs for cache, and i'm not clear about expiration: https://docs.openstack.org/openstacksdk/latest/user/config/configuration.html | 22:04 |
| *** yamamoto has quit IRC | 22:05 | |
| corvus | what does the expiration dictionary mean? | 22:05 |
| clarkb | corvus: the keys are the different cached objects and the values are ttls in seconds | 22:05 |
| clarkb | -1 is a special ttl that means never expire looks like | 22:05 |
| clarkb | (I'm not sure what the valid list of keys is there) | 22:06 |
| corvus | looks like that's the extent of documentation about expiration | 22:06 |
| dhinesh | :fungi: after some digging, got access to my old CI setup, will continue to work on that. Thanks for help. | 22:08 |
| corvus | i'm editing clouds.yaml on bridge to see if tweaking the cache params makes startup faster | 22:08 |
| corvus | so far it's not | 22:09 |
| *** bobh has quit IRC | 22:10 | |
| *** diablo_rojo has quit IRC | 22:12 | |
| *** r-daneel has quit IRC | 22:12 | |
| *** spartakos has quit IRC | 22:14 | |
| *** rfolco has quit IRC | 22:15 | |
| *** spartakos has joined #openstack-infra | 22:15 | |
| *** gema has quit IRC | 22:18 | |
| *** gema has joined #openstack-infra | 22:20 | |
| *** gema has quit IRC | 22:20 | |
| *** gema has joined #openstack-infra | 22:20 | |
| *** spartakos has quit IRC | 22:23 | |
| *** hashar has quit IRC | 22:25 | |
| *** spartakos has joined #openstack-infra | 22:27 | |
| *** dave-mccowan has quit IRC | 22:38 | |
| *** jamesmcarthur has joined #openstack-infra | 22:44 | |
| *** jamesmcarthur has quit IRC | 22:48 | |
| corvus | clarkb: our 2 vcpus may have an impact too. | 22:51 |
| corvus | clarkb: it's the constructed inventory plugin that's taking so long, not openstack. | 22:51 |
| clarkb | that is unexpected | 22:52 |
| *** rlandy has quit IRC | 22:55 | |
| *** caphrim007_ has joined #openstack-infra | 22:58 | |
| *** caphrim00_ has joined #openstack-infra | 22:59 | |
| *** eernst has quit IRC | 23:01 | |
| *** caphrim007 has quit IRC | 23:01 | |
| fungi | weird | 23:01 |
| corvus | i'm trying to profile it now | 23:01 |
| *** caphrim007_ has quit IRC | 23:02 | |
| *** caphrim00_ has quit IRC | 23:03 | |
| clarkb | corvus: reading docs on constructed inventory it uses the fact cache to builds its variable set. That probably explains why we read the entire fact cache for every ansible run | 23:05 |
| clarkb | regardless of what host we run against | 23:05 |
| corvus | clarkb: yep | 23:06 |
| clarkb | I wonder if we shouldn't do a more traditional inventory | 23:06 |
| clarkb | and use globbing and other pattern matching | 23:06 |
| corvus | clarkb: i'm not convinced it's the cache that's slow | 23:07 |
| corvus | the slowest call is self._add_host_to_composed_groups(self.get_option('groups') | 23:08 |
| corvus | this is the profile: http://paste.openstack.org/show/729716/ | 23:08 |
| corvus | let me git github links | 23:09 |
| corvus | https://github.com/ansible/ansible/blob/devel/lib/ansible/plugins/inventory/constructed.py#L121 | 23:09 |
| corvus | which is https://github.com/ansible/ansible/blob/devel/lib/ansible/plugins/inventory/__init__.py#L293 | 23:10 |
| corvus | i'm wondering if it's the fact that we're compiling that template for every group for every host | 23:10 |
| corvus | how many hosts do we have? 100? we have 55 groups. that's like 5500 template compilations. | 23:11 |
| clarkb | ~130 hosts | 23:12 |
| corvus | 7150 | 23:12 |
| *** sthussey has quit IRC | 23:13 | |
| clarkb | it is reading the yaml entry and evaluating it in a dynamic jinja2 template huh | 23:13 |
| clarkb | and then it does that hosts * groups times | 23:13 |
| corvus | yeah, if i comment out this line it's super fast: https://github.com/ansible/ansible/blob/devel/lib/ansible/plugins/inventory/__init__.py#L301 | 23:14 |
| corvus | and it's still loading all the facts from disk | 23:14 |
| corvus | so it's the jinja templating that's killing it | 23:14 |
| clarkb | I wonder if we can memoize/cache jinja2 intermediate results | 23:15 |
| clarkb | then only reevaluate with new variables | 23:15 |
| *** ansmith has joined #openstack-infra | 23:16 | |
| *** ijw has joined #openstack-infra | 23:17 | |
| *** spartakos has quit IRC | 23:17 | |
| *** mriedem_mowtown is now known as mriedem | 23:19 | |
| corvus | clarkb: i think i'm going to stop digging into this now. i think we've got good data we can take into next week and discuss inventory/grouping ideas. | 23:21 |
| clarkb | corvus: I think we could get away with the yaml inventory fwiw, we'd jsut be left using patterns | 23:21 |
| *** ijw has quit IRC | 23:21 | |
| corvus | clarkb: the yaml inventory does not call that method, so that's promising. | 23:22 |
| openstackgerrit | Merged openstack-infra/puppet-ptgbot master: Make the FC SIG freed slots actually reservable https://review.openstack.org/600665 | 23:23 |
| corvus | clarkb: that looks mostly like good ole compiled regexes | 23:23 |
| *** mriedem has quit IRC | 23:28 | |
| *** bobh has joined #openstack-infra | 23:45 | |
| *** bobh has quit IRC | 23:49 | |
| *** anteaya has quit IRC | 23:59 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!