| *** jamesmcarthur has joined #openstack-infra | 00:08 | |
| *** jamesmcarthur has quit IRC | 00:33 | |
| *** zhangfei has joined #openstack-infra | 00:40 | |
| *** longkb has joined #openstack-infra | 00:43 | |
| *** felipemonteiro has joined #openstack-infra | 01:03 | |
| *** jamesmcarthur has joined #openstack-infra | 01:06 | |
| *** felipemonteiro has quit IRC | 01:11 | |
| *** felipemonteiro has joined #openstack-infra | 01:13 | |
| *** jamesmcarthur has quit IRC | 01:36 | |
| *** jamesmcarthur has joined #openstack-infra | 01:42 | |
| *** jamesmcarthur has quit IRC | 01:48 | |
| *** hongbin has joined #openstack-infra | 02:04 | |
| *** zhangfei has quit IRC | 02:14 | |
| *** zhangfei has joined #openstack-infra | 02:14 | |
| *** dklyle has joined #openstack-infra | 02:42 | |
| *** david-lyle has quit IRC | 02:43 | |
| *** dklyle has quit IRC | 02:43 | |
| *** dklyle has joined #openstack-infra | 02:43 | |
| *** felipemonteiro has quit IRC | 02:52 | |
| *** egonzalez has quit IRC | 02:55 | |
| *** egonzalez has joined #openstack-infra | 02:56 | |
| *** neilsun has joined #openstack-infra | 02:58 | |
| *** annp has joined #openstack-infra | 03:12 | |
| *** niska has quit IRC | 03:18 | |
| *** Bhujay has joined #openstack-infra | 03:18 | |
| openstackgerrit | Ian Wienand proposed openstack-infra/zuul-sphinx master: Add config option to find other roles https://review.openstack.org/593477 | 03:22 |
|---|---|---|
| *** felipemonteiro has joined #openstack-infra | 03:23 | |
| *** gcb_ has joined #openstack-infra | 03:23 | |
| *** Bhujay has quit IRC | 03:26 | |
| *** zhangfei has quit IRC | 03:30 | |
| *** niska has joined #openstack-infra | 03:30 | |
| *** felipemonteiro has quit IRC | 03:38 | |
| *** ramishra has joined #openstack-infra | 03:39 | |
| openstackgerrit | Ian Wienand proposed openstack-infra/system-config master: Generate roles documentation https://review.openstack.org/593478 | 03:43 |
| openstackgerrit | Ian Wienand proposed openstack-infra/system-config master: Generate roles documentation https://review.openstack.org/593478 | 03:48 |
| *** longkb has quit IRC | 03:55 | |
| *** longkb has joined #openstack-infra | 03:56 | |
| *** felipemonteiro has joined #openstack-infra | 03:56 | |
| *** felipemonteiro has quit IRC | 03:57 | |
| *** felipemonteiro has joined #openstack-infra | 04:07 | |
| *** felipemonteiro has quit IRC | 04:10 | |
| *** Bhujay has joined #openstack-infra | 04:16 | |
| AJaeger | ianw: thanks for approving the afsmon change, could you review the followup in project-config as well, please? https://review.openstack.org/592671 | 04:17 |
| AJaeger | infra-manual reviewers, please review https://review.openstack.org/593138 and https://review.openstack.org/593344 to update docs for in-repo move | 04:18 |
| openstackgerrit | Ian Wienand proposed openstack-infra/system-config master: Generate roles documentation https://review.openstack.org/593478 | 04:25 |
| openstackgerrit | Ian Wienand proposed openstack-infra/system-config master: Move to doc/requirements.txt https://review.openstack.org/593483 | 04:25 |
| openstackgerrit | Ian Wienand proposed openstack-infra/system-config master: [DNM] testing zuul-sphinx updated installation https://review.openstack.org/593484 | 04:25 |
| *** felipemonteiro has joined #openstack-infra | 04:28 | |
| *** jaosorior has joined #openstack-infra | 04:31 | |
| openstackgerrit | Ian Wienand proposed openstack-infra/zuul-sphinx master: Open role readme files in utf-8 mode https://review.openstack.org/579474 | 04:33 |
| openstackgerrit | Ian Wienand proposed openstack-infra/zuul-sphinx master: Add config option to find other roles https://review.openstack.org/593477 | 04:33 |
| openstackgerrit | Ian Wienand proposed openstack-infra/system-config master: [DNM] testing zuul-sphinx updated installation https://review.openstack.org/593484 | 04:34 |
| ianw | AJaeger: ^^^ it's a bit annoying that locally "tox -e docs" runs under python3, but then in the gate it goes back to running under python2 as it's not under tox | 04:38 |
| openstackgerrit | Ian Wienand proposed openstack-infra/zuul-sphinx master: Add config option to find other roles https://review.openstack.org/593477 | 04:42 |
| openstackgerrit | Ian Wienand proposed openstack-infra/system-config master: Update to latest docs PTI https://review.openstack.org/593483 | 04:46 |
| openstackgerrit | Ian Wienand proposed openstack-infra/system-config master: Generate roles documentation https://review.openstack.org/593478 | 04:46 |
| openstackgerrit | Ian Wienand proposed openstack-infra/system-config master: [DNM] testing zuul-sphinx updated installation https://review.openstack.org/593484 | 04:46 |
| *** felipemonteiro has quit IRC | 04:51 | |
| *** annp has quit IRC | 05:00 | |
| AJaeger | ianw: we could do the same change for the infra docs jobs that dhellmann did for PTI jobs. | 05:01 |
| AJaeger | ianw: so, updating publish-openstack-sphinx-docs-infra with tox-docs instead of publish-sphinx-docs. | 05:03 |
| AJaeger | ianw: want a change? | 05:04 |
| *** hongbin has quit IRC | 05:06 | |
| openstackgerrit | Ian Wienand proposed openstack-infra/system-config master: Add kerberos-client role https://review.openstack.org/593486 | 05:10 |
| openstackgerrit | Ian Wienand proposed openstack-infra/system-config master: Add openafs-client role https://review.openstack.org/593487 | 05:10 |
| *** gcb_ has quit IRC | 05:10 | |
| *** longkb has quit IRC | 05:12 | |
| *** longkb has joined #openstack-infra | 05:12 | |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config master: Create publish-openstack-tox-docs-infra https://review.openstack.org/593488 | 05:14 |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/openstack-zuul-jobs master: Switch infra docs job to new PTI https://review.openstack.org/593489 | 05:14 |
| AJaeger | ianw: ^ | 05:16 |
| openstackgerrit | Ian Wienand proposed openstack-infra/system-config master: Update to latest docs PTI https://review.openstack.org/593483 | 05:18 |
| openstackgerrit | Ian Wienand proposed openstack-infra/system-config master: Generate roles documentation https://review.openstack.org/593478 | 05:18 |
| openstackgerrit | Ian Wienand proposed openstack-infra/system-config master: Add kerberos-client role https://review.openstack.org/593486 | 05:18 |
| openstackgerrit | Ian Wienand proposed openstack-infra/system-config master: Add openafs-client role https://review.openstack.org/593487 | 05:18 |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config master: Create publish-openstack-tox-docs-infra https://review.openstack.org/593488 | 05:20 |
| openstackgerrit | Ian Wienand proposed openstack-infra/openstack-zuul-jobs master: Test OpenAFS client install https://review.openstack.org/589335 | 05:21 |
| *** e0ne has joined #openstack-infra | 05:26 | |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config master: Create publish-tox-docs-infra https://review.openstack.org/593488 | 05:27 |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/openstack-zuul-jobs master: Switch infra docs job to new PTI https://review.openstack.org/593489 | 05:27 |
| AJaeger | ianw: updated - and this will need some cleanup afterwards to use new templates ^ | 05:27 |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config master: Use new infra tox-docs templates https://review.openstack.org/593490 | 05:32 |
| AJaeger | ianw: and one more ^ - and then we can clean up. another time if we get buyin from infra-root on these ^ | 05:33 |
| AJaeger | dhellmann: please review as well ^ | 05:35 |
| openstackgerrit | Sampath Priyankara (samP) proposed openstack-infra/irc-meetings master: Change Masakari Team Meeting Time https://review.openstack.org/593493 | 05:39 |
| *** oanson has joined #openstack-infra | 05:43 | |
| *** janki has joined #openstack-infra | 05:44 | |
| *** jamesmcarthur has joined #openstack-infra | 05:48 | |
| *** jamesmcarthur has quit IRC | 05:49 | |
| johnsom | FYI, either packethost or nova is having a problem bringing up a devstack for the gates: http://logs.openstack.org/55/593455/1/gate/neutron-lbaasv2-dsvm-api/e803592/logs/screen-n-cpu.txt.gz#_Aug_20_01_53_36_431400 | 05:50 |
| *** psachin has joined #openstack-infra | 05:58 | |
| *** e0ne has quit IRC | 06:00 | |
| *** apetrich has joined #openstack-infra | 06:01 | |
| *** aluria has joined #openstack-infra | 06:18 | |
| *** Dobroslaw has joined #openstack-infra | 06:20 | |
| *** jtomasek has joined #openstack-infra | 06:23 | |
| openstackgerrit | Colleen Murphy proposed openstack-infra/system-config master: Turn on the future parser for etherpad.o.o https://review.openstack.org/590029 | 06:24 |
| openstackgerrit | Colleen Murphy proposed openstack-infra/system-config master: Turn on the future parser for ethercalc.o.o https://review.openstack.org/590030 | 06:24 |
| openstackgerrit | Colleen Murphy proposed openstack-infra/system-config master: Turn on the future parser for eavesdrop.o.o https://review.openstack.org/590048 | 06:24 |
| openstackgerrit | Colleen Murphy proposed openstack-infra/system-config master: Turn on the future parser for paste.o.o https://review.openstack.org/590492 | 06:25 |
| openstackgerrit | Colleen Murphy proposed openstack-infra/system-config master: Turn on the future parser for planet.o.o https://review.openstack.org/590493 | 06:25 |
| openstackgerrit | Colleen Murphy proposed openstack-infra/system-config master: Turn on the future parser for logstash-worker01 https://review.openstack.org/590499 | 06:25 |
| openstackgerrit | Colleen Murphy proposed openstack-infra/system-config master: Turn on the future parser for all logstash workers https://review.openstack.org/590500 | 06:25 |
| openstackgerrit | Colleen Murphy proposed openstack-infra/system-config master: Turn on the future parser for firehose.o.o https://review.openstack.org/591500 | 06:25 |
| openstackgerrit | Colleen Murphy proposed openstack-infra/system-config master: Turn on the future parser for review and review01 https://review.openstack.org/591516 | 06:25 |
| openstackgerrit | Colleen Murphy proposed openstack-infra/system-config master: Turn on the future parser for grafana.o.o https://review.openstack.org/592640 | 06:25 |
| openstackgerrit | Colleen Murphy proposed openstack-infra/system-config master: Turn the future parser on for health.o.o https://review.openstack.org/592647 | 06:25 |
| *** dpawlik has joined #openstack-infra | 06:33 | |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/openstack-zuul-jobs master: Switch infra docs job to new PTI https://review.openstack.org/593489 | 06:37 |
| *** dpawlik has quit IRC | 06:38 | |
| *** dpawlik has joined #openstack-infra | 06:40 | |
| *** zhangfei has joined #openstack-infra | 06:42 | |
| *** e0ne has joined #openstack-infra | 06:42 | |
| *** e0ne has quit IRC | 06:43 | |
| openstackgerrit | Nguyen Hai proposed openstack-infra/project-config master: remove job settings for glance repositories https://review.openstack.org/593525 | 06:43 |
| *** pcaruana has joined #openstack-infra | 06:46 | |
| *** ginopc has joined #openstack-infra | 06:47 | |
| *** ykarel has joined #openstack-infra | 06:57 | |
| dmellado | AJaeger: cmurphy any chance that zuul's not picking up patches to be put on gate queue as of now? | 07:03 |
| dmellado | There are two patches that I'd really like to have merged this morning before cutting the release | 07:04 |
| dmellado | and after w+1 nothing happened xD | 07:04 |
| dmellado | https://review.openstack.org/#/c/592563/ | 07:04 |
| AJaeger | you need to merge https://review.openstack.org/#/c/584421 first, those are stacked on top of each other - aren't they? | 07:05 |
| *** rcernin has quit IRC | 07:05 | |
| AJaeger | dmellado: that's one change - what's the other? | 07:05 |
| dmellado | hold on, 2nd on failed on one of our gates | 07:06 |
| dmellado | damn | 07:06 |
| dmellado | AJaeger: oh, probably, I still didn't have my coffee, lemme check | 07:06 |
| * AJaeger sends coffee to dmellado ;) | 07:06 | |
| dmellado | that was the one, sorry on the noise! | 07:07 |
| dmellado | I'll recheck that one and W+1 it after it passes CI, thanks AJaeger | 07:07 |
| AJaeger | you're welcome, dmellado. | 07:08 |
| AJaeger | dmellado: My personal check list for these is: Is there an unmerged depends-On? Is it stacked on another one? | 07:09 |
| dmellado | sounds like a reasonable one, as long as you get the coffee before xD | 07:12 |
| AJaeger | ;) | 07:13 |
| *** f1 has joined #openstack-infra | 07:16 | |
| *** jaosorior has quit IRC | 07:17 | |
| *** annp has joined #openstack-infra | 07:24 | |
| *** jpena|off is now known as jpena | 07:27 | |
| *** tosky has joined #openstack-infra | 07:30 | |
| *** janki has quit IRC | 07:31 | |
| *** shardy has joined #openstack-infra | 07:33 | |
| *** jaosorior has joined #openstack-infra | 07:37 | |
| *** rpittau has joined #openstack-infra | 07:41 | |
| *** yamahata has quit IRC | 07:48 | |
| *** adriant has quit IRC | 07:59 | |
| *** adriant has joined #openstack-infra | 07:59 | |
| *** snapiri has joined #openstack-infra | 08:02 | |
| *** dbecker has joined #openstack-infra | 08:16 | |
| *** dtantsur|afk is now known as dtantsur | 08:25 | |
| *** electrofelix has joined #openstack-infra | 08:28 | |
| *** hashar has joined #openstack-infra | 08:30 | |
| *** dingyichen has quit IRC | 08:47 | |
| *** e0ne has joined #openstack-infra | 08:58 | |
| *** jpich has joined #openstack-infra | 09:00 | |
| *** fresta has joined #openstack-infra | 09:04 | |
| *** fresta_ has joined #openstack-infra | 09:05 | |
| *** fresta- has quit IRC | 09:06 | |
| *** dpawlik has quit IRC | 09:07 | |
| *** stakeda has joined #openstack-infra | 09:07 | |
| *** dpawlik has joined #openstack-infra | 09:07 | |
| *** fresta has quit IRC | 09:09 | |
| openstackgerrit | Merged openstack-infra/irc-meetings master: Change Masakari Team Meeting Time https://review.openstack.org/593493 | 09:11 |
| *** Douhet has quit IRC | 09:19 | |
| *** Douhet has joined #openstack-infra | 09:20 | |
| egonzalez | hi, ask.o.o is down | 09:28 |
| *** dpawlik has quit IRC | 09:29 | |
| *** sshnaidm is now known as sshnaidm|afk | 09:29 | |
| *** dpawlik has joined #openstack-infra | 09:31 | |
| *** dpawlik_ has joined #openstack-infra | 09:32 | |
| *** dpawlik has quit IRC | 09:32 | |
| frickler | egonzalez: yep, I just noticed and restarted it | 09:33 |
| frickler | infra-root: started apache2 on ask.o.o once more, seems to die regularly now, maybe someone has time to investigate a bit further | 09:34 |
| *** ginopc has quit IRC | 09:35 | |
| openstackgerrit | Merged openstack-infra/project-config master: [Neutron] Fix graphs with number of jobs running recently https://review.openstack.org/593070 | 09:51 |
| *** ykarel is now known as ykarel|lunch | 09:54 | |
| *** psachin has quit IRC | 09:58 | |
| AJaeger | config-core, these changes are needed by release team - care to review, please? https://review.openstack.org/591837 https://review.openstack.org/591836 https://review.openstack.org/592667 | 10:06 |
| *** longkb has quit IRC | 10:06 | |
| openstackgerrit | Merged openstack-infra/project-config master: Add sphinx-docs job to Monasca projects https://review.openstack.org/566722 | 10:12 |
| *** dtantsur is now known as dtantsur|brb | 10:27 | |
| *** shardy has quit IRC | 10:31 | |
| *** shardy has joined #openstack-infra | 10:31 | |
| *** ykarel|lunch is now known as ykarel | 10:35 | |
| *** vdrok has quit IRC | 10:39 | |
| *** vdrok has joined #openstack-infra | 10:40 | |
| *** gbutnaru has quit IRC | 10:46 | |
| *** stakeda has quit IRC | 10:48 | |
| *** ginopc has joined #openstack-infra | 10:50 | |
| *** e0ne has quit IRC | 10:51 | |
| *** holser_ has joined #openstack-infra | 11:01 | |
| *** jpena is now known as jpena|lunch | 11:03 | |
| *** ansmith has quit IRC | 11:13 | |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/zuul master: WIP web: rewrite interface in react https://review.openstack.org/591604 | 11:13 |
| *** Bhujay has quit IRC | 11:17 | |
| *** holser_ has quit IRC | 11:20 | |
| *** holser_ has joined #openstack-infra | 11:20 | |
| *** fresta has joined #openstack-infra | 11:28 | |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/zuul master: WIP web: rewrite interface in react https://review.openstack.org/591604 | 11:28 |
| *** zhangfei has quit IRC | 11:28 | |
| *** fresta- has joined #openstack-infra | 11:28 | |
| *** larainema has quit IRC | 11:29 | |
| *** fresta_ has quit IRC | 11:31 | |
| *** fresta has quit IRC | 11:32 | |
| AJaeger | config-core, could you review https://review.openstack.org/#/c/592289/ , please? I like to retire a repo | 11:32 |
| *** dave-mccowan has joined #openstack-infra | 11:34 | |
| *** Bhujay has joined #openstack-infra | 11:40 | |
| *** e0ne has joined #openstack-infra | 11:47 | |
| *** rh-jelabarre has joined #openstack-infra | 11:49 | |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/zuul master: web: fix multiple builds query parameters for a single column https://review.openstack.org/593605 | 12:00 |
| openstackgerrit | Colleen Murphy proposed openstack-infra/project-config master: Add publish-to-pypi job to keystone-tempest-plugin https://review.openstack.org/593606 | 12:00 |
| *** jcoufal has joined #openstack-infra | 12:00 | |
| *** jroll has quit IRC | 12:00 | |
| *** jroll has joined #openstack-infra | 12:01 | |
| *** weshay|ruck is now known as weshay | 12:03 | |
| *** panda is now known as panda|lunch | 12:03 | |
| *** sshnaidm|afk is now known as sshnaidm | 12:04 | |
| *** jamesdenton has joined #openstack-infra | 12:06 | |
| *** jamesden_ has joined #openstack-infra | 12:07 | |
| *** pcarver has joined #openstack-infra | 12:08 | |
| *** jamesdenton has quit IRC | 12:08 | |
| *** jamesden_ is now known as jamesdenton | 12:08 | |
| pcarver | Has anyone written any tool for visualizing Zuul configuration? | 12:09 |
| pcarver | e.g. something that parses the YAML config and outputs graphviz or similar that can be used to view it as a flow chart / dependency graph | 12:10 |
| *** trown|outtypewww is now known as trown | 12:13 | |
| *** Bhujay has quit IRC | 12:16 | |
| *** Bhujay has joined #openstack-infra | 12:16 | |
| *** jpena|lunch is now known as jpena | 12:19 | |
| *** needssleep is now known as TheJulia | 12:23 | |
| *** jcoufal_ has joined #openstack-infra | 12:31 | |
| *** jcoufal has quit IRC | 12:31 | |
| *** ccamacho has joined #openstack-infra | 12:33 | |
| *** jcoufal has joined #openstack-infra | 12:33 | |
| *** rlandy has joined #openstack-infra | 12:36 | |
| *** jcoufal_ has quit IRC | 12:36 | |
| fungi | frickler: did the apache logs go silent around 06:25-06:30 utc and did it leave a stale pidfile behind? those are the symptoms i noticed a couple days ago when i restarted it | 12:36 |
| *** jbadiapa has joined #openstack-infra | 12:37 | |
| fungi | i have a feeling the coming xenial upgrade may fix inexplicable apache crashes on ask.o.o (we know it fixes some other apache segfaults at least), but the lack of any real error message here makes it hard to know | 12:38 |
| fungi | pcarver: none to my knowledge. fair warning, if you're talking about the openstack community's zuul configuration, that would be a quite massive graph | 12:38 |
| *** tpsilva has joined #openstack-infra | 12:39 | |
| fungi | i think the new configuration api is making a start toward being able to browse configuration and follow links for inheritance and variation in other playbooks/roles (even across git repositories) | 12:39 |
| fungi | across branches too, presumably | 12:40 |
| *** kgiusti has joined #openstack-infra | 12:42 | |
| *** sthussey has joined #openstack-infra | 12:44 | |
| *** dtantsur|brb is now known as dtantsur | 12:45 | |
| *** ansmith has joined #openstack-infra | 12:53 | |
| *** jamesmcarthur has joined #openstack-infra | 12:56 | |
| *** eharney has quit IRC | 13:00 | |
| mordred | pcarver, fungi: tristanC has done some work to show visualization. we're still discussing what shape it'll wind up being (to fungi's point, showing a job graph of ALL of the jobs in openstack would be too big) | 13:06 |
| tristanC | pcarver: fungi: mordred: well the job page already lists ALL the jobs, the graph I proposed should be shorter as it display a dendogram tree, it's: https://review.openstack.org/537869 | 13:13 |
| tristanC | though, it's based of the REST api, maybe there is value for an offline mode where zuul load and dump the graph of it's configuration locally | 13:15 |
| *** jamesmcarthur has quit IRC | 13:15 | |
| AJaeger | cmurphy: https://docs.openstack.org/infra/manual/creators.html#give-openstack-permission-to-publish-releases is also needed | 13:16 |
| cmurphy | oh right | 13:16 |
| *** janki has joined #openstack-infra | 13:16 | |
| cmurphy | AJaeger: done | 13:17 |
| frickler | fungi: yes to both, stopped yesterday morning, i.e. on the next log rotation after your restart it seems | 13:18 |
| AJaeger | cmurphy: LGTM | 13:19 |
| AJaeger | config-core, could you review https://review.openstack.org/#/c/592289/ , please? I like to move forward with retiring that repo | 13:21 |
| *** mriedem has joined #openstack-infra | 13:23 | |
| *** ginux has joined #openstack-infra | 13:26 | |
| *** rkukura has joined #openstack-infra | 13:26 | |
| *** ginux is now known as Guest95044 | 13:26 | |
| *** zul has joined #openstack-infra | 13:27 | |
| *** ginopc has quit IRC | 13:27 | |
| *** Guest95044 is now known as ginopc | 13:27 | |
| *** bobh has joined #openstack-infra | 13:34 | |
| openstackgerrit | Merged openstack-infra/project-config master: Create publish-tox-docs-infra https://review.openstack.org/593488 | 13:35 |
| mordred | fungi, corvus, clarkb: http://paste.openstack.org/show/728423/ is the list of hosts with connectivity issues from bridge | 13:38 |
| *** eharney has joined #openstack-infra | 13:38 | |
| fungi | mordred: thanks for summarizing. most of those are expected/explained. i think the one i'm not as sure about is git.o.o (perhaps puppet has been broken on it for a while?) | 13:40 |
| mordred | fungi: yah - it seems that way - it doesn't have the ssh keys | 13:41 |
| mordred | fungi: I was fixing that one by hand- but also need to update the sshd config there - is 'systemctl restart ssh' safe? | 13:42 |
| *** rkukura has quit IRC | 13:42 | |
| *** jamesmcarthur has joined #openstack-infra | 13:45 | |
| mordred | fungi: ok - I fixed config and restarted on git.o.o | 13:51 |
| *** efried_pto is now known as efried | 13:56 | |
| pabelanger | ianw: mind a review on https://review.openstack.org/591788/ cleaning up some legacy opensuse nodesets | 13:59 |
| dhellmann | mordred : we've run into a required-projects/tox-siblings question with the test jobs in one of the ironic projects | 14:00 |
| *** jlv-sick is now known as jlvillal | 14:00 | |
| dhellmann | there is apparently a tox-python35-jobs-neutron template with variants of the unit test jobs that bring in neutron from master | 14:01 |
| dhellmann | do we need that any more? is that the only way to do that? if so, someone needs to add a version for 36 | 14:01 |
| *** Bhujay has quit IRC | 14:02 | |
| fungi | mordred: yeah, generally restarting sshd won't terminate established connections (but config changes also don't take effect for established connections so you need to make a second connection to test, ideally before disconnecting your first session, you know, just in case) | 14:02 |
| mordred | fungi: ++ | 14:02 |
| mordred | dhellmann: looking | 14:03 |
| dhellmann | mordred : more discussion in #openstack-ironic | 14:03 |
| *** pbourke has quit IRC | 14:04 | |
| *** pbourke has joined #openstack-infra | 14:06 | |
| *** chandankumar has quit IRC | 14:12 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/system-config master: Run base and bridge playbooks in run_all.sh https://review.openstack.org/593420 | 14:13 |
| openstackgerrit | Monty Taylor proposed openstack-infra/system-config master: Start running puppet cron on bridge.openstack.org https://review.openstack.org/591152 | 14:13 |
| openstackgerrit | Monty Taylor proposed openstack-infra/system-config master: Add non-working hosts to disabled list https://review.openstack.org/593636 | 14:13 |
| *** jamesmcarthur has quit IRC | 14:23 | |
| *** jamesmcarthur has joined #openstack-infra | 14:24 | |
| *** panda|lunch is now known as panda | 14:25 | |
| *** e0ne has quit IRC | 14:28 | |
| openstackgerrit | Dmitry Tantsur proposed openstack-infra/openstack-zuul-jobs master: Add openstack-python36-jobs-neutron template https://review.openstack.org/593643 | 14:32 |
| dtantsur | dhellmann: ^^^ | 14:32 |
| dtantsur | I think we don't have to change project-config, since the actual job is the same | 14:32 |
| dhellmann | I don't think you can add required-projects from within a project template | 14:33 |
| dhellmann | dtantsur : ^ | 14:33 |
| dhellmann | I might be wrong? | 14:33 |
| dtantsur | dhellmann: it's a copy-paste from python35 variant | 14:33 |
| dhellmann | oh, we have that for the 35 | 14:33 |
| dhellmann | yeah | 14:33 |
| dhellmann | ok, cool, that's easier than i thought :-) | 14:33 |
| dtantsur | :) | 14:33 |
| dtantsur | can I use depends-on with such change? | 14:34 |
| dhellmann | thanks, dtantsur | 14:34 |
| dtantsur | np | 14:34 |
| *** e0ne has joined #openstack-infra | 14:35 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/system-config master: Manage sshd config with ansible https://review.openstack.org/593647 | 14:35 |
| openstackgerrit | Monty Taylor proposed openstack-infra/system-config master: Remove ssh v1 options from sshd_config https://review.openstack.org/593648 | 14:35 |
| *** r-daneel has joined #openstack-infra | 14:37 | |
| *** aojea has joined #openstack-infra | 14:38 | |
| *** shachar has joined #openstack-infra | 14:40 | |
| *** snapiri has quit IRC | 14:40 | |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/nodepool master: Implement an Amazon EC2 driver https://review.openstack.org/535558 | 14:42 |
| *** shachar is now known as snapiri | 14:42 | |
| AJaeger | fungi, could youreview https://review.openstack.org/593138 and https://review.openstack.org/593344 to update docs for in-repo move, please? | 14:45 |
| *** chandankumar has joined #openstack-infra | 14:47 | |
| *** jiapei has joined #openstack-infra | 14:48 | |
| *** eharney has quit IRC | 14:55 | |
| openstackgerrit | Jeremy Stanley proposed openstack-infra/infra-manual master: Clarify project creation now that jobs are in-repo https://review.openstack.org/593138 | 14:56 |
| openstackgerrit | Doug Hellmann proposed openstack-infra/project-config master: remove job settings for requirements repositories https://review.openstack.org/593660 | 14:57 |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/infra-manual master: Clarify project creation now that jobs are in-repo https://review.openstack.org/593138 | 14:58 |
| *** rpioso|afk is now known as rpioso | 14:58 | |
| AJaeger | fungi: just found a double for in my text and fixed - want to +2A or shall I? ^ Thanks for your update! | 14:58 |
| AJaeger | thanks, fungi | 14:59 |
| fungi | if anybody has a moment to review a minor git-review change, https://review.openstack.org/584607 would be good to get inyw! | 14:59 |
| fungi | er, you're welcome | 14:59 |
| fungi | on 584607 i've been running from git-review master branch tip with that change cherry-picked and haven't seen any issues | 15:00 |
| mordred | fungi: done. I've been thinking recently - 'git review -d https://review.openstack.org/584607' would be nice to support - I accidentaly do it frequently myself | 15:01 |
| fungi | mordred: there's a change out there for that too | 15:02 |
| mordred | fungi: oh yeah? | 15:02 |
| * mordred goes to look/review | 15:02 | |
| fungi | pretty sure dhellmann wrote it (unless i'm imagining) | 15:02 |
| dhellmann | yes | 15:02 |
| dhellmann | I think it might not be complete? | 15:02 |
| dhellmann | in terms of the variations of urls | 15:02 |
| dhellmann | oh, someone fixed it up so ianw was happy with it | 15:03 |
| dhellmann | mordred : https://review.openstack.org/#/c/424823/ | 15:03 |
| mordred | yah - it looks great | 15:03 |
| mordred | ZOMG that patch is magic and changes my life | 15:05 |
| mordred | fungi: I have verified that it works as expected locally (it does yay) and have added a second +2 - do you want me to hold off on the +A part? | 15:06 |
| fungi | nah, it's fine to approve | 15:06 |
| fungi | i'm heading out to grab an early lunch, and then can help with more ansipuppetification once i'm back and west-coasters are caffeinated | 15:07 |
| openstackgerrit | Merged openstack-infra/infra-manual master: Clarify project creation now that jobs are in-repo https://review.openstack.org/593138 | 15:09 |
| *** eharney has joined #openstack-infra | 15:09 | |
| *** pcaruana has quit IRC | 15:11 | |
| corvus | you'll be waiting a while if you're waiting on me to caffeinate... | 15:11 |
| mordred | corvus: I've been waiting for you to caffeinate for years ... | 15:12 |
| corvus | there was that one time... | 15:13 |
| mordred | fungi, corvus: ifyou get bored, https://review.openstack.org/#/c/555271/ is a manpage update documented the usepushurl option that I use from git-review that nobody else seems to know about :) | 15:13 |
| corvus | mordred: but if we merge that, everyone will start using it and it won't be cool anymore | 15:15 |
| mordred | that's a good point | 15:16 |
| mordred | corvus, fungi, clarkb: Ok - I think I have finally wrapped my head around what's going on with host keys and review.openstack.org from bridge | 15:16 |
| *** jchhatbar has joined #openstack-infra | 15:16 | |
| mordred | it's working from ansible because ansible is ssh-ing to the host using an IP address - the ip address is correct in the local authorized_keys file | 15:16 |
| mordred | sshing to review.openstack.org using openssh is hitting the host key conflict because the localy saved authorized_keys entry for the hosname review.openstack.org _did_ change- but we never noticed because we almost never perform that action | 15:17 |
| *** janki has quit IRC | 15:19 | |
| * Shrews sends mordred a "I was using usepushurl first" fanboy t-shirt | 15:20 | |
| openstackgerrit | Merged openstack-infra/git-review master: Update default gerrit namespace https://review.openstack.org/584607 | 15:21 |
| corvus | they changed *back* to refs/for ? | 15:26 |
| openstackgerrit | Merged openstack-infra/git-review master: Fix git review -d M,N with later gerrit https://review.openstack.org/541791 | 15:26 |
| corvus | that's awesome | 15:26 |
| *** aojea has quit IRC | 15:27 | |
| *** xarses_ has quit IRC | 15:28 | |
| *** stewie925 has joined #openstack-infra | 15:28 | |
| stewie925 | tosky: thank you! | 15:29 |
| stewie925 | hi infra team - am requesting help with a pep8 error issue I foundin my gerrit - http://paste.openstack.org/show/728430/ | 15:30 |
| openstackgerrit | Merged openstack-infra/git-review master: support review URLs as download arguments https://review.openstack.org/424823 | 15:31 |
| mordred | stewie925: that's not actually an error - it's fine (see line 10) | 15:33 |
| mordred | stewie925: it's a result of pip being confused by the config of the git repos on the box - it's not an actual issue | 15:33 |
| mordred | stewie925: the issues in your patch start a bit lower: http://logs.openstack.org/28/590028/7/check/openstack-tox-pep8/3f3018e/job-output.txt.gz#_2018-08-18_01_30_41_890815 | 15:34 |
| stewie925 | thank you !!! | 15:35 |
| *** xarses has joined #openstack-infra | 15:35 | |
| *** xarses has quit IRC | 15:36 | |
| *** xarses has joined #openstack-infra | 15:36 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/git-review master: Update tox docs environment to use sphinx-build https://review.openstack.org/593669 | 15:38 |
| openstackgerrit | Monty Taylor proposed openstack-infra/git-review master: Start using reno for releasenotes https://review.openstack.org/593670 | 15:38 |
| mordred | fungi, corvus: ^^ | 15:38 |
| *** gyee has joined #openstack-infra | 15:38 | |
| mordred | it occured to me that with git-review being slow moving- having release notes would be a good idea | 15:39 |
| openstackgerrit | Monty Taylor proposed openstack-infra/git-review master: Add additional release notes https://review.openstack.org/593671 | 15:44 |
| *** ramishra has quit IRC | 15:44 | |
| mordred | corvus, fungi: sorry- one more ^^ added two more things that have landed that are worthy of note | 15:45 |
| corvus | mordred: what should i be doing for the ansible stuff? | 15:45 |
| mordred | corvus: I rebased the outstanding update-cfg-mgmt stack so that it was less confusing - so it might be worth reviewing it - I also added a few more patches | 15:46 |
| mordred | corvus: I tested https://review.openstack.org/#/c/593029 over the weekend and it seems to have worked fine | 15:46 |
| mordred | corvus: oh - and I split the 'enable cron' patch in two- one to add the new playbooks to run_all - and one that only adds the cron | 15:47 |
| mordred | corvus: other than that - I think we just want to test running base.yaml against everything - and we should be good to go | 15:48 |
| corvus | mordred: in 593029 i don't really understand why installing clouds.yaml is its own playbook? | 15:48 |
| mordred | corvus: probably no good reason - think we should just collapse that into a play in base? | 15:48 |
| corvus | mordred: yeah, or... i guess we haven't really started talking about what it would look like to have a fully ansible managed host yet, eh? | 15:49 |
| corvus | (playbook per host-group or... i dunno, something else?) | 15:50 |
| mordred | yah. the spec talks about playbook per host-group or per 'service' or something like that | 15:50 |
| mordred | but it's definitely handwavey at the moment and we haven't gotten there yet | 15:51 |
| corvus | is there a reason we need to do this now for this? | 15:51 |
| corvus | it just feels like it's pretty normal host configuration | 15:52 |
| mordred | no - it was just a piece that was done - I can put it at the end of the stack if you prefer | 15:52 |
| corvus | mordred: maybe let's do that? cause adding it as a role to a host-group playbook makes the most sense to me | 15:53 |
| mordred | kk | 15:53 |
| mordred | corvus: that said - it might be worth looking at the new patch for sshd config I put up this morning - since that one *is* something we might want to move earlier in the stack (since it's the sshd config) | 15:53 |
| corvus | mordred: 593646 changes later, change #1 is retired | 15:54 |
| *** rkukura has joined #openstack-infra | 15:54 | |
| mordred | corvus: you mean move it later in the stack? not sure I follow | 15:56 |
| corvus | mordred: no it was an idle observation about https://review.openstack.org/1 sorry | 15:56 |
| corvus | i'm reviewing 593647 now | 15:56 |
| mordred | corvus: ah - heh | 15:57 |
| corvus | mordred: all others lgtm | 16:00 |
| *** jpena is now known as jpena|off | 16:01 | |
| *** rpioso is now known as rpioso|afk | 16:01 | |
| mordred | cool | 16:02 |
| mordred | corvus: I think when I move the CA patch later I should move the sshd_config earlier | 16:02 |
| mordred | so that we can run a test of base.yaml with it while we're watching it? | 16:02 |
| *** hashar has quit IRC | 16:03 | |
| corvus | mordred: just so i'm clear -- those are two unrelated things, you're just saying move the sshd earlier for our convenience while you're already restacking things? | 16:04 |
| corvus | mordred: (you're not saying that there is some relationship between the ca and ssh patches, right?) | 16:05 |
| mordred | corvus: yes - that's right | 16:06 |
| corvus | mordred: then yes, that sounds great! | 16:06 |
| openstackgerrit | Monty Taylor proposed openstack-infra/system-config master: Remove infracloud references https://review.openstack.org/593108 | 16:06 |
| openstackgerrit | Monty Taylor proposed openstack-infra/system-config master: Add non-working hosts to disabled list https://review.openstack.org/593636 | 16:06 |
| openstackgerrit | Monty Taylor proposed openstack-infra/system-config master: Manage sshd config with ansible https://review.openstack.org/593647 | 16:06 |
| openstackgerrit | Monty Taylor proposed openstack-infra/system-config master: Run base and bridge playbooks in run_all.sh https://review.openstack.org/593420 | 16:06 |
| openstackgerrit | Monty Taylor proposed openstack-infra/system-config master: Start running puppet cron on bridge.openstack.org https://review.openstack.org/591152 | 16:06 |
| openstackgerrit | Monty Taylor proposed openstack-infra/system-config master: Install limestone CA on openstack using hosts https://review.openstack.org/593029 | 16:06 |
| openstackgerrit | Monty Taylor proposed openstack-infra/system-config master: Remove ssh v1 options from sshd_config https://review.openstack.org/593648 | 16:06 |
| mordred | corvus: done! | 16:06 |
| mordred | corvus: so I think I should fetch down 593420 - then run a couple of spot-tests (one on ubuntu, one on centos) of the base playbook so we can verify ssh still works afterwards | 16:07 |
| mordred | corvus: then we can run base everywere - then run run_all - yeah? | 16:07 |
| mordred | and if all of that is happy, we can land the cronjob patch and call this project complete | 16:08 |
| corvus | mordred: yep that sounds righ | 16:09 |
| mordred | ossum | 16:10 |
| corvus | mordred: i think the ca patch is better in base.yaml, but would be best in hostgroup playbooks -- what do you think? | 16:11 |
| mordred | corvus: I'm logged in to review-dev already - so I'm going to run against it | 16:11 |
| mordred | corvus: I agree that it would be better in hostgroup playbooks - I think I've been hoping we could switch to zuul-triggering instead of cron before we grow more things than run_all.sh | 16:11 |
| *** bobh_ has joined #openstack-infra | 16:12 | |
| corvus | mordred: can we just WIP 593029 then? i'd hate to set a precedent of putting non-fundamental per-host stuff in base.yaml | 16:12 |
| mordred | corvus: we could - the reason I did it is that it's a piece that's needed for bridge (that we did by hand) that's in puppet - but that is also shared with the nodepool hosts - so it's an awkward little guy | 16:14 |
| mordred | corvus: that said - the file is on bridge, so it's not an active issue | 16:15 |
| mordred | so sure | 16:15 |
| *** hashar has joined #openstack-infra | 16:15 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/system-config master: Remove ssh v1 options from sshd_config https://review.openstack.org/593648 | 16:16 |
| *** bobh has quit IRC | 16:16 | |
| mordred | corvus: ^^ also rebased that so it doesn't depend on the limestone CA patch | 16:16 |
| corvus | mordred: okay, that explains the urgency at least :) | 16:16 |
| *** e0ne has quit IRC | 16:16 | |
| corvus | maybe we can go ahead and set up the hostgroup playbooks soon | 16:16 |
| mordred | yah | 16:17 |
| *** ykarel is now known as ykarel|away | 16:17 | |
| *** ginopc has quit IRC | 16:18 | |
| *** strigazi has quit IRC | 16:19 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/system-config master: Manage sshd config with ansible https://review.openstack.org/593647 | 16:20 |
| *** shardy has quit IRC | 16:21 | |
| mordred | corvus: ^^ running against review-dev01 illuminated an issue | 16:21 |
| openstackgerrit | Monty Taylor proposed openstack-infra/system-config master: Run base and bridge playbooks in run_all.sh https://review.openstack.org/593420 | 16:21 |
| openstackgerrit | Monty Taylor proposed openstack-infra/system-config master: Start running puppet cron on bridge.openstack.org https://review.openstack.org/591152 | 16:21 |
| openstackgerrit | Monty Taylor proposed openstack-infra/system-config master: Remove ssh v1 options from sshd_config https://review.openstack.org/593648 | 16:21 |
| corvus | mordred: ++ | 16:23 |
| *** strigazi has joined #openstack-infra | 16:24 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/system-config master: Manage sshd config with ansible https://review.openstack.org/593647 | 16:26 |
| openstackgerrit | Monty Taylor proposed openstack-infra/system-config master: Run base and bridge playbooks in run_all.sh https://review.openstack.org/593420 | 16:26 |
| openstackgerrit | Monty Taylor proposed openstack-infra/system-config master: Start running puppet cron on bridge.openstack.org https://review.openstack.org/591152 | 16:26 |
| openstackgerrit | Monty Taylor proposed openstack-infra/system-config master: Remove ssh v1 options from sshd_config https://review.openstack.org/593648 | 16:26 |
| mordred | corvus: you're going to enjoy ps 6 | 16:26 |
| mordred | corvus: I changed one of the comments in the file on review-dev so that I could see the handler fire | 16:27 |
| *** stewie925 has quit IRC | 16:28 | |
| mordred | I am now happy about how it's running on ubuntu - will try git01.openstack.org now | 16:29 |
| corvus | ps6 of which? | 16:29 |
| mordred | corvus: 593647 | 16:30 |
| *** yamahata has joined #openstack-infra | 16:31 | |
| corvus | mordred: that's only up to ps4. you *must* have big things planned! | 16:31 |
| corvus | mordred: but ps4 is delightful | 16:31 |
| *** mriedem has quit IRC | 16:32 | |
| mordred | corvus: hahah | 16:32 |
| mordred | corvus: yeah - I meants ps4 | 16:32 |
| corvus | mordred: what's the chroot bit about? | 16:32 |
| mordred | corvus: it was logic we had in the puppet module - I believe because of the way the puppet tests run in a chroot? | 16:34 |
| mordred | oh- actually- I think it was there for DIB | 16:34 |
| corvus | that makes sense | 16:35 |
| corvus | i mean, clearly we should not restart sshd in a chroot, i just wondered when that would happen :) | 16:35 |
| corvus | so we could probably remove that, but it's currently harmless? | 16:35 |
| *** trown is now known as trown|lunch | 16:35 | |
| mordred | corvus: yah I agree with both things | 16:36 |
| mordred | corvus: ok - I have tested the stack up through 593420,6 by running base against review-dev and git01 (to get ubuntu and centos) | 16:36 |
| mordred | I'm happy that it's installing the right file, and that it restarts and doesnt' restart ssh appropriately | 16:36 |
| mordred | corvus: which means I think we're back to where we left off on friday - that the next step is just running the base playbook with no --limit | 16:37 |
| fungi | okay, seeing what i missed while i was enjoying my lunch | 16:39 |
| corvus | sounds like we're ready. want to do that now or wait for clarkb, fungi? | 16:39 |
| mordred | we have a fungi - | 16:39 |
| mordred | and I *think* we've spot-checked everything we need to spot-check | 16:39 |
| mordred | I'm game to run it if y'all are | 16:39 |
| AJaeger | config-core, anybody to review https://review.openstack.org/592289 to retire rst2bash, please? And https://review.openstack.org/593489 and https://review.openstack.org/593490 to use "tox -e docs" for intra docs, please? | 16:40 |
| corvus | mordred: you're running git01 again? | 16:40 |
| mordred | corvus: yah - just to make sure we get 1 changed item this time | 16:40 |
| mordred | (last time I'd made a local change in sshd_config to trigger the handler - so that time should have been - and was - a noop) | 16:41 |
| AJaeger | thanks, corvus and mordred ! | 16:42 |
| mordred | if that command looks sensible, I think we're ready to rock-and-roll | 16:42 |
| mordred | incidentally, we're logging to /var/log/ansible.log now - so it's available for tailing or whatnot | 16:43 |
| corvus | mordred: ah excellent, that was my only comment (that i thought we should caputure this run). but we are. so lgtm. | 16:44 |
| mordred | kk. I'm gonna hit the button | 16:44 |
| mordred | corvus: it is not pleased with corvustest :) | 16:44 |
| corvus | neither am i | 16:45 |
| fungi | corvus: mordred: pretty sure we don't have a clarkb this week (at least it seems like his aforementioned beach vacation plans suggested we should not expect one) | 16:45 |
| mordred | oh right | 16:45 |
| fungi | i can pretend to be clarkb if it helps, but i'm not sure i look good in glasses | 16:46 |
| corvus | using --diff with the logfile is :( -- lots of color escape codes | 16:46 |
| fungi | anyway, starting to look into that stack | 16:46 |
| openstackgerrit | Merged openstack-infra/openstack-zuul-jobs master: Switch infra docs job to new PTI https://review.openstack.org/593489 | 16:46 |
| corvus | storyboard-dev looked unhappy? | 16:47 |
| mordred | yeah | 16:47 |
| corvus | unreachable apparently | 16:47 |
| mordred | yeah - don't know why - it'sworing for me from bridge | 16:48 |
| mordred | maybe we need to add the connect-retries setting that we set in zuul | 16:48 |
| mordred | [ssh_connection]\nretries = 3 | 16:49 |
| fungi | i'm struggling to think of a reason bridge.o.o would have trouble ssh'ing to storyboard-dev.o.o | 16:49 |
| openstackgerrit | Merged openstack-infra/project-config master: Retire rst2bash (step 2) https://review.openstack.org/592289 | 16:51 |
| *** ccamacho has quit IRC | 16:51 | |
| corvus | should we try it with a --limit option? | 16:51 |
| corvus | see if it's systemic or a one-off (in which case, maybe the retries is the solution) | 16:51 |
| mordred | corvus: yah | 16:51 |
| *** bobh_ has quit IRC | 16:52 | |
| *** jchhatba_ has joined #openstack-infra | 16:52 | |
| *** jchhatba_ has quit IRC | 16:52 | |
| mordred | corvus, fungi: I ran "ansible storyboard-dev.openstack.org -m setup" in another shell from bridge and it worked fine | 16:53 |
| mordred | so I think retries | 16:53 |
| fungi | still, strange | 16:53 |
| mordred | also - I just noticed in the scrolling that I think we're installing those sources.list files to the wrong place | 16:54 |
| corvus | /etc/apt/sources.list isn't right? | 16:55 |
| mordred | hrm. maybe we're not and the diff output was just confusing | 16:55 |
| *** jchhatbar has quit IRC | 16:55 | |
| mordred | yeah - I saw this: | 16:55 |
| mordred | --- before: /etc/apt/sources.list | 16:55 |
| mordred | +++ after: /opt/system-config/playbooks/roles/base-repos/files/sources.list.trusty.x86_64 | 16:55 |
| mordred | and was worried - but I just spot-checked nl02 and it seems correct | 16:55 |
| corvus | mordred: yeah, that is confusing; the line above is reassuring though: | 16:56 |
| corvus | "dest": "/etc/apt/sources.list" | 16:56 |
| mordred | ++ | 16:56 |
| mordred | and I guess in terms of a diff header that's actually a nice line | 16:56 |
| mordred | tells us where it came from | 16:56 |
| openstackgerrit | Merged openstack-infra/git-review master: Document usepushurl in the man page https://review.openstack.org/555271 | 16:56 |
| corvus | yeah; the alternative is the ansible tempfile (which is what is reported in the changed line above, which is less useful) | 16:57 |
| openstackgerrit | Merged openstack-infra/system-config master: haproxy: don't manage rsyslog service https://review.openstack.org/592637 | 16:57 |
| *** sshnaidm is now known as sshnaidm|off | 16:58 | |
| ssbarnea | git-review: can we get a +W on https://review.openstack.org/#/c/559488/ ? is already many months old and nobody argued against | 16:59 |
| openstackgerrit | Monty Taylor proposed openstack-infra/git-review master: Add additional release notes https://review.openstack.org/593671 | 16:59 |
| mordred | fungi: ^^ updated per your comment | 16:59 |
| ssbarnea | wow, is this really happening, i see lots of activity on git-review repo recently. | 17:03 |
| fungi | well, i particular i wanted to get that namespace fix merged because it had a bunch of other fixes blocking on it | 17:03 |
| openstackgerrit | Merged openstack-infra/project-config master: Use new infra tox-docs templates https://review.openstack.org/593490 | 17:03 |
| openstackgerrit | Merged openstack-infra/git-review master: Update tox docs environment to use sphinx-build https://review.openstack.org/593669 | 17:03 |
| mordred | ssbarnea: that patch lgtm - but I'd like for fungi to look at it too | 17:04 |
| corvus | much output | 17:04 |
| *** mriedem has joined #openstack-infra | 17:05 | |
| *** e0ne has joined #openstack-infra | 17:05 | |
| *** jpich has quit IRC | 17:05 | |
| *** neilsun has quit IRC | 17:05 | |
| mordred | corvus: ok - graphite.o.o changed significantly - the other changes are all just ordering of the variable | 17:05 |
| corvus | i'll look at the log for graphite | 17:06 |
| mordred | corvus: I can still ssh in to graphite from bridge as root, so that's good | 17:06 |
| ssbarnea | mordred: sure. if we manage to cleanup the list of pending review on git-review it would be really great, merging safe one and abandoning ancient ones (like with no update in >1 year) it will really be great. | 17:07 |
| mordred | ssbarnea: yah. I just got excited by the patch from electrofelix to allow git review -d https://review.openstack.org/#/c/559488 :) | 17:07 |
| corvus | mordred: there's a lot of openstack api stuff in the log | 17:08 |
| corvus | mordred: is it at all possible the openstack inventory plugin is outputting api responses to the log? | 17:08 |
| mordred | looking | 17:08 |
| mordred | corvus: can you give me a string to look for? | 17:09 |
| corvus | mordred: RAX-PUBLIC-IP-ZONE-ID | 17:09 |
| mordred | corvus: why yes! it does appear that we are logging the inventory opertions | 17:09 |
| corvus | (i grepped for graphite, and one of the lines i got is long enough to make 'less' difficult to use; i imagine it's the full inventory in json on one line) | 17:10 |
| mordred | yup. I imagine it's that too | 17:10 |
| *** ykarel|away has quit IRC | 17:10 | |
| *** yamahata has quit IRC | 17:10 | |
| mordred | corvus: we should definitely clean thatup | 17:10 |
| *** ykarel|away has joined #openstack-infra | 17:10 | |
| mordred | it shouldn't be logging anything sensitive, so it's more about making that log usable than about ... oh - you know what? | 17:11 |
| mordred | I'd be willing to be it's because we have --verbose on | 17:11 |
| mordred | s/be/bet/ | 17:11 |
| mordred | since those flags affect python logging loggers and I thin they set things on the root logger | 17:12 |
| corvus | mordred: i don't see anything objectionable on graphite -- it seems to have installed a lot of keys because the old ones had the puppet header. | 17:13 |
| corvus | mordred: that's the only notable thing i see changed | 17:13 |
| mordred | yah- the main thing was the diff for /etc/ssh/sshd_config was larger than the other hosts | 17:13 |
| corvus | oh i'll look closer at that | 17:13 |
| *** mriedem has quit IRC | 17:14 | |
| *** e0ne has quit IRC | 17:15 | |
| corvus | mordred: curious; permitrootlogin was 'yes' | 17:15 |
| corvus | and passwordauthentication was 'yes' | 17:16 |
| corvus | that was not our sshd config | 17:16 |
| mordred | corvus: he graphite file seemed to be just completely not what our normal file is | 17:16 |
| mordred | yah | 17:16 |
| fungi | ssbarnea: until seeing 559488 i didn't realize gerrit 2.15 introduces the ability to mark changes as private. aside from the release notes and the "view private changes" acl permission, i can't find much documentation about it. do you happen to know if it's possible to disable the private setting in newer gerrit? | 17:16 |
| corvus | i wonder if it was subjected to a rescue or something? | 17:16 |
| mordred | with rackspace overwriting our file? | 17:16 |
| *** dtantsur is now known as dtantsur|afk | 17:17 | |
| corvus | ya | 17:17 |
| corvus | (though i would think puppet should write it back?) | 17:17 |
| mordred | I'd expect that to happen, yeah | 17:17 |
| mordred | corvus: we should really stop running nova-agent on those boxes | 17:17 |
| corvus | i thought we did :( | 17:17 |
| mordred | it's running on grpahite | 17:18 |
| mordred | it's not running on review-dev | 17:18 |
| *** ykarel|away has quit IRC | 17:18 | |
| corvus | i don't see any puppet-user syslog entries on graphite | 17:19 |
| mordred | have we just not been puppeting graphite tihs whole time for some reason? | 17:19 |
| corvus | er, wait, i think i failed grep | 17:20 |
| fungi | the images rackspace provides have nova-agent set up ahead of time (which is the main reason they don't put distro provided images in their catalog). it also implies xen-agent, which i think may be necessary to support device hotplug on xen, but that also might have changed in recent years | 17:20 |
| corvus | yeah, it's been running. i don't see any interesting output | 17:20 |
| mordred | WEIRD | 17:20 |
| mordred | ok - so nb0[1-3] had ansible failures | 17:21 |
| mordred | I want to re-run storyboard-dev since it didn't run most of the run due to the connection issue - then let's look at nb0* ? | 17:22 |
| fungi | ahh, looks like there are kernel tasks for xenwatch and xenbus, and some loaded kernel modules for xen drivers, so i guess that's what handles hotplug these days | 17:22 |
| mordred | corvus: unless you want to investigate what might have happened with nb0* ? | 17:22 |
| openstackgerrit | Doug Hellmann proposed openstack-infra/project-config master: remove job settings for Technical Committee repositories https://review.openstack.org/593706 | 17:22 |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config master: Update and fix tox-docs jobs https://review.openstack.org/593712 | 17:24 |
| openstackgerrit | Ildiko Vancsa proposed openstack-infra/irc-meetings master: OpenStack Upstream Institute meeting time update https://review.openstack.org/593713 | 17:24 |
| mordred | corvus, fungi SO - I think nb01 is getting erroneously tagged as being a SuSE node | 17:26 |
| mordred | because it has zypper installed | 17:26 |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/openstack-zuul-jobs master: Remove unused infra publish templates https://review.openstack.org/593714 | 17:26 |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config master: Remove obsolete infra publish jobs https://review.openstack.org/593715 | 17:27 |
| AJaeger | config-core, I screwed up with infra publishing, please check https://review.openstack.org/593712 | 17:27 |
| *** hemna_ has joined #openstack-infra | 17:28 | |
| mordred | "ansible_nodename": "nb01", | 17:28 |
| mordred | "ansible_os_family": "Debian", | 17:28 |
| mordred | "ansible_pkg_mgr": "zypper", | 17:28 |
| fungi | mordred: er, yikes. you can install dpkg/apt on centos too if you want, and rpm on gentoo, and... | 17:31 |
| mordred | yeah | 17:31 |
| mordred | I'm reporting theissue and also looking for a workaround | 17:31 |
| fungi | determining what the distro is by looking at what package managers are installed is bunk | 17:31 |
| *** hashar is now known as hasharDinner | 17:32 | |
| mordred | well - it's not determining distro - it's the "package" module (which you can use to call the 'right' package manager) figuring out which package manager to use | 17:32 |
| mordred | but I agree - it's still a bit bunk | 17:32 |
| fungi | ahh, yeah still no way to know which of those is the "preferred" package manager for a given system | 17:36 |
| corvus | wow, sorry, stepped afk for a sec | 17:41 |
| *** e0ne has joined #openstack-infra | 17:42 | |
| *** jamesmcarthur has quit IRC | 17:42 | |
| corvus | mordred: what did you do to get the nb01 fact output? | 17:42 |
| *** yamahata has joined #openstack-infra | 17:43 | |
| mordred | corvus: I looked two different ways | 17:43 |
| *** jamesmcarthur has joined #openstack-infra | 17:43 | |
| mordred | the first was "ansible nb01.openstack.org -m setup| grep pkg_mgr" | 17:43 |
| mordred | the second was "vim /var/cache/ansible/facts/nb01.openstack.org" | 17:44 |
| mordred | the package action plugin is doing this: | 17:44 |
| corvus | neat. the first one vomited the facts into the log, which was helpful in this case :) | 17:44 |
| mordred | https://github.com/ansible/ansible/blob/devel/lib/ansible/plugins/action/package.py#L48 | 17:45 |
| corvus | mordred: hrm. seems like we can override it an the task level, but i don't imagine that's a sufficient solution | 17:47 |
| mordred | yeah | 17:47 |
| *** trown|lunch is now known as trown | 17:48 | |
| openstackgerrit | Nate Johnston proposed openstack-infra/elastic-recheck master: Add elastic-recheck neutron DNSIntegrationTests:test_server_with_fip https://review.openstack.org/593722 | 17:49 |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/openstack-zuul-jobs master: Add build-tox-docs template https://review.openstack.org/593723 | 17:50 |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config master: Update infra-manual publishing https://review.openstack.org/593724 | 17:50 |
| corvus | mordred, fungi: https://github.com/ansible/ansible/blob/devel/lib/ansible/module_utils/facts/system/pkg_mgr.py#L110 | 17:50 |
| corvus | so basically, it has exceptions for redhat, but not debuntu | 17:51 |
| corvus | (cause it really does start by just finding the first one in this list: https://github.com/ansible/ansible/blob/devel/lib/ansible/module_utils/facts/system/pkg_mgr.py#L26 ) | 17:51 |
| corvus | wait no the last one in that list | 17:51 |
| *** e0ne has quit IRC | 17:51 | |
| corvus | thus zypper > apt-get > dnf > yum | 17:52 |
| *** diablo_rojo_ has joined #openstack-infra | 17:53 | |
| *** diablo_rojo_ has quit IRC | 17:53 | |
| *** diablo_rojo has joined #openstack-infra | 17:53 | |
| *** rkukura has quit IRC | 17:54 | |
| fungi | ssbarnea: https://gerrit-review.googlesource.com/Documentation/rest-api-config.html seems to mention a disable_private_changes option which i can't find in the configuration docs. wonder whether it actually exists and they just neglected to document it, or the documentation there is referring to a control they never got around to implementing? | 17:56 |
| openstackgerrit | Nate Johnston proposed openstack-infra/elastic-recheck master: Add elastic-recheck neutron DNSIntegrationTests:test_server_with_fip https://review.openstack.org/593722 | 17:58 |
| corvus | fungi, ssbarnea: or is it a plugin? | 17:59 |
| *** electrofelix has quit IRC | 17:59 | |
| *** jamesmcarthur has quit IRC | 18:00 | |
| fungi | mmm, the only think in that vein i see at https://gerrit.googlesource.com/plugins/ is reject-private-submit which seems to only disable merging of private changes | 18:01 |
| fungi | here we go! https://gerrit-review.googlesource.com/Documentation/config-gerrit.html#change.disablePrivateChanges | 18:02 |
| fungi | "If set to true, users are not allowed to create private changes." | 18:02 |
| *** r-daneel has quit IRC | 18:06 | |
| *** jiapei has quit IRC | 18:08 | |
| *** r-daneel has joined #openstack-infra | 18:08 | |
| AJaeger | config-core, please review https://review.openstack.org/593712 to fix infra docs publishing. | 18:08 |
| openstackgerrit | Merged openstack-infra/git-review master: `work-in-progress` and `private` workflow in Gerrit 2.15 https://review.openstack.org/559488 | 18:10 |
| AJaeger | infra-root, do we want to move the wheel-mirror build jobs in requirements repo - or in project-config? See https://review.openstack.org/#/c/593660/1/zuul.d/projects.yaml | 18:11 |
| corvus | fungi: you want to disable for openstack? | 18:11 |
| *** e0ne has joined #openstack-infra | 18:14 | |
| *** raissa has joined #openstack-infra | 18:14 | |
| fungi | corvus: i foresee challenges with the way we use gerrit now, that either private changes won't really be private because of replication (similar to how drafts weren't effectively private) leading to a false sense of security, and challenges with either keeping zuul from acting on private changes (thus exposing them) or preventing it from choking when it encounters references to changes it can't | 18:14 |
| fungi | access (like we got with changes depending on draft changes) | 18:14 |
| fungi | though those zuul points are likely bugs/missing features on the zuul end if so | 18:15 |
| openstackgerrit | David Shrewsbury proposed openstack-infra/project-config master: Add a job to push pbrx container images https://review.openstack.org/593725 | 18:17 |
| openstackgerrit | Monty Taylor proposed openstack-infra/system-config master: Have ansible retry ssh connections https://review.openstack.org/593728 | 18:18 |
| openstackgerrit | Monty Taylor proposed openstack-infra/system-config master: Add local copy of pkg_mgr fact module https://review.openstack.org/593729 | 18:18 |
| openstackgerrit | Monty Taylor proposed openstack-infra/system-config master: Swap the precedence order of zypper and apt-get https://review.openstack.org/593730 | 18:18 |
| mordred | corvus: ^^ | 18:18 |
| openstackgerrit | David Shrewsbury proposed openstack-infra/project-config master: Add a job to push pbrx container images https://review.openstack.org/593725 | 18:21 |
| mordred | corvus: that patch fixes it in my local testing | 18:23 |
| openstackgerrit | David Shrewsbury proposed openstack-infra/zuul master: Add container push job https://review.openstack.org/593125 | 18:24 |
| openstackgerrit | David Shrewsbury proposed openstack-infra/zuul master: Add container push job https://review.openstack.org/593125 | 18:25 |
| openstackgerrit | David Shrewsbury proposed openstack-infra/zuul master: Add container push job https://review.openstack.org/593125 | 18:27 |
| *** rpioso|afk is now known as rpioso | 18:30 | |
| *** jamesmcarthur has joined #openstack-infra | 18:34 | |
| *** r-daneel has quit IRC | 18:37 | |
| *** r-daneel has joined #openstack-infra | 18:37 | |
| corvus | fungi: yes. i agree. our earlier experience suggests that to be the case. | 18:37 |
| *** yankcrime has quit IRC | 18:38 | |
| mordred | corvus: wow. zypper seems to be in xenial but not bionic | 18:38 |
| pabelanger | we use zypper to build opensuse images | 18:38 |
| pabelanger | on xenial | 18:38 |
| *** jamesmcarthur has quit IRC | 18:38 | |
| pabelanger | mordred: it is a dependency for DIB | 18:39 |
| corvus | mordred: well, um, that's one way to solve the problem i guess. | 18:39 |
| fungi | interesting. it was in artful and is in cosmic, but skipped bionic? | 18:40 |
| fungi | i wonder if it was temporarily kicked out of debian/testing and then reintroduced | 18:40 |
| fungi | oh, yep, in jessie and buster but not stretch | 18:41 |
| fungi | https://tracker.debian.org/news/719494/zypper-removed-from-testing/ | 18:41 |
| pabelanger | Oh, so if zypper isn't on bionic, we cannot build opensuse images for nodepool-builder when we update | 18:42 |
| pabelanger | unless we add package from ppa | 18:42 |
| fungi | unless we can get it backported from cosmic | 18:42 |
| fungi | or that, yeah | 18:42 |
| pabelanger | yah | 18:42 |
| *** e0ne has quit IRC | 18:42 | |
| fungi | looks like the problem was that libzypp didn't keep up with the libstdc++ v5 transition | 18:43 |
| fungi | looks like it had rc bugs open from 2015/2016 up until this year | 18:44 |
| fungi | at which point it was reintroduced | 18:44 |
| mordred | corvus: better fix coming | 18:45 |
| openstackgerrit | Monty Taylor proposed openstack-infra/system-config master: Ensure apt is used on ubuntu hosts with zypper https://review.openstack.org/593730 | 18:48 |
| mordred | corvus: ^^ that's whatI submitted upstream in https://github.com/ansible/ansible/pull/44413 | 18:49 |
| mordred | corvus: how do you feel about me pulling that patch onto bridge and trying base on nb0* again | 18:49 |
| openstackgerrit | David Shrewsbury proposed openstack-infra/project-config master: Add a job to push pbrx container images https://review.openstack.org/593725 | 18:50 |
| *** yamahata has quit IRC | 18:52 | |
| openstackgerrit | David Shrewsbury proposed openstack-infra/project-config master: Add a job to push pbrx container images https://review.openstack.org/593725 | 18:53 |
| mordred | corvus, fungi: I've got a re-try teed up on bridge with those changes | 18:57 |
| *** hasharDinner is now known as hashar | 18:58 | |
| AJaeger | infra-root, please review https://review.openstack.org/593712 to fix infra docs publishing. Sorry, for breaking it ;( | 19:03 |
| corvus | mordred: ++ | 19:04 |
| corvus | mordred: i'm going to grab lunch, biab. | 19:05 |
| mordred | corvus: kk | 19:06 |
| *** f1 has quit IRC | 19:06 | |
| mordred | corvus: when you get back, I think we'll be in a position to proceed | 19:06 |
| *** e0ne has joined #openstack-infra | 19:06 | |
| fungi | mnaser: so coming back around on the salsa.debian.org gitlab storage thread from last week, it looks like the nearly impossible to find gitlab ce docs do say it uses fog at least for interacting with lfs storage https://docs.gitlab.com/ce/workflow/lfs/lfs_administration.html#storing-lfs-objects-in-remote-object-storage | 19:07 |
| mnaser | fungi: first thing i read "Introduced in GitLab Premium 10.0. Brought to GitLab Core in 10.7." ugh | 19:08 |
| fungi | yeah | 19:08 |
| mnaser | https://github.com/fog/fog-openstack#initial-setup | 19:09 |
| mnaser | i guess it can be used with those connection_params | 19:09 |
| fungi | clearly there were 7 minor releases of gitlab ee before they decided ce might be worthy of getting that | 19:09 |
| mnaser | fungi: i dont know enough about the depths of the debian project but anyone can feel free to reach out to me :) | 19:10 |
| * fungi shakes fist at github for stealing control of both / and ^G in his browser | 19:12 | |
| fungi | mnaser: so i guess https://github.com/fog/fog-openstack/blob/master/docs/storage.md suggests that it does swift too (even though the initial setup doc doesn't say anything about it) | 19:17 |
| mordred | fungi: if you get a sec, https://review.openstack.org/#/c/593108 through https://review.openstack.org/#/c/593730 are ready for landing I believe | 19:18 |
| mnaser | fungi: guess we can probably host it for them then :> | 19:18 |
| fungi | thanks mordred! | 19:18 |
| mordred | fungi: I have verified them on bridge - and nb* is now working | 19:18 |
| fungi | i see, awesome | 19:18 |
| mordred | (probably easier to work backwards from https://review.openstack.org/#/c/593730 - there are two paths upwards from 593108 | 19:19 |
| mordred | also - I gotta say - being able to override that one file until the patch comes out in a release is pretty nice | 19:19 |
| pabelanger | mordred: couldn't we just set an inventory variable for ansible_pkg_mgr in nodepool to apt? I don't think facts override them | 19:21 |
| AJaeger | pabelanger, mnaser, could either of you review https://review.openstack.org/593712 , please? | 19:23 |
| *** e0ne has quit IRC | 19:26 | |
| mnaser | AJaeger: is ther a change that tests that | 19:26 |
| AJaeger | mnaser: no ;( | 19:27 |
| AJaeger | mnaser: that's why I broke it ;) | 19:27 |
| mordred | pabelanger: nope. the underlying package code actually checks ansible_facts.pkg_mgr | 19:28 |
| mordred | pabelanger: incidentally, getting at facts via the ansible_facts dict is The Future - as their trying to distinguish between system-provided fact information and user-set variables | 19:29 |
| mordred | pabelanger: but there's a big pile of scrollback in #ansible-devel if you wanna read all about it | 19:29 |
| openstackgerrit | David Shrewsbury proposed openstack-infra/project-config master: Add a job to push pbrx container images https://review.openstack.org/593725 | 19:29 |
| mordred | pabelanger: (and I wish - that would have been much easier) | 19:29 |
| AJaeger | mnaser: once that change is merged, we could e.g. merge https://review.openstack.org/593643 to test it | 19:29 |
| mnaser | AJaeger: looks good | 19:30 |
| AJaeger | thanks, mnaser | 19:31 |
| *** EvilienM is now known as EmilienM | 19:31 | |
| openstackgerrit | David Shrewsbury proposed openstack-infra/project-config master: Add a job to push pbrx container images https://review.openstack.org/593725 | 19:31 |
| *** eharney has quit IRC | 19:32 | |
| pabelanger | mordred: ah, I thought ansible_pkg_mgr was a thing | 19:32 |
| mordred | pabelanger: it is - it's the 'old' shorthand - and still works to read the value for now | 19:32 |
| mordred | pabelanger: but, unfortunately for us, the underlying code in the package action plugin actually looks at the 'new' fancy variable name 'ansible_facts.pkg_mgr' *first* | 19:33 |
| pabelanger | ack | 19:34 |
| *** jamesmcarthur has joined #openstack-infra | 19:36 | |
| *** jtomasek has quit IRC | 19:36 | |
| *** yamahata has joined #openstack-infra | 19:39 | |
| openstackgerrit | Merged openstack-infra/project-config master: Update and fix tox-docs jobs https://review.openstack.org/593712 | 19:42 |
| AJaeger | I get lots of "[WARNING] Ansible is being run in a world writable directory (/home/zuul/src/git.openstack.org/openstack-infra/project-config), ignoring it as an ansible.cfg source. For more information see https://docs.ansible.com/ansible/devel/reference_appendices/config.html#cfg-in-world-writable-dir" - see | 19:45 |
| AJaeger | http://logs.openstack.org/12/593712/1/gate/openstack-zuul-jobs-linters/e8b3109/ara-report/result/95a60c8d-f312-4085-b4a4-2cce8b1df228/ | 19:45 |
| AJaeger | is that a problem? Haven't seen that before and see it only in streaming logs and ara report | 19:45 |
| AJaeger | mnaser: want to +2A https://review.openstack.org/#/c/593643/ to test doc publishing, please? | 19:45 |
| *** r-daneel_ has joined #openstack-infra | 19:45 | |
| mnaser | AJaeger: done | 19:46 |
| *** r-daneel has quit IRC | 19:47 | |
| *** r-daneel_ is now known as r-daneel | 19:47 | |
| AJaeger | thanks, mnaser | 19:47 |
| openstackgerrit | Goutham Pacha Ravi proposed openstack-infra/project-config master: remove job settings for manila repositories https://review.openstack.org/593898 | 19:50 |
| openstackgerrit | Sean McGinnis proposed openstack-infra/project-config master: Remove operations-guide jobs https://review.openstack.org/593899 | 19:50 |
| pabelanger | AJaeger: yes, we need to stop / start zuul executor and delete our git cache, they are currently world readable in zuul | 19:51 |
| pabelanger | I was talking to fungi last week about maybe doing that | 19:51 |
| pabelanger | AJaeger: https://review.openstack.org/585068/ is fix in zuul | 19:52 |
| corvus | pabelanger: executors have been restarted | 19:53 |
| corvus | pabelanger: i did not delete the cache | 19:53 |
| corvus | pabelanger: maybe you could just run a chmod on it or something? | 19:54 |
| corvus | mordred: back | 19:54 |
| openstackgerrit | Merged openstack-infra/openstack-zuul-jobs master: Add openstack-python36-jobs-neutron template https://review.openstack.org/593643 | 19:54 |
| pabelanger | corvus: yah, maybe we can just delete cache on a single executor to confirm they recreated properly, then chmod the rest. | 19:54 |
| fungi | presumably recursive chmod should be entirely non-impacting | 19:54 |
| pabelanger | I can help do that tomorrow, I don't have much time left for todya | 19:55 |
| pabelanger | today* | 19:55 |
| corvus | pabelanger: ok. if you want to delete the cache, just be careful -- it takes a long time to repopulate. we would only want to do one at a time. deleting one and chmoding the others is probably fine. | 19:55 |
| pabelanger | corvus: ack | 19:55 |
| fungi | sounds like a fine compromise | 19:55 |
| fungi | at least that way we get confirmation it works to populate itself with the correct permissions (beyond what ci tests already ran against the change in question before it merged) | 19:56 |
| AJaeger | pabelanger: ah, thanks | 19:56 |
| *** sileht has joined #openstack-infra | 19:57 | |
| corvus | https://object-storage-ca-ymq-1.vexxhost.net/swift/v1/86bbbcfa8ad043109d2d7af530225c72/logs_82/592582/2/check/tempest-full/a77af5a/ is a tempest-full run with logs stored in swift | 19:58 |
| *** timburke has quit IRC | 19:58 | |
| mordred | corvus, fungi: y'all think I should do another base run - or just run run_all ? | 19:58 |
| mordred | oh - actually - we need to land those patches before I can actually run run_all | 19:58 |
| *** HenryG_ is now known as HenryG | 19:58 | |
| fungi | yeah, sorry, still looking at them | 19:58 |
| pabelanger | corvus: excellect! | 19:59 |
| corvus | mordred: you tested the new one on nb? | 19:59 |
| mordred | no worries- pabelanger knocked em in | 19:59 |
| mordred | corvus: I did | 19:59 |
| mordred | corvus: everything is green now | 19:59 |
| corvus | mordred: cool, then i'd say merge the patches then run_all | 19:59 |
| mordred | cool. pabelanger has approved the stack, so as soon as they land, we'll pull and run-all things | 19:59 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul-jobs master: htmlify: don't log https://review.openstack.org/593901 | 20:00 |
| mordred | corvus, fungi, pabelanger: fwiw, the upstream fix has been merged into the ansible devel branch | 20:01 |
| mordred | and has been submitted as a backport to the stable-2.6 branch | 20:01 |
| pabelanger | mordred: great | 20:01 |
| *** r-daneel_ has joined #openstack-infra | 20:02 | |
| mordred | so we'll DEFINITELY be able to remove that module_utils file when 2.7 comes out - but might be able to when the next 2.6 bugfix release comes out | 20:02 |
| corvus | mordred, pabelanger: quick patch: https://review.openstack.org/593901 removes the super verbose stuff at the end of https://object-storage-ca-ymq-1.vexxhost.net/swift/v1/86bbbcfa8ad043109d2d7af530225c72/logs_82/592582/2/check/tempest-full/a77af5a/job-output.txt | 20:02 |
| corvus | mordred: \o/ | 20:02 |
| *** Dmitrii-Sh_ has joined #openstack-infra | 20:02 | |
| *** r-daneel has quit IRC | 20:03 | |
| *** r-daneel_ is now known as r-daneel | 20:03 | |
| fungi | mordred: on 593729 i see that file is gplv3 | 20:04 |
| mordred | corvus: oh - wait - there are a few that still need +A | 20:04 |
| mordred | fungi: yeah - it's just a straight copy from the ansible repo | 20:04 |
| *** diablo_rojo has quit IRC | 20:04 | |
| fungi | we already have another gplv3 file in that repo, but no copying file | 20:04 |
| fungi | do we need a copying file? | 20:04 |
| pabelanger | corvus: is there another role to be used to make datetime linkable again for job-output.txt? | 20:04 |
| mordred | fungi: that's a good question | 20:04 |
| mordred | fungi: probably wouldn't hurt to add one - but we also dont' produce tarball releases of that repo, so I *think* I'm comfortable with the file just having a header | 20:05 |
| fungi | yeah, it was a grey area for me too | 20:05 |
| corvus | pabelanger: the htmlify role | 20:06 |
| corvus | pabelanger: i did not link to it above because it is the output of the htmlify role itself that i wanted to show you, which is not in the htmlified version (because it hadn't run yet) | 20:06 |
| corvus | pabelanger: https://object-storage-ca-ymq-1.vexxhost.net/swift/v1/86bbbcfa8ad043109d2d7af530225c72/logs_82/592582/2/check/tempest-full/a77af5a/job-output.html#l34775 is the end of the htmlified output | 20:07 |
| fungi | mordred: is 593636 duplicating entries from the emergency disable list into inventory/groups.yaml? | 20:07 |
| fungi | do we still have a separate emergency disable list? | 20:07 |
| mordred | it shouldn't be? yes - we stil have a separate emergency file | 20:07 |
| * mordred goes to look | 20:07 | |
| fungi | and if so, was the tooling skipping it? | 20:08 |
| pabelanger | corvus: ah, neat. I see now. Both .html and .txt | 20:08 |
| pabelanger | ++ | 20:08 |
| mordred | fungi: yah - those hosts are not in the emergency file | 20:08 |
| AJaeger | yeah, infra docs publishing worked again - thanks for reviews mordred and mnaser. | 20:08 |
| AJaeger | https://docs.openstack.org/infra/openstack-zuul-jobs/project-templates.html just got updated... | 20:08 |
| *** Dmitrii-Sh has quit IRC | 20:08 | |
| AJaeger | corvus: +2A ^ | 20:08 |
| AJaeger | config-core, these changes are needed by release team - care to review, please? https://review.openstack.org/591837 https://review.openstack.org/591836 https://review.openstack.org/592667 | 20:08 |
| *** v1k0d3n has quit IRC | 20:08 | |
| *** Dmitrii-Sh_ is now known as Dmitrii-Sh | 20:09 | |
| fungi | interesting. wiki-upgrade-test.openstack.org and wiki.openstack.org were in the emergency disable list on puppetmaster | 20:09 |
| ssbarnea | pabelanger: i am glad to see we are finally getting real html for logs. do we have a bug or blueprint about this? i am curious about next steps. | 20:09 |
| mordred | fungi: yah - those were also in the disabled file already | 20:09 |
| fungi | mordred: looks like the wiki entries are in there on bridge too, yeah | 20:10 |
| mordred | fungi: yah | 20:10 |
| fungi | i should have realized the file would be at that same path | 20:10 |
| pabelanger | ssbarnea: I'll defer to corvus for swift logs | 20:10 |
| mordred | we should maybe be better about onlyusing emergency for emergencies and getting stuff disabled in the disabled group for things that are going to be disabled for longer | 20:10 |
| ssbarnea | to be more precice I have few wishlists myself and i want to see if planned direction is in sync with my log-dreams ;) | 20:10 |
| *** v1k0d3n has joined #openstack-infra | 20:10 | |
| ssbarnea | be looking for: ANSI coloring, bookmarks with line numbering (and timestamp as tooltip instead of taking 30% of the screen width). | 20:11 |
| mordred | ssbarnea: current effort is more just about moving where we're uploading things. however, we've got thoughts/desires/plans for adding some nicer log viewing to the zuul dashboard | 20:12 |
| ssbarnea | mordred: storage is something of different nature, unrelated to the format of the logs. | 20:13 |
| mordred | yup | 20:13 |
| mordred | swift storage is about getting us off of being worried that we're going to have a volume go down again | 20:14 |
| mordred | it is, as you rightly point out, mostly orthogonal to adding richer display of logs | 20:14 |
| ssbarnea | while playing with ansi2html myself I found it buggy in many aspects and I was planning to write an alternative that would address its current limitations (and also working as a streaming wrapper). still, this will take some time. | 20:16 |
| fungi | mordred: agreed about longer disablement moving into the non-emergency list | 20:17 |
| *** kgiusti has left #openstack-infra | 20:18 | |
| ssbarnea | another reason being that license controvery with ansi2html, i want something BSD/MIT, avoiding dangers of GPL. still, if I succeed, the generated html files should look more like travis console logs. | 20:19 |
| *** jtomasek has joined #openstack-infra | 20:21 | |
| mordred | oh - I don't know about that | 20:21 |
| *** diablo_rojo has joined #openstack-infra | 20:21 | |
| mordred | I find the travis logs completely unreadable | 20:21 |
| openstackgerrit | Merged openstack-infra/system-config master: Remove infracloud references https://review.openstack.org/593108 | 20:21 |
| mordred | ssbarnea: however, making something that has richer display, such as collapsable sections - or control over visibilty of the timestamps for job-output.txt.gz - I totally agree | 20:23 |
| corvus | i reject "dangers of gpl" as a premise | 20:24 |
| *** jtomasek has quit IRC | 20:24 | |
| mordred | ssbarnea: main thing to keep in mind is that whatever that thing is is going to want to be in javascript as part of zuul-dashboard- so I wouldn't spend too much effort on command line utilities if you are thinking towards a future of updates to job-output.txt.gz | 20:24 |
| *** eharney has joined #openstack-infra | 20:25 | |
| corvus | ssbarnea: see http://lists.zuul-ci.org/pipermail/zuul-discuss/2018-July/000501.html | 20:26 |
| mordred | job-output.txt.gz itself shares emission code with the live log streaming code which needs to be able work with text-based clients - once the log is finished and the json file containing the rich data about the whole log is available, rendering that for the browser in more advanced ways would be great | 20:26 |
| mordred | corvus: that's easier than me re-typing that in channel :) | 20:26 |
| corvus | mordred: yep. only thing i'd add is that in all cases, we still want the unadorned plain text log to be fetchable (for local download/grep/etc). | 20:27 |
| mordred | yup | 20:27 |
| mordred | agree | 20:27 |
| mordred | grep, it turns out, is our friend | 20:27 |
| openstackgerrit | Merged openstack-infra/zuul-jobs master: htmlify: don't log https://review.openstack.org/593901 | 20:31 |
| *** jtomasek has joined #openstack-infra | 20:31 | |
| smcginnis | Anyone know if there is a new *-pti equivalent for the publish-openstack-sphinx-docs-direct job? | 20:32 |
| ssbarnea | mordred: nope, i am not looking for a JS live processor, it will never work, it must be a wrapper as there are some runtime aspects that are impossible to store inside a plain text file. it would produce dual output (plain and html), so it will be grep friendly. I do not plan to implement anything ansible specific. For example I want to print stderr output using different colour in the logs and processing stdout and stderr well | 20:33 |
| ssbarnea | is a real challenge. I want to use the same too for sharing console execution (instead of saving the output to local file to stream it a web server), something like seashells. (like a tee for the web). | 20:33 |
| *** jtomasek has quit IRC | 20:34 | |
| *** ansmith has quit IRC | 20:34 | |
| fungi | ssbarnea: the crux of the challenge is that we'd like to not only store logs in object storage, but serve them from object storage too so they don't have to funnel through some smart frontend somewhere | 20:34 |
| fungi | serving the content directly to the client from the place it's stored is massively less bandwidth and resource intensive | 20:35 |
| *** jcoufal has quit IRC | 20:35 | |
| mordred | ssbarnea: I think we're missing each other about a piece of the puzzle | 20:36 |
| mordred | ssbarnea: I'm definitey not suggesting a javascript wrapper around the text file | 20:36 |
| mordred | ssbarnea: I'm talking about a javascript renderer around the log-output.json file - which has ALL of the possible data in structured format | 20:36 |
| *** rkukura has joined #openstack-infra | 20:37 | |
| ssbarnea | fungi: this is going to be a big challenge because object storage solutions like S3 or swift do not allow streaming (serving an object while still apending to it). I know for sure that S3 does not support this and will probably never. | 20:37 |
| ssbarnea | mordred: it is an interesting idea, not sure if browsers will be ever be able to cope with this, I plan to find a solution that allows you to browse the longs on your mobile phone! | 20:38 |
| openstackgerrit | Merged openstack-infra/system-config master: Add non-working hosts to disabled list https://review.openstack.org/593636 | 20:38 |
| mordred | ssbarnea: but - also unfortunately for your desire, we capture stdout and stderr interleaved in the first place which is done on purpose as the best way available to us to be able to show events in the order they occurred | 20:38 |
| ssbarnea | nice typo, longs instead of logs, there is some truth in it .... | 20:38 |
| fungi | ssbarnea: the log streaming interface in zuul is different, as those logs have not been uploaded while the job is underway and the logs are being appended | 20:38 |
| mordred | ssbarnea: yes - I agree with that goal | 20:38 |
| mordred | ssbarnea: (of viewing on phone) | 20:38 |
| notmyname | (ssbarnea: FWIW, I've heard a request for that functionality in swift before, and I have a vague idea of how we could actually do it. but there's a few big things that have to get done first, so it's definitely a long-term thing) | 20:39 |
| mordred | and yes - what fungi said - the log streaming is done via a log streaming process that interacts with the zuul_stream interface | 20:39 |
| mordred | ssbarnea: all of that said - there are a bunch of moving pieces here architecturally I'd be happy to dig in to with you if it's an area you're thinking you want to poke at | 20:40 |
| ssbarnea | anyway, i will let you see my POC as soon I have it ready, as I said is has nothing to do with ansible or zuul, is will be a generic wrapper. The original need was to find an opensource equivalend to seashells for sharing terminal output. (seashells server being closed source). | 20:40 |
| mordred | but I'm going to warn you - ansible callback plugins get involved | 20:40 |
| openstackgerrit | Merged openstack-infra/system-config master: Manage sshd config with ansible https://review.openstack.org/593647 | 20:41 |
| openstackgerrit | Merged openstack-infra/system-config master: Run base and bridge playbooks in run_all.sh https://review.openstack.org/593420 | 20:41 |
| openstackgerrit | Merged openstack-infra/system-config master: Have ansible retry ssh connections https://review.openstack.org/593728 | 20:41 |
| openstackgerrit | Merged openstack-infra/system-config master: Add local copy of pkg_mgr fact module https://review.openstack.org/593729 | 20:41 |
| openstackgerrit | Doug Hellmann proposed openstack-infra/project-config master: remove job settings for sahara repositories https://review.openstack.org/593938 | 20:41 |
| ssbarnea | that is what I was hoping for, to get some feedback once I have a prototype, there were lots of approaches over the last 20 years, but somehow I failed to find one that covers all important use-cases, all of them had serious issues at some point, otherwise I would have adopted one :D | 20:42 |
| mordred | ssbarnea: what? things are complicated? ;) | 20:44 |
| *** r-daneel has quit IRC | 20:44 | |
| ssbarnea | mordred: i am not against ansible callback plugins, almost two years ago I was really into writing a better output callback for ansible *but* after finding more about current architecture and chatting with bcoca I realised that ansible needs a LOT of time to get the logging fixed, anyone that worked on that code knows what I am saying.... that area is a mess. | 20:45 |
| *** holser_ has quit IRC | 20:45 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/system-config master: Ensure apt is used on ubuntu hosts with zypper https://review.openstack.org/593730 | 20:45 |
| ssbarnea | but is getting better, slowly. | 20:45 |
| mordred | ssbarnea: yah - this is true. unfortunately for live-streaming it is unpossible to not use callback plugins | 20:45 |
| fungi | did our stdout streaming implementation (mostly) make it upstream into ansible finally? | 20:46 |
| mordred | not even close to | 20:47 |
| corvus | fungi: no, mordred has an awesome idea of how to do that better. it's partially implemented. | 20:47 |
| mordred | I do have a half-implemented set of patches to change how our stremaing is working | 20:47 |
| mordred | and once that's working it is upstreamable | 20:47 |
| corvus | it is much more upstreamable i think. | 20:47 |
| corvus | mordred and i seem to be in strict agreement on this. :) | 20:47 |
| fungi | neat! | 20:48 |
| mordred | yah - now we just have to get it to work :) | 20:48 |
| corvus | deets | 20:48 |
| fungi | bah, details | 20:48 |
| fungi | if there's one thing i've learned about software, it doesn't need to work... just needs a catch name! | 20:49 |
| fungi | er, catchy | 20:49 |
| ssbarnea | time for me to go to sleep, cc me on any output related work, i will be pleased to help if possible. | 20:49 |
| *** r-daneel has joined #openstack-infra | 20:50 | |
| openstackgerrit | Duc Truong proposed openstack-infra/irc-meetings master: Change Senlin meeting time https://review.openstack.org/593939 | 20:50 |
| ssbarnea | on the funny side, I already have a name for my wrapper, "tee for the web", can anyone guess? It is really catchy. | 20:50 |
| mordred | ssbarnea: my brain is not giving me any reasonable answers ... | 20:52 |
| ssbarnea | the tool is going to be called "wee", and you could do "cmd | wee",... I wonder if someone would find it offensive. | 20:52 |
| mordred | hahahaha | 20:52 |
| *** holser_ has joined #openstack-infra | 21:01 | |
| *** trown is now known as trown|outtypewww | 21:02 | |
| *** jamesmcarthur has quit IRC | 21:04 | |
| *** raissa has quit IRC | 21:05 | |
| *** jamesmcarthur has joined #openstack-infra | 21:07 | |
| *** jamesmcarthur has quit IRC | 21:12 | |
| *** jamesmcarthur has joined #openstack-infra | 21:17 | |
| openstackgerrit | Ian Wienand proposed openstack-infra/system-config master: Update to latest docs PTI https://review.openstack.org/593483 | 21:22 |
| openstackgerrit | Ian Wienand proposed openstack-infra/system-config master: Generate roles documentation https://review.openstack.org/593478 | 21:22 |
| openstackgerrit | Ian Wienand proposed openstack-infra/system-config master: Add kerberos-client role https://review.openstack.org/593486 | 21:22 |
| openstackgerrit | Ian Wienand proposed openstack-infra/system-config master: Add openafs-client role https://review.openstack.org/593487 | 21:22 |
| openstackgerrit | Merged openstack-infra/system-config master: Ensure apt is used on ubuntu hosts with zypper https://review.openstack.org/593730 | 21:23 |
| openstackgerrit | Merged openstack-infra/zuul master: Map file comment line numbers https://review.openstack.org/590442 | 21:24 |
| *** jamesmcarthur has quit IRC | 21:26 | |
| mordred | fungi, corvus, pabelanger: ok. the stack has landed. I think it's time to pull master on bridge and run run_all.sh - yeah? | 21:26 |
| mordred | ianw: ooh,you seem to be up to - I can ping you in those questions too :) | 21:27 |
| *** ansmith has joined #openstack-infra | 21:27 | |
| fungi | yep, seems we're ready? | 21:27 |
| *** rkukura has quit IRC | 21:28 | |
| mordred | I think so | 21:28 |
| fungi | aside from the cr-2 and w-1 changes in that topic, all that's left is the cron stack | 21:28 |
| corvus | i'm ready? | 21:28 |
| *** jamesmcarthur has joined #openstack-infra | 21:28 | |
| corvus | i just wanted to end with '?' cause everyone else did | 21:28 |
| fungi | are you? ;) | 21:28 |
| corvus | i believe i'm as ready as everyone else. ?. | 21:29 |
| fungi | let's get on with it then!?!.; | 21:29 |
| mordred | ok - here we go?!?! | 21:29 |
| mordred | ianw: btw - there is a root screen session on bridge.openstack.org- and also things are logging to /var/log/ansible.log if you're in the mood to watch or tail | 21:29 |
| ianw | ok, i'm around but i have no idea what's going on :) | 21:32 |
| fungi | that's me most days | 21:33 |
| mordred | corvus, fungi: nothing seems to be actually logging in to /var/log/ansible.log - oh, that's because we fixed it and it's actually /var/log/ansible/ansible.log now | 21:37 |
| mordred | ianw: we're running run_all.sh by hand with all the new ansible stuff in it before turning cron back on | 21:38 |
| ianw | heh well that explains why it looked like nothing was happening too :) | 21:38 |
| ianw | the log file | 21:38 |
| fungi | it's all in the log file. you know, the other one | 21:38 |
| *** holser_ has quit IRC | 21:38 | |
| mordred | ianw: I was getting realy disturbed :) | 21:39 |
| fungi | i had to step away for a sec to roll the trash out to the curb. is it already done? or am i looking at the wrong screen session? | 21:39 |
| ianw | btw while it's all scrolling the zypper not on bionic has been a known issue. i don't know what to do about it, my appetite for fixing huge amounts of c++ was very low | 21:40 |
| ianw | fungi: the correct log file seems to still be logging | 21:40 |
| fungi | mmm | 21:42 |
| fungi | i only see one root screen session on bridge per `screen -list` | 21:43 |
| fungi | 5763.pts-5.bridge (08/16/18 18:15:37) (Attached) | 21:44 |
| fungi | seems the screen session was showing activity but not in the window which was displaying for me (does screen -x now do independent window switching between connections?) | 21:46 |
| mordred | wow. I didn't know that | 21:46 |
| mordred | if so - that's pretty cool | 21:46 |
| mordred | so - it took about 15 minutes just to run base | 21:47 |
| corvus | what's our parallelism? | 21:47 |
| *** jamesmcarthur has quit IRC | 21:47 | |
| mordred | we might want to do some experiments with -f 10 values | 21:47 |
| fungi | yeah, if we're targeting a 15-minute pulse still, might be nice to get it completing in well under that | 21:48 |
| *** hashar has quit IRC | 21:48 | |
| mordred | hrm. puppet failed on git04 | 21:48 |
| fungi | if we can't, this might be the time to think about changing the pulse frequency too | 21:48 |
| fungi | and now git02 | 21:49 |
| fungi | i wonder if all the centos servers are going to do this | 21:49 |
| mordred | where does puppet log on those servers? | 21:49 |
| *** rcernin has joined #openstack-infra | 21:50 | |
| fungi | ahh, the create_cgitrepos exec failed | 21:51 |
| mordred | fungi: where did you see that? | 21:51 |
| fungi | /var/log/messages on centos | 21:51 |
| fungi | look for puppet-user | 21:51 |
| fungi | Aug 20 21:47:45 git04 puppet-user[8554]: (/Stage[main]/Openstack_project::Git_backend/Exec[create_cgitrepos]) Dependency Vcsrepo[/etc/project-config] has failures: true | 21:51 |
| fungi | that was on 04 | 21:52 |
| mordred | I see it now | 21:52 |
| mordred | Aug 20 21:47:43 git04 puppet-user[8554]: Execution of '/usr/bin/git fetch origin' returned 128: fatal: unable to access 'https://git.openstack.org/openstack-infra/project-config/': Encountered end of file | 21:52 |
| mordred | well that's less exciting | 21:52 |
| fungi | oh, yep | 21:53 |
| fungi | i just saw that was a dependency failure and scrolled up | 21:53 |
| corvus | http://grafana.openstack.org/d/MYvSHcSiz/git-load-balancer?orgId=1 | 21:53 |
| corvus | something just happened to several git backends and the load balancer noticed | 21:53 |
| *** jamesmcarthur has joined #openstack-infra | 21:54 | |
| fungi | i wonder if we were that something which just happened | 21:54 |
| mordred | we're frequently something which happens | 21:54 |
| corvus | seems to have been a blip, but according to the history, that basically never happens | 21:56 |
| fungi | well, if we run again and there's another blip... correlation | 21:56 |
| mordred | ++ | 21:56 |
| corvus | mordred: default is -f 5 right? | 21:58 |
| fungi | mmm, one other thing i see in the git backend syslog | 21:58 |
| fungi | we're doing apache restarts? | 21:58 |
| corvus | fungi: that would do it | 21:58 |
| mordred | we are? | 21:58 |
| openstackgerrit | Ian Wienand proposed openstack-infra/system-config master: [DNM] testing zuul-sphinx updated installation https://review.openstack.org/593484 | 21:59 |
| fungi | i see apache loading modules and configuration, at least | 21:59 |
| mordred | oh - well - I did see that ansible reported "changed" for selinux-policy install - which in theory restarts systemd | 21:59 |
| mordred | so maybe that also caused apache to restart? | 21:59 |
| fungi | Aug 20 21:47:45 git04 systemd: Starting The Apache HTTP Server... | 22:00 |
| fungi | so, yeah, i'm gonna go with that's what happened | 22:00 |
| mordred | yah | 22:00 |
| corvus | Aug 20 21:47:39 git01 puppet-user[7999]: (/Stage[main]/Httpd/File[/etc/httpd/conf.d/cgit.conf]/ensure) removed | 22:00 |
| corvus | that may have triggered the restart? | 22:01 |
| fungi | saw that. did it trigger a refresh? | 22:01 |
| openstackgerrit | Ian Wienand proposed openstack-infra/openstack-zuul-jobs master: Test OpenAFS client install https://review.openstack.org/589335 | 22:02 |
| mordred | corvus: also - to your question from earlier - yes, -f 5 is the default | 22:02 |
| mordred | corvus: however, in run_all.sh we explicitly pass -f 10 | 22:02 |
| corvus | Aug 20 21:49:10 git01 puppet-user[7999]: (/Stage[main]/Httpd/Service[httpd]) Triggered 'refresh' from 1 events | 22:02 |
| mordred | except for on afs - where we pass -f 1 | 22:02 |
| corvus | mordred: we haven't added bridge to cacti have we? | 22:03 |
| mordred | corvus: no. we should do that | 22:03 |
| corvus | i'll do that | 22:03 |
| openstackgerrit | James E. Blair proposed openstack-infra/system-config master: Add bridge to cacti https://review.openstack.org/593965 | 22:04 |
| corvus | yep. we add our ansible bastion to cacti by putting in in the puppet hiera. | 22:04 |
| mordred | corvus: we also haven't added snmp to bridge | 22:04 |
| fungi | i agree, other than the ensure removed on /etc/httpd/conf.d/cgit.conf (is that a new removal?) i don't see what else would have triggered the httpd service refresh | 22:04 |
| corvus | mordred: oh, i guess that's the next thing to add to base, huh? | 22:05 |
| corvus | mordred: we probably need an snmp module ala exim? | 22:05 |
| fungi | snmp configuration is pretty basic, so sounds good | 22:05 |
| mordred | corvus: and firewall rules - and yes - snmp and firewall rules were next on my list | 22:05 |
| corvus | fungi: yeah: Aug 20 21:49:10 git01 puppet-user[7999]: (/Stage[main]/Httpd/Service[httpd]) Triggered 'refresh' from 1 events | 22:05 |
| corvus | fungi: i have no idea what that file was or why it was removed or why it was removed just now. | 22:05 |
| *** jamesmcarthur has quit IRC | 22:06 | |
| mordred | corvus, fungi: well, git.openstack.org/cgit still works - so that's good | 22:06 |
| corvus | yeah, as do the per-site versions | 22:06 |
| corvus | git.zuul-ci.org etc | 22:06 |
| corvus | or, well, airship does. starlingx returns the whole openstack list. | 22:07 |
| corvus | not sure if that's what it did before or not. | 22:07 |
| *** jamesmcarthur has joined #openstack-infra | 22:09 | |
| corvus | er, sorry i took over the bridge conn. i'll go back to my station now. :) | 22:10 |
| corvus | mordred: maybe try -f 20? | 22:10 |
| corvus | mordred: i'll start on an snmp role | 22:12 |
| fungi | https://git.starlingx.io/ shows me just the stx-.* repos... i wonder if we have one misconfigured backend in the mix | 22:13 |
| fungi | i'll see if i can find one | 22:13 |
| *** jamesmcarthur has quit IRC | 22:14 | |
| corvus | oh | 22:14 |
| corvus | fungi: i got the hostname wrong: http://git.starlingx.org/cgit | 22:14 |
| fungi | hah | 22:14 |
| fungi | that explains it | 22:14 |
| corvus | it explains *that*. it raises other questions. :) | 22:15 |
| fungi | yeah, seems git.starlingx.org is a cname to git.openstack.org | 22:16 |
| mordred | fungi: that seems to not be the best configuration :) | 22:17 |
| mordred | corvus: shall I try another run but with -f 20? or should we wait for snmp? | 22:18 |
| dtroyer | I didn't realise starlingx.org was a thing… I thought it was all starlingx.io | 22:18 |
| corvus | mordred: i'll probably have snmp ready in 15m -- up to you | 22:18 |
| mordred | corvus: ok. I'm looking at iptables to see how bad it'll be to knock that together | 22:19 |
| fungi | dtroyer: yeah, i've asked the osf folks who maintain the dns for that if they can explain why there's a git cname in it | 22:20 |
| fungi | i mean, i have the requisite access to remove the record, but am hesitant to do so until i find out why it's there | 22:20 |
| fungi | http://www.starlingx.org/ is also a broken cname | 22:21 |
| dtroyer | fungi: AFAIK we've only documented/used starlingx.io, I'll see if I can find anyone who knows otherwise | 22:21 |
| corvus | (ftr, i just assumed .org -- i did not get it from docs or anything) | 22:22 |
| corvus | (that was just me typing into a browser location field) | 22:22 |
| ianw | mordred: firefox has decided you are a highlight and every time i open a tab you should be glaring at me -- https://imgur.com/a/NT7IvEl | 22:22 |
| mordred | ianw: hahahahaha | 22:22 |
| corvus | i have that picture of mordred hanging on my office wall to inspire me in my work | 22:23 |
| dtroyer | corvus: I figured… and it's likely others will do that too, maybe we want to support the redirection? not sure I have an opinion... | 22:23 |
| fungi | creepy | 22:23 |
| corvus | dtroyer: if so, i'm sure we can come up with a simple static server to do redirects like that | 22:23 |
| *** r-daneel has quit IRC | 22:24 | |
| corvus | fungi: *inspiring* | 22:24 |
| dtroyer | wfm | 22:24 |
| fungi | inspiringly creepy | 22:24 |
| corvus | [descriptions of my office may or may not reflect reality] | 22:24 |
| fungi | my office itself does not reflect reality | 22:25 |
| corvus | fungi: i assume yours is bigger on the inside | 22:25 |
| fungi | it's one of the reasons i can stand to be in it for long periods | 22:25 |
| fungi | corvus: yes, but the outside is stuck camoflaged as a pipe organ | 22:25 |
| corvus | i've always imagined an entire data center sort of tucked into the corner in the 4th dimension | 22:25 |
| fungi | tends to be a bit conspicuous on the streets of londontown | 22:26 |
| mordred | we're not supporting centos6 anymore right? | 22:28 |
| fungi | i'm certainly not. dunno about you | 22:28 |
| ianw | no | 22:28 |
| mordred | aewsome | 22:32 |
| ianw | finger://ze08.openstack.org/acbc8156cda7433985287a31180f1ef7 : POST_FAILURE in 46s ... what's the deal with that? | 22:32 |
| openstackgerrit | James E. Blair proposed openstack-infra/system-config master: Add snmpd role and add it to base https://review.openstack.org/593969 | 22:34 |
| ianw | pabelanger: do any recent changes to epel enabling come to mind? the kerberos roles worked in zuul-jobs, now i've moved them to system-config and it fails missing kstart package, which is apparently epel | 22:34 |
| corvus | ianw: you can grep for acbc8156cda7433985287a31180f1ef7 in /var/logs/zuul/executor-debug.log on ze08 to find the error | 22:35 |
| ianw | also all the tumbleweed tests seem to fail in validate-host as well | 22:36 |
| ianw | corvus: hrm ... http://paste.openstack.org/show/728466/ | 22:37 |
| mordred | corvus, fungi: halfway through writing the iptables role - but the start/stop thing is a little bit of a mess | 22:40 |
| mordred | would y'all mind looking at https://github.com/geerlingguy/ansible-role-firewall - it seems like it might actually do what we're wanting - although it does it a bit differently | 22:41 |
| fungi | mordred: you mean because stop isn't a thing and start/restart/reload just apply the current ruleset? | 22:41 |
| mordred | fungi: yah - our puppet has a hack in place to cause puppet to do a start/stop rather than restart | 22:43 |
| mordred | the role there seems to accomplish the same thing by writing out a start script and making its own systemd unit file that runs the script on start and does a flush on stop | 22:43 |
| mordred | the parameters it takes are remarkably similar to the ones we use | 22:44 |
| mordred | so it seems like it might be a useable replacement | 22:44 |
| *** jamesmcarthur has joined #openstack-infra | 22:45 | |
| fungi | except this is basically replacing the iptables-persistent/netfilter-persistent implementation on debuntu i guess? i don't see where it's installing any packages | 22:45 |
| fungi | also the firewall management tools it knows how to remove/disable aren't the ones we're using, so i think we'd have some extra work to do around that as well | 22:46 |
| mordred | yah - it only installs iptables | 22:46 |
| mordred | I can also just finish the role I'm writing | 22:46 |
| corvus | mordred: it doesn't have something like our 'allowed_hosts': http://git.openstack.org/cgit/openstack-infra/system-config/tree/manifests/site.pp#n211 | 22:47 |
| mordred | I was mostly just concerned with the fact that we apparently have to hack around iptables-persistent not providing a working restart | 22:47 |
| fungi | it might be better if we didn't treat iptables-persistent like a "service" and just called its initscript to reload configuration | 22:47 |
| corvus | mordred: so if we did that, we'd have to put all of those into iptablem command line things. which is fine. we used to do that with puppet. | 22:48 |
| corvus | at least it'd be a list :) | 22:48 |
| corvus | fungi, mordred: yeah, that's an option -- our notify handler could just do that rather than trying to do something crazy with service | 22:48 |
| mordred | fungi: what's the init script restart command we sould call? | 22:48 |
| mordred | and is it the same on xenial? | 22:49 |
| mordred | (with the systemd) | 22:49 |
| mordred | I like that idea though | 22:49 |
| *** jamesmcarthur has quit IRC | 22:50 | |
| corvus | ianw: 2018-08-20 22:03:45,325 DEBUG zuul.AnsibleJob: [build: acbc8156cda7433985287a31180f1ef7] Ansible output: b'execvp ansible-playbook: No such file or directory' | 22:50 |
| corvus | ianw: that's also from that job | 22:50 |
| corvus | ianw: something went very wrong? | 22:50 |
| fungi | i think the initscript name changed between trusty and xenial? but on xenial and trusty pretty sure it's netfilter-persistent now. checking | 22:50 |
| corvus | ianw: Aug 20 22:04:04 ze08 puppet-user[12318]: (/Stage[main]/Zuul/Exec[install_zuul]) Triggered 'refresh' from 1 events | 22:51 |
| fungi | mordred: confirmed... on bionic and xenial it's /etc/init.d/netfilter-persistent while on trusty it's /etc/init.d/iptables-persistent instead | 22:51 |
| corvus | ianw: perhaps we upgraded ansible during that build | 22:52 |
| mordred | fungi: awesome | 22:52 |
| mordred | so - I don't think I ahve the brainpower to finish writing this iptables role tonight | 22:52 |
| fungi | start|restart|reload|force-reload perform load_rules | 22:53 |
| mordred | I'll pick it back up tomorrow | 22:53 |
| fungi | so any of those as an argument will do the trick | 22:53 |
| corvus | ianw: 2.5.8 released 4 days ago, and puppet has been frozen for longer than that, so i think that's the issue | 22:53 |
| fungi | stop does nothing at all, just returns | 22:53 |
| mordred | corvus: ah - yah - that sounds like it | 22:53 |
| mordred | corvus, fungi: how are y'all feeling about enabling the cron job? | 22:53 |
| fungi | it also has special "save" and "flush" arguments which write out the rules from what's running, and wipe all rules. neither are called under normal operation | 22:54 |
| corvus | mordred: ask ianw? :) but from my pov, i think it's tested enough to warrant enabling. | 22:54 |
| *** yamahata has quit IRC | 22:54 | |
| fungi | i'm good merging it, and will be around and awake and reasonably sober for a while still | 22:55 |
| mordred | fungi, ianw: cool - I'll let one of you do the honors of +A - we'll need to manually run run_all one more time to pick it up once it's merged | 22:56 |
| fungi | looking | 22:56 |
| mordred | https://review.openstack.org/#/c/591152/ | 22:56 |
| fungi | mordred: where is the git pull for updated system-config? | 22:57 |
| fungi | i don't see it in there | 22:57 |
| fungi | looks like it runs /opt/system-config/production/run_all.sh without updating /opt/system-config/production | 22:58 |
| *** yamahata has joined #openstack-infra | 22:58 | |
| mordred | fungi: run_all.sh runs update-system-config playbook as its first place | 23:07 |
| mordred | play | 23:07 |
| *** tosky has quit IRC | 23:08 | |
| fungi | in fact, it looks like /opt/system-config/production/run_all.sh doesn't exist at all. there's a /opt/system-config/run_all.sh though | 23:10 |
| ianw | corvus: sorry, just back from school run, so it was just bad timing with the update run? | 23:12 |
| fungi | mordred: so assuming that's a typo in 591152 i think i mostly grok the rest... but this means that changes to run_all.sh will always be delayed by one pulse because update-system-config.yaml isn't run directly from cron before the script is invoked and the script doesn't seem to attempt to re-exec itself, right? | 23:13 |
| *** sthussey has quit IRC | 23:14 | |
| *** agopi_ has joined #openstack-infra | 23:17 | |
| *** agopi has quit IRC | 23:20 | |
| fungi | ianw: not yet, i had some questions about 591152 so it's not merged yet | 23:20 |
| ianw | fungi: oh sorry i'm talking about the weird internal ansible issues that manifest as the POST_FAILURES. i think that's because of the prior manual runs updating ansible on exectuors | 23:21 |
| *** tpsilva has quit IRC | 23:22 | |
| mordred | fungi: ah! we need to rebase that patch, yes | 23:24 |
| openstackgerrit | Monty Taylor proposed openstack-infra/system-config master: Start running puppet cron on bridge.openstack.org https://review.openstack.org/591152 | 23:26 |
| mordred | fungi: and yes- changes to run_all.sh will always be delayed by one pulse | 23:27 |
| mordred | fungi: I think the solution to that for in the future will be when zuul is running it | 23:27 |
| fungi | wfm, just making sure i was following correctly | 23:28 |
| mordred | thanks for catching the production there ... I'd forgotten we hadn't rebased that one when we did the production-ectomy | 23:28 |
| *** markvoelker has joined #openstack-infra | 23:29 | |
| fungi | np | 23:30 |
| fungi | thanks for writing this massive pile of changes | 23:30 |
| fungi | and for keeping them in such discrete, reviewable chunks | 23:31 |
| openstackgerrit | Monty Taylor proposed openstack-infra/system-config master: WIP Add iptables role https://review.openstack.org/593973 | 23:31 |
| *** markvoelker has quit IRC | 23:32 | |
| mordred | fungi: with this bad-boy it's the only way I can remember what's going on :) | 23:33 |
| openstackgerrit | Monty Taylor proposed openstack-infra/system-config master: Move roles into roles dir https://review.openstack.org/590753 | 23:34 |
| mordred | ianw: there you go - that should go hand-in-hand with your roles patches | 23:34 |
| ianw | mordred: ^ is that ready to go? | 23:35 |
| mordred | ianw: yah. should be a noop change | 23:35 |
| ianw | i actually put up https://review.openstack.org/#/c/593478/ yesterday to build the role documentation | 23:36 |
| mordred | ooh sweet | 23:36 |
| ianw | needs some things in zuul-sphinx though, cause the roles were nested in | 23:36 |
| ianw | my initial thought was that we were separating them between "generic" and fairly infra-control-plane specific | 23:37 |
| mordred | oh, no - sorry - I just hadn't finished moving them - I think we can put them all in there - then if any of them wind up being nice - like your afs-client and the exim role - once we're happy with them and everythign is stable we could split out ansible-role repos for them | 23:38 |
| mordred | I think the main concern is not creating new repos-per-role while we're still figuring things out | 23:38 |
| mordred | cause then that's a lot of work | 23:39 |
| *** agopi_ is now known as agopi | 23:39 | |
| ianw | ok, yeah ... i'm not sure the puppet-* repos have been that successful. there's not a lot of eyes on them and discoverability is low. IMO it seems better to work with a monorepo while practical, as i think the cool kids like to call it | 23:43 |
| ianw | i'll rebase the docs change. i still the the python2 zuul-sphinx utf-8 fixes are required | 23:44 |
| mordred | sweet. I'm excited about that docs change | 23:45 |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!