| fungi | yeah, with the ability to securely supply secrets directly in configuration, i don't suppose we need to bother with having sysadmins create them at this point | 00:01 |
|---|---|---|
| fungi | yay us! | 00:01 |
| tonyb | Nice work | 00:01 |
| tonyb | automating/documenting yourselves out ot (that) job :) | 00:01 |
| *** wolverineav has joined #openstack-infra | 00:03 | |
| *** markvoelker has quit IRC | 00:06 | |
| *** rascasoft has joined #openstack-infra | 00:09 | |
| *** jamesmcarthur has joined #openstack-infra | 00:10 | |
| clarkb | re zuul-preview, https://review.openstack.org/#/c/636238/ has plenty +2's I'll let you decide if/when the zuul-discuss discussion reaches a conclusion to approve? | 00:15 |
| clarkb | corvus: er ^ was for you | 00:15 |
| *** jamesmcarthur has quit IRC | 00:16 | |
| *** eernst has joined #openstack-infra | 00:16 | |
| corvus | clarkb: ack thanks, will do | 00:17 |
| *** david-lyle has joined #openstack-infra | 00:18 | |
| *** eernst_ has joined #openstack-infra | 00:18 | |
| *** manjeets has quit IRC | 00:18 | |
| *** manjeets has joined #openstack-infra | 00:19 | |
| clarkb | as for restarting the scheduler I've apparently got to guide the kids through valentines cards as this is a required homework activity from school (who knew) so I'll probably have to do that tomorrow as I won't be bale to look at computer shortly | 00:19 |
| *** tosky has quit IRC | 00:20 | |
| fungi | i still need to go obtain at least one obligatory valentines card. i suppose i'll find time to make that happen at some point tomorrow | 00:20 |
| *** eernst has quit IRC | 00:21 | |
| *** dklyle has quit IRC | 00:21 | |
| fungi | and pretend i don't know that i'm just propping up the greeting card industry and their money-making holiday invention schemes | 00:21 |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/zuul master: web: remove build and job_name filter from the buildset route https://review.openstack.org/636504 | 00:22 |
| openstackgerrit | Merged openstack-infra/project-config master: Remove openstack/instack https://review.openstack.org/635221 | 00:24 |
| openstackgerrit | Merged openstack-infra/project-config master: Move networking-ansible jobs to its project https://review.openstack.org/635836 | 00:25 |
| *** sdake has quit IRC | 00:26 | |
| clarkb | the next puppet run (~20 minutes?) will apply the logging config change on zuul01 | 00:27 |
| *** rascasoft has quit IRC | 00:27 | |
| clarkb | but ya I need to go deal with cardstock | 00:27 |
| fungi | and then we want to perform a scheduler restart after that? | 00:27 |
| clarkb | fungi: yup | 00:27 |
| fungi | just the scheduler? | 00:27 |
| clarkb | that should get us the improved logging from corvus' change as well as the logging config debug logging for github3 | 00:27 |
| clarkb | fungi: ya the scheduler is the only thing that talks to github I think | 00:28 |
| clarkb | (github talks to web, but that isn't where we changed logging) | 00:28 |
| fungi | do zuul-web and/or zuul-fingergw also need restarting when the scheduler restarts? | 00:28 |
| fungi | or are they tolerant of scheduler restarts these days? | 00:28 |
| clarkb | fungi: In the past zuul-web has needed it, unsure if that is still required. Pretty easy to restart it if so | 00:28 |
| fungi | so just do our pipeline dump, scheduler stop, scheduler start, pipeline reenqueue dance and also double-check the webui is still returning content? | 00:29 |
| clarkb | that sounds right to me | 00:29 |
| clarkb | fungi: grepping for github_io in /etc/zuul/logging.conf should confirm it was updated as expected too | 00:31 |
| fungi | thanks | 00:31 |
| openstackgerrit | Merged openstack-infra/project-config master: Add new python 3 Neutron jobs to grafana dashboard https://review.openstack.org/636359 | 00:31 |
| openstackgerrit | Merged openstack-infra/project-config master: Announce changes to governance-{sigs,website} on IRC https://review.openstack.org/635625 | 00:31 |
| *** armax has joined #openstack-infra | 00:32 | |
| *** rascasoft has joined #openstack-infra | 00:41 | |
| *** eernst_ has quit IRC | 00:42 | |
| *** sdake has joined #openstack-infra | 00:46 | |
| *** openstackgerrit has quit IRC | 00:52 | |
| *** jamesmcarthur has joined #openstack-infra | 00:53 | |
| *** ekcs has joined #openstack-infra | 00:54 | |
| *** jamesmcarthur has quit IRC | 00:54 | |
| *** mriedem is now known as mriedem_away | 00:57 | |
| *** rascasoft has quit IRC | 00:57 | |
| *** mriedem_away has quit IRC | 01:00 | |
| *** markvoelker has joined #openstack-infra | 01:03 | |
| *** rascasoft has joined #openstack-infra | 01:03 | |
| *** jamesmcarthur has joined #openstack-infra | 01:04 | |
| *** gyee has quit IRC | 01:07 | |
| *** hwoarang has quit IRC | 01:09 | |
| *** gyee has joined #openstack-infra | 01:10 | |
| *** hwoarang has joined #openstack-infra | 01:13 | |
| *** sdake has quit IRC | 01:14 | |
| *** rascasoft has quit IRC | 01:16 | |
| *** sdake has joined #openstack-infra | 01:17 | |
| *** diablo_rojo has quit IRC | 01:17 | |
| *** rascasoft has joined #openstack-infra | 01:27 | |
| *** stakeda has joined #openstack-infra | 01:27 | |
| ianw | well that's service; acme.sh can now get certs via rax dns integration -> https://github.com/Neilpang/acme.sh/commit/d30b441ede3d2907c0645cf4dea78d740c2a6f08 | 01:29 |
| *** sdake has quit IRC | 01:29 | |
| *** jamesmcarthur has quit IRC | 01:30 | |
| *** krasmussen has quit IRC | 01:30 | |
| pabelanger | that is alot of sh | 01:32 |
| *** sdake has joined #openstack-infra | 01:32 | |
| ianw | pabelanger: i've seen a lot of sh in my day :) they've made some good choices | 01:33 |
| *** jamesmcarthur has joined #openstack-infra | 01:33 | |
| pabelanger | :) | 01:34 |
| ianw | #status wrt per prior update, removed acme-opendev.org from openstackci user's domains in rax | 01:34 |
| openstackstatus | ianw: unknown command | 01:34 |
| fungi | infra-root: pbr freeze says we've installed corvus's logging improvements on zuul01 now, and a grep in the logging.conf indicates clarkb's github3 debug logging change is applied so i'll proceed with the pipeline dump and scheduler restart now unless there are objections | 01:34 |
| ianw | #status log wrt per prior update, removed acme-opendev.org from openstackci user's domains in rax | 01:34 |
| openstackstatus | ianw: finished logging | 01:35 |
| *** rascasoft has quit IRC | 01:35 | |
| *** markvoelker has quit IRC | 01:35 | |
| ianw | it does seem that it's a pretty common trope to use this CNAME thing where you point, say, _acme-challenge.review.opendev.org -> acme-opendev.org and put the txt records in there | 01:38 |
| ianw | it seems like a good idea because you don't fiddle production dns, so any screwups will only affect cert renewals | 01:38 |
| fungi | that seems racy if you're setting more than one service at a time | 01:39 |
| *** hamzy has joined #openstack-infra | 01:39 | |
| fungi | er, setting up | 01:39 |
| *** sdake has quit IRC | 01:39 | |
| ianw | fungi: AIUI, you can have multiple txt records ... it's what's done for wildcards | 01:40 |
| fungi | though something along those lines might make it easier to use dynamic dns updating without risking opening your primary domain up to it | 01:40 |
| fungi | ianw: oh, you mean round-robin (not wildcards presumably)? | 01:41 |
| fungi | if acme supports finding at least one record with the expected challenge response then i guess that would work | 01:41 |
| ianw | fungi: no i mean for acme v2 wildcard support; you add two txt records | 01:43 |
| fungi | oh, not dns wildcard records | 01:43 |
| ianw | no, sorry, wildcard certs | 01:43 |
| fungi | makes sense. they would want to verify you can create arbitrary dns names within the scope of the wildcard, yes | 01:44 |
| ianw | but your point is taken .. you can't just assume put in txt record / clear txt record without taking some not of which one is yours | 01:44 |
| ianw | s/not/note/ | 01:44 |
| *** rascasoft has joined #openstack-infra | 01:45 | |
| *** gyee has quit IRC | 01:47 | |
| *** armax has quit IRC | 01:51 | |
| *** ramishra has joined #openstack-infra | 01:52 | |
| *** jamesmcarthur has quit IRC | 01:53 | |
| *** jamesmcarthur has joined #openstack-infra | 01:54 | |
| ianw | the very nice thing about acme.sh is that it requires almost nothing (unlike a full virtualenv for certbot, etc). so an ansible job on bridge could run on the host and get the token, then add it to bind (on the acme only domain, which isn't version controlled), then go back to the host to renew and there's your cert | 01:58 |
| clarkb | oh thats an interesting point re acme dedicated zone | 01:59 |
| clarkb | we could bypass code review there | 01:59 |
| ianw | yeah, all that would be touching it is this theoretical ansible job to add/remove txt records | 01:59 |
| fungi | yep, either serve it directly from the broker and delegate to it from the parent zone, or set it up as a ddns writeable sandbox zone | 02:00 |
| ianw | and the actual CNAME redirect on the "real" domain *is* still version controlled | 02:00 |
| fungi | okay, hearing no objections, i'll give the release channel a heads up and start in on the zuul scheduler restart | 02:01 |
| clarkb | fungi we've made boxes and are now coloring carss | 02:02 |
| *** lathiat has joined #openstack-infra | 02:02 | |
| ianw | *and* with acme.sh having rax dns support, i think we could have a pretty trivial cron job on bridge.o.o that just dumps cert files into secret storage, for distribution via normal channels | 02:02 |
| fungi | have fun! | 02:02 |
| ianw | for existing hosts. but nice to just have one tool | 02:02 |
| *** jamesmcarthur has quit IRC | 02:05 | |
| ianw | i was never really that happy with v1 of the spec. i'm feeling a lot more confident about these ideas | 02:05 |
| fungi | zuul looks like it's still checking gerrit repos, so giving it a minute | 02:05 |
| fungi | before i reenqueue everything | 02:05 |
| fungi | and submitting cat jobs | 02:05 |
| fungi | the status page is still just saying "fetching info..." so presumably zuul-web will need a restart too | 02:06 |
| fungi | looks like the scheduler has reached a steady state, so reenqueuing check and gate now | 02:08 |
| fungi | reenqueue is completed, status page is still contentless, so i'll restart zuul-web next | 02:11 |
| fungi | that seems to have solved it, after i deleted the stale pidfile which was preventing it from starting | 02:14 |
| fungi | #status log zuul scheduler restarted for debug logging enhancements | 02:14 |
| openstackstatus | fungi: finished logging | 02:15 |
| *** jamesmcarthur has joined #openstack-infra | 02:16 | |
| *** wolverineav has quit IRC | 02:18 | |
| *** wolverineav has joined #openstack-infra | 02:19 | |
| *** hongbin has joined #openstack-infra | 02:21 | |
| *** rascasoft has quit IRC | 02:21 | |
| *** armax has joined #openstack-infra | 02:23 | |
| *** wolverineav has quit IRC | 02:24 | |
| *** rascasoft has joined #openstack-infra | 02:28 | |
| *** jamesmcarthur has quit IRC | 02:28 | |
| *** ekcs has quit IRC | 02:37 | |
| *** jamesmcarthur has joined #openstack-infra | 02:59 | |
| *** jamesmcarthur has quit IRC | 03:06 | |
| *** sdake has joined #openstack-infra | 03:06 | |
| *** rascasoft has quit IRC | 03:10 | |
| *** hwoarang has quit IRC | 03:19 | |
| *** hwoarang has joined #openstack-infra | 03:20 | |
| *** weshay has quit IRC | 03:21 | |
| *** rascasoft has joined #openstack-infra | 03:24 | |
| *** rlandy has quit IRC | 03:24 | |
| *** wolverineav has joined #openstack-infra | 03:25 | |
| *** jamesmcarthur has joined #openstack-infra | 03:26 | |
| *** jamesmcarthur has quit IRC | 03:30 | |
| *** rascasoft has quit IRC | 03:34 | |
| *** rascasoft has joined #openstack-infra | 03:39 | |
| *** eernst has joined #openstack-infra | 03:52 | |
| *** spsurya has joined #openstack-infra | 03:55 | |
| *** udesale has joined #openstack-infra | 03:55 | |
| *** hwoarang has quit IRC | 04:00 | |
| *** hwoarang has joined #openstack-infra | 04:03 | |
| *** ykarel|away has joined #openstack-infra | 04:04 | |
| *** eernst has quit IRC | 04:07 | |
| *** rascasoft has quit IRC | 04:07 | |
| *** ykarel|away is now known as ykarel | 04:08 | |
| *** rascasoft has joined #openstack-infra | 04:11 | |
| *** sdake has quit IRC | 04:15 | |
| *** rascasoft has quit IRC | 04:22 | |
| *** openstackgerrit has joined #openstack-infra | 04:24 | |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/zuul master: web: add buildsets page https://review.openstack.org/630041 | 04:24 |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/zuul master: web: add /{tenant}/buildset/{uuid} route https://review.openstack.org/630078 | 04:24 |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/zuul master: web: add buildset page https://review.openstack.org/630079 | 04:25 |
| *** rascasoft has joined #openstack-infra | 04:32 | |
| *** janki has joined #openstack-infra | 04:32 | |
| *** rascasoft has quit IRC | 04:42 | |
| *** armax has quit IRC | 04:48 | |
| *** rascasoft has joined #openstack-infra | 04:49 | |
| *** hongbin has quit IRC | 04:55 | |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/zuul master: web: add /{tenant}/buildset/{uuid} route https://review.openstack.org/630078 | 05:08 |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/zuul master: web: add buildset page https://review.openstack.org/630079 | 05:08 |
| tonyb | I have created what I intend to be the service account for the sandbox 'bot'. It's called openstack-bot-reviewer | 05:13 |
| tonyb | Just incase I've done soemthing terrible I thought I'd let y'all know the name so if it shows up you know what it is and who to blame ;P | 05:14 |
| tonyb | tomorrow I'll finish the implementation and work to getting it added to project-config | 05:15 |
| *** zul has quit IRC | 05:26 | |
| *** rascasoft has quit IRC | 05:28 | |
| *** udesale has quit IRC | 05:36 | |
| *** rascasoft has joined #openstack-infra | 05:41 | |
| *** udesale has joined #openstack-infra | 05:42 | |
| *** udesale has quit IRC | 05:48 | |
| *** udesale has joined #openstack-infra | 05:48 | |
| *** psachin has joined #openstack-infra | 05:49 | |
| *** rascasoft has quit IRC | 05:49 | |
| *** rascasoft has joined #openstack-infra | 05:53 | |
| *** markvoelker has joined #openstack-infra | 06:00 | |
| *** markvoelker has quit IRC | 06:04 | |
| *** rascasoft has quit IRC | 06:06 | |
| openstackgerrit | OpenStack Proposal Bot proposed openstack-infra/project-config master: Normalize projects.yaml https://review.openstack.org/636551 | 06:08 |
| *** sshnaidm is now known as sshnaidm|afk | 06:22 | |
| *** sdake has joined #openstack-infra | 06:28 | |
| *** jtomasek has joined #openstack-infra | 06:30 | |
| *** kjackal has joined #openstack-infra | 06:39 | |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/zuul master: zuul-runner: add quick-start integration test https://review.openstack.org/635701 | 06:41 |
| *** rascasoft has joined #openstack-infra | 06:41 | |
| *** yboaron_ has joined #openstack-infra | 06:42 | |
| *** wolverineav has quit IRC | 06:45 | |
| *** wolverineav has joined #openstack-infra | 06:46 | |
| *** quiquell|off is now known as quiquell|rover | 06:49 | |
| *** rascasoft has quit IRC | 06:50 | |
| *** diablo_rojo has joined #openstack-infra | 06:50 | |
| *** wolverineav has quit IRC | 06:51 | |
| *** rascasoft has joined #openstack-infra | 06:55 | |
| *** markvoelker has joined #openstack-infra | 07:00 | |
| *** apetrich has joined #openstack-infra | 07:06 | |
| *** sdake has quit IRC | 07:07 | |
| *** apetrich has quit IRC | 07:08 | |
| *** adriant has quit IRC | 07:11 | |
| *** adriant has joined #openstack-infra | 07:11 | |
| *** udesale has quit IRC | 07:12 | |
| *** kjackal has quit IRC | 07:18 | |
| *** wolverineav has joined #openstack-infra | 07:18 | |
| *** udesale has joined #openstack-infra | 07:19 | |
| *** wolverineav has quit IRC | 07:23 | |
| *** udesale has quit IRC | 07:24 | |
| *** udesale has joined #openstack-infra | 07:25 | |
| *** jamesmcarthur has joined #openstack-infra | 07:26 | |
| *** gfidente has joined #openstack-infra | 07:28 | |
| *** aojea has joined #openstack-infra | 07:29 | |
| *** jamesmcarthur has quit IRC | 07:31 | |
| *** markvoelker has quit IRC | 07:34 | |
| *** e0ne has joined #openstack-infra | 07:36 | |
| openstackgerrit | Merged openstack-infra/project-config master: Normalize projects.yaml https://review.openstack.org/636551 | 07:37 |
| openstackgerrit | Quique Llorente proposed openstack-infra/zuul master: Mark as unsafe commit message at inventory https://review.openstack.org/633930 | 07:44 |
| *** kopecmartin|off is now known as kopecmartin | 07:46 | |
| openstackgerrit | Quique Llorente proposed openstack-infra/zuul master: Mark as unsafe commit message at inventory https://review.openstack.org/633930 | 07:47 |
| *** ccamacho has joined #openstack-infra | 07:48 | |
| *** slaweq has joined #openstack-infra | 07:49 | |
| *** kjackal has joined #openstack-infra | 07:51 | |
| *** rpittau has joined #openstack-infra | 07:57 | |
| openstackgerrit | Ian Wienand proposed openstack-infra/infra-specs master: [wip] letsencrypt spec https://review.openstack.org/587283 | 08:06 |
| *** diablo_rojo has quit IRC | 08:10 | |
| *** yboaron_ has quit IRC | 08:15 | |
| openstackgerrit | Ian Wienand proposed openstack-infra/infra-specs master: [wip] letsencrypt spec https://review.openstack.org/587283 | 08:16 |
| openstackgerrit | Quique Llorente proposed openstack-infra/zuul master: Mark as unsafe commit message at inventory https://review.openstack.org/633930 | 08:28 |
| *** markvoelker has joined #openstack-infra | 08:31 | |
| *** wolverineav has joined #openstack-infra | 08:31 | |
| *** wolverineav has quit IRC | 08:35 | |
| *** ykarel is now known as ykarel|lunch | 08:36 | |
| *** tkajinam has quit IRC | 08:36 | |
| *** pgaxatte has joined #openstack-infra | 08:40 | |
| openstackgerrit | Quique Llorente proposed openstack-infra/zuul master: Mark as unsafe commit message at inventory https://review.openstack.org/633930 | 08:42 |
| *** dmsimard has quit IRC | 08:44 | |
| *** dmsimard has joined #openstack-infra | 08:45 | |
| *** tosky has joined #openstack-infra | 08:53 | |
| *** xek has joined #openstack-infra | 08:54 | |
| *** jpena|off is now known as jpena | 08:54 | |
| *** jpich has joined #openstack-infra | 09:03 | |
| openstackgerrit | Felix Schmidt proposed openstack-infra/zuul master: Retrieve full list of jobs with details per tenant via API https://review.openstack.org/635714 | 09:03 |
| *** markvoelker has quit IRC | 09:04 | |
| *** e0ne has quit IRC | 09:06 | |
| *** e0ne has joined #openstack-infra | 09:07 | |
| *** ociuhandu has quit IRC | 09:07 | |
| *** yboaron_ has joined #openstack-infra | 09:08 | |
| *** yamamoto has quit IRC | 09:10 | |
| *** yamamoto has joined #openstack-infra | 09:12 | |
| *** yboaron_ has quit IRC | 09:14 | |
| *** yboaron_ has joined #openstack-infra | 09:14 | |
| openstackgerrit | Jiao Pengju proposed openstack-infra/project-config master: Update karbor to use storyboard https://review.openstack.org/636574 | 09:18 |
| *** ociuhandu has joined #openstack-infra | 09:20 | |
| *** ociuhandu has quit IRC | 09:25 | |
| *** yboaron_ has quit IRC | 09:26 | |
| *** yboaron_ has joined #openstack-infra | 09:26 | |
| *** derekh has joined #openstack-infra | 09:29 | |
| openstackgerrit | Nguyen Hai proposed openstack/gertty master: fix tox python3 overrides https://review.openstack.org/579757 | 09:32 |
| *** luizbag has joined #openstack-infra | 09:33 | |
| *** ociuhandu has joined #openstack-infra | 09:34 | |
| *** e0ne has quit IRC | 09:34 | |
| *** baysao has joined #openstack-infra | 09:38 | |
| *** ociuhandu has quit IRC | 09:38 | |
| *** aojea has quit IRC | 09:40 | |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/zuul master: web: add buildsets page https://review.openstack.org/630041 | 09:42 |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/zuul master: web: add /{tenant}/buildset/{uuid} route https://review.openstack.org/630078 | 09:42 |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/zuul master: web: add buildset page https://review.openstack.org/630079 | 09:42 |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/nodepool master: Implement an OpenShift Pod provider https://review.openstack.org/590335 | 09:46 |
| *** aojea has joined #openstack-infra | 09:47 | |
| *** ykarel|lunch is now known as ykarel | 09:49 | |
| *** sshnaidm|afk is now known as sshnaidm | 09:59 | |
| *** adrianreza has joined #openstack-infra | 09:59 | |
| *** markvoelker has joined #openstack-infra | 10:02 | |
| *** markmcd has joined #openstack-infra | 10:07 | |
| *** baysao has quit IRC | 10:11 | |
| *** ociuhandu has joined #openstack-infra | 10:12 | |
| *** wolverineav has joined #openstack-infra | 10:20 | |
| *** markmcd has left #openstack-infra | 10:22 | |
| *** yamamoto has quit IRC | 10:30 | |
| *** gfidente has quit IRC | 10:31 | |
| *** markvoelker has quit IRC | 10:34 | |
| strigazi | Hello infra-core, I can't unblock this https://review.openstack.org/#/c/561858/ | 10:42 |
| strigazi | oh, it just started the gate jobs | 10:43 |
| *** e0ne has joined #openstack-infra | 10:46 | |
| *** e0ne_ has joined #openstack-infra | 10:50 | |
| *** gfidente has joined #openstack-infra | 10:51 | |
| *** e0ne has quit IRC | 10:52 | |
| *** wolverineav has quit IRC | 10:52 | |
| frickler | infra-root: seems I cannot ssh into wiki.o.o, can someone check whether my account exists there? or is this host special somehow? someone reported login issues in #openstack and I wanted to check logs | 10:52 |
| zbr | frickler: i was able to login there using ubuntu openid. | 10:55 |
| jhesketh | frickler: it doesn't appear that you have a user on that host | 10:56 |
| jhesketh | frickler: might be worth checking if bridge is managing it properly | 10:56 |
| frickler | there also seem to be no data for it on cacti, which may or may not be related | 10:58 |
| *** rascasoft has quit IRC | 10:59 | |
| *** rascasoft has joined #openstack-infra | 11:01 | |
| *** ociuhandu has quit IRC | 11:02 | |
| *** ociuhandu has joined #openstack-infra | 11:03 | |
| *** yamamoto has joined #openstack-infra | 11:05 | |
| mordred | frickler: yeah - I concur with jhesketh - and wow, the users on that host are a real blast from the past | 11:07 |
| frickler | logging into wiki-dev01 works, so I'm assuming the wiki host is just too old | 11:07 |
| cmurphy | i think it's not really puppeted or we turned off puppet for it | 11:08 |
| mordred | I believe we have it in the disabled list because it was only ever halfway properly puppeted | 11:08 |
| mordred | yeah | 11:08 |
| cmurphy | jinx | 11:08 |
| mordred | maybe we should at least run the base.yaml playbook on it | 11:08 |
| mordred | to keep up with root accounts | 11:08 |
| cmurphy | while you're logging into things infra-root could i request this directory get cleaned out https://specs.openstack.org/openstack/keystone-specs/specs/keystone/backlog/ it has a lot of specs left over that we've since moved to other directories http://git.openstack.org/cgit/openstack/keystone-specs/tree/specs/keystone/backlog and it seems to show up in search engines and confuse our users | 11:11 |
| frickler | mordred: almost completely unrelated: do you plan to add the k8s nodes into cacti? might be useful to be able to watch the load on them. or would you rather do some kind of monitoring inside the pods? | 11:13 |
| *** udesale has quit IRC | 11:13 | |
| frickler | mordred: also did you make progress with the dns issues during k8s-master failure? or was the result that we just treat that as an acceptable risk? | 11:14 |
| *** yamamoto has quit IRC | 11:14 | |
| frickler | infra-root: (sorry for more spam) zuul01 is sending two mails per hour with errors like "ls: cannot access '/var/lib/zuul/backup/zuul.openstack.org-http_*': No such file or directory" | 11:16 |
| mordred | cmurphy: hrm. I can't ssh into specs.openstack.org | 11:20 |
| mordred | oh. yeah I can | 11:20 |
| mordred | I can't ssh in to it from wiki.openstack.org | 11:20 |
| mordred | cmurphy: the whole directory? | 11:21 |
| mordred | frickler: it's a good (and hard) question re: cacti. I think it would be a good idea to - however, the way the ansible from upstream works, the firewall stuff is all managed with openstack security groups - which means our normal ansible setup for opening the correct firewall ports for the snmp doesn't apply | 11:23 |
| mordred | frickler: I haven't come up with a "good" solution yet | 11:24 |
| mordred | frickler: for dns failures during k8s-master failure - I think it's "acceptable risk" for now. the answer will be getting the k8s master set up properly HA, which is going to take a little more work | 11:24 |
| cmurphy | mordred: no just the files that aren't in git | 11:25 |
| mordred | frickler: as long as the k8s master doesn't STAY down things should continue working | 11:25 |
| mordred | cmurphy: ok. so delete the files that are not also in that git dir | 11:27 |
| cmurphy | mordred: yes please | 11:27 |
| mordred | kk. | 11:27 |
| frickler | mordred: it looked to me like it was a basic design decision of infraly to run single master, so would you consider forking it? | 11:27 |
| *** yamamoto has joined #openstack-infra | 11:28 | |
| mordred | cmurphy: mordred@static:/srv/static/specs/keystone-specs/specs/keystone/backlog$ rm application-credentials.html json-web-token.html | 11:28 |
| mordred | frickler: yeah - if they're not interested in accepting support for ha master | 11:29 |
| mordred | frickler: I filed https://github.com/infraly/k8s-on-openstack/issues/47 - but he hasn't responded yet | 11:29 |
| *** markvoelker has joined #openstack-infra | 11:30 | |
| frickler | mordred: ah, I didn't see that, will add a watch ;) | 11:30 |
| cmurphy | mordred: also centralized-policies-delivering-mechanism.html centralized-policies-fetch-cache.html drop-driver-version.html fernet-key-store.html microversions.html online-schema-migration.html policy-default.html project-tags.html unified-delegation.html | 11:38 |
| mordred | cmurphy: hrm. I may be looking in th wrong place | 11:38 |
| mordred | cmurphy: cause I don't see those files | 11:39 |
| *** ccamacho has quit IRC | 11:39 | |
| * cmurphy tries force refresh | 11:39 | |
| cmurphy | i see them in https://specs.openstack.org/openstack/keystone-specs/specs/keystone/backlog/ | 11:40 |
| *** janki has quit IRC | 11:40 | |
| *** janki has joined #openstack-infra | 11:40 | |
| mordred | oh! hang on | 11:41 |
| mordred | cmurphy: /src/static/specs/openstack/keystone-specs - not /src/static/specs/keystone-specs | 11:41 |
| mordred | cmurphy: your list makes much more sense to me now | 11:41 |
| cmurphy | yay | 11:42 |
| mordred | cmurphy: check now - better? | 11:43 |
| cmurphy | mordred: hmm not really | 11:45 |
| AJaeger | cmurphy, mordred : Everything 2017 you should be able to delete... | 11:45 |
| AJaeger | mordred: delete everything that has no 2019 timestamp in that directory | 11:45 |
| AJaeger | or delete everything and ask cmurphy to publish again ;) | 11:46 |
| cmurphy | heh | 11:46 |
| AJaeger | cmurphy: just push a dummy commit and self-merge - and blame me ;) | 11:46 |
| mordred | k. I think I finally found the right dir | 11:47 |
| mordred | it was /srv/static/specs/openstack/keystone-specs/specs/keystone/backlog not /srv/static/specs/openstack/keystone-specs/specs/backlog | 11:47 |
| cmurphy | mordred: yes! except also still need application-credentials.html and json-web-tokens.html rm'd | 11:48 |
| mordred | done | 11:48 |
| cmurphy | yay tyty | 11:49 |
| AJaeger | mordred: your first good deed for the day is done ;) | 11:49 |
| *** wolverineav has joined #openstack-infra | 11:49 | |
| mordred | \o/ | 11:50 |
| *** rpittau has quit IRC | 11:52 | |
| *** roman_g has joined #openstack-infra | 11:53 | |
| *** e0ne has joined #openstack-infra | 11:56 | |
| *** e0ne has quit IRC | 11:57 | |
| *** e0ne_ has quit IRC | 11:57 | |
| *** hamzy has quit IRC | 11:59 | |
| *** panda is now known as panda|lunch | 12:00 | |
| *** sdake has joined #openstack-infra | 12:03 | |
| *** markvoelker has quit IRC | 12:04 | |
| *** ginopc has joined #openstack-infra | 12:04 | |
| *** gfidente has quit IRC | 12:06 | |
| *** e0ne has joined #openstack-infra | 12:08 | |
| *** ccamacho has joined #openstack-infra | 12:09 | |
| fungi | frickler: jhesketh: correct, wiki.openstack.org is a cname to wiki-upgrade-test.openstack.org and has puppet disabled because we still never found time to completely manage mediawiki and all the plugins and configuration from puppet. wiki-dev01.openstack.org _is_ puppeted, as you observed, but there are still some kinks which need to be worked out with it before we could consider using a puppeted | 12:12 |
| fungi | deployment to replace the current production wiki | 12:12 |
| *** e0ne has quit IRC | 12:12 | |
| *** hamzy has joined #openstack-infra | 12:13 | |
| *** sdake has quit IRC | 12:16 | |
| mordred | yay fungi is up with knowledge | 12:18 |
| openstackgerrit | Helen Walsh proposed openstack-infra/project-config master: PowerMax/ VMAX - add create permission to core group https://review.openstack.org/636608 | 12:19 |
| fungi | it was "puppeted" in a sense long ago, because the only manifest applied to it was our server manifest to create accounts and configure exim. i guess running our base playbook against it could accomplish something similar | 12:21 |
| *** e0ne has joined #openstack-infra | 12:22 | |
| *** wolverineav has quit IRC | 12:23 | |
| *** yamamoto has quit IRC | 12:28 | |
| *** rpittau has joined #openstack-infra | 12:29 | |
| *** jpena is now known as jpena|lunch | 12:34 | |
| *** yamamoto has joined #openstack-infra | 12:36 | |
| *** udesale has joined #openstack-infra | 12:50 | |
| *** rlandy has joined #openstack-infra | 12:55 | |
| *** gfidente has joined #openstack-infra | 12:57 | |
| *** janki has quit IRC | 13:00 | |
| *** markvoelker has joined #openstack-infra | 13:01 | |
| *** trown|outtypewww is now known as trown | 13:06 | |
| *** rh-jelabarre has joined #openstack-infra | 13:06 | |
| quiquell|rover | fungi, mordred: Is this ok for merge ? https://review.openstack.org/#/c/633930/ | 13:06 |
| quiquell|rover | AJaeger: ^ | 13:06 |
| *** mriedem has joined #openstack-infra | 13:08 | |
| *** kjackal has quit IRC | 13:08 | |
| *** kjackal has joined #openstack-infra | 13:10 | |
| *** e0ne has quit IRC | 13:11 | |
| *** panda|lunch is now known as panda | 13:12 | |
| *** jamesmcarthur has joined #openstack-infra | 13:14 | |
| *** e0ne has joined #openstack-infra | 13:17 | |
| *** gcutrini has joined #openstack-infra | 13:19 | |
| *** e0ne_ has joined #openstack-infra | 13:20 | |
| *** wolverineav has joined #openstack-infra | 13:20 | |
| *** e0ne has quit IRC | 13:23 | |
| *** ykarel is now known as ykarel|afk | 13:23 | |
| *** janki has joined #openstack-infra | 13:28 | |
| quiquell|rover | join #openstack-telemetry | 13:28 |
| quiquell|rover | damn | 13:28 |
| *** jamesmcarthur has quit IRC | 13:29 | |
| *** weshay has joined #openstack-infra | 13:31 | |
| *** gcutrini has quit IRC | 13:32 | |
| *** markvoelker has quit IRC | 13:34 | |
| *** weshay is now known as weshay|ruck | 13:35 | |
| *** jpena|lunch is now known as jpena | 13:37 | |
| *** eharney has quit IRC | 13:37 | |
| *** luizbag has quit IRC | 13:39 | |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/nodepool master: Implement an OpenShift Pod provider https://review.openstack.org/590335 | 13:39 |
| *** e0ne_ has quit IRC | 13:40 | |
| *** e0ne has joined #openstack-infra | 13:45 | |
| *** gcutrini has joined #openstack-infra | 13:50 | |
| *** jamesmcarthur has joined #openstack-infra | 13:50 | |
| *** e0ne_ has joined #openstack-infra | 13:50 | |
| *** e0ne has quit IRC | 13:51 | |
| *** wolverineav has quit IRC | 13:53 | |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/zuul master: executor: add log_stream_port and log_stream_file settings https://review.openstack.org/535538 | 13:55 |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/zuul master: executor: add log_stream_port and log_stream_file settings https://review.openstack.org/535538 | 13:55 |
| *** e0ne_ has quit IRC | 14:04 | |
| *** sdake has joined #openstack-infra | 14:07 | |
| *** sdake has quit IRC | 14:09 | |
| *** ykarel|afk is now known as ykarel | 14:11 | |
| *** sdake has joined #openstack-infra | 14:11 | |
| *** sdake has quit IRC | 14:13 | |
| *** sdake has joined #openstack-infra | 14:14 | |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/nodepool master: Implement a Runc driver https://review.openstack.org/535556 | 14:15 |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/nodepool master: Implement an OpenShift Pod provider https://review.openstack.org/590335 | 14:17 |
| *** rfolco has quit IRC | 14:18 | |
| *** eharney has joined #openstack-infra | 14:22 | |
| *** sdake has quit IRC | 14:23 | |
| *** dave-mccowan has joined #openstack-infra | 14:25 | |
| *** jamesmcarthur has quit IRC | 14:26 | |
| *** rlandy is now known as rlandy|afk | 14:26 | |
| *** e0ne has joined #openstack-infra | 14:29 | |
| *** dave-mccowan has quit IRC | 14:30 | |
| *** markvoelker has joined #openstack-infra | 14:31 | |
| *** eernst has joined #openstack-infra | 14:32 | |
| *** e0ne_ has joined #openstack-infra | 14:35 | |
| *** e0ne has quit IRC | 14:35 | |
| *** eernst has quit IRC | 14:36 | |
| *** rfolco has joined #openstack-infra | 14:37 | |
| *** lpetrut has joined #openstack-infra | 14:38 | |
| *** gcutrini has quit IRC | 14:40 | |
| *** e0ne_ has quit IRC | 14:41 | |
| *** zul has joined #openstack-infra | 14:43 | |
| *** yamamoto has quit IRC | 14:44 | |
| *** sthussey has joined #openstack-infra | 14:47 | |
| *** ekultails has joined #openstack-infra | 14:49 | |
| *** yamamoto has joined #openstack-infra | 14:49 | |
| *** yamamoto has quit IRC | 14:49 | |
| *** yamamoto has joined #openstack-infra | 14:50 | |
| *** yamamoto has quit IRC | 14:50 | |
| *** wolverineav has joined #openstack-infra | 14:50 | |
| *** e0ne has joined #openstack-infra | 14:50 | |
| *** e0ne_ has joined #openstack-infra | 14:55 | |
| *** jamesmcarthur has joined #openstack-infra | 14:55 | |
| *** e0ne has quit IRC | 14:55 | |
| *** gcutrini has joined #openstack-infra | 14:57 | |
| *** yamamoto has joined #openstack-infra | 15:00 | |
| *** yamamoto has quit IRC | 15:00 | |
| *** verdurin has quit IRC | 15:02 | |
| *** markvoelker has quit IRC | 15:04 | |
| *** verdurin has joined #openstack-infra | 15:05 | |
| *** gcutrini has quit IRC | 15:15 | |
| *** yboaron_ has quit IRC | 15:19 | |
| *** gcutrini has joined #openstack-infra | 15:19 | |
| *** gcutrini has quit IRC | 15:20 | |
| *** janki has quit IRC | 15:21 | |
| *** wolverineav has quit IRC | 15:23 | |
| *** smarcet has joined #openstack-infra | 15:25 | |
| openstackgerrit | Jakub Libosvar proposed openstack-infra/project-config master: Remove rtd_web_hook from networking-ansible https://review.openstack.org/636651 | 15:26 |
| *** rossella_s has quit IRC | 15:26 | |
| *** dayou has quit IRC | 15:29 | |
| *** luizbag has joined #openstack-infra | 15:32 | |
| *** armax has joined #openstack-infra | 15:35 | |
| *** rkukura has joined #openstack-infra | 15:37 | |
| openstackgerrit | sebastian marcet proposed openstack-infra/openstackid-resources master: Update API code to work with Presentation Moderators collection (+N) https://review.openstack.org/636190 | 15:45 |
| *** ekultails has quit IRC | 15:46 | |
| *** ekultails has joined #openstack-infra | 15:47 | |
| *** quiquell|rover is now known as quiquell|off | 15:48 | |
| *** Adri2000 has quit IRC | 15:48 | |
| *** ekultails has quit IRC | 15:48 | |
| *** ekultails has joined #openstack-infra | 15:48 | |
| *** Adri2000 has joined #openstack-infra | 15:51 | |
| *** dayou has joined #openstack-infra | 15:52 | |
| *** gcutrini has joined #openstack-infra | 15:53 | |
| *** ociuhandu has quit IRC | 15:55 | |
| *** ociuhandu has joined #openstack-infra | 15:55 | |
| *** ykarel is now known as ykarel|away | 15:58 | |
| *** lpetrut has quit IRC | 16:00 | |
| *** markvoelker has joined #openstack-infra | 16:01 | |
| *** sdake has joined #openstack-infra | 16:05 | |
| *** ianychoi has quit IRC | 16:08 | |
| openstackgerrit | Matthieu Huin proposed openstack-infra/zuul master: CLI: fail if trying to enqueue/dequeue a change for the wrong project https://review.openstack.org/636662 | 16:11 |
| *** walshh__ has joined #openstack-infra | 16:14 | |
| walshh__ | Hi, to the to the infra-root sysadmins. Could I be added to the new devstack-plugin-vmax-release. Details in https://review.openstack.org/#/c/635890/ | 16:16 |
| walshh__ | Thank you in advance | 16:16 |
| *** lewo has quit IRC | 16:16 | |
| *** lewo has joined #openstack-infra | 16:17 | |
| clarkb | walshh__: done | 16:17 |
| clarkb | you may need to hard refresh the browser window with branches if you are wanting to create a branch there | 16:18 |
| *** walshh__ has quit IRC | 16:20 | |
| *** wolverineav has joined #openstack-infra | 16:20 | |
| *** walshh__ has joined #openstack-infra | 16:23 | |
| *** gfidente has quit IRC | 16:26 | |
| *** ramishra has quit IRC | 16:29 | |
| *** yamamoto has joined #openstack-infra | 16:30 | |
| *** markvoelker has quit IRC | 16:34 | |
| *** diablo_rojo has joined #openstack-infra | 16:37 | |
| *** yamamoto has quit IRC | 16:38 | |
| *** smarcet has quit IRC | 16:42 | |
| *** sdake has quit IRC | 16:46 | |
| *** sdake has joined #openstack-infra | 16:47 | |
| *** e0ne has joined #openstack-infra | 16:47 | |
| *** e0ne_ has quit IRC | 16:48 | |
| *** jamesmcarthur has quit IRC | 16:51 | |
| *** wolverineav has quit IRC | 16:53 | |
| clarkb | pabelanger: corvus: now that I've sorted out voip.ms account stuff the next item on my pbx list is to upgrade it to xenial. Do I need to prevent the new server from trying to connect to the voip.ms sip server or can they both connect for a short period of time? | 16:53 |
| *** gyee has joined #openstack-infra | 16:55 | |
| *** gcutrini has left #openstack-infra | 16:56 | |
| *** gfidente has joined #openstack-infra | 16:56 | |
| *** ykarel|away has quit IRC | 16:58 | |
| *** sdake has quit IRC | 16:58 | |
| *** pgaxatte has quit IRC | 16:59 | |
| *** eernst has joined #openstack-infra | 17:00 | |
| *** jamesmcarthur has joined #openstack-infra | 17:00 | |
| *** rlandy|afk is now known as rlandy | 17:00 | |
| *** wolverineav has joined #openstack-infra | 17:01 | |
| *** eernst has quit IRC | 17:03 | |
| *** smarcet has joined #openstack-infra | 17:03 | |
| *** ekcs has joined #openstack-infra | 17:04 | |
| corvus | clarkb: they will probably fight over the registration, and therefore it will be unpredictable which receives calls | 17:04 |
| clarkb | ok I can give the new server bad credentials then via hiera I think | 17:05 |
| openstackgerrit | Clark Boylan proposed openstack-infra/system-config master: Prep for pbx upgrade to xenial https://review.openstack.org/636681 | 17:07 |
| clarkb | infra-root ^ is next step in the pbx upgrade if you have a moment for a quick review | 17:07 |
| *** ykarel|away has joined #openstack-infra | 17:09 | |
| *** ociuhandu_ has joined #openstack-infra | 17:11 | |
| *** eumel8 has joined #openstack-infra | 17:13 | |
| *** luizbag_ has joined #openstack-infra | 17:15 | |
| *** luizbag has quit IRC | 17:17 | |
| *** gfidente is now known as gfidente|afk | 17:18 | |
| *** psachin has quit IRC | 17:18 | |
| *** ekcs has quit IRC | 17:19 | |
| *** wolverineav has quit IRC | 17:20 | |
| *** jamesmcarthur has quit IRC | 17:22 | |
| *** udesale has quit IRC | 17:23 | |
| *** rpittau has quit IRC | 17:24 | |
| *** e0ne has quit IRC | 17:25 | |
| *** kopecmartin is now known as kopecmartin|off | 17:26 | |
| *** ykarel|away has quit IRC | 17:31 | |
| pabelanger | clarkb: I agree with corvus | 17:31 |
| *** markvoelker has joined #openstack-infra | 17:32 | |
| clarkb | pabelanger: thanks. I'll just give it dummy credentials when I boot it then switch when everything is ready. That should avoid any problems | 17:32 |
| *** jamesmcarthur has joined #openstack-infra | 17:34 | |
| *** geguileo has joined #openstack-infra | 17:34 | |
| *** dtantsur is now known as dtantsur|afk | 17:37 | |
| *** ociuhand_ has joined #openstack-infra | 17:39 | |
| openstackgerrit | Merged openstack-infra/zuul-jobs master: Rework upload-forge role to use module https://review.openstack.org/635941 | 17:39 |
| *** smarcet has quit IRC | 17:42 | |
| *** ociuhandu has quit IRC | 17:42 | |
| openstackgerrit | Clark Boylan proposed openstack-infra/system-config master: Prep for pbx upgrade to xenial https://review.openstack.org/636681 | 17:44 |
| *** smarcet has joined #openstack-infra | 17:44 | |
| *** michaelxin has quit IRC | 17:45 | |
| *** ociuhand_ has quit IRC | 17:48 | |
| *** gfidente|afk has quit IRC | 17:51 | |
| *** jpich has quit IRC | 17:52 | |
| clarkb | another question I'm pretty sure I kno the answer to is I can create it as pbx01.opendev.org cname pbx.openstack.org to it and SIP shouldn't care right? it doesn't do vhosting or certs which would get mad about that | 17:54 |
| openstackgerrit | Merged openstack-infra/project-config master: Rework to use new upload-forge module https://review.openstack.org/635965 | 17:54 |
| *** derekh has quit IRC | 17:55 | |
| pabelanger | clarkb: should be fine, there is a setting in sip.conf where you can actually setup external DNS if needed, but in our case, i don't think we care much | 17:57 |
| pabelanger | we are only accepting incoming calls | 17:58 |
| *** markvoelker has quit IRC | 18:04 | |
| *** bhavikdbavishi has joined #openstack-infra | 18:07 | |
| *** agopi has quit IRC | 18:09 | |
| *** jpena is now known as jpena|off | 18:13 | |
| openstackgerrit | Jan Kundrát proposed openstack-infra/nodepool master: runc: Use correct interpreter for Ansible https://review.openstack.org/636702 | 18:16 |
| openstackgerrit | Jan Kundrát proposed openstack-infra/nodepool master: runc: Support IPv6 https://review.openstack.org/636703 | 18:16 |
| openstackgerrit | Jan Kundrát proposed openstack-infra/nodepool master: runc: Use the hypervisor's zuul-console location https://review.openstack.org/636704 | 18:16 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul master: Re-use the github PR object when fetching reviews https://review.openstack.org/636705 | 18:16 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul master: Add comment about extra issues request https://review.openstack.org/636706 | 18:17 |
| *** sdake has joined #openstack-infra | 18:19 | |
| *** e0ne has joined #openstack-infra | 18:22 | |
| *** ijw has joined #openstack-infra | 18:23 | |
| *** wolverineav has joined #openstack-infra | 18:23 | |
| *** betherly has joined #openstack-infra | 18:25 | |
| *** smarcet has quit IRC | 18:25 | |
| *** smarcet has joined #openstack-infra | 18:26 | |
| *** wolverineav has quit IRC | 18:27 | |
| *** quiquell|off is now known as quiquell|rover | 18:28 | |
| *** ociuhandu has joined #openstack-infra | 18:29 | |
| *** betherly has quit IRC | 18:29 | |
| *** wolverineav has joined #openstack-infra | 18:29 | |
| *** yamamoto has joined #openstack-infra | 18:31 | |
| *** verdurin has quit IRC | 18:31 | |
| *** ociuhandu has quit IRC | 18:33 | |
| *** trown is now known as trown|lunch | 18:34 | |
| *** kjackal has quit IRC | 18:34 | |
| *** kjackal has joined #openstack-infra | 18:34 | |
| *** aojea has quit IRC | 18:35 | |
| openstackgerrit | Merged openstack-infra/zone-opendev.org master: Add kubernetes nodes to DNS https://review.openstack.org/635485 | 18:36 |
| *** sdake has quit IRC | 18:37 | |
| *** verdurin has joined #openstack-infra | 18:39 | |
| openstackgerrit | James E. Blair proposed openstack-infra/system-config master: Correct paths in rook playbook https://review.openstack.org/636713 | 18:40 |
| corvus | clarkb, fungi: ^ | 18:40 |
| *** bhavikdbavishi has quit IRC | 18:42 | |
| clarkb | corvus: does the local connection preserve PWD? | 18:42 |
| clarkb | corvus: wondering if we should make that a fully rooted path | 18:42 |
| corvus | i'll check | 18:42 |
| corvus | oh, it's the playbook directory, so that's wrong | 18:43 |
| openstackgerrit | James E. Blair proposed openstack-infra/system-config master: Correct paths in rook playbook https://review.openstack.org/636713 | 18:44 |
| corvus | the only issue was they aren't called "rook-" | 18:44 |
| corvus | i missed that in my initial analysis | 18:44 |
| clarkb | ya those paths look good now relative to the playbook that is running. Just not sure if playbook preservers that path on a local instantiation | 18:45 |
| * clarkb assumes it will work and approves | 18:46 | |
| *** ijw has quit IRC | 18:46 | |
| *** betherly has joined #openstack-infra | 18:47 | |
| *** betherly has quit IRC | 18:47 | |
| *** agopi has joined #openstack-infra | 18:50 | |
| *** mriedem is now known as mriedem_away | 18:52 | |
| *** kjackal has quit IRC | 18:52 | |
| *** kjackal_v2 has joined #openstack-infra | 18:52 | |
| corvus | clarkb: yes, i tested that | 18:54 |
| clarkb | https://review.openstack.org/#/c/636681/ passes tests now for the pbx updates if anyone has time for a quick review | 18:56 |
| *** markvoelker has joined #openstack-infra | 19:01 | |
| *** quiquell|rover is now known as quiquell|off | 19:02 | |
| *** wolverineav has quit IRC | 19:04 | |
| *** wolverineav has joined #openstack-infra | 19:08 | |
| *** luizbag_ has quit IRC | 19:14 | |
| *** ociuhandu has joined #openstack-infra | 19:16 | |
| openstackgerrit | Clark Boylan proposed openstack-infra/zuul master: Don't request PR issue data https://review.openstack.org/636728 | 19:18 |
| *** ijw has joined #openstack-infra | 19:26 | |
| *** eernst has joined #openstack-infra | 19:27 | |
| *** eernst has quit IRC | 19:28 | |
| *** ijw has quit IRC | 19:28 | |
| *** ociuhandu has quit IRC | 19:28 | |
| *** eharney has quit IRC | 19:31 | |
| openstackgerrit | Merged openstack-infra/system-config master: Correct paths in rook playbook https://review.openstack.org/636713 | 19:32 |
| openstackgerrit | Clark Boylan proposed openstack-infra/system-config master: Disable github rate limit logging https://review.openstack.org/636734 | 19:33 |
| *** markvoelker has quit IRC | 19:34 | |
| *** eernst has joined #openstack-infra | 19:42 | |
| *** eernst has quit IRC | 19:42 | |
| *** eernst has joined #openstack-infra | 19:42 | |
| *** irclogbot_1 has quit IRC | 19:47 | |
| *** e0ne has quit IRC | 19:47 | |
| *** trown|lunch is now known as trown | 19:52 | |
| openstackgerrit | Clark Boylan proposed openstack-infra/zuul master: Don't request PR issue data https://review.openstack.org/636728 | 19:53 |
| *** jamesmcarthur has quit IRC | 19:55 | |
| *** ijw has joined #openstack-infra | 19:57 | |
| *** irclogbot_1 has joined #openstack-infra | 20:00 | |
| *** ijw has quit IRC | 20:01 | |
| *** smarcet has joined #openstack-infra | 20:02 | |
| *** wolverineav has quit IRC | 20:03 | |
| *** panda is now known as panda|off | 20:04 | |
| fungi | corvus: oh, spotted another domain with dkim signatures invalidated by openstack-discuss: ovh.com is also setting c=simple/simple and mailman's header reindenting runs afoul of that (same as for dell.com) | 20:04 |
| openstackgerrit | Merged openstack-infra/project-config master: Add zuul-preview project https://review.openstack.org/636238 | 20:06 |
| *** jamesmcarthur has joined #openstack-infra | 20:07 | |
| corvus | fungi: ack. we should probably figure out a resolution to that soon. i think we probably have enough data now. but i can't do so atm. | 20:08 |
| openstackgerrit | James E. Blair proposed openstack-infra/project-config master: Add a job to use the intermediate registry https://review.openstack.org/634347 | 20:09 |
| clarkb | ovh seems to host their own email, maybe they are willing to divulge info on what setup they are using that causes the whitespace oddness | 20:10 |
| corvus | unfortunately, they are unaware that there is a problem right now, since we're accepting their messages | 20:11 |
| *** jamesmcarthur has quit IRC | 20:12 | |
| *** zaro has joined #openstack-infra | 20:12 | |
| fungi | clarkb: the simpler solution is to probably ask the likes of ovh and dell to consider setting c=relaxed/relaxed in their dkim-signature headers so that whitespace adjustments in transit don't invalidate their signatures | 20:14 |
| clarkb | fungi: ya was mostly thinking if it is an issue in $smtp service then maybe we can fix it there and everyone gets the update | 20:15 |
| fungi | pnnl's issue is likely a bigger challenge to solve, as they seem to mangle their references header in non-rfc-compliant ways, so when mailman normalizes that it renders the signature invalid | 20:15 |
| clarkb | but that is definitely a straightforward easy fix | 20:15 |
| *** jamesmcarthur has joined #openstack-infra | 20:16 | |
| *** smarcet has quit IRC | 20:24 | |
| *** eharney has joined #openstack-infra | 20:25 | |
| *** ociuhandu has joined #openstack-infra | 20:31 | |
| *** markvoelker has joined #openstack-infra | 20:31 | |
| *** wolverineav has joined #openstack-infra | 20:33 | |
| *** jamesmcarthur has quit IRC | 20:35 | |
| *** eernst has quit IRC | 20:35 | |
| *** wolverineav has quit IRC | 20:38 | |
| *** eernst has joined #openstack-infra | 20:41 | |
| openstackgerrit | Clark Boylan proposed openstack-infra/zuul master: First draft PR caching by commit sha https://review.openstack.org/636751 | 20:50 |
| *** rosmaita has joined #openstack-infra | 20:51 | |
| *** roman_g has quit IRC | 20:51 | |
| *** jtomasek has quit IRC | 21:02 | |
| *** ociuhandu has quit IRC | 21:04 | |
| *** mriedem_away is now known as mriedem | 21:04 | |
| *** wolverineav has joined #openstack-infra | 21:04 | |
| *** markvoelker has quit IRC | 21:05 | |
| *** xek has quit IRC | 21:07 | |
| openstackgerrit | Ian Wienand proposed openstack-infra/system-config master: [dnm] testing... https://review.openstack.org/636759 | 21:11 |
| corvus | clarkb, fungi: https://review.openstack.org/634347 is ready, and should let me start testing the docker registry stuff | 21:12 |
| openstackgerrit | Clark Boylan proposed openstack-infra/zuul master: First draft PR caching by commit sha https://review.openstack.org/636751 | 21:12 |
| clarkb | corvus: cool I'll take a look after lunch | 21:12 |
| fungi | similarly after i get these okonomiyaki turned out | 21:13 |
| openstackgerrit | Ian Wienand proposed openstack-infra/system-config master: [dnm] testing... https://review.openstack.org/636759 | 21:13 |
| openstackgerrit | Ian Wienand proposed openstack-infra/system-config master: [dnm] testing... https://review.openstack.org/636759 | 21:23 |
| *** jamesmcarthur has joined #openstack-infra | 21:25 | |
| *** jamesmcarthur has quit IRC | 21:30 | |
| *** ijw has joined #openstack-infra | 21:30 | |
| *** whoami-rajat has quit IRC | 21:32 | |
| *** memoussati has joined #openstack-infra | 21:36 | |
| *** wolverineav has quit IRC | 21:37 | |
| openstackgerrit | Merged openstack-infra/system-config master: Update to gitea v1.6.3 https://review.openstack.org/629942 | 21:39 |
| memoussati | Good evening to you all, I'm new around here and if possible have a question. | 21:43 |
| ianw | memoussati: always feel free to ask away, if anyone knows the answer we'll respond :) | 21:45 |
| fungi | memoussati: i find it entirely possible that you have a question. it's also possible (perhaps even probable, though don't count on it) that we have an answer. best to just ask and we'll let you know | 21:45 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul master: WIP: alternative github sha cache https://review.openstack.org/636764 | 21:47 |
| memoussati | I encountered a parsing bug in puppet-neutron when using multiple host_routes in one subnet. It was however essential that we use multiple host_routes and documentation would assume that it is possible. We wrote a fix and I was planning on making a PR. Can I assume that if I can't find a story on storyboard regarding this issue that it's not a known bug? | 21:48 |
| memoussati | Thanks for the kind words, ianw and fungi :) | 21:48 |
| memoussati | Version we use is stable/ocata but there were no changes to the parse function of host_routes in master (yet). | 21:50 |
| *** ijw has quit IRC | 21:51 | |
| memoussati | And if you need more context, I hoped I explained the issue with the parsing correctly here: https://github.com/hostnet/puppet-neutron/pull/1 . | 21:51 |
| fungi | memoussati: this sounds like something you want to bring up with the maintainers of the puppet openstack project. you can find them in #puppet-openstack | 21:55 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul master: Re-use the github PR object when fetching reviews https://review.openstack.org/636705 | 21:56 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul master: Add comment about extra issues request https://review.openstack.org/636706 | 21:56 |
| memoussati | Ah thank you, fungi! I followed the Contributor Guide and it brought me here! I will ask around there! | 21:58 |
| fungi | you're welcome. openstack is made up of lots (70+ at last count) of separate projects each managed by their own teams | 21:59 |
| rosmaita | fungi: do you know how often the jobs that publish to docs.o.o run? are they periodic or publish-on-change? | 22:02 |
| *** markvoelker has joined #openstack-infra | 22:02 | |
| *** wolverineav has joined #openstack-infra | 22:03 | |
| *** trown is now known as trown|outtypewww | 22:06 | |
| *** wolverineav has quit IRC | 22:08 | |
| fungi | rosmaita: at least for most projects, when a change merges a docs build is enqueued into the (high priority) post pipeline which should publish within minutes thereafter | 22:09 |
| fungi | rosmaita: you should be able to find them in the builds tab at https://zuul.openstack.org/ | 22:10 |
| fungi | if your docs aren't updating, odds are docs builds have started failing for you | 22:12 |
| rosmaita | fungi: is this the right job: https://zuul.openstack.org/builds?job_name=publish-openstack-sphinx-docs | 22:13 |
| rosmaita | looks like it hasn't run for cinder in a while | 22:14 |
| rosmaita | https://zuul.openstack.org/builds?job_name=publish-openstack-sphinx-docs&project=openstack%2Fcinder | 22:14 |
| *** ijw has joined #openstack-infra | 22:16 | |
| corvus | fungi, clarkb: had tasty food? want to wash it down with https://review.openstack.org/634347 ? | 22:21 |
| clarkb | yup already reviewing it | 22:22 |
| *** openstackgerrit has quit IRC | 22:22 | |
| clarkb | trying to convince myself that the order of operations between pre and post playbooks is right | 22:23 |
| clarkb | for some reason my gut is telling me they are backwards, but now double checkign roles | 22:23 |
| corvus | clarkb: pretty sure they're right, so let me know if you need convincing :) | 22:23 |
| clarkb | no theyr are right | 22:24 |
| clarkb | we pull in pre to grab prebuilt images from other jobs | 22:24 |
| corvus | exactly | 22:24 |
| clarkb | then push at the end of our job to expose our image to other jobs | 22:24 |
| clarkb | and the specifics are zuul return artifacts set by the image builds | 22:24 |
| clarkb | is blocking for an earlier change's images to build something still to be implemented? | 22:25 |
| smcginnis | rosmaita: Looks like those runs were only on stable too. | 22:25 |
| corvus | clarkb: nope, that's in place (provides/requires) | 22:25 |
| clarkb | (we are just within a buildset for now right?) | 22:25 |
| corvus | we can cross buildsets because of provides/requires | 22:26 |
| clarkb | ah | 22:26 |
| corvus | it's a brave new world | 22:26 |
| clarkb | rosmaita: might be easier to go in the other direction. Find a change that should've updated docs and we can see what jobs ran for it | 22:26 |
| smcginnis | rosmaita: Ah - https://review.openstack.org/#/c/613131/ | 22:26 |
| smcginnis | Oh, didn't see what repo that was. Nevermind. | 22:27 |
| rosmaita | clarkb: https://review.openstack.org/#/c/624424/ | 22:27 |
| clarkb | corvus: last question, for the secret https://review.openstack.org/#/c/634347/5/zuul.d/secrets.yaml say we have to move the job like I did with base-jobs where would one find the data to put into that secret? is it part of the registry config on the server? | 22:27 |
| rosmaita | smcginnis: i wonder if we did something bad to .zuul.yaml in master recently | 22:27 |
| rosmaita | i will check history | 22:27 |
| corvus | clarkb: it's in ansible hostvars on bridge | 22:27 |
| clarkb | corvus: thanks | 22:28 |
| *** kjackal_v2 has quit IRC | 22:29 | |
| rosmaita | smcginnis: i don't see anything in cinder/.zuul.yaml | 22:30 |
| rosmaita | also, i think our docs venv in tox.ini isn't actually used by the gate docs tests | 22:31 |
| clarkb | rosmaita: that ran openstack-tox-docs which is part of the publish-openstack-docs-pti template | 22:31 |
| clarkb | so it ran at least half of the expected jobs | 22:31 |
| clarkb | the next question is if maybe that got caught in the zuul restart that fungi did yesterday | 22:31 |
| rosmaita | that could be | 22:31 |
| fungi | also possible, though i think i recall post being empty at the time | 22:32 |
| rosmaita | the patch merged like 23 hours ago | 22:32 |
| clarkb | it may have change merged at 3:08pm PST and fungi restarted things around 5:30pm iirc | 22:32 |
| clarkb | so if it ended up queued for a bit it is possible | 22:32 |
| fungi | post is high priority now i thought? | 22:32 |
| fungi | should have completed almost instantly after merge | 22:32 |
| clarkb | fungi: ya and we do supercedent so queues are shorter so likely not that | 22:33 |
| fungi | 2.5 hours seems an awfully long time given (unless we changed it again) post is at the same priority as gate | 22:33 |
| *** eandersson has quit IRC | 22:33 | |
| fungi | and the jobs in it usually run way, waaaaaay faster too | 22:33 |
| rosmaita | fungi: clarkb: what do you think of what smcginnis noticed on https://zuul.openstack.org/builds?job_name=publish-openstack-sphinx-docs&project=openstack%2Fcinder | 22:34 |
| fungi | i lean toward we've somehow configured that job not to run for changes to master | 22:34 |
| rosmaita | doesn't seem to be running for master | 22:34 |
| rosmaita | what fungi said | 22:34 |
| rosmaita | is that something we could have done by mistake on our end? | 22:34 |
| clarkb | publish-openstack-tox-docs is the job | 22:34 |
| *** markvoelker has quit IRC | 22:34 | |
| clarkb | fwiw | 22:34 |
| rosmaita | ok, i am looking at wrong job | 22:35 |
| clarkb | http://zuul.openstack.org/builds?project=openstack%2Fcinder&pipeline=post&branch=master&job_name=publish-openstack-tox-docs and it last ran January 23 | 22:35 |
| *** agopi_ has joined #openstack-infra | 22:35 | |
| *** gouthamr has quit IRC | 22:35 | |
| *** gouthamr has joined #openstack-infra | 22:36 | |
| *** cburgess has quit IRC | 22:36 | |
| clarkb | other projects have run that job today | 22:36 |
| clarkb | so it isn't globally broken | 22:36 |
| fungi | corvus: yes on tasty food. this time i was tasked with making vegetarian okonomiyaki so had to step up my game with more flavorful liquids in the batter | 22:36 |
| *** ijw has quit IRC | 22:37 | |
| corvus | fungi: that sounds like mostlynomiyaki. | 22:37 |
| *** agopi has quit IRC | 22:37 | |
| *** ijw has joined #openstack-infra | 22:38 | |
| smcginnis | Nothing has run in the cinder post pipeline since January 23. Does that seem right? | 22:38 |
| smcginnis | http://zuul.openstack.org/builds?project=openstack%2Fcinder&branch=master&pipeline=post | 22:38 |
| *** cburgess has joined #openstack-infra | 22:38 | |
| fungi | corvus: too true, too true | 22:38 |
| *** eernst has quit IRC | 22:39 | |
| fungi | smcginnis: unless all of cinder's core reviewers went on vacation at the same time, no that sounds fairly unright | 22:39 |
| clarkb | smcginnis: considering cinder has merged code since then: no | 22:39 |
| *** agopi__ has joined #openstack-infra | 22:40 | |
| rosmaita | clarkb: here is another doc change: https://review.openstack.org/#/c/635009/ | 22:40 |
| rosmaita | 8 days ago | 22:41 |
| *** ijw has quit IRC | 22:42 | |
| *** agopi__ is now known as agopi | 22:42 | |
| *** agopi_ has quit IRC | 22:42 | |
| *** jamesmcarthur has joined #openstack-infra | 22:43 | |
| fungi | so what is the name of the job we think these should be triggering? i think we can find the nak for them in the scheduler debug logs, right? | 22:43 |
| smcginnis | Only thing to touch zuul.yaml in the cinder repo even close to then was renaming integrated-gate-py35 to integrated-gate-py3 | 22:44 |
| *** eernst has joined #openstack-infra | 22:44 | |
| rosmaita | fungi: clarkb said publish-openstack-tox-docs | 22:44 |
| *** memoussati has quit IRC | 22:44 | |
| corvus | [ping me if you need me to jump in] | 22:44 |
| *** openstackgerrit has joined #openstack-infra | 22:44 | |
| openstackgerrit | Merged openstack-infra/project-config master: Add a job to use the intermediate registry https://review.openstack.org/634347 | 22:44 |
| clarkb | ya publish-openstack-docs-pti has publish-openstack-tox-docs in its post pipeline | 22:45 |
| clarkb | and nova et al have run that job in the last day | 22:45 |
| clarkb | so pretty sure it is that job not running for some reason | 22:45 |
| fungi | corvus: unless you just happen to know the answer, no, please keep working on what you're already working on while we dig ;) | 22:45 |
| openstackgerrit | Kendall Nelson proposed openstack-infra/storyboard master: Add documentation for private stories https://review.openstack.org/636235 | 22:45 |
| corvus | nope. just didn't want to barge in, but didn't want to ignore :) | 22:46 |
| clarkb | requirements limits this job to running against the master branch | 22:46 |
| fungi | thanks! | 22:46 |
| clarkb | but I can't find any other limitations of the job (granted code search only looks at master) | 22:46 |
| *** eernst has quit IRC | 22:47 | |
| *** jamesmcarthur has quit IRC | 22:48 | |
| *** smarcet has joined #openstack-infra | 22:48 | |
| fungi | it's running quite regularly for openstack/python-brick-cinderclient-ext too | 22:50 |
| clarkb | fungi: when not debugging ^ can you review https://review.openstack.org/#/c/636681/ so I can replace pbx server in the near future? | 22:50 |
| fungi | yup | 22:50 |
| *** jamesmcarthur has joined #openstack-infra | 22:51 | |
| fungi | that's the only project with "cinder" in its name which has triggered the job today though | 22:51 |
| smcginnis | os-brick ran on the 8th. That seems normal with the amount of activity there. | 22:53 |
| smcginnis | python-cinderclient ran at 01:06 today. | 22:53 |
| openstackgerrit | James E. Blair proposed openstack-infra/system-config master: Reduce timeouts in run_all.sh https://review.openstack.org/636775 | 22:54 |
| clarkb | what is curious is it is all cinder post jobs not just this one job | 22:54 |
| fungi | okay, fair, my definition of "today" was since the zuul logs were rotated at ~06:30z | 22:54 |
| smcginnis | Relative "today" based on location. ;) | 22:55 |
| *** eernst has joined #openstack-infra | 22:55 | |
| corvus | (^ that change about timeouts is because bridge hung for 2 hours today, so stuff that landed 3 hours ago still isn't in place) | 22:55 |
| fungi | "one standard logrotate day" | 22:55 |
| smcginnis | Relative "today" based on cron. | 22:56 |
| smcginnis | So any way to figure out what happened after 2019-01-23T14:11:33 | 22:57 |
| *** eernst has quit IRC | 22:57 | |
| *** eernst has joined #openstack-infra | 22:57 | |
| clarkb | smcginnis: its probably going to be easier to tackle it the other direction since that was like 3 weeks ago (too many logs to go through) | 22:58 |
| *** wolverineav has joined #openstack-infra | 22:59 | |
| clarkb | eg answer the question of "why does cinder run no post jobs" | 22:59 |
| *** eandersson has joined #openstack-infra | 22:59 | |
| fungi | seeing what i can glean from scheduler debug logs for the most recent commit to merge | 22:59 |
| *** eernst has quit IRC | 23:00 | |
| *** eernst_ has joined #openstack-infra | 23:00 | |
| fungi | wow, been a while since my cinder clone got a ref update | 23:01 |
| clarkb | 2019-02-12 23:09:56,759 ERROR zuul.Pipeline.openstack.post: Error freezing job graph for <QueueItem 0x7f6a25e4acf8 for <Branch 0x7f69d7e792e8 openstack/cinder refs/heads/master updated 5656aa92506090e055d0093678c5f92bda23c24e..2775a4139ee7acefe990be0dd4d64a23d52da1cd> in post> | 23:01 |
| *** david-lyle has quit IRC | 23:02 | |
| fungi | yup | 23:02 |
| *** dklyle has joined #openstack-infra | 23:02 | |
| corvus | there should be more detail around that | 23:02 |
| clarkb | 2019-02-12 23:09:56,762 DEBUG zuul.Pipeline.openstack.post: Invalid config for change <Branch 0x7f69d7e792e8 openstack/cinder refs/heads/master updated 5656aa92506090e055d0093678c5f92bda23c24e..2775a4139ee7acefe990be0dd4d64a23d52da1cd> | 23:02 |
| clarkb | corvus: ok I'll -A to my grep | 23:02 |
| fungi | using grep -A and -B to avoid loading it all in memory, yes | 23:03 |
| clarkb | Exception: Project openstack/cinder is not allowed to run job publish-loci-cinder | 23:04 |
| fungi | i saw a bunch of those earlier when looking at debug logs too. didn't realize at the time they were preventing all jobs | 23:04 |
| *** eernst_ has quit IRC | 23:05 | |
| fungi | you'll find them for more projects than cinder too | 23:05 |
| smcginnis | Would that be related to refactoring out the opendev/openstack things? | 23:05 |
| clarkb | smcginnis: I don't think so. Probably more related to use of secrets | 23:05 |
| *** tkajinam has joined #openstack-infra | 23:05 | |
| clarkb | but now I have to refresh my memory on that | 23:06 |
| *** jamesmcarthur has quit IRC | 23:07 | |
| clarkb | https://git.openstack.org/cgit/openstack/loci/tree/.zuul.d/cinder.yaml | 23:07 |
| *** rascasoft has quit IRC | 23:08 | |
| fungi | it's been there since their .zuul.yaml was imported | 23:08 |
| fungi | so presumably something about that job changed recently | 23:08 |
| fungi | that hasn't been touched in the cinder repo since september | 23:09 |
| corvus | i think it may be this: https://zuul-ci.org/docs/zuul/releasenotes.html#security-issues | 23:09 |
| corvus | https://review.openstack.org/632566 | 23:10 |
| fungi | ohh | 23:10 |
| smcginnis | Timing is about right. | 23:10 |
| *** slaweq has quit IRC | 23:11 | |
| fungi | so the repo where that playbook/secret lives needs to set allowed-projects to include the projects using it? | 23:11 |
| corvus | sorry about that; i thought i looked for jobs like that, but either i just imagined i did, or my search was imperfect. | 23:11 |
| clarkb | "If a job with secrets is defined in an untrusted-project, allowed-projects is automatically set to that project only, and can not be overridden." | 23:11 |
| fungi | "...can not be overridden." | 23:12 |
| fungi | okay, so the playbook and secret need to be relocated to a trusted/config repo | 23:12 |
| clarkb | or into cinder | 23:12 |
| fungi | which, given it's a job which is meant to be run for lots of projects )presumably also responsible for the glance issues), not that | 23:13 |
| clarkb | another option is change the trigger mechansim (run it periodically or somehow trigger loci pipelines when cinder et al merge) | 23:14 |
| corvus | yeah. we have contradictory use-cases here. in this case, loci wants to say it's okay for cinder to use this job in a post-review pipeline. the contradictory use-case is that we want shade to be able to test against vexxhost using real creds in a trusted-check pipeline. i can't see a way to reconcile the two (the key to all of this is that 'allowed-projects' is basically toothless in a world | 23:15 |
| corvus | with depends-on) | 23:15 |
| corvus | the current state of zuul is safer, but doesn't support the loci job as written. | 23:16 |
| corvus | i feel like moving the loci jobs to project-config is warranted considering what they do. | 23:16 |
| cloudnull | qq - i hope - I'm not sure what to look for, but it looks like our deployment guide docs are out of date: https://docs.openstack.org/project-deploy-guide/openstack-ansible/latest/configure.html vs https://raw.githubusercontent.com/openstack/openstack-ansible/master/deploy-guide/source/configure.rst - so im curious if anyone can point me in the righ | 23:17 |
| cloudnull | t direction to figuring out why that might not be publishing ? | 23:17 |
| smcginnis | cloudnull: Could be an issue just discovered. | 23:18 |
| cloudnull | orly | 23:18 |
| smcginnis | Though I actually wouldn't expect that with openstack-ansible. | 23:18 |
| corvus | i think that only the -push jobs would need to be moved. | 23:18 |
| fungi | cloudnull: if your projects also run the loci image publish job, then perhaps | 23:19 |
| corvus | so just this: https://git.openstack.org/cgit/openstack/loci/tree/.zuul.d/cinder.yaml#n22 and the secret | 23:19 |
| clarkb | thinking out loud here could loci define an abstract push job without a secret then add a final job with secret to either project-config or cinder? | 23:19 |
| cloudnull | fungi i dont think we run any loci specific jobs | 23:19 |
| fungi | hogepodge: ^ as sitting ptl for loci, i expect we want your input on moving that job | 23:19 |
| clarkb | but that way the bulk of the code still lives in loci | 23:19 |
| corvus | clarkb: that's similar to what i was suggesting. i don't think the abstract job gives us anything though -- since we need the push playbook and the secret both in project-config. | 23:20 |
| clarkb | oh right the playbook has to be colocated with the secret | 23:20 |
| fungi | cloudnull: what repository does that document live in? | 23:20 |
| cloudnull | openstack/openstack-ansible | 23:21 |
| fungi | cloudnull: i would start here http://zuul.openstack.org/builds?project=openstack%2Fopenstack-ansible&pipeline=post | 23:22 |
| fungi | cloudnull: looks like publish-openstack-tox-docs ran for 3f9a6b3 merging to master at 20:15:33 today | 23:22 |
| cloudnull | maybe we're publishing to the wrong place? | 23:23 |
| fungi | and thinks it completed successfullyt | 23:23 |
| cloudnull | could be ? | 23:23 |
| * cloudnull has never looked at any of the doc things | 23:23 | |
| cloudnull | should we see the change immediately once published? | 23:25 |
| cloudnull | or do the live docs take a while to reflect the change? | 23:26 |
| corvus | cloudnull, fungi: the job fungi mentioned, publish-openstack-tox-docs, publishes here: https://docs.openstack.org/openstack-ansible/latest/ | 23:26 |
| fungi | cloudnull: within 5 minutes or so, i think (we run a frequent cron job to perform an afs release) | 23:26 |
| cloudnull | it looks like http://logs.openstack.org/3f/3f9a6b37256e873b55d5dbe2bd0e0bda45c05dff/post/publish-openstack-tox-docs/4f7b854/job-output.txt.gz#_2019-02-13_20_19_50_755630 did things ? | 23:26 |
| fungi | yeah https://docs.openstack.org/project-deploy-guide/openstack-ansible/latest/configure.html also says "updated: 2018-10-02 09:58" at the bottom | 23:27 |
| corvus | cloudnull: the url you mentioned was last published by this job: http://zuul.openstack.org/build/35a7d0b895c34af38b8df5c4efc0a1b3 | 23:27 |
| *** jamesmcarthur has joined #openstack-infra | 23:27 | |
| cloudnull | oh so the deploy guide job is simply not running ? | 23:28 |
| corvus | https://docs.openstack.org/project-deploy-guide/openstack-ansible/latest/.root-marker | 23:28 |
| corvus | tells you that, if you're curious | 23:28 |
| *** mattw4 has joined #openstack-infra | 23:28 | |
| cloudnull | ah that's cool! | 23:29 |
| * cloudnull TIL | 23:29 | |
| fungi | cloudnull: if it was separate from the docs build for that repo, then yes october seems to be the last it ran | 23:29 |
| cloudnull | interesting. | 23:29 |
| * cloudnull down the rabbit hole I go | 23:29 | |
| *** wolverineav has quit IRC | 23:30 | |
| cloudnull | thanks! smcginnis fungi corvus | 23:30 |
| corvus | http://zuul.openstack.org/builds?job_name=publish-deploy-guide&project=openstack%2Fopenstack-ansible | 23:30 |
| *** wolverineav has joined #openstack-infra | 23:30 | |
| corvus | it's running for branches other than master | 23:30 |
| cloudnull | greatly appreciate the assistance | 23:30 |
| fungi | cloudnull: just returning the favor, as always!!! | 23:30 |
| * cloudnull wonders is this is something we missed when we updated our jobs | 23:30 | |
| *** markvoelker has joined #openstack-infra | 23:31 | |
| *** hwoarang has quit IRC | 23:32 | |
| *** jamesmcarthur has quit IRC | 23:33 | |
| *** hwoarang has joined #openstack-infra | 23:33 | |
| tonyb | corvus, fungi: I'll send an email to the list describeing what I got up to with the bot-reviewer yesterday. I have a a question for infra at: https://review.openstack.org/#/c/636466/6/playbooks/bot-reviewer.yaml | 23:38 |
| *** tosky has quit IRC | 23:39 | |
| corvus | tonyb: i don't see a question | 23:41 |
| tonyb | Thanks! I don't knwo if you knwo this but you actually have to post comments not just write them | 23:44 |
| tonyb | fixed now | 23:45 |
| corvus | infra-root: i've noticed that the gitea-pxc playbook has hung twice | 23:45 |
| corvus | (it's also succeeded once) | 23:45 |
| corvus | it's hung right now, and strace says: | 23:45 |
| corvus | rt_sigsuspend([], 8 | 23:45 |
| corvus | oh wait, i think that was the timeout proc | 23:46 |
| * clarkb reads what gitea-pxc playbook does | 23:47 | |
| corvus | yeah, the actual ansible proc is in a select loop | 23:47 |
| corvus | [pid 13560] clock_gettime(CLOCK_MONOTONIC, {tv_sec=17028697, tv_nsec=273789429}) = 0 | 23:47 |
| corvus | [pid 13560] select(0, NULL, NULL, NULL, {tv_sec=0, tv_usec=1000}) = 0 (Timeout) | 23:47 |
| corvus | it's doing that as fast as possible | 23:47 |
| corvus | it's not at the same place as last time | 23:47 |
| clarkb | which playbook exactly? I see a gitea-playbook.yaml ? | 23:48 |
| corvus | previously it was at TASK [Set up gitea-db startup scripts configmap] ******************************* | 23:48 |
| corvus | now it's at TASK [Set up gitea-db galera replication service] ****************************** | 23:48 |
| *** hwoarang has quit IRC | 23:48 | |
| corvus | clarkb: kubernetes/percona-xtradb-cluster/pxc-playbook.yaml | 23:48 |
| *** jamesmcarthur has joined #openstack-infra | 23:49 | |
| *** hwoarang has joined #openstack-infra | 23:49 | |
| clarkb | that all uses the k8s ansible module | 23:49 |
| clarkb | I wonder if it polls for completion after applying new k8s state? | 23:49 |
| clarkb | https://github.com/ansible/ansible/blob/devel/lib/ansible/modules/clustering/k8s/k8s.py is the code | 23:52 |
| clarkb | which immediately indirects itself elsewhere /me digs up the real code | 23:52 |
| corvus | i'm trying to see if i can find k8s api logs | 23:52 |
| *** jamesmcarthur has quit IRC | 23:53 | |
| clarkb | ansible 2.8's version of the module added a wait parameter | 23:54 |
| clarkb | https://github.com/ansible/ansible/blob/devel/lib/ansible/modules/clustering/k8s/k8s.py#L62 | 23:54 |
| corvus | clarkb: any idea how to see the kube-apiserver logs on opendev-k8s-master.opendev.org ? | 23:55 |
| *** agopi is now known as agopi|out | 23:55 | |
| clarkb | corvus: docker log $container name? | 23:55 |
| clarkb | * docker logs $container | 23:56 |
| corvus | that shows some logs, but nothing useful | 23:56 |
| *** jamesmcarthur has joined #openstack-infra | 23:56 | |
| clarkb | corvus: https://kubernetes.io/docs/tasks/debug-application-cluster/debug-cluster/#master | 23:56 |
| clarkb | that has some file paths you might check | 23:56 |
| *** smarcet has quit IRC | 23:56 | |
| corvus | yeah those don't exist | 23:56 |
| corvus | cause journald | 23:57 |
| corvus | maybe that docker log is all we get :( | 23:57 |
| clarkb | maybe journalctl has the data? | 23:58 |
| corvus | clarkb: any suggestions there? i tried the 2 journalctl commands i know, but i don't know what to expect from this configuration. | 23:58 |
| clarkb | ya I'm similarly lost seems to be mostly docker daemon complaining about something | 23:59 |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!