| openstackgerrit | Merged opendev/base-jobs master: Test local log download script generation https://review.opendev.org/716426 | 00:04 |
|---|---|---|
| ianw | sigh, i overlooked the missing += in https://opendev.org/openstack/diskimage-builder/src/commit/ee9ad32b6fd766dce04d7276c6a7eb489e24741f/diskimage_builder/elements/pip-and-virtualenv/install.d/pip-and-virtualenv-source-install/04-install-pip#L34 | 00:06 |
| ianw | so fedora-31 pip-and-virtualenv is borked. i wonder if we just declare f31 as a "plain" image and use it to figure out the workarounds we need in jobs | 00:07 |
| openstackgerrit | Ian Wienand proposed openstack/diskimage-builder master: pip-and-virtualenv : fix fedora install https://review.opendev.org/716795 | 00:09 |
| openstackgerrit | Ian Wienand proposed openstack/diskimage-builder master: pip-and-virtualenv : fix fedora install https://review.opendev.org/716795 | 00:14 |
| ianw | of course, it also should be tested, so i have to look into why the gate didn't fail on it | 00:14 |
| openstackgerrit | Ian Wienand proposed openstack/project-config master: Fedora 31 : remove pip-and-virtualenv https://review.opendev.org/716798 | 00:17 |
| openstackgerrit | Ian Wienand proposed opendev/system-config master: [dnm] base-test test https://review.opendev.org/716799 | 00:22 |
| mordred | clarkb: yes - I agree - we should put in semaphores. I'll get that in in the morning | 00:33 |
| *** mlavalle has quit IRC | 00:36 | |
| openstackgerrit | Ian Wienand proposed opendev/base-jobs master: base-test: fix typo in log download artifact https://review.opendev.org/716801 | 00:45 |
| mordred | clarkb, ianw: actually - I think we might need to combine zuul semaphore with on-host lock | 00:58 |
| mordred | because I think we want the shared semaphore to help us run in parallel but protect from over concurrency | 00:59 |
| clarkb | can weuse two semaphores for that? | 00:59 |
| mordred | but then if we do the same lock trick we have going on in run_all with the actual playbook run, then we'll protect against double-running teh same playbook - and we can do that in playbook | 00:59 |
| mordred | we can't | 00:59 |
| clarkb | ah | 00:59 |
| mordred | but I think we can easily use the on-host lock in the main playbook and be covered across the board | 01:00 |
| ianw | could we request a zuul feature to get multiple locks? :) | 01:00 |
| ianw | although, they'd have to be ranked i guess ... | 01:01 |
| openstackgerrit | Monty Taylor proposed opendev/system-config master: Use flock to prevent double-run https://review.opendev.org/716804 | 01:03 |
| mordred | ianw, clarkb : something like that ^^ | 01:03 |
| mordred | (didn't rebase because I figured I should get an ack before flooding the channel with it :) ) | 01:03 |
| * mordred goes to eat food now | 01:04 | |
| ianw | seems right | 01:10 |
| openstackgerrit | Merged opendev/base-jobs master: base-test: fix typo in log download artifact https://review.opendev.org/716801 | 01:13 |
| openstackgerrit | Merged openstack/project-config master: Fedora 31 : remove pip-and-virtualenv https://review.opendev.org/716798 | 01:19 |
| openstackgerrit | Ian Wienand proposed opendev/base-jobs master: base-test: closing quote on download url https://review.opendev.org/716812 | 01:56 |
| openstackgerrit | Ian Wienand proposed opendev/base-jobs master: base: add bulk log download script https://review.opendev.org/716813 | 01:56 |
| *** ysandeep|away is now known as ysandeep|rover | 01:57 | |
| openstackgerrit | Merged opendev/base-jobs master: base-test: closing quote on download url https://review.opendev.org/716812 | 02:14 |
| *** pramchan has joined #opendev | 02:28 | |
| *** ysandeep|rover is now known as ysandeep|rover|b | 03:26 | |
| openstackgerrit | Ian Wienand proposed openstack/diskimage-builder master: pip-and-virtualenv : fix fedora install https://review.opendev.org/716795 | 03:31 |
| *** pramchan has quit IRC | 03:50 | |
| openstackgerrit | Ian Wienand proposed openstack/project-config master: Add Fedora 31 to main config file https://review.opendev.org/716837 | 04:02 |
| *** ysandeep|rover|b is now known as ysandeep|rover | 04:19 | |
| ianw | ohhh, i see why i missed pip-and-virtualenv failure in gate. that test had to move to bionic only, because fedora can only build there now. however, we had the bionic tests are non-voting | 04:32 |
| *** ykarel|away is now known as ykarel | 04:33 | |
| openstackgerrit | Ian Wienand proposed openstack/diskimage-builder master: Bionic functional tests should be voting https://review.opendev.org/716839 | 04:34 |
| *** hashar has joined #opendev | 04:58 | |
| *** DSpider has joined #opendev | 06:21 | |
| *** mnasiadka has quit IRC | 06:21 | |
| *** mnasiadka has joined #opendev | 06:22 | |
| *** jaicaa has quit IRC | 06:33 | |
| *** jaicaa has joined #opendev | 06:35 | |
| *** frickler_ is now known as frickler | 07:05 | |
| *** hashar has quit IRC | 07:06 | |
| *** rpittau|afk is now known as rpittau | 07:17 | |
| frickler | ianw: kevinz: seems we see mirror failures in linaro-us, maybe we should switch to accessing the mirror by ipv6 only, or do some special routing like we did for fn earlier? https://a1f557d8e7f24035a796-8b5cc018a02fed487ca2a6d59a4ee9d5.ssl.cf1.rackcdn.com/716839/1/check-arm64/dib-functests-arm64-bionic/129eb08/logs/ubuntu-minimal_bionic-arm64-build-succeeds.FAIL.log | 07:22 |
| frickler | infra-root: I noticed two weird things about the "ara icon gives 401" issue: a) the README contains a link to a relative path like "doc/some.img", but the link gitea renders points to https://opendev.org/recordsansible/doc/some.img", note the missing "ara" component in the path | 07:26 |
| frickler | b) when trying to access git repos that do not exist but for an existing tenant, gitea returns a 401 instead of a "not found" http://paste.openstack.org/show/791439/ | 07:28 |
| *** ysandeep|rover is now known as ysandeep|rover|l | 07:43 | |
| ianw | frickler: ohh, good point on the mirror, i'll try to sync up with kevinz tomorrow we're in more similar tz's | 07:53 |
| *** tosky has joined #opendev | 08:02 | |
| *** ykarel is now known as ykarel|lunch | 08:08 | |
| *** ralonsoh has joined #opendev | 08:12 | |
| *** ysandeep|rover|l is now known as ysandeep|rover | 08:33 | |
| *** hashar has joined #opendev | 08:35 | |
| *** ykarel|lunch is now known as ykarel | 09:42 | |
| *** rpittau is now known as rpittau|bbl | 10:14 | |
| *** hashar is now known as hasharLunch | 10:17 | |
| openstackgerrit | Merged zuul/zuul-jobs master: Do not end host if correct go version is installed https://review.opendev.org/716607 | 10:24 |
| openstackgerrit | Merged zuul/zuul-jobs master: go-jobs: improve testing https://review.opendev.org/716668 | 10:24 |
| *** ysandeep|rover is now known as ysandeep|brb | 10:34 | |
| openstackgerrit | Merged openstack/project-config master: Add Fedora 31 to main config file https://review.opendev.org/716837 | 10:39 |
| *** ysandeep|brb is now known as ysandeep|rover | 11:25 | |
| *** ykarel is now known as ykarel|afk | 12:03 | |
| *** hasharLunch is now known as hashar | 12:11 | |
| *** rpittau|bbl is now known as rpittau | 12:12 | |
| *** ykarel|afk is now known as ykarel | 12:13 | |
| openstackgerrit | Radosław Piliszek proposed opendev/system-config master: Do not canonicalize URLs to registry.npmjs.org https://review.opendev.org/716969 | 12:29 |
| *** ykarel is now known as ykarel|afk | 12:33 | |
| *** ykarel|afk is now known as ykarel | 12:55 | |
| *** njohnston has quit IRC | 13:08 | |
| openstackgerrit | Monty Taylor proposed opendev/system-config master: Add zone keys to zuulcd user https://review.opendev.org/716781 | 13:14 |
| openstackgerrit | Monty Taylor proposed opendev/system-config master: Grab logs in the case of errors and use flock https://review.opendev.org/716804 | 13:14 |
| openstackgerrit | Monty Taylor proposed opendev/system-config master: Run service-bridge in zuul https://review.opendev.org/716745 | 13:14 |
| openstackgerrit | Monty Taylor proposed opendev/system-config master: Migrate gitea-lb to zuul https://review.opendev.org/716746 | 13:14 |
| openstackgerrit | Monty Taylor proposed opendev/system-config master: Run letsencrypt in zuul https://review.opendev.org/716747 | 13:14 |
| openstackgerrit | Monty Taylor proposed opendev/system-config master: Run nodepool in zuul https://review.opendev.org/716770 | 13:14 |
| openstackgerrit | Monty Taylor proposed opendev/system-config master: Run meetpad in zuul https://review.opendev.org/716771 | 13:14 |
| openstackgerrit | Monty Taylor proposed opendev/system-config master: Run mirror-update in zuul https://review.opendev.org/716772 | 13:14 |
| mordred | ianw, clarkb, fungi : I've added a patch that adds flock around the playbook call - but the more I think about it the more I think we might want to do single-semaphore-per-playbook and keep things stricly linear rather than allowing parallel jobs | 13:16 |
| mordred | mostly because flock itself will throw the error, and with file matchers and dependency trees - we might miss the run of a child job if a double-run of a parent job got failed by the flock - and I think reasoning about whetheror not that's going to happen is going to be almost impossible | 13:16 |
| mordred | hrm. althugh flock does fail with a specific error code ... so I suppose we could do a failed_when condition on the playbook task | 13:17 |
| mordred | so that ansible didn't consider a flock failure to be a job failure and would still run child jobs | 13:18 |
| mordred | will taht be too confusing though I wonder? maybe let's just do zuul semaphores for now, keep it 1 playbook at a time (which is how run_all works anyway) and we can think about increasing parallelism later if we want | 13:19 |
| *** ykarel is now known as ykarel|afk | 13:29 | |
| openstackgerrit | Monty Taylor proposed opendev/system-config master: Grab logs in the case of errors https://review.opendev.org/716804 | 13:31 |
| openstackgerrit | Monty Taylor proposed opendev/system-config master: Run service-bridge in zuul and semaphore everything https://review.opendev.org/716745 | 13:31 |
| openstackgerrit | Monty Taylor proposed opendev/system-config master: Migrate gitea-lb to zuul https://review.opendev.org/716746 | 13:31 |
| openstackgerrit | Monty Taylor proposed opendev/system-config master: Run letsencrypt in zuul https://review.opendev.org/716747 | 13:31 |
| openstackgerrit | Monty Taylor proposed opendev/system-config master: Run nodepool in zuul https://review.opendev.org/716770 | 13:31 |
| openstackgerrit | Monty Taylor proposed opendev/system-config master: Run meetpad in zuul https://review.opendev.org/716771 | 13:31 |
| openstackgerrit | Monty Taylor proposed opendev/system-config master: Run mirror-update in zuul https://review.opendev.org/716772 | 13:31 |
| openstackgerrit | Monty Taylor proposed opendev/system-config master: Run nameserver in zuul https://review.opendev.org/716764 | 13:31 |
| openstackgerrit | Monty Taylor proposed openstack/project-config master: Run service-nameserver playbook on zone file changes https://review.opendev.org/717004 | 13:35 |
| openstackgerrit | Monty Taylor proposed opendev/system-config master: Run meetpad in zuul https://review.opendev.org/716771 | 13:39 |
| openstackgerrit | James E. Blair proposed zuul/zuul-jobs master: Support multiple matchers when parsing tox output https://review.opendev.org/716263 | 13:42 |
| openstackgerrit | James E. Blair proposed zuul/zuul-jobs master: Don't silently ignore exceptions when parsing tox output https://review.opendev.org/716766 | 13:43 |
| openstackgerrit | James E. Blair proposed zuul/zuul-jobs master: Strip source dir from file comments https://review.opendev.org/716264 | 13:43 |
| openstackgerrit | James E. Blair proposed zuul/zuul-jobs master: DNM: Debug sphinx message https://review.opendev.org/716722 | 13:43 |
| mordred | infra-root: with all of these prod playbooks, and the non-zero number of image jobs, I'm starting to think we should make a .zuul.d in system-config and start breaking some of these up by subject area - any thoughts? | 14:17 |
| corvus | mordred: no objection | 14:18 |
| *** ysandeep|rover is now known as ysandeep|away | 14:26 | |
| mordred | corvus: cool. I think I'll do it at the end of this stack - because otherwise rebase hell | 14:26 |
| *** ykarel|afk is now known as ykarel | 14:28 | |
| clarkb | mordred: single lock seems fine as does zuul.d | 14:31 |
| mordred | cool | 14:34 |
| openstackgerrit | Tobias Henkel proposed zuul/zuul-jobs master: Ignore absolute paths after stripping work dir https://review.opendev.org/717042 | 14:58 |
| *** mlavalle has joined #opendev | 15:16 | |
| *** ysandeep|away is now known as ysandeep | 15:18 | |
| mordred | clarkb: ok. you ready to see a cool patch series? | 15:19 |
| openstackgerrit | Monty Taylor proposed opendev/system-config master: Run mirror-update in zuul https://review.opendev.org/716772 | 15:20 |
| openstackgerrit | Monty Taylor proposed opendev/system-config master: Run nameserver in zuul https://review.opendev.org/716764 | 15:20 |
| openstackgerrit | Monty Taylor proposed opendev/system-config master: Run mirror in zuul https://review.opendev.org/717048 | 15:20 |
| openstackgerrit | Monty Taylor proposed opendev/system-config master: Run static in zuul https://review.opendev.org/717049 | 15:20 |
| openstackgerrit | Monty Taylor proposed opendev/system-config master: Add file matchers for roles used via include_role https://review.opendev.org/717050 | 15:20 |
| openstackgerrit | Monty Taylor proposed opendev/system-config master: Run backup in zuul https://review.opendev.org/717051 | 15:20 |
| openstackgerrit | Monty Taylor proposed opendev/system-config master: Run registry in zuul https://review.opendev.org/717052 | 15:20 |
| openstackgerrit | Monty Taylor proposed opendev/system-config master: Run zuul-preview in zuul https://review.opendev.org/717053 | 15:20 |
| openstackgerrit | Monty Taylor proposed opendev/system-config master: Run review and review-dev in zuul https://review.opendev.org/717054 | 15:20 |
| openstackgerrit | Monty Taylor proposed opendev/system-config master: Run gitea in zuul https://review.opendev.org/717055 | 15:20 |
| openstackgerrit | Monty Taylor proposed opendev/system-config master: Run AFS in zuul https://review.opendev.org/717056 | 15:20 |
| openstackgerrit | Monty Taylor proposed opendev/system-config master: Run remote-puppet-else in zuul https://review.opendev.org/717057 | 15:20 |
| openstackgerrit | Monty Taylor proposed opendev/system-config master: Remove run_all.sh and ansible cron job https://review.opendev.org/717058 | 15:20 |
| openstackgerrit | Monty Taylor proposed opendev/system-config master: Remove ansible-cron role https://review.opendev.org/717059 | 15:20 |
| mordred | BOOM | 15:20 |
| clarkb | mordred: that and meetpad testing is top of my list htis mornign after the community meeting and tea | 15:30 |
| AJaeger | config-core, here's a simple description update for openstack-zuul-jobs: https://review.opendev.org/714728 - please review | 15:37 |
| yoctozepto | config-core: https://review.opendev.org/716969 - could I get approval on that? kolla CI is fubar atm due to many issues, please make me happy by fixing one :D | 15:39 |
| openstackgerrit | Monty Taylor proposed openstack/project-config master: Add hourly periodic pipeline https://review.opendev.org/717063 | 15:41 |
| openstackgerrit | Monty Taylor proposed opendev/system-config master: Run zuul and nodepool related deploys hourly https://review.opendev.org/717064 | 15:43 |
| yoctozepto | thanks clarkb, appreciated | 15:43 |
| AJaeger | mordred: is that a pipeline only for system-config and other OpenDev things? Meaning: Do you want random project X use the hourly pipeline? | 15:43 |
| clarkb | infra-root fwiw I just joined our test meetpad (which should now be using etherpad-dev) and it loaded first time for me | 15:43 |
| clarkb | I need to set up headest and mic and join several times across browsers to test it further but that seems to have helped | 15:44 |
| clarkb | do still get the cookie warning | 15:44 |
| mordred | AJaeger: I don't really want random projects using that | 15:45 |
| AJaeger | mordred: then let's update the description and warn them off ;) | 15:45 |
| mordred | AJaeger: it's really just a band-aid for us for a while until we figure out how to more actively trigger on zuul publiccation | 15:45 |
| mordred | ++ | 15:45 |
| AJaeger | mordred: I want to be sure that we can remove it in a few weeks without breaking projects | 15:46 |
| corvus | clarkb: what pad? | 15:46 |
| clarkb | corvus: the one we were using before (but its got different content due to server switch) | 15:46 |
| corvus | clarkb: meetpadtest1? | 15:46 |
| clarkb | corvus: I dropped in order to finish up this community update meeting | 15:46 |
| clarkb | corvus: yup | 15:46 |
| corvus | ah | 15:47 |
| clarkb | I think you need to separeatly add a cert verification exception | 15:47 |
| corvus | well, it works in firefox 74 and chromium 80 for me | 15:47 |
| openstackgerrit | Monty Taylor proposed openstack/project-config master: Add hourly periodic pipeline https://review.opendev.org/717063 | 15:47 |
| corvus | clarkb: yes | 15:47 |
| mordred | AJaeger: how's that look? ^^ | 15:47 |
| fungi | could the cookie warning be because the etherpad cookie (which is used to persist your author color and nickname preferences across sessions) is coming from a different domain? | 15:47 |
| *** ykarel is now known as ykarel|away | 15:47 | |
| clarkb | fungi: possible :) maybe we should try the opendev.org domain anyway :) | 15:47 |
| clarkb | I want to test if this is consistently working to load the pad now then I can work on that | 15:48 |
| fungi | i suspect that could solve it for most browsers, anyway | 15:48 |
| openstackgerrit | Sorin Sbarnea proposed zuul/zuul-jobs master: Add support for RedHat platforms on install-podman https://review.opendev.org/716578 | 15:48 |
| fungi | but yeah, one thing at a time | 15:48 |
| openstackgerrit | Monty Taylor proposed opendev/system-config master: Run zuul and nodepool related deploys hourly https://review.opendev.org/717064 | 15:48 |
| AJaeger | mordred: now I'm happy | 15:49 |
| corvus | clarkb, fungi: i still get the cookie error on ff74 on my laptop (but not on my desktop) | 15:49 |
| fungi | any particular pad folks are testing on? i don't have camera/mic/speakers hooked up but i can at least test the pad embedding bits | 15:51 |
| clarkb | fungi: meetpadtest1 is the one I'll be using once headset is set up | 15:51 |
| fungi | fwiw, i'm not getting a cookie warning but i also don't see the etherpad displayed | 15:54 |
| fungi | aha, because eff privacy badger saw it trying to connect to another domain | 15:55 |
| fungi | i'll tell it to allow connections to etherpad-dev.openstack.org and see if that helps | 15:55 |
| clarkb | first load worked. second did not. third did | 15:56 |
| clarkb | I sort of intentionally went slow clicking through cookie warnings on the third | 15:56 |
| *** rpittau is now known as rpittau|afk | 15:56 | |
| fungi | yay, i can see the pad! | 15:56 |
| fungi | "failed to access your microphone!" and "failed to access your camera!" | 15:57 |
| fungi | yeah, there aren't any | 15:57 |
| clarkb | fungi: you can tell pulseaudio to pipe your music mix in | 15:57 |
| clarkb | (set up a monitor of output and set that to mic input :) ) | 15:57 |
| clarkb | I'm going to start putting together a vhost update for etherpad-dev to server opendev.org etherpad | 15:58 |
| fungi | so anyway, no cookie warning for me but i just realized i've been testing over plain http not https | 16:00 |
| clarkb | I just did another reload and clicked through cookies quickly. That first went to unhappy pad error then it quickly went to working on its own | 16:00 |
| fungi | i'm still not getting the cookie warning even on https | 16:01 |
| clarkb | mordred: etherpad-dev is now serving from poendev.org | 16:06 |
| clarkb | you'll need a new ssl verification exception | 16:06 |
| mordred | clarkb: meetpad is configured to point to it | 16:06 |
| clarkb | excellent | 16:06 |
| mordred | meetpadtest2 works for me completely | 16:08 |
| fungi | clarkb: force refreshing meetpad, my browser still says it's fetching from etherpad-dev.openstack.org | 16:14 |
| clarkb | fungi: oh maybe mine is broken that way too? | 16:14 |
| fungi | wait, no | 16:15 |
| clarkb | mordred is going to restart services again to be sure they have the right urls | 16:15 |
| mordred | restarted | 16:15 |
| clarkb | fwiw I think that the cookie issue was domain related | 16:15 |
| clarkb | I haven't seen those anymore | 16:15 |
| fungi | i think privacy badger may have been confused because after i told it to start blocking etherpad-dev.openstack.org again when at meetpad.opendev.org, it still showed me the pad on refresh | 16:15 |
| fungi | also privacy badger now claims "No trackers detected. Hooray for privacy!" | 16:16 |
| fungi | sp it does seem to be same-domain now | 16:16 |
| fungi | curious if folks still get the cookie warnings | 16:16 |
| clarkb | fungi: no cookies are fixed by domain match | 16:17 |
| corvus | oh i didn't know we were testing, i'll hop on | 16:17 |
| clarkb | I'm pretty confident of that at this point | 16:17 |
| clarkb | corvus: we are on meetpadtest2 | 16:17 |
| * fungi moves there | 16:19 | |
| fungi | i see the great and powerful oz | 16:19 |
| fungi | but no etherpad | 16:19 |
| fungi | it was working for me on meetpadtest1 though | 16:20 |
| fungi | the brady bunch view also doesn't show the pad, just cameras | 16:21 |
| *** ysandeep is now known as ysandeep|away | 16:31 | |
| mnaser | -tc | 16:33 |
| mnaser | oops | 16:33 |
| corvus | clarkb: https://meetpad.opendev.org/etherpad/p/aoeuaotnh | 16:52 |
| corvus | "GET /p/aoeuaotnh HTTP/1.1" 400 137 "-" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:74.0) Gecko/20100101 Firefox/74.0" | 16:53 |
| openstackgerrit | Sorin Sbarnea proposed zuul/zuul-jobs master: Add support for RedHat platforms on install-podman https://review.opendev.org/716578 | 16:54 |
| fungi | 2020-04-02 16:51:16,195 INFO zuul.Scheduler: Tenant reconfiguration beginning for openstack due to projects {(<Project openstack/mistral-tempest-plugin>, 'master'), (<Project x/tobiko>, 'master')} | 17:06 |
| fungi | still underway | 17:07 |
| fungi | out reconfiguration events generally only i notice our earlier reconfiguration events are taking around 20 minutes to complete as well | 17:07 |
| * fungi tries again | 17:08 | |
| fungi | a number of our recent reconfiguration events are taking around 20 minutes to complete... is that expected or indicative of a performance problem? | 17:09 |
| fungi | looks like we wind up using a lot of buffer/cache memory but active memory is fairly low | 17:10 |
| fungi | the current one just completed, ~18 minutes. the one before it took a solid 20 minutes | 17:11 |
| fungi | but the ones before that are in the 1-5 minute range | 17:11 |
| clarkb | I don't think that is expected | 17:14 |
| clarkb | 1-5 minutes is more expected | 17:14 |
| clarkb | total startup time is like 10 minutes | 17:14 |
| clarkb | when starting zuul scheduler | 17:14 |
| clarkb | (so that should be a worst case) | 17:14 |
| *** hashar is now known as hasharAway | 17:28 | |
| fungi | we're reconfiguring a lot too. there's another underway already | 17:28 |
| fungi | i wonder if this is the final rush of python 2.7 job removals or something | 17:29 |
| mordred | fungi: yay | 17:30 |
| clarkb | not sure if message got through via jitsi, but I need to eat somethign and then I think my chance for dry bike ride is nowish so have to decide if I'm doing that | 17:31 |
| clarkb | back in a bit | 17:31 |
| openstackgerrit | Merged opendev/system-config master: Do not canonicalize URLs to registry.npmjs.org https://review.opendev.org/716969 | 17:46 |
| openstackgerrit | Monty Taylor proposed opendev/system-config master: Remove run_all.sh and ansible cron job https://review.opendev.org/717058 | 17:50 |
| openstackgerrit | Merged opendev/system-config master: Run base playbook with 50 forks https://review.opendev.org/716727 | 17:53 |
| clarkb | wow I got distracted by email | 17:58 |
| clarkb | actually finding food now | 17:58 |
| *** xavinux has joined #opendev | 18:21 | |
| *** xavinux has left #opendev | 18:23 | |
| *** diablo_rojo has quit IRC | 18:26 | |
| *** ralonsoh has quit IRC | 18:28 | |
| *** diablo_rojo has joined #opendev | 18:29 | |
| openstackgerrit | Merged opendev/system-config master: Rename bridge.yaml to install-ansible.yaml https://review.opendev.org/716731 | 18:30 |
| openstackgerrit | Merged opendev/system-config master: Reorder jobs in job list "chronologicaly" and add files https://review.opendev.org/716734 | 18:30 |
| openstackgerrit | Monty Taylor proposed opendev/system-config master: Remove ansible-cron role https://review.opendev.org/717059 | 19:10 |
| openstackgerrit | Monty Taylor proposed opendev/system-config master: Run zuul and nodepool related deploys hourly https://review.opendev.org/717064 | 19:10 |
| *** hasharAway is now known as hashar | 19:14 | |
| openstackgerrit | Monty Taylor proposed opendev/system-config master: Add cron jobs that were managed by puppet https://review.opendev.org/717112 | 19:48 |
| openstackgerrit | Monty Taylor proposed opendev/system-config master: Trigger everything on inventory changes https://review.opendev.org/717114 | 19:58 |
| clarkb | mordred: ok bike ride complete. I never got around to reviewing the zuul triggered jobs so I guess I should start there | 20:37 |
| mordred | clarkb: \o/ | 20:38 |
| mordred | clarkb: I have many for you | 20:38 |
| mordred | clarkb: it's mostly me wanting to make sure you're not bored | 20:38 |
| clarkb | mordred: for the zone ssh keys, is there a change to update the acls for those and explicitly make them infra root? | 20:42 |
| clarkb | (I think havign that reminder for now will be useful) | 20:42 |
| clarkb | if not I can push those changes up | 20:43 |
| clarkb | mordred: in https://review.opendev.org/#/c/716771/5/.zuul.yaml the dependencies list relists things listed in the base job. Are those not additive? eg if we need to add new deps we have to define them all? | 20:44 |
| clarkb | the change before that for nodepool doesn't relist so I think we may not need to? but unsure | 20:45 |
| mordred | clarkb: I believe they are not additive | 20:45 |
| clarkb | k | 20:45 |
| clarkb | unrelated github is apparently having problems | 20:45 |
| mordred | the nodepool one doesn't re-list because it doesn't add any new deps | 20:45 |
| clarkb | if jobs fail to talk to github that is likely why (or people start being confused about github) | 20:45 |
| clarkb | mordred: rgr | 20:45 |
| clarkb | infra-root if anyone else would like to review too the stack starts at https://review.opendev.org/#/c/716781/2 | 20:46 |
| clarkb | but maybe we land up through the le or gitea-lb change and see that everything is happy then we can land more? | 20:47 |
| mordred | ++ | 20:47 |
| mordred | yeah - I mean - it's a lot of changes up in there | 20:47 |
| mordred | landing in batches is a great idea | 20:47 |
| mordred | clarkb: there's also a couple of changes on the same topic in project-config | 20:47 |
| *** mrunge_ has joined #opendev | 20:48 | |
| clarkb | mordred: should we use zuuls new jitter functionality in https://review.opendev.org/#/c/717063/2/zuul.d/pipelines.yaml maybe I'm just searchign for an excuse to use that functionality :) | 20:49 |
| *** mrunge has quit IRC | 20:49 | |
| mordred | clarkb: sure! you have link? | 20:50 |
| clarkb | I can find one! | 20:50 |
| mordred | ah - found it | 20:50 |
| clarkb | https://zuul-ci.org/docs/zuul/reference/drivers/timer.html#attr-pipeline.trigger.timer.time the very last bit there | 20:51 |
| openstackgerrit | Monty Taylor proposed openstack/project-config master: Add hourly periodic pipeline https://review.opendev.org/717063 | 20:52 |
| clarkb | mordred: thats 2 minutes :) but +2 | 20:53 |
| openstackgerrit | Monty Taylor proposed openstack/project-config master: Run service-nameserver playbook on zone file changes https://review.opendev.org/717004 | 20:54 |
| openstackgerrit | Monty Taylor proposed openstack/project-config master: Add hourly periodic pipeline https://review.opendev.org/717063 | 20:54 |
| clarkb | gerrit has discovered you need github to be up to build gerrit | 20:54 |
| openstackgerrit | Monty Taylor proposed openstack/project-config master: Be clear that zone-zuul-ci.org is owned by infra-core https://review.opendev.org/717121 | 20:54 |
| mordred | clarkb: whoops | 20:54 |
| fungi | heh | 20:54 |
| openstackgerrit | Monty Taylor proposed openstack/project-config master: Add hourly periodic pipeline https://review.opendev.org/717063 | 20:55 |
| mordred | clarkb: fixed the comment | 20:55 |
| mordred | clarkb: and there's the acl update at the bottom of that stack | 20:55 |
| clarkb | mordred: I think zone-opendev needs a similar acl change (sorry I wasn't clear that I think both need that) | 20:55 |
| openstackgerrit | Monty Taylor proposed openstack/project-config master: Be clear that zone repos are owned by infra-core https://review.opendev.org/717121 | 20:56 |
| openstackgerrit | Monty Taylor proposed openstack/project-config master: Run service-nameserver playbook on zone file changes https://review.opendev.org/717004 | 20:56 |
| openstackgerrit | Monty Taylor proposed openstack/project-config master: Add hourly periodic pipeline https://review.opendev.org/717063 | 20:56 |
| mordred | clarkb: how's that all look? | 20:56 |
| clarkb | all the +2s | 20:57 |
| clarkb | looks great, thanks | 20:57 |
| mordred | woot! | 20:57 |
| * mordred is excited for our new zuul cd overlords | 20:57 | |
| ianw | is it done? | 21:04 |
| clarkb | ianw: no, but steady progress | 21:05 |
| fungi | rackspace just notified us of suspected problems on the host where ze07 resides | 21:10 |
| fungi | i expect it will experience an outage, usually does when we get these notifications | 21:11 |
| *** dpawlik has quit IRC | 21:12 | |
| clarkb | fungi: want to +A https://review.opendev.org/#/c/715972 ? | 21:22 |
| clarkb | I think at this point we can create the new mailing list | 21:22 |
| fungi | done | 21:23 |
| clarkb | and I'm not seeing any strong preference for an alternative to -meeting but do see preference for -meeting. Considering that maybe we should move forward as proposed by AJaeger (give it 2 days) and create the new cahnnel too? | 21:23 |
| fungi | wfm | 21:24 |
| fungi | i can start approving those as well | 21:24 |
| fungi | done, done, done | 21:26 |
| clarkb | !! | 21:26 |
| openstack | clarkb: Error: "!" is not a valid command. | 21:26 |
| clarkb | heh sorry openstack I was just excited | 21:26 |
| fungi | openstack was not as excited as you were | 21:28 |
| fungi | also see if the new channel topic seems sufficient | 21:28 |
| clarkb | fungi: looks perfect to me | 21:28 |
| clarkb | could add meeting time in there too, but I've been good about sending those emails | 21:28 |
| fungi | things will also happen in there outside of scheduled meetings, so i could go either way on that | 21:30 |
| fungi | tried to keep it brief for now | 21:30 |
| clarkb | ++ | 21:30 |
| openstackgerrit | Ian Wienand proposed openstack/project-config master: Add Fedora 31 to launchers https://review.opendev.org/717131 | 21:35 |
| ianw | clarkb: ^ so i'm basically going to give up on pip-and-virtualenv for fedora >=31 (if i can). it's broken due to package name changes, etc. and it's causing more hacks on the existing hacks which seems crazy | 21:36 |
| clarkb | ianw: and basically roll forward with the plan we have for removing pip and virtualenv? | 21:37 |
| ianw | i missed it in the gate, because f31 can only build on bionic, so i moved the fedora pip-and-virtualenv functional tests to bionic, but didn't notice that the bionic tests were marked as non-voting | 21:37 |
| ianw | so it *was* failing in the gate, if you knew to look :/ | 21:37 |
| ianw | clarkb: yep, i think let's let this be the canary for getting things setup without pip-and-virtualenv | 21:38 |
| ianw | i think we ditched the devstack test-matrix stuff which was relying on a pre-install of virtualenv. i'll try it when we get the nodes but, maybe, there's not too much to do. can then try something unit-testy | 21:39 |
| clarkb | wfm | 21:39 |
| fungi | #status log ze07 rebooted at 21:13 by the provider citing unspecified hypervisor host issues; ticket 200402-ord-0001040 | 21:42 |
| openstackstatus | fungi: finished logging | 21:42 |
| ianw | i really need to look at the dib gate testing and move it to the container -- really nobody is using bionic to build anything as such (afaik) | 21:43 |
| openstackgerrit | Merged openstack/project-config master: Replace incident channel with opendev-meeting https://review.opendev.org/716038 | 21:43 |
| ianw | clarkb / corvus : following up on the bulk-log download change yesterday -- i believe it's all working, using the role @ https://review.opendev.org/#/c/716813/ . see comments for examples | 21:45 |
| *** DSpider has quit IRC | 21:47 | |
| mordred | ianw: https://review.opendev.org/#/q/topic:infra-prod-zuul if you get bored ;) | 21:48 |
| *** hashar has quit IRC | 21:49 | |
| mordred | ianw: there's a response to your review on https://review.opendev.org/#/c/716747/ - and I think clarkb and I were thinking we could land up through and including that patch and then wait to make sure it's all good | 21:49 |
| ianw | mordred: sure, i'll pull it up now and trade you for a glance over my syntax in https://review.opendev.org/717131 | 21:49 |
| mordred | done and +@ | 21:50 |
| ianw | yeah, was just catching up and i think the global semaphore is probably the KISS solution | 21:50 |
| mordred | +2 | 21:50 |
| mordred | ++ | 21:50 |
| mordred | I thnik it's not as efficient runtime as it could be - but it's the safer choice | 21:50 |
| ianw | cause we've written everything so far assuming that | 21:50 |
| mordred | yup | 21:51 |
| clarkb | ok /me dives into more reviews | 21:53 |
| ianw | mordred: minor on https://review.opendev.org/#/c/716804/3 -- is the rundir left over from the flock change? doesn't seem used? | 21:53 |
| mordred | ianw: ++ - I'll push up a change to remote it | 21:54 |
| clarkb | ianw: test of the download logs script lgtm | 21:55 |
| clarkb | though i didn't actually run it against the test job just confirmed the artifact is tehre | 21:55 |
| openstackgerrit | Monty Taylor proposed opendev/system-config master: Remove leftover /var/run dir creation https://review.opendev.org/717132 | 21:55 |
| clarkb | also zoom zoom fedora 31 | 21:56 |
| mordred | \o/ | 21:56 |
| clarkb | ianw: have we updated the mirroring for fedora yet to include f31? | 21:56 |
| clarkb | seems like that will be an important next step if not | 21:56 |
| ianw | clarkb: yeah, we did that a while ago ... it's been a long time coming :) | 21:57 |
| ianw | mordred: definite ++ on splitting up zuul.d for system-config in future changes | 21:57 |
| openstackgerrit | Merged opendev/system-config master: Add a service discussion mailing list for OpenDev https://review.opendev.org/715972 | 21:58 |
| ianw | one other thought on letsencrypt -- the periodic job essentially overrides the files: matcher, right? i don't think i'd ever considered that before | 21:59 |
| fungi | i don't know about overriding, but periodic trigger has no files | 22:00 |
| mordred | yeah- so they _all_ should run on periodic | 22:00 |
| mordred | but that's similar to run_all is now | 22:00 |
| ianw | yep, that's what we want there for renewals | 22:01 |
| fungi | we'll get warnings from certcheck soon enough anyway if they don't | 22:01 |
| ianw | yep, i mean there's weeks of latency for another run to catch it anyway ... just the theory of it :) | 22:02 |
| clarkb | ya certcheck has been useful anyway as we've seen apache reload leave stale worker processes around | 22:06 |
| clarkb | I've got a todo somewhere to figure out how to make them all timeout | 22:06 |
| clarkb | but it is request count based not wall clock based iirc | 22:06 |
| fungi | right, you set max requests per worker and then they get recycled after handling that many | 22:07 |
| fungi | and right now ours are set to never recycle | 22:07 |
| openstackgerrit | Merged openstack/project-config master: Add Fedora 31 to launchers https://review.opendev.org/717131 | 22:15 |
| ianw | it would be good for https://review.opendev.org/#/c/717114/1/.zuul.yaml if there was some way to merge your files: matchers from a parent | 22:15 |
| ianw | yaml does everything else under the sun, but not a good dict merge | 22:16 |
| fungi | nor does it do a good job of spotting duplicate keys in an associative array | 22:19 |
| mordred | nor of roasting chicken | 22:22 |
| ianw | for something like 717131 can we trigger the nodepool job from project-config? | 22:22 |
| fungi | mmm, rotisserie yaml | 22:23 |
| mordred | ianw: yes! | 22:23 |
| mordred | ianw: and we certainly should | 22:24 |
| clarkb | now I'm hungry | 22:24 |
| fungi | just wokked up some beef yakisoba here | 22:25 |
| fungi | 'twas excellent | 22:25 |
| openstackgerrit | Monty Taylor proposed openstack/project-config master: Trigger infra-prod-service-nodepool on nodepool changes https://review.opendev.org/717135 | 22:26 |
| mordred | ianw: ^^ | 22:26 |
| ianw | cool | 22:28 |
| openstackgerrit | Monty Taylor proposed openstack/project-config master: Run remote-puppet-else when zuul/main.yaml changes https://review.opendev.org/717136 | 22:28 |
| mordred | ianw: also that | 22:28 |
| mordred | ianw: I'm certain the depends-on sequencing is not right there :) | 22:28 |
| ianw | theres ... a lot going on :) | 22:28 |
| openstackgerrit | Merged opendev/system-config master: Replace incident channel with opendev-meeting https://review.opendev.org/716039 | 22:30 |
| openstackgerrit | Monty Taylor proposed openstack/project-config master: Trigger infra-prod-service-nodepool on nodepool changes https://review.opendev.org/717135 | 22:31 |
| openstackgerrit | Monty Taylor proposed openstack/project-config master: Run service-nameserver playbook on zone file changes https://review.opendev.org/717004 | 22:31 |
| openstackgerrit | Monty Taylor proposed openstack/project-config master: Run remote-puppet-else when zuul/main.yaml changes https://review.opendev.org/717136 | 22:31 |
| mordred | ianw: ok. I think that fixes the depends-on | 22:31 |
| openstackgerrit | Monty Taylor proposed openstack/project-config master: Add hourly periodic pipeline https://review.opendev.org/717063 | 22:32 |
| *** openstackstatus has quit IRC | 22:56 | |
| *** openstackstatus has joined #opendev | 22:56 | |
| *** ChanServ sets mode: +v openstackstatus | 22:56 | |
| *** openstack has joined #opendev | 22:58 | |
| *** ChanServ sets mode: +o openstack | 22:58 | |
| ianw | mordred: i think i caught up? i think it all looks ok ... just something we'll want to monitor as i goes in i guess | 23:00 |
| *** tosky has quit IRC | 23:00 | |
| openstackgerrit | Merged opendev/system-config master: Add zone keys to zuulcd user https://review.opendev.org/716781 | 23:14 |
| openstackgerrit | Merged opendev/system-config master: Grab logs in the case of errors https://review.opendev.org/716804 | 23:14 |
| ianw | clarkb: with the changes to drop the test matrix, the bionic-plain image is deep into tempest on devstack anyway ... that's a good sign | 23:38 |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!