| calcmandan | do any auto manufacturers use cobol for the computers in the cars these days? | 00:28 |
|---|---|---|
| fungi | i highly doubt it. did they ever? most people i know working on automotive software use very low-level languages | 00:43 |
| ianw | somehow i doubt cobol and CAN is a popular combination | 00:44 |
| fungi | can? i'd need an entire six pack before i write cobol | 00:45 |
| openstackgerrit | Ian Wienand proposed zuul/zuul-jobs master: ensure-pip: Add role https://review.opendev.org/717639 | 01:32 |
| openstackgerrit | Ian Wienand proposed zuul/zuul-jobs master: ensure-pip: export ensure_pip_virtualenv_command https://review.opendev.org/718224 | 01:32 |
| openstackgerrit | Ian Wienand proposed zuul/zuul-jobs master: fetch-zuul-cloner: use ensure-pip https://review.opendev.org/717882 | 01:32 |
| openstackgerrit | Ian Wienand proposed zuul/zuul-jobs master: fetch-subunit-output test: use ensure-pip https://review.opendev.org/718225 | 01:32 |
| openstackgerrit | Ian Wienand proposed zuul/zuul-jobs master: ensure-tox: use ensure-pip role https://review.opendev.org/717663 | 01:32 |
| openstackgerrit | Ian Wienand proposed zuul/zuul-jobs master: Update Fedora to 31 https://review.opendev.org/717657 | 01:32 |
| openstackgerrit | Ian Wienand proposed zuul/zuul-jobs master: ensure-pip: export ensure_pip_virtualenv_command https://review.opendev.org/718224 | 01:45 |
| openstackgerrit | Ian Wienand proposed zuul/zuul-jobs master: fetch-zuul-cloner: use ensure-pip https://review.opendev.org/717882 | 01:45 |
| openstackgerrit | Ian Wienand proposed zuul/zuul-jobs master: fetch-subunit-output test: use ensure-pip https://review.opendev.org/718225 | 01:45 |
| openstackgerrit | Ian Wienand proposed zuul/zuul-jobs master: ensure-tox: use ensure-pip role https://review.opendev.org/717663 | 01:45 |
| openstackgerrit | Ian Wienand proposed zuul/zuul-jobs master: Update Fedora to 31 https://review.opendev.org/717657 | 01:45 |
| *** corvus has quit IRC | 01:48 | |
| *** corvus has joined #opendev | 01:49 | |
| *** ysandeep|away is now known as ysandeep|rover | 01:52 | |
| *** mrunge_ has joined #opendev | 02:02 | |
| *** mrunge has quit IRC | 02:03 | |
| ianw | [repo-update|http://mirror.bhs1.ovh.openstack.org/opensuse/update/leap/15.1/oss/] Valid metadata not found at specified URL | 02:28 |
| ianw | hrm suse builds failing | 02:28 |
| ianw | there seem to be a lot of vos releases in progress | 02:32 |
| ianw | nothing actually looks ridiculously over time | 02:37 |
| *** ysandeep|rover is now known as ysandeep|afk | 03:09 | |
| *** ysandeep|afk is now known as ysandeep | 04:00 | |
| *** logan- has quit IRC | 04:45 | |
| *** ykarel|away is now known as ykarel | 04:47 | |
| *** logan_ has joined #opendev | 04:47 | |
| *** logan_ is now known as logan- | 04:48 | |
| openstackgerrit | Ian Wienand proposed opendev/system-config master: mirror-update: stats for vos release of mirrors https://review.opendev.org/718600 | 04:50 |
| *** _mlavalle_1 has joined #opendev | 04:50 | |
| *** mlavalle has quit IRC | 04:53 | |
| *** ysandeep is now known as ysandeep|rover | 04:56 | |
| *** sgw has quit IRC | 05:01 | |
| ianw | AJaeger: i believe our suse mirror is up-to-date, but we're seeing errors in jobs https://zuul.opendev.org/t/zuul/build/25c9843d1d7f41b68d2c962cf4274cf8/log/job-output.txt#279 | 05:22 |
| ianw | "Valid metadata not found at specified URL" | 05:22 |
| ianw | which leads me to think it's related to the upstream mirror | 05:22 |
| AJaeger | let's ask dirk - but he's only on #openstack-infra, so I pinged him there | 05:23 |
| openstackgerrit | Merged zuul/zuul-jobs master: tox: Don't inline python warnings https://review.opendev.org/718554 | 05:39 |
| *** jaicaa has quit IRC | 05:52 | |
| *** jaicaa has joined #opendev | 05:55 | |
| *** dpawlik has joined #opendev | 06:07 | |
| *** dpawlik has quit IRC | 06:37 | |
| *** dpawlik has joined #opendev | 06:39 | |
| *** roman_g has quit IRC | 06:41 | |
| *** _mlavalle_1 has quit IRC | 06:41 | |
| *** sshnaidm|afk is now known as sshnaidm | 06:55 | |
| *** rpittau|afk is now known as rpittau | 07:18 | |
| *** dirk has joined #opendev | 07:39 | |
| *** tosky has joined #opendev | 07:42 | |
| *** ralonsoh has joined #opendev | 07:45 | |
| *** zbr has quit IRC | 07:46 | |
| *** zbr has joined #opendev | 07:46 | |
| *** ysandeep|rover is now known as ysandeep|lunch | 08:10 | |
| openstackgerrit | Ivan Kolodyazhny proposed openstack/project-config master: Add publish-to-pypi jobs for new Vitrage xstatic-* projects https://review.opendev.org/716630 | 09:05 |
| *** ykarel is now known as ykarel|lunch | 09:09 | |
| *** dpawlik has quit IRC | 09:10 | |
| *** ysandeep|lunch is now known as ysandeep|rover | 09:18 | |
| *** hashar has joined #opendev | 09:20 | |
| *** dpawlik has joined #opendev | 09:49 | |
| *** ykarel|lunch is now known as ykarel | 09:51 | |
| *** rpittau is now known as rpittau|bbl | 10:13 | |
| dirk | frickler: so, continuing the discussion here.. | 10:20 |
| dirk | frickler: the situation is that there was an issue with how the root certificates are stored which are used for validating the update channels. this is fixed for many weeks, but the image hasn't been rebuilt sicne then, so it has the old certificates still | 10:21 |
| dirk | frickler: the image is not rebuilding properly due to an unrelated reason, which is already fixed in dib via https://review.opendev.org/#/c/716437/2 | 10:21 |
| dirk | however there was no dib release since then, so the fix is n ot effective | 10:21 |
| dirk | just getting a dib release would make nodepool images build again and then the problem above should be gone | 10:22 |
| dirk | I've been lobbying for a new dib release already previously, it seems now I have even better reasons for getting it ;) | 10:22 |
| dirk | ianw: ^^ | 10:22 |
| frickler | dirk: ah, thanks for double checking. I'll have to defer on ian for releasing dib, though | 10:32 |
| *** hashar has quit IRC | 10:50 | |
| *** hashar has joined #opendev | 11:08 | |
| *** hashar is now known as hasharLunch | 11:09 | |
| *** hasharLunch is now known as hashar | 11:39 | |
| *** ysandeep|rover is now known as ysandeep|coffee | 11:47 | |
| openstackgerrit | Bernard Cafarelli proposed openstack/project-config master: Update Grafana dashboards for stable Neutron releases https://review.opendev.org/718676 | 11:49 |
| *** dpawlik has quit IRC | 11:56 | |
| *** rpittau|bbl is now known as rpittau | 11:57 | |
| *** ysandeep|coffee is now known as ysandeep|rover | 12:00 | |
| *** dpawlik has joined #opendev | 12:08 | |
| *** roman_g has joined #opendev | 12:13 | |
| dirk | frickler: a recheck succeeded - https://zuul.opendev.org/t/zuul/build/4c71706b3aa349ffb2be314c3246d56d - WTF ?! | 12:21 |
| openstackgerrit | Sean McGinnis proposed openstack/project-config master: Extract only series name in release job template https://review.opendev.org/718689 | 12:30 |
| openstackgerrit | Tristan Cacqueray proposed zuul/zuul-jobs master: dhall-diff: add new job https://review.opendev.org/718694 | 13:01 |
| openstackgerrit | Tristan Cacqueray proposed zuul/zuul-jobs master: dhall-diff: add new job https://review.opendev.org/718694 | 13:11 |
| *** ykarel is now known as ykarel|afk | 13:11 | |
| *** mrunge_ is now known as mrunge | 13:14 | |
| openstackgerrit | Tristan Cacqueray proposed zuul/zuul-jobs master: dhall-diff: add new job https://review.opendev.org/718694 | 13:22 |
| *** vblando has joined #opendev | 13:24 | |
| *** cmurphy has joined #opendev | 13:25 | |
| openstackgerrit | Tristan Cacqueray proposed zuul/zuul-jobs master: dhall-diff: add new job https://review.opendev.org/718694 | 13:29 |
| openstackgerrit | Tristan Cacqueray proposed zuul/zuul-jobs master: dhall-diff: add new job https://review.opendev.org/718694 | 13:34 |
| *** sgw has joined #opendev | 13:44 | |
| openstackgerrit | Thierry Carrez proposed openstack/project-config master: Introduce job for granular GitHub mirroring https://review.opendev.org/718479 | 13:45 |
| ttx | clark, fungi, AJaeger ^ I fixed the comments. Hopefully got the syntax right for release-test | 13:46 |
| AJaeger | ttx, let's try it ;) | 13:47 |
| openstackgerrit | James E. Blair proposed opendev/system-config master: Use SafeLoader in irc_checks https://review.opendev.org/718558 | 13:52 |
| corvus | fungi: mind landing this? https://review.opendev.org/714126 | 13:53 |
| openstackgerrit | Tristan Cacqueray proposed zuul/zuul-jobs master: dhall-diff: add new job https://review.opendev.org/718694 | 14:00 |
| *** ykarel|afk is now known as ykarel | 14:02 | |
| *** lpetrut has joined #opendev | 14:19 | |
| openstackgerrit | Merged openstack/project-config master: Extract only series name in release job template https://review.opendev.org/718689 | 14:24 |
| openstackgerrit | Merged opendev/system-config master: Add a note about rename files to project renames doc https://review.opendev.org/714126 | 14:30 |
| clarkb | dirk: AJaeger ianw I can push a dib release today. Maybe I'll wait to see if ianw can ack that plan, but I think it is a holiday weekend already in ianw's corner of the world | 14:32 |
| openstackgerrit | Merged openstack/project-config master: Introduce job for granular GitHub mirroring https://review.opendev.org/718479 | 14:47 |
| AJaeger | config-core, a few small reviews, please: https://review.opendev.org/716630 https://review.opendev.org/717775 https://review.opendev.org/718232 | 14:49 |
| *** ysandeep|rover is now known as ysandeep|afk | 14:51 | |
| clarkb | fungi: following up from yesterday is there a key rotation docs update I can review? | 14:56 |
| openstackgerrit | Paul Albertella proposed zuul/zuul-jobs master: Add Bazel build and install roles https://review.opendev.org/693513 | 14:57 |
| fungi | clarkb: there is not but i'll see if i can piece together the earlier bits others did with the bits i did to come up with something | 14:58 |
| clarkb | fungi: ok, I can help too though I'm not really caught up on the expiration side of things | 14:59 |
| fungi | that was a fairly easy bit to do, just a hard bit to discover | 15:00 |
| fungi | bind maintains a persistent key database, in combination with the keyfiles | 15:01 |
| fungi | so their entries have to be removed from the database before they're removed from disk | 15:01 |
| fungi | and the simplest way to do that is to update them with an immediate expiration (or one coincident with the ttl of the records you're validating with it at least) and reload those | 15:02 |
| fungi | then when they reach the expiration time bind removes them from its database | 15:02 |
| fungi | after which you're free to also delete them from disk | 15:02 |
| openstackgerrit | Merged openstack/project-config master: Add publish-to-pypi jobs for new Vitrage xstatic-* projects https://review.opendev.org/716630 | 15:02 |
| clarkb | fungi: if you did an expiration matching record ttls can you remove them from disk as soon as the expiration is set or do you have to wait for the expiration to occur? | 15:04 |
| clarkb | (maybe we don't know becaues that wasn'ttested) | 15:05 |
| fungi | i think it's probably safest to wait, plus deleting them from disk sooner doesn't help, and deleting them long after their expired (or not at all) is probably also fine | 15:05 |
| fungi | s/their/they're/ | 15:06 |
| openstackgerrit | Merged openstack/project-config master: Retire OSA repo_build and pip_install projects https://review.opendev.org/717775 | 15:08 |
| openstackgerrit | Merged openstack/project-config master: Update grafana for ceph to reflect current jobs https://review.opendev.org/718232 | 15:08 |
| mordred | clarkb: yeah - with expiration, it seems that removing from disk is really more of just a cleanup, not as important | 15:18 |
| mordred | clarkb, fungi : https://review.opendev.org/#/c/718559/ | 15:18 |
| clarkb | mordred: is new etherpad all sorted at this point? | 15:23 |
| clarkb | we can do meetpad testing before we do the cut over too | 15:23 |
| mordred | clarkb: yes, should be | 15:24 |
| mordred | clarkb: and yes, I think it's a great idea to point meetpad at it | 15:25 |
| clarkb | I'm updating the config now | 15:25 |
| fungi | awesome! | 15:26 |
| openstackgerrit | Clark Boylan proposed opendev/system-config master: Switch meetpad to the opendev etherpad https://review.opendev.org/718730 | 15:27 |
| clarkb | mordred: fungi ^ | 15:27 |
| mordred | clarkb, fungi: also - https://review.opendev.org/#/c/717056/ and https://review.opendev.org/#/c/717057/ need re-reviews (and everything else there is actually working nicely, so I think we're ready to move on those) | 15:29 |
| fungi | i suppose i should also do an /etc/hosts hack on my workstation to make sure the etherpad.oppenstack.org name will still work and get redirected normally when we turn it into a cname | 15:29 |
| clarkb | fungi: ya we should do that before updating dns | 15:29 |
| mordred | i checked the hourly and periodic runs and everything is green now | 15:29 |
| mordred | ++ | 15:29 |
| fungi | i assume the image it's using is etherpad-lite 1.8.0 | 15:29 |
| mordred | yeah | 15:30 |
| mordred | fungi: https://opendev.org/opendev/system-config/src/branch/master/playbooks/roles/etherpad/templates/docker-compose.yaml.j2#L20 | 15:31 |
| fungi | oh, yep, indeed | 15:31 |
| fungi | 1.8.1(or .3?) may be up any day now too from the sound of things | 15:31 |
| mordred | cool | 15:32 |
| mordred | should be an easy update :) | 15:32 |
| fungi | glorious | 15:32 |
| clarkb | mordred: sorry this just occured to me. I think the hiera data in system-config should be in the files list for https://review.opendev.org/#/c/717057/7/.zuul.yaml | 15:32 |
| mordred | clarkb: totally agree - fixing | 15:32 |
| mordred | clarkb: or - why don't I do that as a followup | 15:33 |
| mordred | so I don't have to push the rebased stack | 15:33 |
| clarkb | mordred: k | 15:33 |
| *** ykarel is now known as ykarel|away | 15:33 | |
| openstackgerrit | Monty Taylor proposed opendev/system-config master: Trigger remote-puppet-else on hiera changes https://review.opendev.org/718733 | 15:34 |
| mordred | clarkb: thereyago | 15:34 |
| clarkb | mordred: note on https://review.opendev.org/#/c/717058/8 for a cleanup item we shouldn't forget | 15:41 |
| openstackgerrit | Jan Kundrát proposed zuul/zuul-jobs master: docs: Document CORS requirements of Zuul's web dashboard https://review.opendev.org/718740 | 15:46 |
| mordred | clarkb: ++ | 15:49 |
| mordred | clarkb: speaking of cleanup: https://review.opendev.org/#/c/717112/ when you get a moment | 15:51 |
| mordred | fungi: ^^ | 15:51 |
| clarkb | mordred: strings are so much fun | 15:53 |
| *** diablo_rojo has joined #opendev | 15:57 | |
| *** rpittau is now known as rpittau|afk | 15:59 | |
| openstackgerrit | Sean McGinnis proposed openstack/project-config master: Release cycle test template file cleanup https://review.opendev.org/718743 | 15:59 |
| *** dpawlik has quit IRC | 16:06 | |
| fungi | clarkb: the extra \ was due to puppet adding an extra level of escaping, right? | 16:11 |
| clarkb | fungi: yes | 16:11 |
| fungi | cool | 16:12 |
| fungi | and ansible/jinja2 does not suffer that i guess | 16:12 |
| openstackgerrit | Matthew Thode proposed opendev/glean master: write one resolv config https://review.opendev.org/717339 | 16:13 |
| clarkb | fungi: nope, because ansible will replace {{ }} variables in '' or "" quoted strings | 16:15 |
| fungi | good deal | 16:15 |
| clarkb | fungi: with puppet the issue was we had to use "" in order to do var replacement but then that treats \ as an escape | 16:15 |
| clarkb | ansible + yaml doens't have that issue as {{ }} is always replaced | 16:15 |
| fungi | right | 16:15 |
| openstackgerrit | Vladimir Blando proposed openstack/project-config master: Add vexxhost/documentation https://review.opendev.org/718747 | 16:24 |
| *** lpetrut has quit IRC | 16:26 | |
| clarkb | the logstash queue has remained mostly stable over the last couple days | 16:27 |
| clarkb | its still somewhat highe rthan I would prefer but it isn't growing (that just means we've got a delay in indexing but are keeping up overall) | 16:27 |
| mordred | fungi: I think I'm just going to go remove the cron jobs on review now so we can land the change, yeah? | 16:28 |
| mordred | (we can live without them for a couple of hours) | 16:28 |
| clarkb | that should be fine | 16:30 |
| openstackgerrit | Matthew Thode proposed opendev/glean master: write one resolv config https://review.opendev.org/717339 | 16:31 |
| openstackgerrit | Merged opendev/system-config master: Switch meetpad to the opendev etherpad https://review.opendev.org/718730 | 16:34 |
| fungi | mordred: sounds good, yep | 16:38 |
| fungi | i just didn't want to approve that change until that had been done so they didn't get doubled up | 16:38 |
| openstackgerrit | Vladimir Blando proposed openstack/project-config master: Add vexxhost/documentation https://review.opendev.org/718747 | 16:43 |
| *** ysandeep|afk is now known as ysandeep|out | 16:46 | |
| openstackgerrit | Vladimir Blando proposed openstack/project-config master: Add vexxhost/documentation https://review.opendev.org/718747 | 17:03 |
| *** sshnaidm is now known as sshnaidm|afk | 17:12 | |
| openstackgerrit | Merged openstack/project-config master: Release cycle test template file cleanup https://review.opendev.org/718743 | 17:24 |
| openstackgerrit | Merged opendev/system-config master: Add cron jobs that were managed by puppet https://review.opendev.org/717112 | 17:28 |
| openstackgerrit | Matthew Thode proposed opendev/glean master: write one resolv config https://review.opendev.org/717339 | 17:33 |
| clarkb | AJaeger: was there ever a chnage to publish opendev/system-config docs to docs.opendev.org? I don't see it under https://docs.opendev.org/opendev | 17:48 |
| *** mlavalle has joined #opendev | 17:52 | |
| clarkb | opendevians https://etherpad.openstack.org/p/4mEmDBuwoY is a rough draft on governance next steps. I'm presenting that from my viewpoint but want to make sure it is agreeable before sending it more broadly. Let me know what you think | 17:54 |
| AJaeger | clarkb: I don't remember such a change | 17:57 |
| clarkb | AJaeger: thanks I wasn't sure if I had seen that or not. I guess thats a good task for me to figure out | 17:58 |
| AJaeger | clarkb: see what we did for infra-manual. Change the promote job and update openstack-manuals to add redirects | 17:59 |
| AJaeger | clarkb: etherpad LGTM, I added a few "," | 17:59 |
| clarkb | AJaeger: thanks | 17:59 |
| *** sshnaidm|afk is now known as sshnaidm|off | 18:02 | |
| openstackgerrit | Clark Boylan proposed opendev/system-config master: Publish docs updates to docs.opendev.org https://review.opendev.org/718762 | 18:08 |
| clarkb | AJaeger: ^ is it that simple? | 18:08 |
| clarkb | that doesn't do the redirects though | 18:08 |
| openstackgerrit | Monty Taylor proposed opendev/system-config master: Change Etherpad default intro text https://review.opendev.org/718763 | 18:10 |
| openstackgerrit | Monty Taylor proposed opendev/system-config master: Change Etherpad default intro text https://review.opendev.org/718763 | 18:13 |
| openstackgerrit | Monty Taylor proposed opendev/system-config master: Update link to etherpad from OpenDev https://review.opendev.org/718764 | 18:13 |
| clarkb | mordred: we should only land that last change after the cut over? | 18:14 |
| mordred | clarkb: yup | 18:14 |
| *** roman_g has quit IRC | 18:14 | |
| mordred | because data put into etherpad.opendev right now will be deleted | 18:14 |
| mordred | clarkb: I put a -2 on it | 18:15 |
| mordred | clarkb: I guess we need to do eavesdrop.opendev.org next :) | 18:15 |
| mordred | and paste | 18:15 |
| clarkb | all the things | 18:16 |
| *** melwitt has joined #opendev | 18:17 | |
| AJaeger | clarkb: yes, that simple - time to make a template? ;) | 18:19 |
| mordred | corvus: when you have a sec could I interest you in a quick re-revie of https://review.opendev.org/#/c/717056 and https://review.opendev.org/#/c/717057 | 18:19 |
| *** ralonsoh has quit IRC | 18:20 | |
| clarkb | fungi: mordred following up on the tarballs LE cert renewall it seems that we don't have a new warning for that otday so the fixed up ansible from zuul must've corrected that for us | 18:21 |
| mordred | woot | 18:22 |
| mordred | also - don't know if you saw my comment from earlier - but I checked and all the ansibles seem to be running properly | 18:22 |
| mordred | so I thnik we're pretty solid now | 18:22 |
| fungi | clarkb: yep, i'll double check just to be sure | 18:27 |
| fungi | -rw-r----- 1 root letsencrypt 1923 Apr 8 16:21 /etc/letsencrypt-certs/tarballs.opendev.org/tarballs.opendev.org.cer | 18:28 |
| fungi | so it updated yesterday | 18:29 |
| fungi | lgtm | 18:29 |
| AJaeger | clarkb: want to change the theme for system-config as well? | 18:30 |
| clarkb | AJaeger: that is a good idea | 18:30 |
| clarkb | I'm about to eat lunch but will continue to look at that docs stuff | 18:31 |
| AJaeger | clarkb: enjoy! | 18:32 |
| fungi | have we started publishing system-config to docs.opendev.org already? | 18:35 |
| fungi | i guess that's still pending | 18:35 |
| AJaeger | fungi, https://review.opendev.org/718762 - that's what we're discussing | 18:36 |
| fungi | aha, 718762 | 18:36 |
| fungi | yep, just took a closer look at scrollback | 18:36 |
| fungi | been a little scattered today | 18:36 |
| clarkb | fungi: https://etherpad.openstack.org/p/4mEmDBuwoY is related if yo uwant to look at thattoo | 18:37 |
| fungi | i'd already pulled it up, just not read yet | 18:37 |
| fungi | clarkb: this may also be a good opportunity for me to switch gears yet again and double-check the ml config for service-discuss to see what we've done on openstack-discuss and others for dmarc mitigation | 18:42 |
| clarkb | fungi: good idea | 18:43 |
| clarkb | though I can probably do that after lunch if you are busy | 18:44 |
| clarkb | it fits under my todo item of getting that email out :) | 18:44 |
| fungi | meh, there are command-line tools and i can diff, so at least easy to generate the hitlist of what we need to tweak | 18:45 |
| corvus | mordred: +2 on those 2 (looks like it's basically just the ansible forks change) | 18:49 |
| corvus | mordred: i did not +w | 18:49 |
| openstackgerrit | Davlet Panech proposed openstack/project-config master: Add kernel to StarlingX https://review.opendev.org/718772 | 18:58 |
| mordred | corvus: thanks! | 19:04 |
| *** diablo_rojo has quit IRC | 19:16 | |
| *** diablo_rojo has joined #opendev | 19:17 | |
| *** roman_g has joined #opendev | 19:26 | |
| fungi | clarkb: so after diffing the ml configs and discarding the irrelevant differences, i think we need to change the following: clear out subject_prefix and msg_footer, we may also want to turn off bounce_processing and we should probably add a welcome_msg which is just a copy of the info field | 19:44 |
| clarkb | fungi: that sounds reasonable. Are you doing that or should I? | 19:48 |
| fungi | i can do it now | 19:49 |
| fungi | need to let my afternoon tea steep a few anyhow | 19:49 |
| fungi | changes applied | 19:51 |
| *** olaph has joined #opendev | 19:52 | |
| *** olaph has quit IRC | 19:52 | |
| *** olaph has joined #opendev | 19:53 | |
| mordred | clarkb: it JUST occurred to me ... we can combine all-clouds.yaml and clouds.yaml on bridge now | 19:53 |
| mordred | clarkb: because we are not using the openstack dynamic inventory anymore (that's why we had them split) | 19:54 |
| mordred | we can really just put all-clouds in as clouds.yaml and be done with it | 19:54 |
| *** hashar is now known as idoine | 19:58 | |
| clarkb | mordred: oh yup | 19:58 |
| *** idoine is now known as quibble | 19:59 | |
| fungi | that'll be nice | 19:59 |
| corvus | infra-root: fyi: https://freenode.net/news/scheduled-maintenance-services-database-spring-cleaning -- i think our nicks are used enough that shouldn't affect us? | 19:59 |
| clarkb | and the cloud manager thing was already using all clouds right? | 19:59 |
| *** quibble is now known as hashar | 19:59 | |
| clarkb | corvus: ya I've only got hte one registered nick and I use it every day:) | 19:59 |
| corvus | and i think our bot nicks are well used too? | 20:00 |
| fungi | ought to be | 20:00 |
| corvus | (we don't have any bot nicks we're not using, right?) | 20:00 |
| fungi | none that we care about, if we do | 20:01 |
| openstackgerrit | Monty Taylor proposed opendev/system-config master: Get rid of all-clouds.yaml https://review.opendev.org/718788 | 20:01 |
| clarkb | openstackstatus should be active enough but its not feed a constant stream of events/meetings so would be the one I worry about | 20:03 |
| clarkb | but we #status log frequently enough that shouldn't be an issue | 20:03 |
| mordred | clarkb: yeah - everything we were doing started with "set OS_CLIENT_CONFIG_FILE" | 20:03 |
| openstackgerrit | Merged opendev/system-config master: Run AFS in zuul https://review.opendev.org/717056 | 20:08 |
| openstackgerrit | Merged opendev/system-config master: Run remote-puppet-else in zuul https://review.opendev.org/717057 | 20:10 |
| clarkb | infra-root did anyone else want to weigh in on https://etherpad.openstack.org/p/4mEmDBuwoY or should I go ahead and send it to service-discuss? | 20:13 |
| corvus | clarkb: that looks swell | 20:17 |
| fungi | oh, yep, read it, lgtm. thanks clarkb! | 20:19 |
| openstackgerrit | Clark Boylan proposed opendev/system-config master: Switch documentation to alabaster theme https://review.opendev.org/718790 | 20:23 |
| clarkb | there is the doc retheme | 20:23 |
| clarkb | cool I'll get that email sent out now | 20:23 |
| openstackgerrit | Monty Taylor proposed opendev/system-config master: Remove puppet and cron mentions from docs https://review.opendev.org/718791 | 20:27 |
| mordred | #status log All ansible has been migrated to Zuul jobs, nothing is running via run_all.sh any more | 20:28 |
| openstackstatus | mordred: finished logging | 20:28 |
| clarkb | \o/ | 20:30 |
| clarkb | fungi: hrm my email hasn't gone through yet, I wonder if we have OOM issues again? | 20:30 |
| clarkb | maybe I should try and trace that again if so | 20:31 |
| clarkb | we have an OOM from 11 hours ago | 20:31 |
| fungi | clarkb: oh yep, there were oom issues, i noticed them while comparing configs, but then i restarted the services for all the sites | 20:32 |
| fungi | meant to #status log that but have been very distracted today | 20:32 |
| clarkb | fungi: hrm I wonder why my email hasn't shown up yet | 20:32 |
| openstackgerrit | Merged opendev/system-config master: Remove run_all.sh and ansible cron job https://review.opendev.org/717058 | 20:34 |
| fungi | i can take a look in a moment | 20:36 |
| mordred | clarkb: I have cleaned out the log rotate script related to run_all now | 20:43 |
| mordred | clarkb: this can go in now: https://review.opendev.org/#/c/717059 | 20:46 |
| mordred | also - corvus fungi https://review.opendev.org/#/c/717059/ - is a thing clarkb noticed earlier today that should add | 20:47 |
| clarkb | mordred: in 717059 we remove the setting of env var PATH for cron. Any idea if that is more global? | 20:48 |
| *** hashar has quit IRC | 20:48 | |
| clarkb | (possible we need to add that back in somewhere for things like the cloud management cron) | 20:48 |
| clarkb | I've approved the change though as it should be fine to fix that up later if we need it | 20:48 |
| mordred | clarkb: not sure - it's a weird task isn't it? | 20:49 |
| clarkb | ya | 20:49 |
| mordred | clarkb: I'm going to convert cloud-launcher to a zuul job next, which will get us out of the cron business on bridge anyway | 20:49 |
| clarkb | that works too | 20:49 |
| fungi | #status log restarted all mailman sites on lists.openstack.org following oom events around 09:00z | 20:51 |
| openstackstatus | fungi: finished logging | 20:51 |
| fungi | this time i confirmed they stopped properly before i started them again | 20:51 |
| fungi | there are 5 IncomingRunner processes, which matches our site count | 20:52 |
| clarkb | my email went through | 20:53 |
| fungi | it was really my secret ploy to be able to catch up on e-mail ;) | 20:53 |
| openstackgerrit | Monty Taylor proposed opendev/system-config master: Run cloud_launcher from zuul https://review.opendev.org/718798 | 21:07 |
| openstackgerrit | Monty Taylor proposed opendev/system-config master: Stop removing cloud-launcher cron https://review.opendev.org/718799 | 21:07 |
| openstackgerrit | Merged opendev/system-config master: Remove ansible-cron role https://review.opendev.org/717059 | 21:15 |
| openstackgerrit | Monty Taylor proposed opendev/system-config master: Get rid of all-clouds.yaml https://review.opendev.org/718788 | 21:44 |
| openstackgerrit | Monty Taylor proposed opendev/system-config master: Remove puppet and cron mentions from docs https://review.opendev.org/718791 | 21:44 |
| openstackgerrit | Monty Taylor proposed opendev/system-config master: Run cloud_launcher from zuul https://review.opendev.org/718798 | 21:44 |
| openstackgerrit | Monty Taylor proposed opendev/system-config master: Stop removing cloud-launcher cron https://review.opendev.org/718799 | 21:44 |
| openstackgerrit | Merged opendev/system-config master: Trigger everything on inventory changes https://review.opendev.org/717114 | 21:50 |
| openstackgerrit | Monty Taylor proposed opendev/system-config master: Remove puppet and cron mentions from docs https://review.opendev.org/718791 | 21:57 |
| openstackgerrit | Monty Taylor proposed opendev/system-config master: Run cloud_launcher from zuul https://review.opendev.org/718798 | 21:58 |
| openstackgerrit | Monty Taylor proposed opendev/system-config master: Stop removing cloud-launcher cron https://review.opendev.org/718799 | 21:58 |
| corvus | mordred: heh, it looks like we would have gotten an inline comment from zuul duplicating my comment about the '-' but the job didn't quite finish in time. but child changes got an invalid file comment (which is fine and expected) | 22:00 |
| clarkb | infra-root alright I mentioned doing a dib release for dirk this morning. I think ianw is properly out on holiday today so won't be weighing in | 22:00 |
| clarkb | any concerns with doing that now? | 22:00 |
| clarkb | this will apparnetly fix opensuse builds | 22:01 |
| mordred | corvus: aw darn | 22:02 |
| corvus | clarkb: other than "i'm a casual dib maintainer unfamiliar with the current state and it's a holiday weekend", no :) | 22:02 |
| mordred | clarkb: I think it's fine - aiui there's one outstanding change | 22:02 |
| fungi | clarkb: yeah, from scrollback (mostly in #openstack-infra) it seems like ianw's hesitation was around wanting to get more proper pip-and-virtualenv element removal in for opensuse instead of releasing with those workarounds merged? | 22:02 |
| corvus | but, i mean, we released recently, and not much has changed? | 22:02 |
| mordred | and it is "fix opensuse" | 22:02 |
| clarkb | fungi: oh I missed that | 22:03 |
| clarkb | fungi: more than 68bb43535e8f032890687be6a253c3505cb319e5 you mean? | 22:03 |
| clarkb | there are 3 non merge changes and one is fix opensuse | 22:03 |
| clarkb | maybe I should wait if there is more outstanding work | 22:03 |
| corvus | mordred: we're ready to do the etherpad switcheroo, right? do we want to send an email and schedule that for... tomorrow? | 22:04 |
| corvus | (i think we were not worried about the outage, but did want to let people know about it in case of issues, so a 1-day advance notice for a friday maint seems okay to me....) | 22:04 |
| clarkb | corvus: ya I think that should work. | 22:04 |
| clarkb | its well vetted at this point so should be fine, but warning people in case something comes up is a good idea | 22:05 |
| openstackgerrit | Merged opendev/system-config master: Trigger remote-puppet-else on hiera changes https://review.opendev.org/718733 | 22:06 |
| corvus | mordred, clarkb: how about we put it in emergency and shut it down until the switch? because if we're going to announce it, i don't want folks to start using it and then have their data wiped when we migrate. | 22:06 |
| mordred | corvus, clarkb : yeah - I thnk doing it tomorrow sound great | 22:06 |
| clarkb | corvus: that is also probably a good idea | 22:06 |
| corvus | mordred: want me to draft an email? | 22:06 |
| mordred | sure! I'll go put it in emergency and shut it down | 22:06 |
| corvus | mordred: ack, thx | 22:06 |
| * fungi tests the redirection real quick, hadn't gotten around to that yet | 22:07 | |
| corvus | clarkb: this should go to service-announce@lists.opendev -- should we also send it elsewhere since the list is new? | 22:08 |
| clarkb | corvus: ya I think we still need to send to the other lists, but service-announce would be great too | 22:08 |
| mordred | ++ | 22:08 |
| mordred | did we announce a gerrit restart for tomorrow too? | 22:08 |
| clarkb | corvus: you'll get moderated there if we set that up properly (fungi and I are set up as moderators) | 22:08 |
| clarkb | mordred: I don't think so | 22:08 |
| mordred | do we want to? | 22:08 |
| fungi | seems to be working on the apache side of things anyway, though leaves me at a "service unavailable" error on etherpad.opendev.org | 22:08 |
| mordred | fungi: I just turned it off | 22:08 |
| * fungi calls it "good enough" | 22:09 | |
| mordred | fungi: want me to turn it back on? | 22:09 |
| corvus | what time do we want to do it? nowish? 2200 utc? or some other time? | 22:09 |
| clarkb | corvus: I would say earlier in the day like 1600-2000 UTC | 22:09 |
| clarkb | mordred: ++ for gerrit email too | 22:09 |
| fungi | mordred: nah. i was just trying to jump in and test the redirect before you turned it off, but seems i was too slow. no biggie. we can always test later when we're ready to cut over | 22:09 |
| fungi | and i can test apache anyway | 22:09 |
| corvus | mordred, clarkb: k, i'll write an omnibus email for both etherpad and gerrit? | 22:09 |
| mordred | corvus: yeah - I thnik that's a good idea | 22:09 |
| clarkb | corvus: ++ thanks | 22:10 |
| corvus | when gerrit restart? | 22:10 |
| corvus | (let's not do it at the same time :) | 22:10 |
| mordred | and I figure we can likely do both around the same time - they both shoudl be quick | 22:10 |
| mordred | oh | 22:10 |
| mordred | let's not do it at the same time then :) | 22:10 |
| corvus | mordred: 1 hour apart okay? | 22:10 |
| mordred | ++ | 22:10 |
| clarkb | I would say do them both earlier in the day and split them about na hour | 22:10 |
| clarkb | ya | 22:10 |
| * mordred has no plans tomorrow | 22:10 | |
| clarkb | yaI can be around all day tomorrow | 22:10 |
| clarkb | no school stuff for kids | 22:10 |
| fungi | i *had* no plans tomorrow, so this can be my plans ;) | 22:10 |
| corvus | 1600 gerrit; 1700 etherpad? | 22:11 |
| clarkb | corvus: wfm | 22:11 |
| mordred | ++ | 22:11 |
| fungi | sounds great | 22:11 |
| corvus | was there a version upgrade for etherpad? | 22:16 |
| corvus | mordred: and how long does the db import take? | 22:16 |
| clarkb | corvus: yes up to 1.8.0 | 22:16 |
| clarkb | (most recent release) | 22:16 |
| fungi | also we'll want to keep an eye out for 1.8.1 (or .3, it's not clear from their discussions) any day | 22:18 |
| corvus | clarkb, fungi, mordred: https://etherpad.openstack.org/p/tpdNLYKbYn | 22:20 |
| mordred | corvus: I was just thinking I should go do another one real quick | 22:20 |
| mordred | let me go do that and time it | 22:20 |
| corvus | mordred: cool, i'll wait for that before sending (i put 15m in as a placeholder) | 22:21 |
| corvus | have we thought about subscribing the foo-discuss lists to service announce? | 22:21 |
| clarkb | corvus: lgtm other than updating the placeholder time for db migrations | 22:22 |
| corvus | clarkb: have a list of *-discuss lists handy? | 22:23 |
| clarkb | ya I'll add to the etherpad | 22:24 |
| fungi | corvus: lgtm, thanks! | 22:25 |
| corvus | ok, there's the headers copy/pasted from my mail buffer :) | 22:26 |
| openstackgerrit | Oleksandr Kozachenko proposed openstack/project-config master: Add vexxhost/rbac-helm https://review.opendev.org/718824 | 22:27 |
| fungi | i'm wary of rebroadcasting service-announce messages on other lists indefinitely. folks who are admins of those lists can of course choose to subscribe them, i suppose | 22:27 |
| corvus | k. i don't feel strongly about it | 22:28 |
| fungi | crossing the streams makes me a little twitchy, not sure why | 22:28 |
| corvus | seems like it might be an okay way to reach more people and probably within scope of all those lists | 22:29 |
| mordred | so - interestingly enogh - I cannot run mysqldump directly from etherpad.opendev.org | 22:31 |
| mordred | so I'm having to do the dump on etherpad.openstack.org - then we'll have to copy the dump over, then run it in | 22:31 |
| clarkb | mordred: is that because the my.cnf isn't copied over? or network access? | 22:32 |
| mordred | clarkb: mysqldump: Got error: 1045: Access denied for user 'eplite'@'10.208.198.148' (using password: YES) when trying to connect | 22:32 |
| mordred | clarkb: the same creds from etherpad.openstack.org work | 22:32 |
| clarkb | oh this sounds familiar actually | 22:32 |
| clarkb | the ssl issue | 22:33 |
| clarkb | maybe | 22:33 |
| fungi | mysql "localhost" problem? | 22:33 |
| mordred | yeah? cause it would be quicker to do a pipeline dump | mysql | 22:33 |
| fungi | have we been dumping via socket? | 22:33 |
| mordred | it's not that - in neither case are localhost involved | 22:33 |
| fungi | ahh | 22:33 |
| mordred | nah - I can dump over tcp from etherpad.openstack.org | 22:33 |
| clarkb | ya I want to say we ran into similar and it comes across as auth error but really its like you arne't using ssl but it wants to by default or something | 22:33 |
| fungi | oh, right, current etherpad db is in trove | 22:33 |
| clarkb | mordred: ya we set ssl=false in the my.cnf on old server | 22:34 |
| mordred | lemme try that | 22:34 |
| fungi | oho | 22:34 |
| fungi | good memory | 22:35 |
| fungi | (and opaque/misleading error) | 22:35 |
| mordred | clarkb: yeah - that did it | 22:36 |
| mordred | let me restart dump/restore test | 22:37 |
| clarkb | I think I only remember it because the error was so obscure compared to the fix | 22:37 |
| mordred | k. running dump/restore in a pipeline | 22:38 |
| fungi | obviously we can drop that hackaround except for the dump from trove | 22:38 |
| mordred | yeah | 22:38 |
| fungi | yay progress! | 22:38 |
| mordred | ftr, I did an exec into the mariadb service on etherpad.opendev.org and am running: mysqldump --single-transaction -h0e39d6e1213c22be004888d63f24ba61822b46fb.rackspaceclouddb.com -ueplite -p etherpad-lite | mysql -p$MYSQL_ROOT_PASSWORD etherpad-lite | 22:39 |
| mordred | with the password pasted to stdin for the first commant | 22:39 |
| mordred | command | 22:39 |
| mordred | (tomorrow, if etherpad is shut down, --single-transaction can be left off because nobody will be using it - but if we left that off today it would lock all the tables :) ) | 22:41 |
| mnaser | mordred: can i +w https://review.opendev.org/#/c/717136/ ? | 22:42 |
| mordred | mnaser: ++ | 22:44 |
| mnaser | ok cool, done | 22:44 |
| mordred | \o/ | 22:46 |
| mordred | mnaser: look at how magical we are now!@ | 22:46 |
| mnaser | mordred: im loving the post jobs that run project manages :) | 22:46 |
| mordred | \o/ | 22:47 |
| mordred | oh - that reminds me - we should turn log collection back on for manage_projects | 22:47 |
| mordred | we've vetted that one | 22:47 |
| mnaser | mordred: if i can borrow some reviews on https://review.opendev.org/#/c/718824/ and https://review.opendev.org/#/c/718747/ :> | 22:48 |
| openstackgerrit | Monty Taylor proposed opendev/system-config master: Collect logs from manage-projects runs https://review.opendev.org/718827 | 22:48 |
| mordred | clarkb: ^^ | 22:48 |
| mordred | mnaser: done | 22:49 |
| mnaser | mordred: thanks! | 22:49 |
| * mnaser is trying github mirroring for the first time too | 22:50 | |
| corvus | mordred: how's it going? | 22:51 |
| openstackgerrit | Merged openstack/project-config master: Run remote-puppet-else when zuul/main.yaml changes https://review.opendev.org/717136 | 22:53 |
| mordred | corvus: still dumping/restoring | 22:57 |
| mordred | process started at Thu Apr 9 22:38:13 UTC 2020 | 22:58 |
| mordred | so that's about 20 minutes so far | 22:58 |
| openstackgerrit | Merged openstack/project-config master: Add vexxhost/rbac-helm https://review.opendev.org/718824 | 22:59 |
| openstackgerrit | Merged openstack/project-config master: Add vexxhost/documentation https://review.opendev.org/718747 | 22:59 |
| mnaser | i imported the wrong ssh key (apparently macos uses some new openssh key format that our build nodes don't understand) for upload-git-mirror | 23:00 |
| mnaser | would someone be able to run? zuul enqueue --tenant vexxhost --pipeline post --project vexxhost/libvirtd_exporter --change 718826 -- i could just land "noop" changes but that seems, eh. | 23:01 |
| mnaser | the secret was changed in vexxhost/base-jobs so that's why it doesn't really result in another CI run (but hopefully once this works, it just works for everything) | 23:01 |
| clarkb | mnaser: out of curiousity what was the key? because bionic should have a fairly recent openssh | 23:02 |
| corvus | mordred: i'm going to take a break, i'll check back | 23:02 |
| clarkb | mnaser: also post doesn't operate relative to changes. Can you link to the job that ran and failed? We can get the enqueue-ref parameters from there | 23:03 |
| mnaser | clarkb: the header of the key had "-----BEGIN OPENSSH PRIVATE KEY-----" | 23:03 |
| mnaser | some context seems to be here - https://serverfault.com/questions/939909/ssh-keygen-does-not-create-rsa-private-key | 23:03 |
| mnaser | http://zuul.opendev.org/t/vexxhost/build/c9f8f3ff93064198a57e7e640abf85a7 this is the exact build that failed | 23:04 |
| fungi | we had to deal with similar issues on bionic, if memory serves | 23:06 |
| mnaser | i mean, i'm _guessing_ that key probably confused it | 23:06 |
| mordred | dump/restore finished | 23:06 |
| clarkb | mnaser: `sudo zuul enqueue-ref --tenant vexxhost --trigger gerrit --pipeline post --project vexxhost/libvirtd_exporter --ref refs/heads/master --newrev 25ddaad5ba01d702bf3421d6f3a37f41d7ef854d --oldrev 9f73c0f66a396a5faadd71a9fff202dff96c1122` has been run | 23:06 |
| mordred | corvus: so that was 27 minutes - so probably an even half-hour | 23:06 |
| mnaser | clarkb: thank you, i'll watch and see what happens.. | 23:07 |
| *** tosky has quit IRC | 23:08 | |
| mnaser | ok cool, we got beyond that -- remote: error: GH006: Protected branch update failed for refs/heads/master. | 23:10 |
| mnaser | i guess i can just get rid of branch protection and manually mirror it for now until the next time we land something | 23:11 |
| *** diablo_rojo has quit IRC | 23:11 | |
| openstackgerrit | Monty Taylor proposed opendev/system-config master: Run cloud_launcher from zuul https://review.opendev.org/718798 | 23:12 |
| openstackgerrit | Monty Taylor proposed opendev/system-config master: Stop removing cloud-launcher cron https://review.opendev.org/718799 | 23:12 |
| fungi | mnaser: i suppose the issue was something like https://federicofr.wordpress.com/2019/01/02/how-to-convert-openssh-private-keys-to-rsa-pem/ | 23:12 |
| mnaser | fungi: yep, exactly what i ended up doing and it seems to have resolved it | 23:13 |
| fungi | mnaser: yeah, seems my ssh keys on debian/sid have been upgraded to the newer format as well | 23:14 |
| fungi | but you're right, bionic is too old to use them directly | 23:14 |
| fungi | ttx: next time you're about, you may want to make sure whatever platform you generated the github ssh key for openstack on exported it in a format bionic can use ^ | 23:15 |
| mnaser | i assume there is a plan of moving to mirroring via post in openstack fungi ? | 23:16 |
| fungi | mnaser: yeah, it's related to the thread ttx started about cleaning up the github mirror | 23:18 |
| fungi | basically we can't easily selectively mirror from gerrit to github with the current legacy configuration (gerrit replication plugin) so the idea is to turn that off and replicate via zuul job | 23:19 |
| fungi | however i think your idea of working out how to make that doable from an executor-only/nodeless job is an important piece of that | 23:20 |
| corvus | mordred: great, thanks! updating and sending | 23:25 |
| corvus | mnaser, fungi: i think i only have the outline of what you're talking about, but i think either opendev base-jobs or any of the individual tenants with a trusted project could add a job that ran the upload-git-mirror role, then projects could add a secret with pass-to-parent set to true | 23:33 |
| corvus | fungi, clarkb: my service-announce message is moderated | 23:35 |
| clarkb | corvus: k will get to it momentarily | 23:36 |
| mnaser | corvus: arguably I would even say that we would have that job in base jobs, list allowed projects in there with a var that matches the project name | 23:36 |
| mnaser | The only issue is controlling if someone decides to override the project and and override another repo | 23:37 |
| mnaser | so many we need to make it in trusted repo only and define those jobs not in repo | 23:38 |
| corvus | mnaser: i don't think allowed-projects is the right answer; instead you just configure the job to only mirror exactly the triggering repo | 23:38 |
| corvus | mnaser: it's like tho doc promote jobs | 23:38 |
| corvus | (you can't promote someone elses docs, but any repo can add the single doc promote job and it runs on the executor) | 23:38 |
| clarkb | corvus: should be accepted now | 23:38 |
| clarkb | and I see a copy now | 23:39 |
| corvus | mnaser: i'm writing up a slightly more detailed proposal as a response to ttx's thread. | 23:48 |
| corvus | mnaser, fungi: how's this? http://lists.openstack.org/pipermail/openstack-discuss/2020-April/014038.html | 23:53 |
| fungi | sorry, back now, looks like clarkb beat me to the moderation queue | 23:55 |
| fungi | corvus: i believe there's already a job proposed, just a sec... | 23:56 |
| fungi | corvus: nope, merged, looks like i approved it earlier today (have i said i'm scattered?) https://review.opendev.org/718479 | 23:57 |
| fungi | easy to iterate on though, nothing important is using it yet | 23:57 |
| mnaser | corvus, fungi: I like that approach, nodeless would help a lot with efficiency honestly | 23:57 |
| fungi | the idea was to have one central secret in a trusted config repo so that we don't have to reencrypt the same account credentials over and over for every repo in the openstack namespace | 23:57 |
| fungi | checking out the proposal on the ml now | 23:58 |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!