| opendevreview | Steve Baker proposed openstack/diskimage-builder master: Replace kpartx with qemu-nbd in extract-image https://review.opendev.org/c/openstack/diskimage-builder/+/828617 | 00:08 |
|---|---|---|
| opendevreview | Steve Baker proposed openstack/diskimage-builder master: Move grub-install to the end, and skip for partition images https://review.opendev.org/c/openstack/diskimage-builder/+/826976 | 00:08 |
| opendevreview | Ian Wienand proposed zuul/zuul-jobs master: encrypt-file : role to encrypt a file https://review.opendev.org/c/zuul/zuul-jobs/+/828818 | 00:14 |
| opendevreview | Ian Wienand proposed opendev/system-config master: [dnm] flesh out a little idea for exporting logs https://review.opendev.org/c/opendev/system-config/+/828810 | 00:41 |
| opendevreview | Ian Wienand proposed opendev/system-config master: [dnm] flesh out a little idea for exporting logs https://review.opendev.org/c/opendev/system-config/+/828810 | 01:10 |
| opendevreview | Steve Baker proposed openstack/diskimage-builder master: Replace kpartx with qemu-nbd in extract-image https://review.opendev.org/c/openstack/diskimage-builder/+/828617 | 01:22 |
| opendevreview | Steve Baker proposed openstack/diskimage-builder master: Move grub-install to the end, and skip for partition images https://review.opendev.org/c/openstack/diskimage-builder/+/826976 | 01:22 |
| opendevreview | Ian Wienand proposed opendev/system-config master: [dnm] flesh out a little idea for exporting logs https://review.opendev.org/c/opendev/system-config/+/828810 | 01:32 |
| opendevreview | Ian Wienand proposed opendev/system-config master: [dnm] flesh out a little idea for exporting logs https://review.opendev.org/c/opendev/system-config/+/828810 | 01:59 |
| ianw | just looking at nodepool and reading backwards | 02:06 |
| ianw | it seems when we checked before there was no console output; i can see some now | 02:15 |
| ianw | (for nb03) | 02:15 |
| ianw | no oops or anything particularly interesting | 02:15 |
| ianw | intersting, while the console log did work for me, rebooting it gives a 504 | 02:17 |
| opendevreview | Ian Wienand proposed opendev/system-config master: [dnm] flesh out a little idea for exporting logs https://review.opendev.org/c/opendev/system-config/+/828810 | 02:20 |
| opendevreview | Merged opendev/system-config master: Clean up some gitea-lb zuul config https://review.opendev.org/c/opendev/system-config/+/828793 | 02:27 |
| opendevreview | Ian Wienand proposed opendev/system-config master: [dnm] flesh out a little idea for exporting logs https://review.opendev.org/c/opendev/system-config/+/828810 | 02:52 |
| opendevreview | Ian Wienand proposed opendev/system-config master: [dnm] flesh out a little idea for exporting logs https://review.opendev.org/c/opendev/system-config/+/828810 | 03:29 |
| opendevreview | Steve Baker proposed openstack/diskimage-builder master: Replace kpartx with qemu-nbd in extract-image https://review.opendev.org/c/openstack/diskimage-builder/+/828617 | 03:36 |
| opendevreview | Steve Baker proposed openstack/diskimage-builder master: Move grub-install to the end, and skip for partition images https://review.opendev.org/c/openstack/diskimage-builder/+/826976 | 03:36 |
| opendevreview | Ian Wienand proposed opendev/system-config master: [dnm] flesh out a little idea for exporting logs https://review.opendev.org/c/opendev/system-config/+/828810 | 03:49 |
| opendevreview | Ian Wienand proposed opendev/system-config master: [dnm] flesh out a little idea for exporting logs https://review.opendev.org/c/opendev/system-config/+/828810 | 04:24 |
| *** ysandeep|out is now known as ysandeep | 05:30 | |
| opendevreview | Ian Wienand proposed opendev/system-config master: [dnm] flesh out a little idea for exporting logs https://review.opendev.org/c/opendev/system-config/+/828810 | 05:34 |
| *** amoralej|off is now known as amoralej | 07:01 | |
| *** ysandeep is now known as ysandeep|lunch | 07:53 | |
| opendevreview | Ian Wienand proposed opendev/system-config master: [dnm] flesh out a little idea for exporting logs https://review.opendev.org/c/opendev/system-config/+/828810 | 08:32 |
| *** jpena|off is now known as jpena | 08:32 | |
| opendevreview | Ian Wienand proposed opendev/system-config master: [dnm] flesh out a little idea for exporting logs https://review.opendev.org/c/opendev/system-config/+/828810 | 08:58 |
| *** ysandeep|lunch is now known as ysandeep | 09:11 | |
| opendevreview | Ian Wienand proposed opendev/system-config master: [dnm] flesh out a little idea for exporting logs https://review.opendev.org/c/opendev/system-config/+/828810 | 09:36 |
| opendevreview | Ian Wienand proposed opendev/system-config master: [dnm] flesh out a little idea for exporting logs https://review.opendev.org/c/opendev/system-config/+/828810 | 10:28 |
| *** rlandy_ is now known as rlandy|ruck | 10:45 | |
| *** ysandeep is now known as ysandeep|break | 11:02 | |
| *** dviroel_ is now known as dviroel | 11:18 | |
| *** ysandeep|break is now known as ysandeep | 11:58 | |
| *** amoralej is now known as amoralej|lunch | 12:58 | |
| *** ysandeep is now known as ysandeep|afk | 13:09 | |
| *** pojadhav is now known as pojadhav|afk | 13:16 | |
| *** ysandeep|afk is now known as ysandeep | 13:46 | |
| *** pojadhav|afk is now known as pojadhav | 13:49 | |
| opendevreview | Jonathan Rosser proposed zuul/zuul-jobs master: Allow some configure-mirrors repositories to be disabled https://review.opendev.org/c/zuul/zuul-jobs/+/829028 | 13:51 |
| *** lajoskatona_ is now known as lajoskatona | 13:52 | |
| opendevreview | Jonathan Rosser proposed zuul/zuul-jobs master: Allow some configure-mirrors repositories to be disabled https://review.opendev.org/c/zuul/zuul-jobs/+/829028 | 14:02 |
| *** amoralej|lunch is now known as amoralej | 14:03 | |
| *** pojadhav is now known as pojadhav|afk | 14:54 | |
| opendevreview | Jonathan Rosser proposed zuul/zuul-jobs master: Allow some configure-mirrors repositories to be disabled https://review.opendev.org/c/zuul/zuul-jobs/+/829028 | 14:55 |
| *** dviroel is now known as dviroel|lunch | 15:26 | |
| opendevreview | Joan Gilabert proposed zuul/zuul-jobs master: Add patch to list of build dependencies https://review.opendev.org/c/zuul/zuul-jobs/+/829063 | 15:40 |
| *** ysandeep is now known as ysandeep|out | 15:59 | |
| *** amoralej is now known as amoralej|off | 16:12 | |
| opendevreview | Merged opendev/system-config master: Switch translate's IDP to OpenInfraID https://review.opendev.org/c/opendev/system-config/+/824059 | 16:14 |
| clarkb | fungi: thank you for taking care of that | 16:16 |
| clarkb | jrosser: for debian backports at least I thought you had to explicitly opt into those package installs? | 16:16 |
| jrosser | i think that if the repo is configured, then it's good to go | 16:17 |
| clarkb | jrosser: hrm I'm pretty sure in our docker images where we install from backports we have to explicitly indicate the package should come from that repo | 16:18 |
| clarkb | but we stopped doing that once we updated everything to bullseye | 16:18 |
| jrosser | hmm, could be, though it's maybe possible to put a preference for !backports | 16:18 |
| fungi | you have to explicitly request a version of a package from backports | 16:19 |
| fungi | the apt priority on the backports suite is set low enough that it won't get packages auto-installed | 16:20 |
| fungi | that's how it's implemented in debian, by design | 16:20 |
| jrosser | ok, well the debian/ubuntu part of 829028 is maybe overdone | 16:21 |
| clarkb | looks like I have reviews on my ls-members fix. I need to target 3.3 since that is the oldest supported branch (currently pushed to 3.4 for testing) and have some updates to make via reviewer suggestion. Hopefully, we can avoid that confusion in the future if I get this cleaned up an landed | 16:21 |
| jrosser | the actual thing we broke was on centos | 16:21 |
| fungi | if you install "foo" you'll get the one from your distribution release suite, you need to either explicitly tell the install you want packages satisfied from backports by default with a command-line switch, or specify the package name like "foo/something-backports" or tell it to install "foo>some.version" where it can only be satisfied by versions in backports | 16:22 |
| clarkb | I think I'm ok with toggling backports entirely with a flag. I just wanted to make sure I wasn't missing an important behavior change | 16:23 |
| clarkb | since the default remains | 16:23 |
| *** dviroel|lunch is now known as dviroel | 16:24 | |
| fungi | the primary benefit to toggling backports off in the role would be that it no longer downloads the indices, so that's fewer network requests which could break on you, less time spent downloading things you're not going to use, less space occupied on disk (maybe, if we didn't already prepopulate copies of indices in the images), and so on | 16:24 |
| fungi | clarkb: refstack is going to be a little more hairy than zanata, surprisingly... the organization table has a creaetd_by_user field which is user openid as a foreign key constraint :/ | 16:25 |
| clarkb | jrosser: is condition: undocumented? I'm not finding any indication of it being honored at https://docs.ansible.com/ansible/latest/user_guide/playbooks_loops.html#with-items | 16:25 |
| clarkb | fungi: so we'll need to add rows then update foreign keys in a second pass? | 16:25 |
| fungi | i have a feeling the only sane way to fix up refstack is going to be to take it offline, dump the db, sed the dump, import that, and then turn the service back on | 16:26 |
| clarkb | thats annoying it isn't an internal db id foreign key | 16:26 |
| jrosser | clarkb: i use it here https://review.opendev.org/c/zuul/zuul-jobs/+/829028/3/roles/configure-mirrors/tasks/mirror/Debian.yaml#18 | 16:26 |
| clarkb | jrosser: right I'm wondering if ansible documents that anywhere. I can't seem to find it if so | 16:27 |
| jrosser | it's not an ansible thing | 16:27 |
| clarkb | oh wait I see | 16:27 |
| clarkb | you're setting a when based on that | 16:27 |
| *** marios is now known as marios|out | 16:44 | |
| opendevreview | Clark Boylan proposed opendev/system-config master: DNM testing upstream fix for gerrit ls-members https://review.opendev.org/c/opendev/system-config/+/828786 | 16:52 |
| clarkb | That update points at the new stable-3.3 merge into stable-3.4 change upstream of us as the fix moved to stable-3.3 | 16:52 |
| fungi | #status log Updated OpenID provider for the Zanata service on translate.openstack.org from openstackid.org to id.openinfra.dev | 16:59 |
| opendevstatus | fungi: finished logging | 16:59 |
| fungi | i've taken translate out of the disable list now | 16:59 |
| opendevreview | Clark Boylan proposed opendev/system-config master: Stop mirroring Fedora 34 https://review.opendev.org/c/opendev/system-config/+/829115 | 17:18 |
| clarkb | I hadn't seen a change for ^ yet so I've pushed one | 17:18 |
| *** hashar is now known as Guest376 | 17:29 | |
| *** Guest376 is now known as hashar | 17:30 | |
| *** jpena is now known as jpena|off | 17:32 | |
| opendevreview | Clark Boylan proposed opendev/system-config master: Stop using puppet repos that will be retired https://review.opendev.org/c/opendev/system-config/+/829119 | 17:35 |
| clarkb | thats the next step in puppet repo retirements. We need to make sure that puppet applies don't break | 17:36 |
| clarkb | working on a change to set noop jobs for the repos that will depends on that next | 17:36 |
| opendevreview | Clark Boylan proposed openstack/project-config master: Set noop jobs on opendev repos for retirement https://review.opendev.org/c/openstack/project-config/+/829121 | 17:55 |
| clarkb | neat my fix to ls members just merged | 18:15 |
| fungi | yay! | 18:21 |
| fungi | oh, also not sure if you saw, but you got feedback over the weekend on the gitweb hash change | 18:22 |
| clarkb | I haven't seen that yet | 18:22 |
| clarkb | Our local testing of the latest patchset for the the ls members fix didn't run beacuse of a merge issue. The issue was that I think I needed to depends on the 3.3 change as well so that zuul would have it to merge via the 3.4 change. Now that the 3.3 chagne has merged I rechecked and jobs have enqueued. I'll report back if somehow the suggested refactor causes issues | 18:23 |
| clarkb | I've responded and will try to adapt the example now | 18:33 |
| *** timburke__ is now known as timburke | 19:02 | |
| clarkb | fungi: I've updated the gitea weblinks change and I think I've addressed all of davido's comments | 19:33 |
| clarkb | I rechecked your change. If Zuul fails to merge it then we also likely need to add https://gerrit-review.googlesource.com/c/plugins/gitiles/+/330361 to the depends on | 19:34 |
| fungi | thanks, makes sense, and so awesome that we can do that! | 19:36 |
| clarkb | hrm it fails to build the tests. I guess bazel build test doesn't actually compile the tests | 19:44 |
| clarkb | er bazel build release | 19:44 |
| clarkb | ok just pushed a new ps which might kick those jobs out of the queue | 19:56 |
| opendevreview | Julia Kreger proposed openstack/diskimage-builder master: Replace kpartx with qemu-nbd in extract-image https://review.opendev.org/c/openstack/diskimage-builder/+/828617 | 19:56 |
| opendevreview | Steve Baker proposed openstack/diskimage-builder master: Move grub-install to the end, and skip for partition images https://review.opendev.org/c/openstack/diskimage-builder/+/826976 | 20:23 |
| opendevreview | Ian Wienand proposed opendev/system-config master: [dnm] flesh out a little idea for exporting logs https://review.opendev.org/c/opendev/system-config/+/828810 | 20:32 |
| opendevreview | Steve Baker proposed openstack/diskimage-builder master: Replace kpartx with qemu-nbd in extract-image https://review.opendev.org/c/openstack/diskimage-builder/+/828617 | 20:41 |
| ianw | kevinz: thanks, i can talk to nb03.opendev.org again | 20:42 |
| NeilHanlon | ianw: any idea if there's a timeline for being able to bring https://review.opendev.org/c/openstack/project-config/+/828435 in ? (no rush, just checking for my own notes) | 20:58 |
| clarkb | we got a new nodepool reelase right? | 21:03 |
| clarkb | er not relaese just an update | 21:06 |
| clarkb | yup we did | 21:07 |
| clarkb | NeilHanlon: left a comment for one small last item. But I think we can land that once updated and we get a second reviewer (I can +2 once it is updated) | 21:07 |
| fungi | clarkb: yeah, the main thing left when i checked on friday was that we needed to restart the builders, which is how i ended up down the rabbit hole of znode cleanup | 21:10 |
| fungi | and the nb03 outage (now solved) | 21:10 |
| clarkb | fungi: ah, zuul should do that for us automatically so should'ev been done by now | 21:10 |
| clarkb | but good to double check | 21:10 |
| fungi | but we had full filesystems on the builders for long enough i didn't want to restart them until they caught up on the image backlog | 21:11 |
| clarkb | fungi: re the puppet module cleanup https://zuul.opendev.org/t/openstack/build/a9ab175a9fea4c299bfae11af3c6cbcd is failing on mediawiki. Do you want me to skip mediawiki for now or do you want to clean up the puppet for it in system-config first? | 21:11 |
| clarkb | fungi: zuul automatically restarts them once an hour if necessary | 21:11 |
| fungi | oh, yeah i'll push something to rip out the mediawiki bits in system-config | 21:11 |
| fungi | also i didn't realize we auto-restarted the builders, i guess it's not as disruptive as i had imagined | 21:12 |
| clarkb | thanks. I can rebase my change on that one | 21:12 |
| opendevreview | Neil Hanlon proposed openstack/project-config master: Add rockylinux-8 to nodepool configuration https://review.opendev.org/c/openstack/project-config/+/828435 | 21:13 |
| NeilHanlon | clarkb, fungi: yeah i didn't want to be annoying while y'all were fixing stuff heh | 21:13 |
| clarkb | +2 thanks | 21:16 |
| clarkb | fungi: I checked nb01 and it appears to have updated | 21:16 |
| fungi | awesome, thanks | 21:16 |
| *** dviroel is now known as dviroel|out | 21:17 | |
| clarkb | corvus: ianw: it sounds like ianw noticed that streaming logs were not working yesterday. I mentioned that it could be the finger daemon on executors getting OOMKillered but in this case sounds like zuul web may have zk errors? | 21:19 |
| opendevreview | Jeremy Stanley proposed opendev/system-config master: Remove configuration management for wiki servers https://review.opendev.org/c/opendev/system-config/+/829134 | 21:22 |
| opendevreview | Ian Wienand proposed opendev/system-config master: [dnm] flesh out a little idea for exporting logs https://review.opendev.org/c/opendev/system-config/+/828810 | 21:22 |
| fungi | clarkb: 829134 assuming it passes testing | 21:23 |
| clarkb | thanks. I'll rebase momentarily | 21:23 |
| ianw | yes; https://meetings.opendev.org/irclogs/%23opendev/%23opendev.2022-02-13.log.html#t2022-02-13T23:02:30 .. i didn't look much further that checking the "docker logs", sorry | 21:24 |
| opendevreview | Clark Boylan proposed opendev/system-config master: Stop using puppet repos that will be retired https://review.opendev.org/c/opendev/system-config/+/829119 | 21:24 |
| ianw | i did pull up the web browser debug tools, and i saw the socket open, and a message sending the build number on that that never seems to get a response. but it didn't spew any obvious errors there either | 21:24 |
| opendevreview | Merged openstack/diskimage-builder master: Fallback to persistent netifs names with systemd https://review.opendev.org/c/openstack/diskimage-builder/+/828266 | 21:30 |
| clarkb | ianw: when you get a chance can you check https://review.opendev.org/c/opendev/system-config/+/829115 I think we are ready for that (more fedora-34 cleanup) | 21:36 |
| fungi | looks like the wiki removal patch failed puppet-apply tests. i'll dig deeper after i finish making dinner | 21:40 |
| corvus | ianw: the zk errors in the web log are unrelated and benign (they are "expected errors") | 21:42 |
| corvus | looking at the new stats, i don't see anything full. i think the next step in debugging would be a repl session and direct inspection of what's going on. i'm not in a position to do that right now. | 21:43 |
| *** rlandy|ruck is now known as rlandy|ruck|biab | 22:02 | |
| opendevreview | Ian Wienand proposed opendev/system-config master: [dnm] flesh out a little idea for exporting logs https://review.opendev.org/c/opendev/system-config/+/828810 | 22:14 |
| opendevreview | Jeremy Stanley proposed opendev/system-config master: Remove configuration management for wiki servers https://review.opendev.org/c/opendev/system-config/+/829134 | 22:34 |
| opendevreview | Jeremy Stanley proposed opendev/system-config master: Stop using puppet repos that will be retired https://review.opendev.org/c/opendev/system-config/+/829119 | 22:34 |
| fungi | clarkb: ^ turns out the memcached module is also used by openstack-health | 22:34 |
| *** rlandy|ruck|biab is now known as rlandy|ruck | 22:50 | |
| clarkb | ah, all of those tools will be getting cleaned up too. But I think best to not try and add them in here. | 22:51 |
| mordred | fungi: +2 on the above with a question on 829134 | 22:57 |
| fungi | thanks! | 23:13 |
| clarkb | I need to try and catch dpawlik3 one of these mornings. Maybe tomorrow since I won't be doing the school run. To talk about ELK stuff | 23:13 |
| fungi | mordred: replied | 23:16 |
| *** fdegir16 is now known as fdegir | 23:25 | |
| clarkb | I've updated the meeting agenda. Is there anything I'm missing? | 23:25 |
| fungi | nothing critical i can see | 23:26 |
| ianw | ++ | 23:34 |
| *** rlandy|ruck is now known as rlandy|out | 23:39 | |
| opendevreview | Clark Boylan proposed opendev/system-config master: Haproxy check Gitea's actual port for aliveness https://review.opendev.org/c/opendev/system-config/+/829141 | 23:40 |
| clarkb | fungi: ^ that was something you called out last week | 23:40 |
| fungi | oh, thanks! i'd already forgotten i noticed that | 23:44 |
| clarkb | The downside to this is we won't notice if apahce stops working | 23:47 |
| clarkb | But I'm far less concerned about apache stopping than gitea | 23:47 |
| fungi | yeah, we regularly stop gitea for container updates | 23:47 |
| fungi | i'm surprised nobody's spotted this problem before now | 23:47 |
| opendevreview | Ian Wienand proposed zuul/zuul-jobs master: encrypt-file : role to encrypt a file https://review.opendev.org/c/zuul/zuul-jobs/+/828818 | 23:48 |
| clarkb | I guess the other thing to double check is if we can run http checks over a tcp load balancer | 23:48 |
| clarkb | because that would in theory remove things that 500 | 23:48 |
| fungi | right | 23:48 |
| fungi | i thought haproxy let you do that, but i could be wrong | 23:49 |
| clarkb | oh apprently it does | 23:49 |
| fungi | (the mixing layer 7 check with layer 4 forwarding) | 23:49 |
| clarkb | so ya maybe we're better off updating to use that instead | 23:49 |
| fungi | agreed, that wat it catches both | 23:49 |
| clarkb | and we would do something like HEAD / ? | 23:49 |
| fungi | and also other error conditions like gitea is still listening but returning 5xx | 23:50 |
| fungi | yeah, i would just head / | 23:50 |
| fungi | presumably it doesn't care what host header we set | 23:50 |
| clarkb | looks like it does OPTIONS / by default | 23:50 |
| fungi | that's probably fine | 23:50 |
| fungi | i think that gets around needing to care about what vhost you're hitting | 23:51 |
| opendevreview | Clark Boylan proposed opendev/system-config master: Haproxy http checks for Gitea https://review.opendev.org/c/opendev/system-config/+/829141 | 23:58 |
| clarkb | that should be pretty robust I think | 23:58 |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!