| openstackgerrit | Ian Wienand proposed openstack/project-config master: Stop launch fedora-31 nodes nodepool https://review.opendev.org/c/openstack/project-config/+/776510 | 00:16 |
|---|---|---|
| openstackgerrit | Ian Wienand proposed openstack/project-config master: Remove fedora-31 disk image config https://review.opendev.org/c/openstack/project-config/+/776511 | 00:16 |
| openstackgerrit | Merged openstack/project-config master: Stop launch fedora-31 nodes nodepool https://review.opendev.org/c/openstack/project-config/+/776510 | 00:39 |
| openstackgerrit | Ian Wienand proposed opendev/system-config master: Stop mirroring Fedora 31 https://review.opendev.org/c/opendev/system-config/+/783585 | 00:55 |
| *** hamalq has joined #opendev | 00:56 | |
| *** hamalq has quit IRC | 01:01 | |
| ianw | we appear to have acquired a "rolecall vote" column in gerrit? not sure if that was there before ... | 01:08 |
| openstackgerrit | Merged openstack/project-config master: Remove fedora-31 disk image config https://review.opendev.org/c/openstack/project-config/+/776511 | 01:16 |
| openstackgerrit | Merged opendev/system-config master: Stop mirroring Fedora 31 https://review.opendev.org/c/opendev/system-config/+/783585 | 01:47 |
| *** brinzhang has joined #opendev | 01:57 | |
| *** sshnaidm|afk has quit IRC | 02:06 | |
| *** hamalq has joined #opendev | 02:57 | |
| *** hamalq has quit IRC | 03:01 | |
| *** ykarel__ has joined #opendev | 03:40 | |
| *** whoami-rajat has joined #opendev | 03:51 | |
| openstackgerrit | Ian Wienand proposed opendev/system-config master: Remove gem mirroring puppet https://review.opendev.org/c/opendev/system-config/+/783594 | 03:59 |
| *** raukadah is now known as chkumar|ruck | 04:06 | |
| ianw | #status log remove mirror.gem, test.fedora and a few out-of-date backup volumes used during various transitions to free up space on afs servers | 04:07 |
| openstackstatus | ianw: finished logging | 04:07 |
| ianw | /dev/mapper/main-vicepa 4.0T 3.2T 845G 80% /vicepa | 04:07 |
| ianw | we started at 304G | 04:07 |
| ianw | #status log released all wheel mirrors, some of which appeared to be locked | 04:08 |
| openstackstatus | ianw: finished logging | 04:08 |
| *** ykarel__ is now known as ykarel | 04:16 | |
| *** paladox has quit IRC | 04:20 | |
| ianw | I've been seeing periodic " VLDB: no permission access for call" doing this. i've sent off a query @ https://lists.openafs.org/pipermail/openafs-info/2021-March/043081.html | 04:26 |
| ianw | i feel like it must be something to do with the afsdb servers; but they all look healthy and no log errors AFAICS | 04:26 |
| *** ricolin has quit IRC | 04:35 | |
| *** ricolin has joined #opendev | 04:48 | |
| *** hamalq has joined #opendev | 04:58 | |
| *** hamalq has quit IRC | 05:03 | |
| *** marios has joined #opendev | 05:09 | |
| *** ysandeep|away is now known as ysandeep | 05:16 | |
| *** brinzhang_ has joined #opendev | 05:26 | |
| *** brinzhang has quit IRC | 05:29 | |
| openstackgerrit | Merged opendev/system-config master: Add Debian Bullseye to the reprepro config https://review.opendev.org/c/opendev/system-config/+/777968 | 05:30 |
| openstackgerrit | Merged opendev/system-config master: Remove gem mirroring puppet https://review.opendev.org/c/opendev/system-config/+/783594 | 05:30 |
| openstackgerrit | Ian Wienand proposed opendev/system-config master: reprepo debian : fix line-ending https://review.opendev.org/c/opendev/system-config/+/783599 | 05:42 |
| openstackgerrit | Merged opendev/system-config master: reprepro: add dist for Ubuntu UCA Wallaby https://review.opendev.org/c/opendev/system-config/+/782396 | 05:44 |
| ianw | ^ i am running the bullseye update in a scree now | 05:49 |
| ianw | screen even | 05:49 |
| *** ralonsoh has joined #opendev | 06:25 | |
| *** slaweq has quit IRC | 06:36 | |
| *** slaweq has joined #opendev | 06:39 | |
| *** sboyron has joined #opendev | 06:50 | |
| *** hamalq has joined #opendev | 06:59 | |
| *** hamalq has quit IRC | 07:03 | |
| *** hashar has joined #opendev | 07:03 | |
| *** rpittau|afk is now known as rpittau | 07:15 | |
| openstackgerrit | Moshiur Rahman proposed openstack/diskimage-builder master: Fix: IPA image buidling with OpenSuse. https://review.opendev.org/c/openstack/diskimage-builder/+/778723 | 07:22 |
| openstackgerrit | Moshiur Rahman proposed openstack/diskimage-builder master: Fix: IPA image buidling with OpenSuse. https://review.opendev.org/c/openstack/diskimage-builder/+/778723 | 07:31 |
| openstackgerrit | Dmitriy Rabotyagov proposed openstack/project-config master: Add Debian Bullseye nodepool images and wheels https://review.opendev.org/c/openstack/project-config/+/783613 | 07:47 |
| *** ysandeep is now known as ysandeep|lunch | 08:09 | |
| *** tosky has joined #opendev | 08:28 | |
| *** elod has quit IRC | 08:34 | |
| *** elod has joined #opendev | 08:34 | |
| *** lpetrut has joined #opendev | 08:45 | |
| *** ykarel is now known as ykarel|lunch | 08:53 | |
| *** hamalq has joined #opendev | 09:00 | |
| *** hamalq has quit IRC | 09:04 | |
| *** noonedeadpunk_ is now known as noonedeadpunk | 09:08 | |
| *** ysandeep|lunch is now known as ysandeep | 09:26 | |
| *** ykarel|lunch is now known as ykarel | 09:27 | |
| *** hashar is now known as hasharLunch | 09:36 | |
| *** CeeMac has quit IRC | 09:37 | |
| *** dtantsur|afk is now known as dtantsur | 09:41 | |
| *** brinzhang0 has joined #opendev | 09:54 | |
| *** brinzhang_ has quit IRC | 09:58 | |
| *** brinzhang_ has joined #opendev | 10:15 | |
| *** sshnaidm|afk has joined #opendev | 10:18 | |
| *** brinzhang0 has quit IRC | 10:18 | |
| *** artom has joined #opendev | 10:24 | |
| openstackgerrit | Dmitriy Rabotyagov proposed openstack/project-config master: Add Debian Bullseye nodepool images and wheels https://review.opendev.org/c/openstack/project-config/+/783613 | 10:43 |
| openstackgerrit | Moshiur Rahman proposed openstack/diskimage-builder master: Fix: IPA image buidling with OpenSuse. https://review.opendev.org/c/openstack/diskimage-builder/+/778723 | 10:44 |
| openstackgerrit | Moshiur Rahman proposed openstack/diskimage-builder master: Fix: IPA image buidling with OpenSuse. https://review.opendev.org/c/openstack/diskimage-builder/+/778723 | 10:46 |
| *** CeeMac has joined #opendev | 10:46 | |
| openstackgerrit | Dmitriy Rabotyagov proposed openstack/project-config master: Add Debian Bullseye nodepool images and wheels https://review.opendev.org/c/openstack/project-config/+/783613 | 10:51 |
| openstackgerrit | Bharat Kunwar proposed openstack/project-config master: [magnum] Remove Review-Priority labels to reduce noise https://review.opendev.org/c/openstack/project-config/+/783631 | 10:55 |
| openstackgerrit | Dmitriy Rabotyagov proposed openstack/project-config master: Add Debian bullseye wheel cache publish jobs https://review.opendev.org/c/openstack/project-config/+/783633 | 10:56 |
| openstackgerrit | Dmitriy Rabotyagov proposed openstack/project-config master: Add Debian bullseye wheel cache publish jobs https://review.opendev.org/c/openstack/project-config/+/783633 | 10:56 |
| *** roman_g has joined #opendev | 11:00 | |
| *** hamalq has joined #opendev | 11:00 | |
| openstackgerrit | Merged opendev/system-config master: reprepo debian : fix line-ending https://review.opendev.org/c/opendev/system-config/+/783599 | 11:01 |
| *** hamalq has quit IRC | 11:05 | |
| *** dtantsur is now known as dtantsur|brb | 11:28 | |
| ianw | bullseye is still copying. will check in morning | 11:28 |
| *** sshnaidm|afk is now known as sshnaidm|off | 11:45 | |
| *** lpetrut has quit IRC | 11:47 | |
| *** roman_g has quit IRC | 11:53 | |
| *** roman_g has joined #opendev | 11:53 | |
| *** roman_g has quit IRC | 11:53 | |
| *** roman_g has joined #opendev | 11:54 | |
| *** roman_g has joined #opendev | 11:55 | |
| *** roman_g has quit IRC | 11:55 | |
| *** roman_g has joined #opendev | 11:56 | |
| *** roman_g has quit IRC | 11:56 | |
| *** roman_g has joined #opendev | 11:56 | |
| *** roman_g has quit IRC | 11:57 | |
| *** lpetrut has joined #opendev | 11:57 | |
| *** roman_g has joined #opendev | 11:57 | |
| *** roman_g has quit IRC | 11:57 | |
| *** paladox has joined #opendev | 12:32 | |
| *** cloudnull has joined #opendev | 12:41 | |
| *** hamalq has joined #opendev | 13:01 | |
| *** hamalq has quit IRC | 13:06 | |
| auristor | ianw: please check that the UserList on afsdb03.openstack.org matches the ones on afsdb01 and afsdb02 | 13:07 |
| *** dardelean has joined #opendev | 13:13 | |
| *** dtantsur|brb is now known as dtantsur | 13:15 | |
| *** hasharLunch is now known as hashar | 13:17 | |
| *** roman_g has joined #opendev | 13:19 | |
| openstackgerrit | Guillaume Chauvel proposed opendev/gear master: WIP: trying to solve gear+zuul+ssl(tls1.3) https://review.opendev.org/c/opendev/gear/+/781238 | 13:21 |
| *** d34dh0r53 has joined #opendev | 13:32 | |
| *** ykarel_ has joined #opendev | 13:39 | |
| *** ykarel has quit IRC | 13:42 | |
| *** d34dh0r53 has quit IRC | 13:52 | |
| *** ysandeep is now known as ysandeep|afk | 13:53 | |
| *** d34dh0r53 has joined #opendev | 13:55 | |
| *** ysandeep|afk is now known as ysandeep | 13:57 | |
| fungi | ianw: the rollcall vote is on some projects and has been for years (mostly governance and specs type repos). gerrit will dynamically add and remove columns for labels in a review list or dashboard based on which ones are relevant for the changes being listed | 14:05 |
| *** whoami-rajat has quit IRC | 14:09 | |
| openstackgerrit | Merged openstack/project-config master: [magnum] Remove Review-Priority labels to reduce noise https://review.opendev.org/c/openstack/project-config/+/783631 | 14:21 |
| *** lpetrut has quit IRC | 14:34 | |
| *** whoami-rajat_ has joined #opendev | 14:44 | |
| *** marios is now known as marios|call | 14:51 | |
| *** brinzhang_ has quit IRC | 14:52 | |
| corvus | fyi i'm looking at some opendev zuul performance metrics in #zuul (in the context of trying to identify if we're seeing zk-related performance changes) | 14:52 |
| *** brinzhang_ has joined #opendev | 14:53 | |
| *** marios|call is now known as marios | 14:54 | |
| *** ykarel_ is now known as ykarel | 14:55 | |
| fungi | thanks, and yeah what the graph shows now looks similar enough to what i recall seeing other weeks, but it's hard to say for sure with the ebb and flow of openstack release cycles | 15:01 |
| *** hamalq has joined #opendev | 15:02 | |
| *** hamalq has quit IRC | 15:07 | |
| *** diablo_rojo has joined #opendev | 15:10 | |
| clarkb | kevinz isn't in the channel currently but usually will respond to linaro cloud problems | 15:11 |
| clarkb | corvus: ^ I can send an email in a bit once I've caught up on scrollback and email and package updates | 15:12 |
| corvus | keystoneauth1.exceptions.connection.SSLError: SSL exception connecting to https://us.linaro.cloud:5000/v3/auth/tokens: HTTPSConnectionPool(host='us.linaro.cloud', port=5000): Max retries exceeded with url: /v3/auth/tokens (Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: certificate has expired (_ssl.c:1091)'))) | 15:12 |
| corvus | clarkb: ^ that's what's in the logs; welcome back! :) | 15:12 |
| clarkb | thanks! | 15:13 |
| * corvus acquires bagels | 15:13 | |
| fungi | yep, seems the let's encrypt cert being used for that cloud does not renew automatically. this is at least the second time we've seen it expire and need manual intervention | 15:14 |
| *** toomer has joined #opendev | 15:15 | |
| *** toomer has quit IRC | 15:16 | |
| *** toomer has joined #opendev | 15:16 | |
| clarkb | corvus: potentially related/colliding with the zuul metric gathering is zk and/or zuul scheduler upgrades are next on my list of todos | 15:16 |
| *** toomer has quit IRC | 15:17 | |
| clarkb | corvus: do you have a preference on which I look at first? I think zk was my initial preference simply because we should be able to do them in a rolling fashion but could do scheduler first | 15:17 |
| *** toomer has joined #opendev | 15:17 | |
| *** toomer has quit IRC | 15:24 | |
| *** toomer has joined #opendev | 15:25 | |
| corvus | clarkb: no pref; i think as long as we note when the changes happen we can account for that when looking at data; it would be good to do all 3 servers in rapid succesion though, so we're not spending a lot of time with some servers different | 15:26 |
| fungi | a thought, we could add a line to our sslcheck config to start monitoring the api cert for clouds we're worried about, so we know if we're about to see them expire | 15:27 |
| fungi | maybe with a low (7 days?) warning time | 15:27 |
| fungi | since we don't know how close to the wire some of them might get before renewing | 15:27 |
| *** ysandeep is now known as ysandeep|dinner | 15:28 | |
| *** mlavalle has joined #opendev | 15:31 | |
| *** ykarel is now known as ykarel|away | 15:33 | |
| clarkb | fungi: not a bad idea | 15:34 |
| *** hashar has quit IRC | 15:39 | |
| *** ykarel|away has quit IRC | 15:42 | |
| *** Guest76033 is now known as jroll | 15:50 | |
| clarkb | I've sent email to kevin now (not fully caught up though) | 16:00 |
| fungi | thanks! | 16:01 |
| *** rpittau is now known as rpittau|afk | 16:08 | |
| openstackgerrit | Merged zuul/zuul-jobs master: Create a template for ssh-key and size https://review.opendev.org/c/zuul/zuul-jobs/+/773474 | 16:10 |
| clarkb | corvus: thinking out loud here for a zk rolling upgrades I think the rough plan would be boot three new servers. Then land a change that replaces zk01.openstack.org with zk01.opendev.org. Then do the same for 02 then 03. The tricky bit is we've got a few places where we set the server IPs that also need to be updated. I guess we land additional changes for those if they are in other repos (like | 16:13 |
| clarkb | project-config?) | 16:13 |
| clarkb | I want to say we do all the connections with IP addrs and not dns names (though the dns names would be updating too) | 16:13 |
| clarkb | an alternative would be to make a compeltely new cluster and the cut over at once | 16:13 |
| openstackgerrit | Merged openstack/project-config master: Add masakari specs entry https://review.opendev.org/c/openstack/project-config/+/783559 | 16:15 |
| *** hamalq has joined #opendev | 16:15 | |
| corvus | clarkb: i don't think we have any ip addrs in project-config | 16:17 |
| corvus | clarkb: i think the rolling upgrade should work and would be ideal | 16:17 |
| clarkb | oh right they get overwritten out of system-config | 16:17 |
| corvus | clarkb: yeah, if you're thinking of nodepool.yaml, ansible templates that in | 16:18 |
| clarkb | ya and the zuul config but that also substitutes vars in from our inventory looks like | 16:19 |
| clarkb | so as long as we add the new server and remove the old one it should work from that perspective | 16:19 |
| clarkb | while that also updates firewall rules existing tcp connections from the old server will remain open. I think that means we will want to manually stop the zk container on the old side just before the new one is added (to avoid quorum issues) | 16:20 |
| clarkb | corvus: do you think the size of the cluster needs to change based on any of the data you've collected so far? | 16:21 |
| clarkb | or should I replace with the same flavors? | 16:21 |
| *** rfayan has joined #opendev | 16:27 | |
| clarkb | also I suppose it should be fairly safe to add the first new server. Double check that things look good on the clients and the cluster. Then do the next two quickly. | 16:27 |
| clarkb | since we should maintain quorum with two nodes (as if we had a normal outage) | 16:27 |
| corvus | clarkb: i'd keep the same flavors; they appear under-utilized right now; let's reserve that headroom (but i don't think we need more) | 16:28 |
| clarkb | corvus: sounds good. I doubt I'll get to that today, but will try to start booting instances tomorrow | 16:29 |
| *** hamalq has quit IRC | 16:31 | |
| *** hamalq has joined #opendev | 16:31 | |
| *** hamalq has quit IRC | 16:33 | |
| *** hamalq has joined #opendev | 16:33 | |
| *** rfayan has quit IRC | 16:33 | |
| *** rfayan has joined #opendev | 16:34 | |
| *** rfayan has joined #opendev | 16:34 | |
| clarkb | is gearman used at all in the opendev zuul install at this point (trying to catch up) | 16:36 |
| corvus | clarkb: oh yes, we're quite a ways away from removing gearman | 16:40 |
| corvus | in fact, nothing in gearman has been moved to zk at this point, only internal scheduler queues | 16:41 |
| *** marios is now known as marios|out | 16:42 | |
| clarkb | ah | 16:43 |
| openstackgerrit | Ghanshyam Mann proposed opendev/irc-meetings master: Update OpenStack TC meeting chair https://review.opendev.org/c/opendev/irc-meetings/+/783714 | 16:43 |
| *** ysandeep|dinner is now known as ysandeep | 16:47 | |
| openstackgerrit | Jeremy Stanley proposed openstack/project-config master: Add Debian Bullseye nodepool images and wheels https://review.opendev.org/c/openstack/project-config/+/783613 | 16:49 |
| *** marios|out has quit IRC | 16:54 | |
| openstackgerrit | Shturm Svetlana proposed zuul/zuul-jobs master: Fix undefined error for zuul_ssh_key_algorithm https://review.opendev.org/c/zuul/zuul-jobs/+/783717 | 16:54 |
| openstackgerrit | Merged opendev/irc-meetings master: Update OpenStack TC meeting chair https://review.opendev.org/c/opendev/irc-meetings/+/783714 | 16:56 |
| *** ysandeep is now known as ysandeep|away | 16:57 | |
| openstackgerrit | Merged zuul/zuul-jobs master: Fix undefined error for zuul_ssh_key_algorithm https://review.opendev.org/c/zuul/zuul-jobs/+/783717 | 17:18 |
| *** auristor has quit IRC | 17:22 | |
| *** ralonsoh has quit IRC | 17:24 | |
| clarkb | fungi: do you know if anyone has had a chance to look at the gerrit account classifications I produced near the end of last week? | 17:41 |
| clarkb | thats the other todo item near the top of my list for getting done this week if people are comfortable withi t | 17:41 |
| *** roman_g has quit IRC | 17:44 | |
| fungi | clarkb: ahh, no, i meant to take a look | 17:49 |
| openstackgerrit | Clark Boylan proposed opendev/system-config master: Upgrade gitea to 1.13.6 https://review.opendev.org/c/opendev/system-config/+/780923 | 18:05 |
| openstackgerrit | Clark Boylan proposed opendev/system-config master: Force gitea to fail for human verification https://review.opendev.org/c/opendev/system-config/+/781776 | 18:05 |
| clarkb | I've set a zuul autohold on ^ which should test the right version now that we've fixed the test dependencies | 18:06 |
| clarkb | the previous change state to test v1.13.4 looked fine at the time but now there is a v1.13.6 we can update to so I've bumped up the version | 18:07 |
| *** dtantsur is now known as dtantsur|afk | 18:07 | |
| *** auristor has joined #opendev | 18:08 | |
| *** whoami-rajat_ is now known as whoami-rajat | 18:31 | |
| *** frigo has joined #opendev | 18:43 | |
| *** cloudnull has quit IRC | 18:43 | |
| *** frigo has quit IRC | 19:20 | |
| *** cloudnull has joined #opendev | 19:21 | |
| *** frigo has joined #opendev | 19:29 | |
| *** frigo has quit IRC | 19:33 | |
| clarkb | https://162.253.55.248:3081/opendev/system-config lgtm. I think 780923 is ready for review | 19:53 |
| clarkb | infra-root ^ if we can get 780923 reviewed I'm around today to ensure it lands happily | 19:53 |
| *** rfayan has quit IRC | 19:53 | |
| fungi | yep, checks out | 19:54 |
| fungi | another set of eyeballs might be good, but maybe i should single-core approve it while you're around to keep an eye on it | 19:55 |
| clarkb | the minor releases do tend to be pretty safe | 19:56 |
| fungi | yeah, i've approved it now | 19:57 |
| clarkb | cool, I will keep an eye on it | 19:59 |
| fungi | clarkb: what do you think about the topic:lp-integration approach to a short-term restoration of lp bug updates for now? | 20:11 |
| fungi | i figure people can always do manual task assignment | 20:12 |
| fungi | at least that restores the comments | 20:12 |
| clarkb | fungi: I've approved the first one and +2'd the others. Didn't approve the others as I think we needt oapprove them in sequence due to not sharing queues? | 20:17 |
| clarkb | I could be wrong about that though | 20:17 |
| openstackgerrit | Merged opendev/gerritlib master: Run gerritlib-jeepyb-integration on ubuntu-focal https://review.opendev.org/c/opendev/gerritlib/+/782603 | 20:22 |
| clarkb | wow that ran quickly. Should I go ahead and approve the next one? | 20:23 |
| fungi | which change was that? | 20:28 |
| fungi | oh, yeah, feel free | 20:28 |
| clarkb | https://review.opendev.org/c/opendev/jeepyb/+/782538 | 20:29 |
| clarkb | it has been approved | 20:29 |
| *** diablo_rojo has quit IRC | 20:35 | |
| *** whoami-rajat has quit IRC | 20:39 | |
| clarkb | hrm the system-config-run-base job failed on the gitea upgrade | 20:46 |
| clarkb | I'm trying to edit the meeting agenda then will look at that | 20:46 |
| ianw | fungi: huh, must have somehow got a change in my default with the rollcall vote that pulled the column up then | 20:49 |
| ianw | clarkb: not sure if you saw the token auth work for the gitea job yet? | 20:49 |
| *** sboyron has quit IRC | 20:49 | |
| clarkb | ianw: I haven't | 20:50 |
| ianw | clarkb: ok, maybe checkout https://review.opendev.org/c/opendev/system-config/+/782887 before too much digging on the gitea job :) | 20:51 |
| clarkb | ianw: it was the base job that failed not gitea proper | 20:52 |
| *** kopecmartin has quit IRC | 20:56 | |
| *** akahat has quit IRC | 20:56 | |
| clarkb | fungi: https://review.opendev.org/c/opendev/system-config/+/781593 is a quick an easy one if you have a moment (noticed it in my catch up) | 20:58 |
| ianw | fwiw had a poke on that gitea server and looks good | 20:59 |
| *** slaweq has quit IRC | 20:59 | |
| *** fbo has joined #opendev | 20:59 | |
| ianw | auristor: yep, it looks like that is the problem. we don't have that file under configuration control, i guess we should | 21:01 |
| *** kopecmartin has joined #opendev | 21:01 | |
| *** slaweq has joined #opendev | 21:01 | |
| auristor | its not supposed to be configuration controlled. its a private database of the local bosserver process and is only supposed to be modified via bos commands | 21:02 |
| fungi | so we could probably orchestrate it during deployment i guess | 21:03 |
| ianw | auristor: well, what i mean is that our automated configuration management doesn't create it, so it's not there :) | 21:03 |
| fungi | yeah, we blur the lines between configuration management and orchestration | 21:04 |
| auristor | orchestration would be fine. so would copying the file over in the meantime. | 21:08 |
| clarkb | I'ev rechecked teh gitea change. The base failre was the ansible rc -13 issue when trying to install a package. I wish we could figure that one out | 21:09 |
| ianw | i guess if I write something to interface via bos add/list/remove at least someone else might be able to reuse it. although i'm tempted to do much less work and just stick the file in ansible and copy it out | 21:11 |
| ianw | infra-root: the one i'd like to get going (after I look at this UserList issue) is https://review.opendev.org/c/opendev/system-config/+/783183 to add review02.opendev.org ... you can log into the server if you'd like to poke at it | 21:13 |
| ianw | i'd love to get that installed and have a static copy of current data so we can poke at it | 21:14 |
| ianw | then we can come up with a more concrete checklist for switch-over | 21:14 |
| fungi | ianw: i think as long as we have some idea of how to recreate/update that file, it's probably fine | 21:16 |
| clarkb | ianw: any idea if review-test is in emergency or not? wondering it it makes sense to test staging on it? | 21:17 |
| ianw | not too sure, i've left review-test alone. i feel like if staging causes some particular issue with review02 that makes it unsuitable for production, we can always start again (and fix that issue :) at this point | 21:18 |
| clarkb | ya I'm mostly worried about it doing too much but I confirmed not replication targets in your item and thats about all I can think would cause problems | 21:20 |
| clarkb | left some thoughts and hints for other reviewers but +2'd | 21:20 |
| *** akahat has joined #opendev | 21:20 | |
| ianw | thanks, yeah i expect it to be a high-touch merge | 21:20 |
| ianw | i have a bunch of smaller things in system-config, if you're feeling review-y, but nothing major | 21:21 |
| ianw | clarkb: also, if you'd like to discuss glean, we can :) i may need a cup of tea prior :) | 21:21 |
| clarkb | I'm up for it | 21:22 |
| ianw | i have to admit i'm still not 100% clear on the "ignore dhcp changes" | 21:25 |
| ianw | anyway i came up with https://review.opendev.org/q/project:opendev%252Fglean+status:open+topic:exec | 21:25 |
| ianw | what this essentially does is splits things up so that there is a "early" glean service that sets the ssh keys/hostname | 21:25 |
| ianw | and then the udev rules *only* setup the interface they're called for | 21:26 |
| ianw | i don't know if maybe what ironic wants is actually a way to only run the early service, and ignore NIC's (leaving setup to something else) | 21:27 |
| ianw | but, independent of that, i think that series of changes is generally correct to actually divide up what glean is doing to avoid running things multiple times | 21:28 |
| clarkb | ianw: I think there are a cuple of competing issues in ironic which has complicated the ability to think about it. | 21:28 |
| clarkb | One of them is that they would like to be able to use the dhcp-all-interfaces role and simple-init together and have simple-init basically do nothing if there is no config drive | 21:29 |
| clarkb | another is making the cost of running glean lower so that their fake baremetal testing can use glean relaibly with the qemu emulation | 21:29 |
| ianw | the first one is the one i find odd | 21:30 |
| clarkb | ianw: ya I found it odd too, but apparently they added a bunch of functionality to dhcp-all-interfaces that probably should've got into simple-init/glean | 21:31 |
| clarkb | but that puts then in an awkward spot where sometimes they want both things? | 21:31 |
| openstackgerrit | Merged opendev/system-config master: Rename refstack group variables https://review.opendev.org/c/opendev/system-config/+/781593 | 21:33 |
| clarkb | ianw: in particular the difference between what glean and dhcp-all-interfaces does is it allows you to bake in config on the image that will work whether or not you've got a config drive later | 21:33 |
| clarkb | ianw: I suspect that the use case here is that they make a ramdisk image that all the people use and want it to work whether or not config drive is also used | 21:34 |
| clarkb | the first change in the stack seems like a good refactor regardless of use case needs | 21:41 |
| *** slaweq has quit IRC | 21:44 | |
| clarkb | ianw: having skimmed the changes but not given them proper review I think it makes sense from the perspective of making it easier for people to understand what the pieces are that go into this. My one concern is that I think the glean-early + udev rules make make the low resource use case (fake baremetal on qemu) worse? | 21:47 |
| clarkb | I definitely think the first change is safe eitehr way so will go ahead and give that a proper review now, but maybe we run the rest of the stack by dtantsur|afk ? | 21:48 |
| clarkb | ianw: for https://review.opendev.org/c/opendev/glean/+/782010/4/setup.cfg is the files directive just not necessary because pbr is already doing the right thing for us? | 21:55 |
| clarkb | (we still want glean.sh to be installed, but I assume it is handling that via some other mechanism) | 21:56 |
| clarkb | oh wait I may have misinterpreted what that directive does. That is what installs to /usr/local/bin/glean.sh | 21:59 |
| clarkb | but now we're just relying on pbr to put the file in the venv under the module path and we can run it out of there | 21:59 |
| fungi | clarkb: huh... https://review.opendev.org/782538 failed to merge on the same gerritlib test bug which its merged depends-on fixed... looks like https://zuul.opendev.org/t/openstack/build/f0a5a52aadc746db939b93b0c2310481/log/job-output.txt#660 installed gerritlib from a release version? | 22:03 |
| fungi | does that job install from source or release alternately? | 22:04 |
| fungi | that would be weird, but maybe we did something to integrate cross-repo dependencies which only works up to the point they get merged? | 22:05 |
| clarkb | I thought it was always from source. and we landed the gerritlib change first | 22:05 |
| fungi | exactly | 22:05 |
| fungi | hence my confusion on the failure | 22:06 |
| clarkb | https://zuul.opendev.org/t/openstack/build/f0a5a52aadc746db939b93b0c2310481/log/job-output.txt#652 | 22:06 |
| fungi | you can even see it's trying to connect to localhost instead of 127.0.0.1 with ssh-keyscan, so it's not using the patched gerritlib | 22:06 |
| fungi | oh, yep, i misread, it installed form source | 22:07 |
| clarkb | it does that after it installs jeepyb too to ensure jeepyb doesn't override it | 22:07 |
| fungi | so then did the cached source on the executor lack that commit maybe? | 22:07 |
| clarkb | ya maybe it didn't get the newer code somehow | 22:08 |
| fungi | Successfully uninstalled gerritlib-0.10.0 | 22:08 |
| fungi | Successfully installed gerritlib-0.10.0 | 22:08 |
| clarkb | https://zuul.opendev.org/t/openstack/build/f0a5a52aadc746db939b93b0c2310481/log/zuul-info/inventory.yaml#68-75 checkout:master | 22:08 |
| *** cloudnull has quit IRC | 22:09 | |
| fungi | the executors fetch from gerrit not from gitea, right? | 22:10 |
| clarkb | are there logs for the workspace setup that will give us a sha? | 22:10 |
| clarkb | fungi: yes | 22:10 |
| fungi | so stalled gerrit->gitea replication wouldn't be a reason | 22:10 |
| ianw | sorry, back now on a fairly sub-optimal 4g connection. no sync on my cable modem, i'm dreading the isp call "have you tried rebooting your computer" | 22:10 |
| ianw | clarkb: yeah, that first change relies on all the files being on disk and then us referencing them directly, rather than relying on things in /usr/local/bin | 22:11 |
| fungi | opendev.org/opendev/gerritlib checked out to: 99136e1e164baa7b1d9dac4f64c5fb511b813c19 integration test: run manage-projects with debug | 22:11 |
| fungi | clarkb: so that seems to confirm it didn't see the new commit | 22:12 |
| clarkb | https://zuul.opendev.org/t/openstack/build/f0a5a52aadc746db939b93b0c2310481/console#0/3/12/ubuntu-focal | 22:12 |
| fungi | https://zuul.opendev.org/t/openstack/build/f0a5a52aadc746db939b93b0c2310481/log/job-output.txt#320-323 | 22:13 |
| clarkb | ya I think that is the clue we want | 22:13 |
| clarkb | could this be related to zuul's change to make sure an entire buildset uses the same refs for all repos/ | 22:13 |
| fungi | clarkb: i wouldn't think so... the buildset should have been consistent either way | 22:18 |
| clarkb | fwiw I expected that it wouldn't work with depends on which is why I waited. But really odd to see this after a merge | 22:18 |
| fungi | well, it passed check using the same depends-on | 22:20 |
| fungi | so i would be surprised if it tested differently in check and gate in that regard | 22:20 |
| clarkb | oh I see huh | 22:21 |
| clarkb | in that case ya it really seems like it didn't properly update the master checkout which is supposed to happen with a gearman job to the mergers iirc | 22:21 |
| fungi | hence my confusion | 22:22 |
| fungi | anyway, i rechecked to see if it repeats | 22:22 |
| openstackgerrit | Ian Wienand proposed opendev/system-config master: openafs-server-config: install UserList https://review.opendev.org/c/opendev/system-config/+/783755 | 22:30 |
| fungi | ianw: that ^ looks incomplete. intentional? | 22:48 |
| ianw | fungi: ahh, yes that is a gid add failure sorry | 22:49 |
| openstackgerrit | Ian Wienand proposed opendev/system-config master: openafs-server-config: install UserList https://review.opendev.org/c/opendev/system-config/+/783755 | 22:50 |
| fungi | cool, figured that was it | 22:50 |
| ianw | sorry, as predicted currently dealing with someone who wants to know the mac address of my wireless router to fix a non-syncing cable modem :/ | 22:52 |
| *** cloudnull has joined #opendev | 22:52 | |
| ianw | fungi: yeah, we probably want to remove a few old entries from the PTS, and that list | 22:59 |
| clarkb | internet problems are the worst | 23:27 |
| openstackgerrit | Ian Wienand proposed opendev/system-config master: gitea: switch to token auth for project creation https://review.opendev.org/c/opendev/system-config/+/782887 | 23:37 |
| fungi | broadband service repair people, zomg | 23:37 |
| clarkb | waiting patiently for the gitea update to land and I realize there is a full deploy set of jobs in progress due to that refstack change modifying inventory stuff :) | 23:38 |
| clarkb | it should move along pretty quickly at this point, but still /me is impatient | 23:38 |
| openstackgerrit | Ian Wienand proposed opendev/system-config master: gitea: switch to token auth for project creation https://review.opendev.org/c/opendev/system-config/+/782887 | 23:39 |
| openstackgerrit | Merged opendev/system-config master: Upgrade gitea to 1.13.6 https://review.opendev.org/c/opendev/system-config/+/780923 | 23:43 |
| openstackgerrit | Merged opendev/system-config master: openafs-server-config: install UserList https://review.opendev.org/c/opendev/system-config/+/783755 | 23:43 |
Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!