| *** mrunge has quit IRC | 00:29 | |
| *** tosky has quit IRC | 00:35 | |
| *** mrunge has joined #opendev | 00:43 | |
| *** brinzhang_ has joined #opendev | 01:03 | |
| *** dpawlik6 has joined #opendev | 01:04 | |
| *** ysandeep has joined #opendev | 01:06 | |
| *** jpenag has joined #opendev | 01:10 | |
| *** brinzhang has quit IRC | 01:12 | |
| *** gnuoy has quit IRC | 01:12 | |
| *** SotK has quit IRC | 01:12 | |
| *** dpawlik has quit IRC | 01:12 | |
| *** jpena|off has quit IRC | 01:13 | |
| *** ysandeep|PTO has quit IRC | 01:13 | |
| *** arxcruz has quit IRC | 01:13 | |
| *** akahat has quit IRC | 01:13 | |
| *** lourot has quit IRC | 01:13 | |
| *** dpawlik6 is now known as dpawlik | 01:13 | |
| *** SotK has joined #opendev | 01:13 | |
| *** lourot has joined #opendev | 01:19 | |
| *** hamalq has quit IRC | 01:23 | |
| *** ysandeep is now known as ysandeep|bbl | 01:31 | |
| *** mlavalle has quit IRC | 01:43 | |
| openstackgerrit | Ian Wienand proposed opendev/system-config master: Remove references to review-dev https://review.opendev.org/c/opendev/system-config/+/780691 | 03:12 |
|---|---|---|
| *** whoami-rajat_ has joined #opendev | 03:24 | |
| openstackgerrit | Ian Wienand proposed opendev/system-config master: gerrit: download latest mysql connector https://review.opendev.org/c/opendev/system-config/+/776857 | 03:46 |
| openstackgerrit | Ian Wienand proposed opendev/system-config master: gerrit : add mariadb_container option https://review.opendev.org/c/opendev/system-config/+/775961 | 03:46 |
| openstackgerrit | Ian Wienand proposed opendev/system-config master: Create review-staging group https://review.opendev.org/c/opendev/system-config/+/780698 | 03:46 |
| openstackgerrit | Ian Wienand proposed opendev/system-config master: refstack: fix backup script typo https://review.opendev.org/c/opendev/system-config/+/780699 | 03:49 |
| openstackgerrit | Merged opendev/system-config master: gitea-git-repos: update deprecated API path https://review.opendev.org/c/opendev/system-config/+/741562 | 04:00 |
| openstackgerrit | Merged opendev/system-config master: Enable srvr, stat and dump commands in the zk cluster https://review.opendev.org/c/opendev/system-config/+/780303 | 04:10 |
| openstackgerrit | Ian Wienand proposed opendev/system-config master: refstack: fix backup script typo https://review.opendev.org/c/opendev/system-config/+/780699 | 04:13 |
| *** ykarel|away has joined #opendev | 04:32 | |
| *** ykarel|away is now known as ykarel | 04:37 | |
| *** dviroel has quit IRC | 05:02 | |
| *** LowKey has joined #opendev | 05:04 | |
| openstackgerrit | Ian Wienand proposed opendev/system-config master: Create review-staging group https://review.opendev.org/c/opendev/system-config/+/780698 | 05:19 |
| openstackgerrit | Ian Wienand proposed opendev/system-config master: gerrit: download latest mysql connector https://review.opendev.org/c/opendev/system-config/+/776857 | 05:19 |
| openstackgerrit | Ian Wienand proposed opendev/system-config master: gerrit: add mariadb_container option https://review.opendev.org/c/opendev/system-config/+/775961 | 05:19 |
| openstackgerrit | Merged opendev/system-config master: refstack: fix backup script typo https://review.opendev.org/c/opendev/system-config/+/780699 | 05:43 |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/project-config master: Normalize projects.yaml https://review.opendev.org/c/openstack/project-config/+/780706 | 06:12 |
| *** zbr has quit IRC | 06:13 | |
| *** zbr has joined #opendev | 06:15 | |
| *** ysandeep|bbl is now known as ysandeep | 06:20 | |
| *** marios has joined #opendev | 06:33 | |
| openstackgerrit | Merged openstack/diskimage-builder master: replace the link which is in the 06-hpdsa file https://review.opendev.org/c/openstack/diskimage-builder/+/730286 | 06:34 |
| *** sboyron has joined #opendev | 06:47 | |
| *** slaweq has joined #opendev | 06:50 | |
| *** fressi has quit IRC | 06:52 | |
| *** hemanth_n has joined #opendev | 06:59 | |
| *** icey_ has quit IRC | 07:00 | |
| *** gnuoy` has joined #opendev | 07:00 | |
| *** icey has joined #opendev | 07:06 | |
| *** whoami-rajat_ is now known as whoami-rajat | 07:09 | |
| *** eolivare has joined #opendev | 07:32 | |
| *** calcmandan has quit IRC | 07:40 | |
| *** calcmandan has joined #opendev | 07:42 | |
| *** rpittau|afk is now known as rpittau | 07:58 | |
| *** hashar has joined #opendev | 08:05 | |
| *** andrewbonney has joined #opendev | 08:10 | |
| openstackgerrit | Daniel Blixt proposed zuul/zuul-jobs master: WIP: Make build-sshkey handling windows compatible https://review.opendev.org/c/zuul/zuul-jobs/+/780662 | 08:19 |
| *** amoralej|off is now known as amoralej | 08:19 | |
| *** gnuoy` is now known as gnuoy | 08:21 | |
| *** zigo has joined #opendev | 08:28 | |
| *** ykarel is now known as ykarel|lunch | 08:33 | |
| *** fressi has joined #opendev | 08:35 | |
| *** fressi has left #opendev | 08:35 | |
| *** tosky has joined #opendev | 09:00 | |
| *** lpetrut has joined #opendev | 09:02 | |
| *** ysandeep is now known as ysandeep|lunch | 09:11 | |
| *** brinzhang0 has joined #opendev | 09:12 | |
| *** brinzhang_ has quit IRC | 09:16 | |
| openstackgerrit | Merged openstack/project-config master: Normalize projects.yaml https://review.opendev.org/c/openstack/project-config/+/780706 | 09:22 |
| *** ykarel|lunch is now known as ykarel | 09:33 | |
| *** jpenag is now known as jpena | 09:33 | |
| *** akahat has joined #opendev | 09:48 | |
| *** sshnaidm has quit IRC | 09:50 | |
| openstackgerrit | Merged openstack/diskimage-builder master: Change paths for bootloader files in iso element https://review.opendev.org/c/openstack/diskimage-builder/+/777606 | 09:56 |
| *** dtantsur|afk is now known as dtantsur | 09:56 | |
| *** ysandeep|lunch is now known as ysandeep | 10:10 | |
| *** arxcruz has joined #opendev | 10:18 | |
| *** sshnaidm has joined #opendev | 10:27 | |
| *** smcginnis has joined #opendev | 10:48 | |
| *** dviroel has joined #opendev | 10:49 | |
| *** fressi has joined #opendev | 10:58 | |
| *** fressi has joined #opendev | 11:00 | |
| openstackgerrit | Daniel Blixt proposed zuul/zuul-jobs master: WIP: Make build-sshkey handling windows compatible https://review.opendev.org/c/zuul/zuul-jobs/+/780662 | 11:07 |
| *** smcginnis has quit IRC | 11:50 | |
| *** stand has quit IRC | 11:52 | |
| *** smcginnis has joined #opendev | 12:07 | |
| *** ykarel is now known as ykarel|afk | 12:10 | |
| *** eolivare_ has joined #opendev | 12:21 | |
| *** eolivare has quit IRC | 12:24 | |
| *** hemanth_n has quit IRC | 12:25 | |
| rosmaita | clarkb: fungi: i figured out another way to address that problem you helped me with yesterday: https://review.opendev.org/c/openstack/cinder/+/780692 (it's got a -1 from zuul for an unrelated reason) | 12:31 |
| *** jpena is now known as jpena|lunch | 12:39 | |
| *** smcginnis has quit IRC | 12:40 | |
| openstackgerrit | Rico Lin proposed opendev/system-config master: Remove TW User Group ML https://review.opendev.org/c/opendev/system-config/+/584035 | 12:42 |
| openstackgerrit | Aurelien Lourot proposed openstack/project-config master: Add Manila dashboard charm to OpenStack charms https://review.opendev.org/c/openstack/project-config/+/780812 | 12:42 |
| openstackgerrit | Aurelien Lourot proposed openstack/project-config master: Add Manila dashboard charm to OpenStack charms https://review.opendev.org/c/openstack/project-config/+/780812 | 12:45 |
| *** smcginnis has joined #opendev | 12:50 | |
| *** hemanth_n has joined #opendev | 13:12 | |
| *** smcginnis has quit IRC | 13:12 | |
| *** hemanth_n has quit IRC | 13:16 | |
| *** smcginnis has joined #opendev | 13:18 | |
| lourot | ^ fungi o/ hopefully this one will have the post-merge zuul jobs passing (pinging you directly because you have the context). Thanks! | 13:20 |
| *** artom has quit IRC | 13:21 | |
| *** artom has joined #opendev | 13:22 | |
| *** amoralej is now known as amoralej|lunch | 13:23 | |
| *** smcginnis has quit IRC | 13:24 | |
| *** smcginnis has joined #opendev | 13:41 | |
| *** jpena|lunch is now known as jpena | 13:42 | |
| *** yoctozepto has quit IRC | 13:53 | |
| *** ykarel|afk is now known as ykarel | 13:58 | |
| *** eharney has joined #opendev | 13:58 | |
| *** mlavalle has joined #opendev | 14:00 | |
| openstackgerrit | Daniel Blixt proposed zuul/zuul-jobs master: WIP: Make build-sshkey handling windows compatible https://review.opendev.org/c/zuul/zuul-jobs/+/780662 | 14:02 |
| *** amoralej|lunch is now known as amoralej | 14:18 | |
| openstackgerrit | Merged openstack/project-config master: Add Manila dashboard charm to OpenStack charms https://review.opendev.org/c/openstack/project-config/+/780812 | 14:29 |
| *** ysandeep is now known as ysandeep|dinner | 14:37 | |
| slittle1 | fatal: unable to access 'https://opendev.org/openstack/horizon.git/': Empty reply from server | 14:49 |
| clarkb | slittle1: what operation emitted that error? also if you open that url and inspect the ssl cert you can identify which of the backends you are talking to which may help with debugging | 14:50 |
| fungi | slittle1: also when did you see it? might help us correlate to our resource trending | 14:53 |
| slittle1 | seconds before I posted | 14:54 |
| slittle1 | sec .... | 14:54 |
| fungi | i'm going to guess it's gitea06 | 14:56 |
| fungi | cacti graphs show we exhausted all memory there a few minutes ago (8gb ram+8gb swap), probably find an oom in dmesg | 14:56 |
| clarkb | 01 is working (it is my "home" server) | 14:57 |
| clarkb | in the process of testing 02 now, but if cacti shows 06 is unhappy then that seems a likely culprit | 14:57 |
| slittle1 | git clone https://opendev.org/openstack/horizon.git -b stable/train | 14:57 |
| fungi | actually, no oom in gitea06's log since almost a month | 14:58 |
| *** gothicserpent has quit IRC | 14:58 | |
| fungi | but graphs show we topped out swap and went into massive iowait thrash there | 14:58 |
| clarkb | slittle1: and if you instpect the ssl cert for https://opendev.org/ it should show you an altname for the backend server serving requests to you through the load balancer | 14:59 |
| clarkb | if that shows 06 it would confirm fungi's suspicion (unless the load balancer has already moved 06 out of the rotation) | 14:59 |
| slittle1 | Cloning into 'horizon'... hangs/times out | 14:59 |
| slittle1 | wget https://opendev.org/openstack/horizon.git | 14:59 |
| slittle1 | --2021-03-16 10:58:31-- https://opendev.org/openstack/horizon.git | 14:59 |
| slittle1 | Resolving opendev.org (opendev.org)... 38.108.68.124, 2604:e100:3:0:f816:3eff:fe6b:ad62 | 14:59 |
| slittle1 | Connecting to opendev.org (opendev.org)|38.108.68.124|:443... connected. | 14:59 |
| slittle1 | HTTP request sent, awaiting response... 500 Internal Server Error | 14:59 |
| slittle1 | how do I get to the ssl cert ? | 15:00 |
| fungi | oh, if you're trying from the command line you can use openssl s_client | 15:01 |
| clarkb | I just open it in my browser and click on the little button to inspect the site security | 15:01 |
| clarkb | but yes, s_client works n the command line too | 15:01 |
| *** fressi has quit IRC | 15:01 | |
| *** fressi has joined #opendev | 15:02 | |
| *** fressi has left #opendev | 15:02 | |
| fungi | echo | openssl s_client -connect opendev.org:https > /dev/null | 15:02 |
| fungi | it'll report the depth=0 CN to stderr | 15:02 |
| fungi | which will contain the backend server name | 15:03 |
| lourot | fungi, you look busy, but when you have a moment, manage_projects timed out again: https://review.opendev.org/c/openstack/project-config/+/780812 - and so the subsequent service-zuul didn't run. Thanks! | 15:04 |
| clarkb | all of the backends but 06 show a short spike in tcp opens, but they seem to handle it ok. 06 on the other hand shows a longer spike and not handling it ok | 15:04 |
| slittle1 | echo | openssl s_client -connect opendev.org:https > /dev/null | 15:05 |
| slittle1 | depth=2 O = Digital Signature Trust Co., CN = DST Root CA X3 | 15:05 |
| slittle1 | verify return:1 | 15:05 |
| slittle1 | depth=1 C = US, O = Let's Encrypt, CN = R3 | 15:05 |
| slittle1 | verify return:1 | 15:05 |
| slittle1 | depth=0 CN = gitea06.opendev.org | 15:05 |
| slittle1 | verify return:1 | 15:05 |
| slittle1 | DONE | 15:05 |
| fungi | yeah, the depth=0 CN line is what's relevant | 15:05 |
| fungi | lourot: well, on a conference call right now, yeah, but will take a look when the meeting ends | 15:06 |
| clarkb | I wonder if we should take 06 out proactively and have it rebalance (alternative is the LB should notice on its own and rebalance) | 15:07 |
| clarkb | my keys are in another room and I am also on a conf call, so will have to dig in after | 15:08 |
| fungi | clarkb: yeah, looks like it's still basically at max swap, i'll try to remove it from the pool | 15:08 |
| fungi | #status log Temporarily disabled balance_git_https,gitea06.opendev.org in haproxy on gitea-lb01 | 15:11 |
| openstackstatus | fungi: finished logging | 15:11 |
| fungi | slittle1: if you try again, you should see a different backend server now | 15:11 |
| clarkb | and hopeflly the rebalance doesn't end up pointing the firehose at some other server | 15:12 |
| slittle1 | depth=0 CN = gitea02.opendev.org | 15:12 |
| slittle1 | cloning is working ! | 15:12 |
| fungi | clarkb: i wonder if it's manage-projects... we've posited that before... the log shows it's not finished gitea06 yet. correlation is not causation, but we seem to notice this around new project creation events | 15:14 |
| clarkb | fungi: oh! ya. Maybe we should just drop the code that tries to update descriptions | 15:14 |
| clarkb | fungi: and we can run that manually when necessary | 15:14 |
| clarkb | I strongly suspect that is what does it since it hits every project (and possibly with a regression in newer gitea not handling that as well as we'd like) | 15:15 |
| fungi | log says gitea06.opendev.org was handled at 14:34:22 and returned a log of internal server errors | 15:16 |
| fungi | http://cacti.openstack.org/cacti/graph.php?action=view&local_graph_id=66748&rra_id=all | 15:17 |
| fungi | correlation | 15:17 |
| *** fressi has joined #opendev | 15:18 | |
| fungi | even without sending https traffic to it, gitea is still consuming a lot of memory and generating a lot of iowait | 15:28 |
| clarkb | it is probably trying to complete the operations that it was working on | 15:29 |
| *** hemanth_n has joined #opendev | 15:29 | |
| fungi | yeah, swap use is falling, i expect that's a lot of i/o paging stuff back into active memory | 15:30 |
| openstackgerrit | Jeremy Stanley proposed opendev/system-config master: Stop updating Gitea project descriptions for now https://review.opendev.org/c/opendev/system-config/+/780904 | 15:39 |
| fungi | clarkb: ^ is that what you had in mind? | 15:39 |
| fungi | i wonder if we could be smarter about it, if querying the descriptions is lightweight and updating them is not, then a lbyl to avoid making unnecessary updates could help | 15:40 |
| clarkb | fungi: yes ish. I think we should move the description updates under the always_update flag. Also we test that so your change will fail as is. I can have a patchset up in a bit | 15:40 |
| fungi | aha, good point | 15:40 |
| clarkb | fungi: I've noticed a couple of other areas of the code that may not be tuned well for gitea | 15:41 |
| clarkb | I'll note them in the commit message for followup | 15:41 |
| openstackgerrit | Clark Boylan proposed opendev/system-config master: Don't always update gitea project descriptions https://review.opendev.org/c/opendev/system-config/+/780904 | 15:43 |
| clarkb | fungi: ^ something like that? | 15:43 |
| fungi | i'll compare to what i was about to push | 15:43 |
| clarkb | that may make the job run much longer, but I think testing that behavior is a good idea | 15:45 |
| fungi | do we need to put it in the "if create or self.always_update:" block instead? do we actually set the description in the make_gitea_project() method? | 15:46 |
| clarkb | fungi: ya creating a project also sets the description through a different process | 15:47 |
| clarkb | so update and create need to be exclusive | 15:47 |
| fungi | gitea06 has calmed back down now, i'll reenable it in the pool and keep an eye on it | 15:47 |
| clarkb | sounds good, we should keep an eye out for problems that indicate we may want to trigger replication for it | 15:48 |
| clarkb | (gerrit should retry if it fails, but seems like that doesn't awlays work out) | 15:48 |
| fungi | #status log Re-enabled gitea06 in haproxy now that the crisis has passed | 15:51 |
| *** hemanth_n has quit IRC | 15:51 | |
| openstackstatus | fungi: finished logging | 15:51 |
| *** ykarel is now known as ykarel|away | 15:56 | |
| clarkb | I'm going to approve the nl01.opendev.org inventory change now | 16:12 |
| openstackgerrit | Clark Boylan proposed opendev/system-config master: Upgrade gitea to 1.13.4 https://review.opendev.org/c/opendev/system-config/+/780923 | 16:20 |
| clarkb | noticed that we're a couple of minor releases behind and there are some performance related bug fixes. Figure we should update as part of the debugging as well | 16:21 |
| fungi | clarkb: should i do a zuul-scheduler smart-reconfigure (the manage-projects nested tasks are completing, the job around them is just timing out) | 16:23 |
| clarkb | fungi: you should double check that gerrit has updated first since gerrit happens after gitea | 16:24 |
| clarkb | if gerrit also looks good then ya I'd do the smart reconfigure command | 16:24 |
| fungi | TASK [gerrit : Run manage-projects] ... "rc": 0 | 16:27 |
| fungi | plus the thousands of lines of output you would expect | 16:27 |
| fungi | so yeah, it ran | 16:28 |
| *** ykarel|away has quit IRC | 16:29 | |
| clarkb | huh, how did it timeout then? | 16:30 |
| *** LowKey has quit IRC | 16:30 | |
| *** LowKey has joined #opendev | 16:32 | |
| fungi | it's nested, right? | 16:36 |
| fungi | the job timed out but the task on bridge kept running to completion after that | 16:36 |
| clarkb | I would've expected the parent process going away to kill the nested one but ya that is possible | 16:37 |
| clarkb | and you're sure you identified the correct ansible run log? | 16:37 |
| clarkb | I would check that the new projects are in gerrit directly as a safety valve. Zuul will complain excessively if we tell it to manage ap roject that doesn't exist iirc | 16:38 |
| fungi | clarkb: though, i think that also means i don't actually need to reconfigure zuul | 16:38 |
| fungi | Last reconfigured: Tue, Mar 16, 2021 3:28 PM | 16:39 |
| fungi | (that's 15:28 utc) | 16:39 |
| clarkb | looks recent enough | 16:39 |
| fungi | that's a few minutes after manage-projects | 16:40 |
| clarkb | unexpected success, the best kind of success | 16:40 |
| fungi | technically correct | 16:40 |
| lourot | thanks, does that mean that my new projects are fine? | 16:41 |
| *** marios is now known as marios|out | 16:41 | |
| *** kopecmartin has joined #opendev | 16:44 | |
| openstackgerrit | Merged opendev/system-config master: Add nl01.opendev.org to our inventory https://review.opendev.org/c/opendev/system-config/+/780620 | 16:45 |
| fungi | lourot: yes, zuul seems to know about it https://zuul.opendev.org/t/openstack/project/opendev.org/openstack/charm-manila-dashboard | 16:47 |
| fungi | the previous ones probably were as well | 16:48 |
| *** hamalq has joined #opendev | 16:48 | |
| fungi | i think we assumed the scheduler might not have reconfigured after the previous timeout, but it probably did | 16:48 |
| lourot | \o/ thanks a lot! | 16:50 |
| *** marios|out has quit IRC | 16:51 | |
| *** hashar has quit IRC | 16:53 | |
| *** ysandeep|dinner is now known as ysandeep | 16:54 | |
| *** ysandeep is now known as ysandeep|away | 17:03 | |
| *** LowKey has quit IRC | 17:07 | |
| *** roman_g has joined #opendev | 17:20 | |
| roman_g | Hello team. Could I ask you to create a branch for us, please? https://review.opendev.org/admin/repos/airship/docs,branches branch name 'v2.0.0' from current HEAD. Tank you. I'm core of the project, however I don't think I have permissions. OpenDev docs reference https://docs.opendev.org/opendev/infra-manual/latest/drivers.html . | 17:22 |
| roman_g | *Thank you | 17:22 |
| clarkb | roman_g: you can update your project acls then do it yourself | 17:26 |
| clarkb | let me double check that repo | 17:26 |
| clarkb | roman_g: https://opendev.org/openstack/project-config/src/branch/master/gerrit/acls/airship/docs.config#L3 the airship-release group should be able to do it | 17:27 |
| roman_g | Ah, all right. Thank you. | 17:27 |
| roman_g | Thank you, clarkb. :) | 17:27 |
| roman_g | I forgot we have it. | 17:28 |
| *** rpittau is now known as rpittau|afk | 17:29 | |
| *** roman_g has quit IRC | 17:36 | |
| clarkb | hrm the LE deploy job failed so all the other deploy jobs behind it are skipped | 17:40 |
| clarkb | we do run nodepool hourly though so I'll wait for the next hourly pass I guess | 17:40 |
| clarkb | I'll look into why the LE job failed shortly | 17:40 |
| clarkb | nb01.opendev.org failed to have the acme.sh client installed | 17:42 |
| clarkb | I suspect we've run out of disk there again | 17:43 |
| *** eolivare_ has quit IRC | 17:43 | |
| clarkb | yup that suspicion is the case | 17:45 |
| *** dtantsur is now known as dtantsur|afk | 17:47 | |
| *** amoralej is now known as amoralej|off | 17:48 | |
| clarkb | fungi: also the stop doing description updates change failed beacuse the zuul description wasn't updated. I thought my edits to the change would address that but maybe there is a bug in there | 17:49 |
| clarkb | I see the bug | 17:52 |
| fungi | we only call the always_update block under certain circumstances, right? | 17:53 |
| * fungi traces that attribute | 17:53 | |
| openstackgerrit | Clark Boylan proposed opendev/system-config master: Don't always update gitea project descriptions https://review.opendev.org/c/opendev/system-config/+/780904 | 17:54 |
| clarkb | fungi: ^ the elif was never hit because it was exclusive to the previous block which would always run if always_update is set | 17:54 |
| clarkb | and yes we only run always_update in certain circumstances, though the test was updated to set it | 17:54 |
| fungi | ahh, it defaults to true and we also set it true | 17:55 |
| fungi | and yeah, that elif as written will never match | 17:56 |
| clarkb | it defaults to false | 17:56 |
| clarkb | at least at the role level | 17:57 |
| clarkb | it looks like nb01 has managed to take all the image builds and nb02 doesn't have any | 17:57 |
| clarkb | and that is part of the reason why its disk has filled up | 17:57 |
| clarkb | I have stopped the builder on 01 and am cleaning out /opt/dib_tmp and will reboot it | 17:58 |
| clarkb | but then I think I'll leave it off for a bit to see if nb02 can build an image | 17:58 |
| clarkb | the dib tmp cleanup got us 74GB of room. nb02 picking up some image slack would give us more | 17:59 |
| *** lpetrut has quit IRC | 18:00 | |
| clarkb | I've also found a couple of leaked vhd intermediate.bak files | 18:01 |
| clarkb | cleaning this up has us to 151GB of free disk now | 18:01 |
| fungi | clarkb: on line 332 we set always_update=dict(type='bool', default=True) | 18:02 |
| clarkb | fungi: interesting, the role itself sets a false default and then passes that in | 18:03 |
| fungi | yeah | 18:03 |
| fungi | i missed the middle tier | 18:03 |
| fungi | so the module defaults it to true, the role defaults it to false, then our invocation of the role set it to true | 18:04 |
| *** jpena is now known as jpena|off | 18:04 | |
| clarkb | on nb02 I'd like to see if this running centos-7 build succeeds then stop the builder there, do the same dib_tmp cleanup, reboot and restart the service on that host | 18:04 |
| fungi | makes sense | 18:05 |
| clarkb | looking at older build logs on nb02 it looks fairly unhappy | 18:06 |
| clarkb | exec_sudo: losetup: /opt/dib_tmp/dib_image.3w2ZnmAi/image0.raw: failed to set up loop device: No such file or directory exec_sudo /usr/local/lib/python3.7/site-packages/diskimage_builder/block_device/utils.py:135 | 18:06 |
| *** lbragstad has quit IRC | 18:07 | |
| clarkb | that dir does exist but is empty | 18:10 |
| clarkb | so there is no image0.raw | 18:10 |
| fungi | clarkb: after reading trough the gitea issues about argon2, https://github.com/go-gitea/gitea/issues/14702 suggests api tokens... do you happen to know if that's already a feature or are they suggesting adding it? | 18:10 |
| clarkb | fungi: I believe it is already a feature | 18:11 |
| fungi | clarkb: are you expecting the No such file or directory error is due to full filesystem? | 18:11 |
| fungi | or no? | 18:11 |
| clarkb | looking at that image build log it cleans up the image file so no surprise it is gone | 18:11 |
| clarkb | fungi: no this is on nb02 which has plenty of disk | 18:12 |
| clarkb | I don't understand this problem but figure I'll still do the planned cleanup and restart and take it from there | 18:12 |
| clarkb | that same issue just hit the centos-7 build that was running | 18:13 |
| clarkb | I'll stop the builder on 02 now | 18:13 |
| *** LowKey has joined #opendev | 18:17 | |
| openstackgerrit | Kendall Nelson proposed opendev/system-config master: Move PTG bot site to opendev namespace https://review.opendev.org/c/opendev/system-config/+/780942 | 18:21 |
| clarkb | 02 has been cleaned up and rebooted and the service is restarted | 18:25 |
| clarkb | the centos-7 image is being rebuilt there now and we can watch it to see if the problem persists there | 18:25 |
| *** lbragstad has joined #opendev | 18:27 | |
| clarkb | I'm going to take a break before the meeting, back in a bit' | 18:27 |
| openstackgerrit | Jeremy Stanley proposed opendev/puppet-ptgbot master: Redirect from server aliases https://review.opendev.org/c/opendev/puppet-ptgbot/+/780947 | 18:44 |
| fungi | diablo_rojo_phon: you're going to want this ^ too | 18:44 |
| fungi | and then plumb the aliases list in with ptg.openstack.org in it | 18:45 |
| *** ralonsoh has quit IRC | 18:51 | |
| *** andrewbonney has quit IRC | 19:09 | |
| *** rosmaita has left #opendev | 19:20 | |
| openstackgerrit | Kendall Nelson proposed opendev/system-config master: Move PTG bot site to opendev namespace https://review.opendev.org/c/opendev/system-config/+/780942 | 19:26 |
| *** whoami-rajat has quit IRC | 19:26 | |
| *** rosmaita has joined #opendev | 19:31 | |
| rosmaita | cinder just merged a patch that fixes our docs job by modifying tox.ini ... do all the patches that failed that job earlier need to be rebased, or can they just be rechecked (or does it depend?) | 19:33 |
| clarkb | rosmaita: zuul tests the outcome of code merging. This means that one of the very first things it does behind the scenes when running a job is it merges the code as proposed into the target branch. If this fails you get a conflict error. If it succeeds the job runs against that potential future state | 19:34 |
| clarkb | long story short you do not need to rebase unless there is a conflict | 19:34 |
| rosmaita | great, ty | 19:34 |
| *** persia has quit IRC | 19:46 | |
| *** persia has joined #opendev | 19:47 | |
| clarkb | the launcher on nl01.opendev.org has started. I see it declining requests since its max-servers value is 0. This is roughly what I expected to happen which is good. | 19:52 |
| *** persia has quit IRC | 19:52 | |
| clarkb | infra-root https://review.opendev.org/c/openstack/project-config/+/779864 should be ready to go I think now that both nl01.openstack.org and nl01.opendev.org are running | 19:53 |
| *** persia has joined #opendev | 19:54 | |
| clarkb | oh I should've mentioned the gitea upgrade in open discussion | 19:58 |
| clarkb | https://review.opendev.org/c/opendev/system-config/+/780923 it passes testing, though I think we should land https://review.opendev.org/c/opendev/system-config/+/780904 frist then rerun | 19:59 |
| ianw | clarkb: should we document that flag in 780904? it only needs a manual run when a description changes? | 20:00 |
| clarkb | ianw: good idea, it is a preexisting flag but it doesn't exist in the docs | 20:01 |
| *** persia has quit IRC | 20:01 | |
| clarkb | ianw: ya its a preexisting flag that we used to coerce things if tehy somehow got out of sync, we haven't had to use it in a long time | 20:01 |
| clarkb | I figured we could add description updates to that coercion while we sort things out. I'll push a new ps with a role readme update | 20:01 |
| *** persia has joined #opendev | 20:02 | |
| *** tosky has quit IRC | 20:05 | |
| *** tosky has joined #opendev | 20:05 | |
| openstackgerrit | Clark Boylan proposed opendev/system-config master: Don't always update gitea project descriptions https://review.opendev.org/c/opendev/system-config/+/780904 | 20:06 |
| clarkb | ianw: ^ something like that? | 20:06 |
| clarkb | (I updated the same change so you'll need to reapply votes) | 20:07 |
| openstackgerrit | Ian Wienand proposed opendev/system-config master: refstack: cleanup old puppet https://review.opendev.org/c/opendev/system-config/+/780138 | 20:07 |
| clarkb | and now lunch | 20:08 |
| fungi | clarkb: i'm reminded, there was a post by mkopec to openstack-discuss back in november asking for issue link updating. maybe we need to treat it similarly to descriptions if we add that? | 20:09 |
| clarkb | to help me not forget I plan to restart the builder on nb01 towards the end of my day as that will have given nb02 a few hours to make some builds (it is doing that successfully now) | 20:09 |
| clarkb | fungi: they will be treated the same after https://review.opendev.org/c/opendev/system-config/+/780904 | 20:09 |
| fungi | http://lists.openstack.org/pipermail/openstack-discuss/2020-December/019416.html | 20:10 |
| clarkb | basically need an out of band run with always_update set to true | 20:10 |
| fungi | okay,. so always_update does update issue links too? | 20:10 |
| clarkb | fungi: ya thats what the block above you had discussed changing does | 20:10 |
| clarkb | though now I'm wodnering if that prior block would handle the descriptuon too. I seem to recall the description was special though | 20:11 |
| fungi | oh, yep right | 20:11 |
| clarkb | yup the testing that failed on ps2 confirms that description is separate | 20:11 |
| clarkb | otherwise we would've passed testing on ps2 when the if elif was in use (and not if if) | 20:11 |
| fungi | i clearly shouldn't be looking at source code and frying things at the same time | 20:11 |
| clarkb | fungi: are you able to review https://review.opendev.org/c/openstack/project-config/+/779864 while frying things? | 20:13 |
| clarkb | I really need to go find lunch myself now though | 20:13 |
| *** yoctozepto has joined #opendev | 20:14 | |
| fungi | clarkb: yep, lgtm | 20:14 |
| openstackgerrit | Kendall Nelson proposed opendev/system-config master: Move PTG bot site to opendev namespace https://review.opendev.org/c/opendev/system-config/+/780942 | 20:15 |
| *** slaweq has quit IRC | 20:26 | |
| openstackgerrit | James E. Blair proposed opendev/system-config master: Use grafyaml container image https://review.opendev.org/c/opendev/system-config/+/780128 | 20:27 |
| *** slaweq has joined #opendev | 20:29 | |
| clarkb | ianw: looking at the kerberos kdc role change again. Would it make sense to run the service-kerberos playbook twice in testing to ensure that you properly noop on the second pass sicne the files exist? | 20:38 |
| clarkb | the gitea job runs manage projects twice which is what made me think of that | 20:39 |
| clarkb | anyone else want to review https://review.opendev.org/c/openstack/project-config/+/779864 before I approve it to flip nl01.openstack.org nodepool config to nl01.opendev.org? | 20:40 |
| ianw | clarkb: yes, could do as a double-check | 20:52 |
| clarkb | ianw: also you should see my comment on https://review.opendev.org/c/opendev/system-config/+/776857 | 20:54 |
| clarkb | I've approved the project-config change to flip over nl01.openstack.org to nl01.opendev.org in nodepool configs. I'll watch it | 20:56 |
| ianw | clarkb: hrm, good catch. i guess we should just leave that one alone; it was really just a thought as i was figuring out where all the .jar's come from | 20:56 |
| clarkb | ianw: ya, I think we can make that change later, but right now the git history of gerrit itself seems to imply that maybe we aren't ready yet | 20:57 |
| ianw | maybe i'll replace that change with "do not do this!" :) | 20:57 |
| clarkb | wfm :) | 20:57 |
| ianw | what is the best way to ping someone on https://gerrit-review.googlesource.com/c/gerrit/+/297522 ? i guess the ML? | 20:58 |
| clarkb | ianw: ya, I've tried their slack server for questions and pings and they seem tp ush people to the ML there | 20:59 |
| *** sboyron has quit IRC | 21:00 | |
| openstackgerrit | Merged openstack/project-config master: Flip nl01.openstack.org to nl01.opendev.org https://review.opendev.org/c/openstack/project-config/+/779864 | 21:02 |
| ianw | i've successfully imported the trove mysql dump into mariadb, so that fix is really only for testing where we start a fresh db | 21:04 |
| clarkb | I'm tailing the launcher debug logs on both hosts and grepping for 'Active requests' that seems to be the info to watch and see that things have flipped over | 21:09 |
| clarkb | the transition has begun | 21:13 |
| *** mlavalle has quit IRC | 21:13 | |
| clarkb | there might be some quota fights but seems to be happy so far | 21:14 |
| fungi | okay, dinner is eaten and cleaned up, i'm back now for more fun | 21:15 |
| fungi | looks like the flippin' launcher went well | 21:16 |
| clarkb | so far at least, I'm still wating for the old one to be done | 21:17 |
| clarkb | once its active request set falls to zero I think I can stop the service on that server, and start working on changes to clean it up | 21:17 |
| clarkb | then I guess tomorrow plan to launch 02 03 and 04 and get them swapped too | 21:17 |
| fungi | maybe to address the drift on project descriptions, issue links, et cetera we could have a periodic job which just runs with always_update in a weekly pipeline or something | 21:21 |
| clarkb | fungi: we probably also want to try and make those other improvements first as I expect doing the always_update will be disruptive as is | 21:22 |
| clarkb | (but we can currently choose to take that risk if we need to) | 21:22 |
| fungi | it might not be so bad if it happened at a time when there's little other load to contend with | 21:23 |
| clarkb | that is a good point | 21:23 |
| fungi | though if this is related to password hash memory usage, i do wonder why it would hit one server so hard and not impact the other 7, unless there was something else going on at the same time to compound it | 21:24 |
| *** slaweq has quit IRC | 21:24 | |
| clarkb | ya I suspect that isn't the direct cause, but could be something that applies more pressure to an already busy system | 21:26 |
| openstackgerrit | Ian Wienand proposed opendev/system-config master: kerberos-kdc: role to manage Kerberos KDC servers https://review.opendev.org/c/opendev/system-config/+/778840 | 21:32 |
| openstackgerrit | Ian Wienand proposed opendev/system-config master: kerberos: switch servers to Ansible control https://review.opendev.org/c/opendev/system-config/+/779890 | 21:32 |
| openstackgerrit | Ian Wienand proposed opendev/system-config master: kerberos-kdc: add database backups https://review.opendev.org/c/opendev/system-config/+/779891 | 21:32 |
| openstackgerrit | Ian Wienand proposed opendev/system-config master: system-config-run-kerberos: run twice https://review.opendev.org/c/opendev/system-config/+/780979 | 21:32 |
| ianw | hrm, i didn't mean to rebase that | 21:33 |
| fungi | seems it was a trivial rebase at least | 21:47 |
| fungi | i was reviewing the first in the series when that hit, but i based my vote on the previous patchset since gerrit carried over clarkb's vote | 21:48 |
| openstackgerrit | Clark Boylan proposed openstack/project-config master: Add idle configs for nl02-04.opendev.org https://review.opendev.org/c/openstack/project-config/+/780982 | 21:57 |
| openstackgerrit | Clark Boylan proposed openstack/project-config master: Flip nl02-04.openstack.org to nl02-04.opendev.org https://review.opendev.org/c/openstack/project-config/+/780983 | 21:57 |
| openstackgerrit | Clark Boylan proposed openstack/project-config master: Remove unused nl0X.openstack.org config files https://review.opendev.org/c/openstack/project-config/+/780984 | 21:57 |
| openstackgerrit | Clark Boylan proposed openstack/project-config master: Update nodepool zk configs to be a bit less confusing https://review.opendev.org/c/openstack/project-config/+/780985 | 21:57 |
| clarkb | the first change in that stack is safe to land now | 21:57 |
| clarkb | the others need to be landed in coordination with new servers existing and being happy and others need to wait for old servers ot be out of inventory | 21:58 |
| clarkb | but that is all prep work for trying to get through these tomorrow | 21:58 |
| clarkb | nl01.openstack.org is looking idle now. I'll stop its nodepool-launcher | 21:58 |
| openstackgerrit | Clark Boylan proposed opendev/system-config master: Cleanup nl01.openstack.org https://review.opendev.org/c/opendev/system-config/+/780986 | 21:59 |
| clarkb | I think ^ is ready to land whenever we are happy with nl01.opendev.org | 21:59 |
| clarkb | I set the first unsafe change to workinprogress to avoid any confusion | 22:00 |
| clarkb | feel free to review :) | 22:01 |
| clarkb | I'm going to start launching 02-04 now with the intention of getting them ansibled and flipped over tomorrow morning | 22:04 |
| *** stevebaker has quit IRC | 22:07 | |
| clarkb | nb02 just failed an image build with DEBUG diskimage_builder.block_device.utils [-] exec_sudo: umount: /opt/dib_tmp/dib_build.zODdHTpF/mnt/: target is busy. exec_sudo /usr/local/lib/python3.7/site-packages/diskimage_builder/block_device/utils.py:135 | 22:10 |
| clarkb | I wonder if this is genesis of the problem I noticed earlier (it did successfully build two centos images first though) | 22:11 |
| clarkb | ianw: ^ is that familar to you? | 22:11 |
| ianw | clarkb: hrm ... no | 22:11 |
| ianw | that sounds like some daemon we might have started that hasn't shutdown? | 22:12 |
| fungi | i've seen it when processes get left running in a chroot | 22:12 |
| fungi | yeah, like that | 22:12 |
| clarkb | hrm maybe the suse builds are leaking something that causes that? | 22:12 |
| clarkb | it has started a new suse tumbleweed build so we get to see if it persists | 22:12 |
| ianw | 2021-03-16 20:47:13.185 | /tmp/in_target.d/finalise.d/89-boot-settings: line 140: /etc/init.d/boot.local: No such file or directory | 22:12 |
| ianw | that seemed to cause the failure in opensuse-tumbleweed-0000301586.log | 22:13 |
| clarkb | oh there is an earlier failure? I should've checked for that | 22:13 |
| clarkb | thinking out loud here we may want to pause that image build to try and get nb02 to build more images so nb01 can be restarted | 22:14 |
| ianw | there's a lot of | 22:15 |
| ianw | exec_sudo: losetup: /opt/dib_tmp/dib_image.oQ9ri5cB/image0.raw: failed to set up loop device: No such file or directory | 22:15 |
| clarkb | ianw: ya those are teh errors that caused nb01 to fill up bceause nb02 stopped building images | 22:18 |
| clarkb | ianw: what I had done was to stop nb01 and clean it up but then did the same with nb02 hoping it could then build a number of images to take load off of nb01 before starting nb01 again | 22:18 |
| clarkb | the status now is nb01 has had dib_tmp cleaned out and the server was rebooted but nodepool-builder is stopped (I downed the docker compose setup) | 22:19 |
| clarkb | nb02 did the same but I started the service up again and it then managed to build a few images | 22:19 |
| clarkb | I was going to start up nb01 again when nb02 had built a number of images | 22:19 |
| openstackgerrit | Merged opendev/system-config master: Don't always update gitea project descriptions https://review.opendev.org/c/opendev/system-config/+/780904 | 22:21 |
| openstackgerrit | Merged opendev/system-config master: refstack: cleanup old puppet https://review.opendev.org/c/opendev/system-config/+/780138 | 22:21 |
| ianw | we have /dev/loop* ... | 22:22 |
| fungi | i think the error is complaining about the other end | 22:22 |
| fungi | the file to be mounted on the loop device | 22:22 |
| ianw | hrm ... /opt/dib_tmp/dib_image.WVe8UrDr/image0.raw: failed to set up loop device: No such file or directory | 22:23 |
| ianw | i dunno, i don't see how that file can't be there | 22:23 |
| clarkb | ianw: thats from a pre cleanup and restart build though right? | 22:25 |
| ianw | -rw-r--r-- 1 nodepool nodepool 1.6M Mar 16 16:07 ubuntu-focal-0000086000.log | 22:25 |
| *** brinzhang_ has joined #opendev | 22:26 | |
| ianw | -rw-r--r-- 1 nodepool nodepool 1.6M Mar 16 17:48 ubuntu-focal-0000086235.log | 22:26 |
| ianw | also | 22:26 |
| ianw | Create image file [/opt/dib_tmp/dib_image.3w2ZnmAi/image0.raw] | 22:27 |
| ianw | with open(filename, "w") as fd: | 22:27 |
| ianw | fd.seek(size - 1) | 22:27 |
| ianw | fd.write("\0") | 22:27 |
| clarkb | ya those were pre cleanups | 22:27 |
| openstackgerrit | Merged opendev/system-config master: kerberos-kdc: role to manage Kerberos KDC servers https://review.opendev.org/c/opendev/system-config/+/778840 | 22:28 |
| *** lbragstad_ has joined #opendev | 22:29 | |
| ianw | reboot system boot 4.15.0-137-gener Tue Mar 16 18:24 still running | 22:29 |
| ianw | ok, so after 18:24 | 22:29 |
| ianw | so it built centos-7 & 8 and is now stuck on tumbleweed | 22:29 |
| openstackgerrit | Clark Boylan proposed opendev/zone-opendev.org master: Add nl02-04 to DNS https://review.opendev.org/c/opendev/zone-opendev.org/+/780988 | 22:30 |
| clarkb | ianw: ya | 22:30 |
| clarkb | or at least may be stuck on tumbleweed, not sure if it will fail that again | 22:30 |
| ianw | it looks like a systematic error | 22:31 |
| ianw | /tmp/in_target.d/finalise.d/89-boot-settings: line 140: /etc/init.d/boot.local: No such file or directory | 22:31 |
| *** calcmandan_ has joined #opendev | 22:31 | |
| fungi | maybe that element has bitrotted against a recent change in tumbleweed? | 22:32 |
| openstackgerrit | Clark Boylan proposed opendev/system-config master: Add new opendev.org nodepool launchers https://review.opendev.org/c/opendev/system-config/+/780989 | 22:32 |
| ianw | that's a prjoect-config element | 22:32 |
| clarkb | fungi: ya could be | 22:32 |
| clarkb | ok all the changes to do nl02-04 are now up. Just a matter of reviewing them for accuracy and landing them in the right orders | 22:33 |
| ianw | https://opendev.org/openstack/project-config/src/branch/master/nodepool/elements/nodepool-base/finalise.d/89-boot-settings | 22:33 |
| ianw | if [[ "$DISTRO_NAME" =~ (opensuse) ]] ; then | 22:34 |
| ianw | rclocal=/etc/init.d/boot.local | 22:34 |
| ianw | i'm guessing tumbleweed has decided this is no longer cool | 22:34 |
| clarkb | I'm on tumbleweed let me see what I've got if anything | 22:34 |
| clarkb | ianw: I've got /etc/init.d/boot.d | 22:34 |
| clarkb | as a directory | 22:35 |
| clarkb | maybe we can set rclocal to /etc/init.d/boot.d/opendev.local? | 22:35 |
| *** lbragstad has quit IRC | 22:35 | |
| *** LowKey has quit IRC | 22:35 | |
| *** sshnaidm has quit IRC | 22:35 | |
| *** brinzhang0 has quit IRC | 22:35 | |
| *** calcmandan has quit IRC | 22:35 | |
| *** zbr has quit IRC | 22:37 | |
| fungi | so they changed to a run-parts replacement for it i guess | 22:37 |
| clarkb | ya | 22:37 |
| clarkb | that is my guess | 22:37 |
| clarkb | though everything is empty locally | 22:37 |
| clarkb | https://forums.opensuse.org/showthread.php/531768-Where-Art-Thou-Boot-Local says the rc-local.service is what ingests the scripts | 22:37 |
| * clarkb looks to see what that looks like | 22:37 | |
| clarkb | ExecStart=/etc/init.d/boot.local start is what it says | 22:38 |
| ianw | clarkb: do you have a packae "aaa_base"? this seems to be where it may have previously been shipping a boot.local file | 22:38 |
| clarkb | maybe /etc/init.d isn't there at all on the images | 22:38 |
| clarkb | i+ | aaa_base <- yes the i means it is installed | 22:39 |
| clarkb | there is also an aaa_base-extras which is installed | 22:39 |
| ianw | https://build.opensuse.org/package/revisions/openSUSE:Factory/aaa_base | 22:40 |
| ianw | https://build.opensuse.org/request/show/826262 | 22:41 |
| ianw | Let's own /etc/init.d/ as it is gone from package filesystem | 22:41 |
| *** sshnaidm has joined #opendev | 22:41 | |
| ianw | 7 months ago though ... | 22:41 |
| ianw | seems to me it should be creating a boot.local file in the .spec @ https://build.opensuse.org/package/view_file/openSUSE:Factory/aaa_base/aaa_base.spec?expand=1 line 134 | 22:43 |
| clarkb | aaa_base is being installed according to the build log too | 22:44 |
| clarkb | could it be a permissions issue? | 22:44 |
| clarkb | we're trying to redirect straight into the file and not using a tmp file then mv'ing it with sudo | 22:45 |
| *** rosmaita has left #opendev | 22:45 | |
| clarkb | hrm the forwarding.conf update would imply we have privs there though | 22:45 |
| clarkb | as we chown root:root without sudo | 22:46 |
| ianw | yeah it must be as root | 22:46 |
| ianw | looking in /opt/dib_tmp/dib_build.ITSx0k5W/mnt/etc which is the currently building chroot, there is no /etc/init.d | 22:47 |
| clarkb | is it possible aaa_base updated and that spec is not longer current/valid? | 22:48 |
| clarkb | eg the TODO got done? | 22:48 |
| ianw | (201/224) Installing: aaa_base-84.87+git20210308.d7a7d3a-1.1.x86_64 [...........done] | 22:49 |
| ianw | that seems to all match, i'd say | 22:49 |
| *** mlavalle has joined #opendev | 22:50 | |
| clarkb | woudl the rpm pacakge build optimize that out since its a couple of empty files? | 22:50 |
| * clarkb downloads an rpm | 22:50 | |
| clarkb | https://download.opensuse.org/repositories/openSUSE:/Factory/standard/x86_64/aaa_base-84.87+git20210308.d7a7d3a-1.1.x86_64.rpm | 22:51 |
| clarkb | mark /etc/init.d/{boot,after}.local as %config(noreplace) (boo#1179097) <- shows up in the change log of that rpm | 22:52 |
| clarkb | would noreplace imply some sort of perms setting preventing changes? | 22:53 |
| clarkb | there is also a script that deletes those files in some situations | 22:54 |
| ianw | yeah, i've jumped into the chroot for the currently building, and reinstalling aaa_base the directory isn't made | 22:55 |
| ianw | # we have several local files, that changed over the time. Check the | 22:56 |
| ianw | # existing one, if they contain real data. If not, delete them. | 22:56 |
| clarkb | I can't find this script in the package stuff on obs that I see in the rpm | 22:56 |
| clarkb | ya thats teh script I'm talking about | 22:56 |
| clarkb | it is aaa_base.pre | 22:58 |
| clarkb | https://github.com/openSUSE/aaa_base/blob/master/aaa_base.pre found it | 22:58 |
| clarkb | rpm package guide says that pre is run prior to the install | 23:00 |
| clarkb | and it is the install that is making the empty files | 23:00 |
| clarkb | (that just makes me extra confused now( | 23:00 |
| clarkb | ianw: it does install /dev/null to those paths, do we maybe not have dev null present? | 23:01 |
| clarkb | (and maybe it just silently fails and moves on?) | 23:02 |
| clarkb | this is quickly getting outside of my packaging knowledge | 23:02 |
| clarkb | maybe we see if dirk has any good ideas? | 23:03 |
| ianw | yeah, i mean i don't know what changed but i think we have a pretty good clue where to look | 23:04 |
| ianw | we should probably just pause it I guess. we don't test project-config elements in the gate which is a bit of a bummer, why we've never noticed this | 23:05 |
| clarkb | I'm g ood with pausing so that we can focus on getting nb02 to build more images then get nb01 back in the toration | 23:06 |
| ianw | i have to run out for about 20 minutes but can pause it after and file a bug | 23:06 |
| clarkb | ianw: would you prefer I write that change? | 23:06 |
| clarkb | I can write it now | 23:06 |
| clarkb | jup to you | 23:06 |
| *** stevebaker has joined #opendev | 23:14 | |
| clarkb | infra-root https://review.opendev.org/c/opendev/zone-opendev.org/+/780988 https://review.opendev.org/c/openstack/project-config/+/780982 and https://review.opendev.org/c/opendev/system-config/+/780989 (note its parent https://review.opendev.org/c/opendev/system-config/+/780986/) should all be ready to go at this point | 23:24 |
| clarkb | when nl01.opendev.org came online ansible did not start its launcher | 23:24 |
| clarkb | but even if it had the idle configs seemed to work well | 23:24 |
| clarkb | ianw: thinking out loud about the boot.local issue maybe now that everything is using system we should write out a shell script and have a unit run it? | 23:27 |
| fungi | using system...d? | 23:29 |
| fungi | just making sure i grok the suggestion | 23:29 |
| clarkb | oh yes s/system/systemd/ | 23:30 |
| fungi | huh, system-config-run-meetpad failed in the gate for https://review.opendev.org/778308 | 23:31 |
| fungi | checking to see if it's just a fluke | 23:31 |
| fungi | ahh, nothing new | 23:34 |
| fungi | ERROR: for web toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit | 23:34 |
| ianw | sorry back now | 23:48 |
| ianw | trying to fix it is also a valid path forward :) | 23:49 |
Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!