Friday, 2021-07-16

*** odyssey4me is now known as Guest94501:22
*** odyssey4me is now known as Guest95501:53
opendevreviewMerged opendev/system-config master: Remove paste01.openstack.org  https://review.opendev.org/c/opendev/system-config/+/80087903:03
opendevreviewMerged opendev/system-config master: Enable openEuler mirroring  https://review.opendev.org/c/opendev/system-config/+/78487404:26
*** ykarel|away is now known as ykarel04:31
ianwi'm running ^ in a screen 04:39
*** chandankumar is now known as chkumar|rover05:21
*** iurygregory_ is now known as iurygregory06:38
*** amoralej|away is now known as amoralej07:02
*** rpittau|afk is now known as rpittau07:31
*** dmellado_ is now known as dmellado07:41
opendevreviewVishal Manchanda proposed openstack/project-config master: Retire django-openstack-auth  https://review.opendev.org/c/openstack/project-config/+/80045908:07
*** bhagyashris_ is now known as bhagyashris|ruck08:24
*** ykarel is now known as ykarel|lunch09:06
*** odyssey4me is now known as Guest101109:20
opendevreviewMatthias Runge proposed openstack/project-config master: Enable deleting left-overs of panko deprecation  https://review.opendev.org/c/openstack/project-config/+/80024109:56
*** odyssey4me is now known as Guest102510:06
*** ykarel|lunch is now known as ykarel11:11
*** dviroel|out is now known as dviroel11:19
*** amoralej is now known as amoralej|lunch12:05
*** marios_ is now known as marios12:40
*** marios is now known as marios|call13:10
*** amoralej|lunch is now known as amoralej13:13
opendevreviewAnanya Banerjee proposed opendev/elastic-recheck master: Run elastic-recheck container  https://review.opendev.org/c/opendev/elastic-recheck/+/72962313:14
opendevreviewAnanya Banerjee proposed opendev/elastic-recheck master: Run elastic-recheck container  https://review.opendev.org/c/opendev/elastic-recheck/+/72962313:20
*** marios|call is now known as marios13:34
opendevreviewTristan Cacqueray proposed openstack/project-config master: gerritbot: add matrix configuration  https://review.opendev.org/c/openstack/project-config/+/80106313:52
opendevreviewTristan Cacqueray proposed opendev/system-config master: Run matrix-gerritbot on eavesdrop  https://review.opendev.org/c/opendev/system-config/+/80050613:57
opendevreviewTakashi Kajinami proposed openstack/project-config master: tripleo-common-tempest-plugin - Step 2: End project Gating  https://review.opendev.org/c/openstack/project-config/+/80015413:59
opendevreviewTakashi Kajinami proposed openstack/project-config master: tripleo-common-tempest-plugin - Step 4: Remove Project  https://review.opendev.org/c/openstack/project-config/+/80015713:59
opendevreviewTristan Cacqueray proposed opendev/system-config master: Run matrix-gerritbot on eavesdrop  https://review.opendev.org/c/opendev/system-config/+/80050614:01
*** rpittau is now known as rpittau|afk14:08
*** ykarel is now known as ykarel|away14:46
opendevreviewTristan Cacqueray proposed openstack/project-config master: gerritbot: add matrix configuration  https://review.opendev.org/c/openstack/project-config/+/80106314:59
opendevreviewTristan Cacqueray proposed openstack/project-config master: gerritbot: add matrix configuration  https://review.opendev.org/c/openstack/project-config/+/80106315:22
opendevreviewTristan Cacqueray proposed openstack/project-config master: gerritbot: add matrix configuration  https://review.opendev.org/c/openstack/project-config/+/80106315:33
opendevreviewClark Boylan proposed opendev/system-config master: Push a patch to try and prevent gerrit openid deletion  https://review.opendev.org/c/opendev/system-config/+/80083215:59
clarkbThat is an update to get in sync with the current code upstream as well as add useful information to the commit message. I won't be trying to hold that and replace the currently held node with it since the code change is almost nil16:00
opendevreviewTakashi Kajinami proposed openstack/diskimage-builder master: Replace deprecated import of ABCs from collections  https://review.opendev.org/c/openstack/diskimage-builder/+/80109216:02
corvusclarkb: zuul master has a sql migration we haven't run in opendev; it's a column drop/add on the build table.  i think the last time we did something to the build table, it took about 30m for the migration to run.16:27
corvusclarkb: maybe we want to think about doing the zuul restart very late today or even tomorrow (i'm happy to do that).  or we could think about trimming the build table.16:28
clarkboh ya I remember we wondered why it was taking so long to start up16:29
clarkbcorvus: by trimming you mean drop older rows ya?16:29
corvuswe should at least have log messages telling us why it's taking so long this time :)16:30
corvusclarkb: yeah16:30
corvusof course, we'll need to trim some other tables too, so it's a little involved, but we probably have time to prepare those statements.16:30
corvus(we'll need to clean up the related artifact and buildset entries)16:31
clarkbI'm not sure I have a great idea as to which would be preferable at this point16:32
clarkbqueues are not small right now but they also aren't huge16:32
clarkbcorvus: do we maybe want to try this afternoon and just let the half hour migration happen?16:32
corvusthat's sort of how i'm leaning, yes16:33
corvusi'd rather do the cleanup by making a tool or setting in zuul (rather than doing it now by hand)16:33
opendevreviewTristan Cacqueray proposed opendev/system-config master: Run matrix-gerritbot on eavesdrop  https://review.opendev.org/c/opendev/system-config/+/80050616:34
clarkbI'm thinking that since we don't have a large backlog and it is friday we are unlikely to see queues grow16:34
clarkband maybe just doing the restart as planned is easiest (but do it this afternoon as we let some backlog get handled)16:34
corvusclarkb: sounds good16:34
clarkbcorvus: any concern for being able to rollback after the migration if we need to revert zuul changes?16:34
clarkbI guess as long as it isn't the db migration that needs to be rolled back we are fine16:35
*** marios is now known as marios|out16:35
corvusclarkb: yeah, we won't be able to drop back to the most recent tag; we'd have to revert a commit and make a new image.16:36
clarkbok, just want to make sure that is understood, but seems like we have an out16:37
opendevreviewVishal Manchanda proposed openstack/project-config master: Retire django-openstack-auth  https://review.opendev.org/c/openstack/project-config/+/80045916:43
opendevreviewVishal Manchanda proposed openstack/project-config master: Retire django-openstack-auth  https://review.opendev.org/c/openstack/project-config/+/80053216:43
*** amoralej is now known as amoralej|off16:45
clarkbgoogle followed up on my gerrit pushing issues and said it was likely an issue in their replication system17:00
opendevreviewTristan Cacqueray proposed opendev/system-config master: Run matrix-gerritbot on eavesdrop  https://review.opendev.org/c/opendev/system-config/+/80050618:29
corvusclarkb: how you feeling about a restart soonish?20:21
clarkbcorvus: let me look at the zuul satus20:29
clarkboh ya queues are just about empty. I'm up for it if you are20:29
clarkbcorvus: I let the openstack release team know and they don't seem to have any jobs running at the moment20:33
corvuscool, i'll open up some windows...20:34
corvusokay, i ran the pull playbook for good measure20:35
corvusi'll go ahead and restart?20:35
clarkbcorvus: did we double check that the image published properly?20:35
clarkbhttps://zuul.opendev.org/t/zuul/build/501daeaae4d141adbd2ff80b6690921f was successful at least thats probably good enough? or do you want to check dockerhub?20:36
corvusclarkb: i checked that https://review.opendev.org/800112 had a green promote job20:36
corvusthat's the extent of my checking, but i think it's adequate20:36
clarkbok then ya I think we go ahead20:36
corvusrestarting now20:37
corvusi think maybe while we're waiting, i'll manually install the mysqlclient package on the host?20:41
corvuswe don't seem to have it handy20:41
clarkbfine with me20:41
corvusok, i see "ALTER TABLE zuulbuild DROP COLUMN nodename" running20:43
corvuson to 'ALTER TABLE zuul_build ADD COLUMN nodeset VARCHAR(255)'20:53
clarkbprogress :)20:53
clarkbthe debug log shows it doing stuff now I think?20:55
corvusi think that's just queing up events20:55
clarkbmaybe that is just gerrit event stream being handled. Ya20:55
clarkbI think it just moved on to the next step21:00
corvusit's done21:00
clarkbabout 23 minutes?21:01
corvusthat's my read21:01
corvusre-enqueing21:10
clarkbcorvus: one thing I notice is that it had some errors saying unable to load build event. I suspect those are leftover from when we stopped it?21:10
clarkband we had since lost the state in the scheduler to process those?21:11
corvusi didn't see those21:11
clarkbthey came in right after the configs were loaded21:11
clarkb2021-07-16 21:09:46,102 DEBUG zuul.Scheduler: [build: 572bdefa2a594f0f891b09afc5544ee1] Processing result event <zuul.model.BuildCompletedEvent object at 0x7f6ca7f0ff70>21:11
clarkbthen21:12
clarkb2021-07-16 21:09:46,102 ERROR zuul.Scheduler: Unable to find build 572bdefa2a594f0f891b09afc5544ee121:12
corvusclarkb: yep, that makes sense; i think they're harmless21:13
clarkbok21:13
corvuscompletion events from executors shutting down after the scheduler was shut down21:13
clarkbah21:13
corvusre-enqueue finished21:13
clarkbfwiw the pep8 job at the top of the check pipeline seems to be running happily so far21:13
corvus#status log restarted all of zuul on commit 43b7f7f22c74301e830b042357536e6b5357d6e821:13
opendevstatuscorvus: finished logging21:14
corvusthe builds and buildsets pages are less useful right now21:14
corvusi think we need to filter out the in-progress builds21:14
clarkb++21:15
corvusbut they are functioning21:15
clarkband that pep8 job just succeeded21:16
corvuson the status page, you can click on completed builds and go right to the build page now21:17
clarkbthere is a py38 failure and looking at that job it failed due to job run content (so thats a proper fail)21:17
corvusmeanwhile, the buildset page doesn't display the in-progress builds, but it probably should :)21:18
corvusso we got some things backwards :)21:18
corvus(sorry i mean the specific buildset page, not the list of buildsets)21:18
clarkbah ok I was going to say I think the buildest list page is showing in progress items21:19
mordredit's certainly showing some buildsets that are less than exciting :) 21:20
mordredor - sorry - differently exciting21:21
clarkb798918,4 may be our first merge post restart21:22
clarkbin the openstack tenant21:22
corvusmordred: yeah i'm looking at https://zuul.opendev.org/t/openstack/buildset/4293ae47ea2046e083b6f2cbf8ec2822 which is basically having builds added to it as they complete, but i think we should have it show the in-progress builds too21:23
corvusmeanwhile, https://zuul.opendev.org/t/openstack/buildsets probably shouldn't show in-progress by default21:23
corvus(but having an option to see them would be nice)21:23
mordred++21:23
mordredagree with both21:23
mordredon https://zuul.opendev.org/t/openstack/builds - what is the yellow ! telling me?21:24
corvusmordred: that we haven't written code to handle builds without results yet, so it's assumed to be some sort of anomalous result21:24
clarkbhttps://review.opendev.org/c/starlingx/ansible-playbooks/+/798918/ did merge21:24
clarkband so far spot checking failures shows failures due to what ran in the job and not the zuul overlord21:26
* corvus fires up js dev env21:27
clarkbeverything continues to look good. I'm going to take a break. I should be able to review any changes on the js side that come out of this21:35
clarkbin theory we only need to update zuul-web to pick those up too ya?21:35
corvusclarkb: yep21:35
johnsomI have a patch that has two Depends-On, I can see it checks out the correct SHA, but devstack seems to still go download and install the packages. Is there something I should check off the top of your heads?  https://review.opendev.org/c/openstack/designate/+/80072723:09
johnsomDirect link to the proper check out: https://zuul.opendev.org/t/openstack/build/dc7b51cbf2bd4338aa8c4f8f7cef9a71/log/job-output.txt#60023:10
clarkbjohnsom: does your job not disable git cloning and set error on clone?23:11
johnsomclarkb I don't know, it's one of the main tempest jobs for designate.23:12
clarkberror_on_clone is set to true23:12
clarkbhttps://zuul.opendev.org/t/openstack/build/dc7b51cbf2bd4338aa8c4f8f7cef9a71/log/controller/logs/local_conf.txt#40 that is the flag in devstack that is supposed ot prevent devstack from changing what zuul checks out23:13
clarkbthough I guess that is only for the clone step not the checkout step. I think histoircally those two were one and the same but maybe devstack split them?23:13
johnsomIt just seems like it's not installing out of the zuul checkout location. I'm not sure if there is a requirements re-write that should happen, or something installing it before the other parts of devstack run, or .....23:15
clarkbjohnsom: are you sure the problem is that it is checking out a different sha after the fact? I think it may be installing the lib from pypi instead of from source instead23:15
clarkbyou need to set libs from git (I forget the exact form that takes in devstack) to indicate those libraries shouldbe installed fromsource and not pypi23:16
johnsomWhat I want is to use the version zuul checked out due to the depends-on. What I see is the checkout is right, but devstack still pulls the package from pypi23:16
clarkbya I'm caught up now. So way back when it was decided that openstack needed to test with released versions of libs because that is what people would actually install and use rather than doing everything from source23:17
johnsomAh, ok. That is probably missing here. It's so far between when I ever need to pull in special oslo versions I forget the details.23:17
clarkbwhen that change was made all your oslo libs and python clients and so on started to be installed by default from pypi. There is an escape hatch for that called libs from git that allows you to specify that some library should be installed from source instead. You need to toggle that23:17
johnsomThanks!23:17
clarkbone thing to keep in mind is that you probably don't want to set libs from git for those repos permanently. Just do it to test that depends on, then go land the updates to the oslo libs and tag them in new releases. Then bump requirements then your applications can make use of them23:21
johnsomRight, this is making sure I'm doing the right things in these olso patches, so just a one-or-so-off23:21

Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!