Thursday, 2020-03-26

*** zxiiro has quit IRC00:30
*** tosky has quit IRC00:52
*** mlavalle has quit IRC01:22
*** tetsuro has joined #openstack-release01:24
*** diablo_rojo has quit IRC02:01
*** diablo_rojo has joined #openstack-release03:10
*** tetsuro has quit IRC03:46
*** tetsuro has joined #openstack-release03:49
*** ykarel|away is now known as ykarel04:08
*** udesale has joined #openstack-release04:47
*** tonyb[m] has quit IRC04:49
*** tonyb[m] has joined #openstack-release04:49
*** udesale has quit IRC05:10
*** udesale has joined #openstack-release05:12
*** evrardjp has quit IRC05:36
*** evrardjp has joined #openstack-release05:36
*** diablo_rojo has quit IRC05:39
*** armax has quit IRC05:53
*** ykarel is now known as ykarel|meeting06:01
*** ykarel|meeting is now known as ykarel06:49
*** jbadiapa has joined #openstack-release07:01
*** vishalmanchanda has joined #openstack-release07:56
*** evrardjp has quit IRC07:57
*** evrardjp has joined #openstack-release08:01
*** slaweq has joined #openstack-release08:01
*** amoralej|off is now known as amoralej08:02
*** guilhermesp has quit IRC08:06
*** dmellado has quit IRC08:06
*** dmellado has joined #openstack-release08:10
*** rpittau|afk is now known as rpittau08:26
*** toabctl has quit IRC08:30
*** e0ne has joined #openstack-release08:51
*** tosky has joined #openstack-release09:00
*** toabctl has joined #openstack-release09:14
*** e0ne_ has joined #openstack-release09:25
*** e0ne has quit IRC09:25
*** ykarel is now known as ykarel|lunch09:27
*** e0ne_ has quit IRC09:50
*** e0ne has joined #openstack-release09:51
*** ykarel|lunch is now known as ykarel10:14
*** tetsuro has quit IRC10:34
*** dtantsur|afk is now known as dtantsur10:48
*** e0ne has quit IRC10:59
*** e0ne has joined #openstack-release10:59
*** e0ne has quit IRC11:00
*** e0ne has joined #openstack-release11:00
ttxbeh11:37
ttxAll releases failing with <ProtocolError for pypi.org/pypi: 503 Backend is unhealthy>11:38
ttxalthough Pypi reports all systems operational and no unusual error rate11:41
ttxwe are not the only ones seeing issues though11:45
*** rpittau is now known as rpittau|bbl11:45
*** e0ne_ has joined #openstack-release11:46
*** e0ne has quit IRC11:47
*** e0ne has joined #openstack-release11:50
*** e0ne_ has quit IRC11:50
*** e0ne has quit IRC11:53
*** e0ne has joined #openstack-release11:54
openstackgerritMerged openstack/releases master: Add Victoria cycle schedule for manila  https://review.opendev.org/71508311:57
*** e0ne has quit IRC12:14
*** e0ne has joined #openstack-release12:14
*** udesale_ has joined #openstack-release12:21
openstackgerritMerged openstack/releases master: Release manila 8.1.1 off of stable/stein  https://review.opendev.org/71503512:23
openstackgerritMerged openstack/releases master: Release manila 9.1.1 off of stable/train  https://review.opendev.org/71503212:23
*** udesale has quit IRC12:24
openstackgerritMerged openstack/releases master: Release 5.2.0 of python-openstackclient  https://review.opendev.org/71509112:27
*** e0ne has quit IRC12:28
*** e0ne has joined #openstack-release12:28
ttxLooks like it's back to normal now12:32
ttxhttps://status.python.org/incidents/1yd6sfczzfmz12:32
openstackgerritMerged openstack/releases master: Release 0.44 of openstacksdk  https://review.opendev.org/71509012:34
fungittx: does anything need to be reenqueued12:49
fungi?12:49
smcginnisAFS error on the manila release notes, but I think we can ignore that.12:50
smcginnisI haven't seen any other failures. Did I miss something ttx?12:50
smcginnisfungi: In case it's of interest: https://zuul.opendev.org/t/openstack/build/04088a620e904d3d8c7785a7c9095279/log/job-output.txt#162012:50
fungirsync: stat "/afs/.openstack.org/docs/releasenotes/manila/es/_static/js/.jquery-3.2.1.js.JXahrd" failed: No such file or directory (2)12:52
fungithat looks like another rsync race12:52
fungilike a different job had written that temporary location and the current job saw it but then when it tried to delete it was already gone12:53
ttxfungi: no, they failed validation12:54
ttxand now all passed12:54
fungithere's a .root-marker in /afs/.openstack.org/docs/releasenotes/manila/ so there would have to have been more than one job trying to upload manila release notes at the same time12:55
smcginnisfungi: Ah, yep. There was a stein and train release.12:55
smcginnisfungi: We may need some help figuring out why https://review.opendev.org/#/c/714682/ is different locally.12:56
smcginnisfungi: validate job passes locally. Fails to import the project in the gate even though the code appears fine from manual inspection.12:56
fungiERROR: Could not find 2.4.1.0: Command '['git', 'show', '2.4.1.0']' returned non-zero exit status 128.12:59
fungithere's several of those for different tags13:00
fungiis that expected?13:00
ttxyeah that's expected13:00
fungiokay so this is the first obvious sign of trouble? ModuleNotFoundError: No module named 'xstatic.pkg.angular_gettext'13:01
*** amoralej is now known as amoralej|lunch13:01
ttxThe weird thing there is that looking at the code it runs, it clearly finds the "angular_gettext" directory while looking into the xstatic/pkg directory13:02
ttxbut then fails to load the module13:02
ttxlike it finds angular_gettext/ but not angular_gettext/__init__,py13:02
ttxI can't reproduce that locally13:02
ttxso I was wondering if there was not some borked local checkout13:02
fungiyou're testing locally with python3.6?13:03
ttxPython 3.6.9 yes13:03
fungijust want to rule out the possibility that one wheel is broken and others for different interpreter versions are okay13:04
ttxhttps://opendev.org/openstack/releases/src/branch/master/openstack_releases/xstaticutils.py shows the code that fails13:04
ttxIt basically looks into xstatic/pkg13:04
ttxthen tries to load modules for anything that has not __ in the name13:04
ttxso it definitely finds "angular_gettext" in there13:05
ttxwhich shows that the checkout is mostly there13:05
fungixstatic.pkg.angular_gettext seems like it might be something vendored13:06
fungior is that just how all the xstatic packages are organized?13:06
ttxyes it's normal13:07
fungiyeah, i guess that's normal since it's python packaging of javascript13:07
ttxxstatic things are python packages wrapped around JS libs13:07
ttxhttps://opendev.org/openstack/xstatic-angular-gettext/src/branch/master/xstatic/pkg13:07
ttxso here you can find where it gets that "angular_gettext" (it fails loading) from13:07
fungii can play around with the package in ~30 minutes and see if i spot obvious problems13:08
ttxI don;t think it's urgent. it's just puzzling13:08
fungiis this installing it from source or from a release?13:08
ttxshould be a git checkout13:08
*** rpittau|bbl is now known as rpittau13:10
*** zxiiro has joined #openstack-release13:29
*** e0ne_ has joined #openstack-release13:41
*** e0ne has quit IRC13:41
fungii guess the job is doing its own cloning of that repo, since it doesn't appear in the list or projects zuul provided for the build13:48
ttxprobably13:49
fungiyeah, okay, found in the log where it does that13:49
fungii can definitely importlib.import_module('xstatic.pkg.angular_gettext') in a python3.6 venv after installing 6b7ba0f13:54
fungiand it's failing consistently, i take it13:55
fungijudging from prior zuul comments on that change13:56
ttxyeah, there were a bunch of rechecks already13:56
ttxhence the mystery13:56
ttxalso I reviewed the changes since last time it worked and they are pretty minor13:57
ttxat least nothing justifying why that one would fail while the others work13:58
* ttx factchecks that13:59
ttxyeah so xstatic-angular-bootstrap passes that test alright14:00
ttxhttps://zuul.opendev.org/t/openstack/build/a2536a5362e94c71b880168ae72447c5/log/job-output.txt#382214:00
fungii can set an autohold for that job and 714682, just a sec14:00
*** amoralej|lunch is now known as amoralej14:02
fungiokay, once the next failure comes back i can jump on there and see what's different. can add any of your ssh keys too if you like14:03
ttxI would probably not know my way around that node and where to look :)14:05
fungiwe got one, digging around in it now14:16
smcginnisfungi: Let me know if I can help. Not sure what I could add, but following along at home at least.14:21
*** chandan_kumar has joined #openstack-release14:29
*** chandankumar has quit IRC14:29
*** dulek has quit IRC14:30
*** corvus has quit IRC14:30
*** corvus has joined #openstack-release14:31
*** dulek has joined #openstack-release14:33
*** armax has joined #openstack-release14:43
smcginnisfungi: Oh, not to pile on on things, but election related, we need to get https://review.opendev.org/715088 through if you have a chance to take a look at that one.15:05
*** ykarel is now known as ykarel|away15:06
fungismcginnis: i am no longer an election official15:10
fungibut also it seems to have been approved moments ago15:10
smcginnisOK, Andy got it. I thought you still had core rights there.15:10
fungiif i do, it's just because they haven't been cleaned up yet15:11
*** KeithMnemonic has quit IRC15:13
*** armstrong has joined #openstack-release15:19
fungiokay, back to trying to dissect that xstatic job failure, sorry, was stretched between other discussions and security sig meeting15:33
*** priteau has joined #openstack-release15:45
*** udesale_ has quit IRC15:50
*** chandan_kumar is now known as chandankumar15:51
fungioh! so the reason i'm not finding anything useful on the held node is that it failed on a different problem: https://zuul.opendev.org/t/opendev/build/b873ebadfe78421c868c4bc2009b855a/15:54
fungiwe've identified this as bad data coming from some nodes in pypi's cdn network15:54
smcginnisOh, interesting.15:55
smcginnisAh, the pygments issue just landed in upper-constraints too.15:55
fungiwell, i don't think it accounts for the earlier failures though, because pypi's struggles just started today and the verify failures are going back as far as tuesday15:55
ttxyeah, different issue15:56
smcginnis#startmeeting releaseteam16:00
openstackMeeting started Thu Mar 26 16:00:01 2020 UTC and is due to finish in 60 minutes.  The chair is smcginnis. Information about MeetBot at http://wiki.debian.org/MeetBot.16:00
openstackUseful Commands: #action #agreed #help #info #idea #link #topic #startvote.16:00
*** openstack changes topic to " (Meeting topic: releaseteam)"16:00
openstackThe meeting name has been set to 'releaseteam'16:00
*** mlavalle has joined #openstack-release16:00
smcginnisPing list: ttx armstrong diablo_rojo, diablo_rojo_phon16:00
smcginnis#link https://etherpad.openstack.org/p/ussuri-relmgt-tracking16:00
diablo_rojo_phono/16:00
armstrongo/16:00
smcginnisLine 48616:00
elodo/16:00
hberaudo/16:00
ttxo/16:01
smcginnis#topic This weeks tasks16:01
*** openstack changes topic to "This weeks tasks (Meeting topic: releaseteam)"16:01
smcginnisttx got the countdown email updated. We'll cover that on its own.16:02
smcginnisOther task was just to notify infra to be prepared for getting the next cycle's signing key ready.16:02
smcginnisThat has been done. Nothing changes yet, but just being prepared.16:02
smcginnisSo we're in good shape for this week.16:02
smcginnis#topic Next week's tasks16:03
*** openstack changes topic to "Next week's tasks (Meeting topic: releaseteam)"16:03
smcginnisFirst one is the autoreleases for c-w-i.16:03
smcginnisI will take that one.16:03
smcginnisThere's still some opportunity for better automation on that.16:03
smcginnisI have a few fragments of things locally. I'd like to use this as a reason to keep going with that.16:03
smcginnis"Update the feature list and allowed stable branch names in devstack-gate for the new stable branch. For example, https://review.opendev.org/362435 and https://review.opendev.org/363084"16:04
smcginnisThis is a good one to "delegate". :)16:04
ttxI can pick that up if nobody wants it16:04
ttxAlthough I have a task below it already so happy to leave it16:04
smcginnisAny other takers?16:04
ttxnote: we finally went full circle and now have all weekly emails drafted16:05
smcginnis\o/16:05
ttxOnce I post taht process update they will all be living in doc16:05
diablo_rojo_phonNice!16:05
smcginnisttx: I guess we can put you down for that task, but anyone else feel free to speak up and take it instead.16:06
ttx++16:06
smcginnisNext task, sending countdown.16:06
smcginnisNot sure we have to have that called out as a separate task, but good reminder that we are now back to weekly emails.16:07
smcginnisI will get that sent tomorrow.16:07
smcginnisAnd last task for next week i the process update that ttx has taken.16:07
ttx++16:08
smcginnisOur process doc is pretty solid now. But if anyone notices something we just do because we just do it, feel free to point out or add to the process guide to capture anything missing there.16:08
smcginnis#topic Stuck reviews16:08
*** openstack changes topic to "Stuck reviews (Meeting topic: releaseteam)"16:08
smcginnis#link https://review.opendev.org/#/c/714682/ xstatic issue16:08
smcginnisWe have one lib that is getting an odd failure.16:09
smcginnisfungi is helping us out by looking into it.16:09
smcginnisNot clear yet what is going on there.16:09
fungiyeah, i still need a clean run with the failure16:09
fungigoing to set another autohold and hope it too doesn't fall victim to the temporary pypi cdn issue16:09
smcginnisGreat, thanks for working on that.16:10
smcginnisReally hard to tell what's happening with gate behaving differently than local runs.16:10
fungiwill report in this channel once i have ideas16:10
smcginnisJust let us know even if you just need someone to bounce ideas off of.16:10
smcginnisAnd thanks again.16:10
fungiit's my pleasure16:11
smcginnis#link https://review.opendev.org/#/q/status:open+project:openstack/releases+topic:rocky-em Rocky-EM reviews16:11
smcginnisFirst one - OSA. Waiting on a final release from them I believe.16:11
smcginnisThat's a cycle-trailing one, so they wanted to wait for others to go first before wrapping up.16:11
smcginnisI think there was just one patch they had mentioned that they were still waiting on.16:11
smcginnisThough last I looked, it hasn't even merged to master yet I think.16:12
ttxmaybe noonedeadpunk or evrardjp can enlighten us16:12
noonedeadpunkyeah that's true actually16:12
noonedeadpunkActually all functional things are merged now16:12
noonedeadpunkI've pinged patch owner yestarday regarding this doc patch but had no feedback16:13
smcginnisnoonedeadpunk: Ready for a final release? Or need to wait a little longer on that?16:13
noonedeadpunkSo I'm kinda about just to merge new bump https://review.opendev.org/#/c/714926/ and do final release on that.16:13
noonedeadpunkI guess docs will be rendered even without being tagged?16:14
smcginnisnoonedeadpunk: Awesome, thanks. We'll watch for that, then get the rocky-em patch updated once we have the new commit shas.16:14
smcginnisOh, that's true. They still get published, so no need to wait for that for the final release.16:14
noonedeadpunksmcginnis: does it require to do another release to set EM on top of it?16:15
smcginnisReally the only thing would be if someone wanted to build the docs locally from the last official release.16:15
smcginnisnoonedeadpunk: Yeah, we'll need a new patch doing the release, then the em patch can get rebased on top of that.16:15
noonedeadpunkok, so I'm monitoring it as well, so can place it asap16:16
smcginnisThanks!16:16
smcginnisOther outstanding on is heat.16:16
smcginnisI tried pinging ricolin, but no response.16:16
elodmaybe we can just reset the patch to its original state and merge? :/16:17
ttxyeah I would reset to original and set the countdown16:17
smcginnisOK, that sounds reasonable. I will get that updated.16:18
smcginnisLast item -16:18
smcginnis#link https://review.opendev.org/#/c/709748/16:18
ttxit's a classic case where something proposed by one of us just lingers longer16:19
ttxsince we feel self-approving would be inappropriate16:19
smcginnisI've had the issue that if it's not in the sections for each series in the review inbox, it's invisible. :)16:19
smcginnisI can take a look at that after the meeting.16:20
smcginnisQuick looks, looks like it should be good.16:20
ttxhmm, it shows on my review inbox16:20
ttx(I have a "other" section in there :)16:20
smcginnisI tend to ignore the All Releases section since it's been a lot of -W and other patches.16:20
smcginnisI know I shouldn't, but that's what ends up happening for me. :)16:21
ttxI use a custom one16:21
smcginnisDifferent than https://tiny.cc/ReleaseInbox16:21
ttxhttps://bit.ly/2JeILIb16:21
smcginnisI'll see about adding an Other section to that ReleaseInbox one.16:21
smcginnisAnyway...16:22
smcginnis#topic Validate countdown email16:22
*** openstack changes topic to "Validate countdown email (Meeting topic: releaseteam)"16:22
smcginnis#link https://etherpad.openstack.org/p/relmgmt-weekly-emails16:22
smcginnisLine 246?16:22
ttxdown to ~24616:22
smcginnisAll looks good to me.16:24
smcginnisAdded the cycle highlights in there too.16:24
smcginnisI've heard a few teams talking about those already, so that's good.16:25
diablo_rojo_phonThanks smcginnis was about to do that myself :)16:25
diablo_rojo_phonYeah same.16:25
smcginnisHopefully we will get a little more in by the time they are needed this time.16:25
smcginnisOK, I'll send that out tomorrow, my morning.16:26
smcginnisIf anyone has any feedback on the draft, please update it before then.16:26
smcginnis#topic AOB16:26
*** openstack changes topic to "AOB (Meeting topic: releaseteam)"16:26
smcginnisAnything else to discuss in-meeting?16:26
smcginnisFolks are also encouraged to bring up things at any point in the channel too.16:27
smcginnisSo if there's nothing else for now, I think we can wrap early.16:28
smcginnisThanks everyone. Appreciate the time spent on reviews. Stay safe.16:28
fungithanks smcginnis!16:28
ttxthanks!16:28
smcginnis#endmeeting16:28
*** openstack changes topic to "OpenStack Release Managers office - Come here to discuss how to release OpenStack components - Logged at http://eavesdrop.openstack.org/irclogs/%23openstack-release/"16:28
openstackMeeting ended Thu Mar 26 16:28:23 2020 UTC.  Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4)16:28
openstackMinutes:        http://eavesdrop.openstack.org/meetings/releaseteam/2020/releaseteam.2020-03-26-16.00.html16:28
openstackMinutes (text): http://eavesdrop.openstack.org/meetings/releaseteam/2020/releaseteam.2020-03-26-16.00.txt16:28
openstackLog:            http://eavesdrop.openstack.org/meetings/releaseteam/2020/releaseteam.2020-03-26-16.00.log.html16:28
diablo_rojo_phonThanks!16:28
smcginnisfungi: Looks like that time hit the error we wanted.16:30
fungiyeah, i'm jumping into it now16:31
elodttx: i can take over the 'Update the feature list and allowed stable branch names in devstack-gate' task. seems trivial :)16:31
ttxelod: it is, and you'll learn tricks!16:31
ttxfeel free to take it over16:31
elodttx: ok :)16:32
smcginnisttx: Do you have a gerrit dashboard creator file for your dashboard?16:32
ttxno, old-school manm16:32
smcginnisCool, I can pull that out of the URL.16:33
smcginnisgerrit-dashboard-decompile16:33
fungilooks like tox clears out the contents of .tox/validate/tmp/ at some point16:36
openstackgerritMerged openstack/releases master: Use governance_consistency.py for MembershipFreeze  https://review.opendev.org/70974816:44
openstackgerritSean McGinnis proposed openstack/releases master: [heat] Transition Rocky to EM  https://review.opendev.org/70988916:53
smcginnisfungi: So no signs of what might have happened?16:57
smcginnisI wonder if we could manually walk through the script process to recreate it.16:57
fungii'm still trying to understand the job16:57
fungiit's running python globally i guess, not from within a virtualenv?16:57
fungithe traceback cites /usr/lib/python3.6/importlib/__init__.py16:57
smcginnisHmm, that doesn't seem right.17:00
smcginnisIf it installed the package into a venv, but then tries to import it using system packages, that could explain it.17:00
smcginnisNot why others work and only this one fails, but it would at least be something.17:01
fungito be fair, i can't find the venv it might have installed it in either17:01
fungi`src/opendev.org/openstack/releases/.tox/validate/bin/pip list|grep xstatic` returns nothing17:01
smcginnisSo to run the validate script, it needs to have install the releases repo. Can you see where and how that is being done?17:03
smcginnisOh, running tox -e validate I guess, based on that path.17:04
fungiyeah, it's installed in the virtualenv there17:04
smcginnisSo the validate job probably needs to do a 'pip install -e xstaticpackage' somewhere in there I guess.17:05
fungiwell, openstack_releases/xstaticutils.py works around that17:06
fungiit adds the repo_dir to sys.path and then uses importlib.import_module() to pull it from there17:06
ttxremember that in the same change angular-bootstrap is just fine17:07
smcginnisttx: OpenDev+PTG wasn't important enough for the "Upcoming Community Events"? :)17:07
fungiright, for now i'm trying to dissect enough of the tooling to understand what it's doing so i can isolate the failing call17:07
ttxsmcginnis: we can't put a date on it right now17:07
*** sean-k-mooney has joined #openstack-release17:07
sean-k-mooneyhi o/17:07
sean-k-mooneyjust want to check something. os-vif has not merged any patches since m2 so can we just use the m2 release for the non client lib freeze17:08
smcginnissean-k-mooney: Yep. That should be fine then.17:09
sean-k-mooneycool if we have to merge something ill request a release but i dont think that will happen17:09
sean-k-mooneythanks17:09
smcginnissean-k-mooney: Sounds like a plan. Thanks!17:10
smcginnisfungi, ttx: Could it be an issue that there are multiple xstatic.pkg's, so one is overwriting the other?17:10
ttxsmcginnis: not totally impossible17:11
smcginnisNot sure about what they are doing in the __init__.py files: https://opendev.org/openstack/xstatic-angular-gettext/src/branch/master/xstatic/pkg/__init__.py17:11
smcginnisBut would think that would get overwritten to declare a different namespace. Maybe?17:12
smcginnisThen resulting in the import failure.17:12
*** e0ne_ has quit IRC17:12
smcginnisCould be easy to test by doing a patch separating those to just release one at a time.17:12
*** rpittau is now known as rpittau|afk17:12
*** e0ne has joined #openstack-release17:12
fungismcginnis: ttx: we've seen that in the past with packages sharing a package namespace17:13
fungilike is it creating a xstatic/pkg/__init__.py which is overwriting the other?17:14
smcginnisIf I read that right.17:14
smcginnisBut the content should be the same. But what gets loaded may change.17:14
fungii'm getting pulled away by local chores for 5-10 minutes, brb17:15
smcginnisThanks fungi17:15
smcginnisI will put up some test patches separating those releases.17:16
elodttx: should I -W this patch until next week or is it OK to land it sooner? https://review.opendev.org/#/c/715272/17:17
openstackgerritSean McGinnis proposed openstack/releases master: Release xstatic-bootstrap-scss 3.4.1.0  https://review.opendev.org/71527417:20
openstackgerritSean McGinnis proposed openstack/releases master: Release xstatic-angular-gettext 2.4.1.0  https://review.opendev.org/71527517:21
openstackgerritSean McGinnis proposed openstack/releases master: Release xstatic-angular-bootstrap 2.5.0.0  https://review.opendev.org/71527617:21
ttxelod: I think it;s ok this way, we won;t approve it just now anyway17:24
elodttx: cool, thx17:26
ricolinsmcginnis, regrading https://review.opendev.org/#/c/709889, can I create a new rocky version now (with the hash I put in in #4) so we can release the newest rocky commits?17:29
smcginnisricolin: Yes, that would be great. Then we could update the rocky-em patch on that.17:30
*** dtantsur is now known as dtantsur|afk17:32
fungion my held test node i've made a tarball snapshot of the entire ~zuul/src tree so i don't contaminate it, and am rerunning `tox -e validate` to see if the error is reproducible when running manually17:32
smcginnisfungi: Looks like the separate patches are passing.17:32
smcginnisSo it's some kind of overwriting clash.17:33
smcginnisI think.17:33
fungiModuleNotFoundError: No module named 'xstatic.pkg.angular_gettext'17:33
fungiyeah, so i can recreate it rerunning in here at least17:33
smcginnisOh good.17:33
smcginnisAny ideas how that can be fixed?17:34
fungino, i'm not sure what's breaking it yet, though it sounds from your experiments like a conflict between multiple xstatic packages at least17:41
fungii'm installing a text editor to be able to try a few alternatives17:41
smcginnisMy concern is we can get around this, but are our end users going to hit this too when installing these packages together.17:41
*** evrardjp has quit IRC17:49
*** evrardjp has joined #openstack-release17:49
fungiyes, i don't think releasing them in separate single-project batches is an answer17:49
fungijust a means of helping narrow down the cause17:49
fungii've removed the angular-bootstrap release to see what happens17:49
fungiModuleNotFoundError: No module named 'xstatic.pkg.bootstrap_scss'17:49
*** openstack has quit IRC17:49
*** openstack has joined #openstack-release17:50
*** ChanServ sets mode: +o openstack17:50
fungiyeah, okay, so repeating it correctly, it also fails17:52
fungiModuleNotFoundError: No module named 'xstatic.pkg.bootstrap_scss'17:52
fungiso it seems that no two of these three xstatic modules can have releases validated at the same time17:53
fungileading me to believe it's a problem with openstack_releases/xstaticutils.py17:53
smcginnisNow can they be coinstalled and used in normal operation at the same time? Or is it the way that we try to do our validation?17:53
fungithough why folks are unable to reproduce it locally is still a question17:53
fungiwell, we're not coinstalling them in this job17:54
fungithey're technically not being "installed" at all that i can see17:54
fungiat least not with pip and friends17:54
smcginnisRight, just plonked down and added to the sys path.17:54
fungiwhich leads me to suspect this is a namespace-related issue17:55
fungiwhichever one is taking precedence in the sys.path is probably obscuring the other17:55
fungiby taking over the entire xstatic.pkg namespace17:55
smcginnisSo we probably are OK working around it for now by releasing separately.17:57
fungii'll hack up the xstaticutils.py module with some debugging to see if i can tell17:57
smcginnisThen we have an open issue of fixing how we handle things in xstaticutils so we can do this right.17:57
smcginnisAwesome, thanks!17:57
fungii also need to work out why tox is clearing the tempdir18:01
fungii don't see an option to tell it to not do that18:02
noonedeadpunkrocky bump is blocked by pygments thing:(18:03
noonedeadpunkin osa18:03
fungisys.path: ['/home/zuul/src/opendev.org/openstack/releases/.tox/validate/bin', '/usr/lib/python36.zip', '/usr/lib/python3.6', '/usr/lib/python3.6/lib-dynload', '/home/zuul/src/opendev.org/openstack/releases/.tox/validate/lib/python3.6/site-packages', '/home/zuul/src/opendev.org/openstack/releases',18:08
fungi'/home/zuul/src/opendev.org/openstack/releases/.tox/validate/tmp/releases-6veo5lh_/openstack/xstatic-bootstrap-scss']18:08
fungithat's from a run where i removed the openstack/xstatic-angular-gettext release18:08
fungiand it failed saying it couldn't import xstatic.pkg.bootstrap_scss18:09
fungiso there's no xstatic-angular-bootstrap in the sys.path18:09
fungiimplying we're validating them independently from a sys.path perspective18:10
openstackgerritRico Lin proposed openstack/releases master: Release new rocky version for heat repos  https://review.opendev.org/71528218:12
openstackgerritRico Lin proposed openstack/releases master: [heat] Transition Rocky to EM  https://review.opendev.org/70988918:12
*** amoralej is now known as amoralej|off18:30
fungii've tweaked it to use shutil.copytree to make copies of the unpacked modules in /tmp so i can play with them more interactively18:36
*** armstrong has quit IRC18:37
fungihttp://paste.openstack.org/show/791207/18:42
fungidoesn't fail for me when trying to emulate what xstaticutils.py does with a copytree() of the repo_dir from immediately prior to the call which raises that exception18:43
fungii wonder if there could be some kind of caching at work18:44
*** diablo_rojo has joined #openstack-release18:47
smcginnisfungi: Did you try to import each one?18:50
fungiyeah, importing them in a fresh python interpreter process seems to be fine18:51
fungii'm fiddling around with importlib.reload() on the namespaces first18:52
fungimaybe they're being cached18:52
fungiscore!!!18:55
smcginnisYeah!18:55
funginow to see if i can whittle it down a bit18:55
smcginnisSo some obscure import_lib internal behavior?18:56
fungistill no clue why this doesn't surface in local runs18:56
fungibut it's another piece of the puzzle at least18:56
smcginnisstephenfin: Any ideas on https://review.opendev.org/#/c/715282/19:17
smcginnisstephenfin: Heat sdist build fails now. Looks like it's another side effect of https://review.opendev.org/#/c/567559/419:17
smcginnisstephenfin: Past issues have been with it scanning branches before release notes were present. This one is failing on a recent tag though, so not sure why reno complains about it.19:18
*** e0ne has joined #openstack-release19:18
*** e0ne has quit IRC19:39
openstackgerritJeremy Stanley proposed openstack/releases master: Bust import caching of xstatic package namespaces  https://review.opendev.org/71530120:08
fungismcginnis: ttx: ^ that's the simplest solution i've been able to arrive at which gets `tox -e validate` working for the change in question20:09
smcginnisInteresting. Well, if that does it, then good enough!20:10
openstackgerritSean McGinnis proposed openstack/releases master: Release xstatic deliverables  https://review.opendev.org/71468220:17
smcginnisttx: So here's an interesting thing.20:20
*** e0ne has joined #openstack-release20:20
smcginnisttx: I updated the xstatic releases: https://review.opendev.org/#/c/714682/20:21
smcginnisSince Akihiro is the author, the PTL approved check left +1.20:21
smcginnisttx: Thinking that should actually check the Committer and not the Owner.20:21
smcginnisNot a big concern, but someone could modify a patch originally submitted by the PTL to be something completely different, and it would look like the PTL approves it.20:22
*** e0ne has quit IRC20:29
*** e0ne has joined #openstack-release20:29
*** priteau has quit IRC20:41
*** e0ne has quit IRC20:42
openstackgerritSean McGinnis proposed openstack/releases master: Release xstatic deliverables  https://review.opendev.org/71468220:44
openstackgerritSean McGinnis proposed openstack/releases master: Bust import caching of xstatic package namespaces  https://review.opendev.org/71530120:55
openstackgerritJeremy Stanley proposed openstack/releases master: Bust import caching of xstatic package namespaces  https://review.opendev.org/71530121:08
fungismcginnis: that should fix it21:08
fungioh, you already pushed that, sorry21:08
smcginnisNo worries, I should have checked. Figured you would have gotten pulled down another rabbit hole by now. ;)21:11
*** jbadiapa has quit IRC21:23
openstackgerritSean McGinnis proposed openstack/releases master: Release xstatic deliverables  https://review.opendev.org/71468221:26
openstackgerritMerged openstack/releases master: Bust import caching of xstatic package namespaces  https://review.opendev.org/71530121:42
openstackgerritMonty Taylor proposed openstack/releases master: Release 0.45.0 of openstacksdk  https://review.opendev.org/71531721:49
mordredsmcginnis: ^^21:49
fungismcginnis: yeah, so i'm fairly confident what was happening is that the first xstatic module to get imported left stale implicit xstatic and xstatic.pkg namespaces, which caused subsequent imports of other modules to fail because they weren't found in the already cached namespaces22:00
fungionly reloading one or the other of those did not seem to solve the problem22:00
fungireloading both seemed to work though22:00
fungiand they had to be explicitly imported and then reloaded. i tried just reloading and catching nameerror exceptions but that was insufficient22:01
fungiand only importing the namespaces each time but not reloading them also didn't help22:01
mordreddiablo_rojo, ttx, smcginnis: if anybody is around who can approve releases - 715317 turns out to also be blocking opendev ops changes22:55
*** slaweq has quit IRC22:59
*** prometheanfire has quit IRC23:04
*** slaweq has joined #openstack-release23:10
*** slaweq has quit IRC23:15
*** tetsuro has joined #openstack-release23:33
*** tetsuro has quit IRC23:35
*** tetsuro has joined #openstack-release23:37
*** vishalmanchanda has quit IRC23:39
*** tetsuro has quit IRC23:47
*** tosky has quit IRC23:53

Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!