Monday, 2020-07-20

*** irclogbot_3 has quit IRC00:16
*** tetsuro has joined #openstack-nova00:19
*** irclogbot_3 has joined #openstack-nova00:20
*** ociuhandu has joined #openstack-nova00:25
*** ociuhandu has quit IRC00:30
*** tetsuro has quit IRC00:39
*** tetsuro has joined #openstack-nova00:41
openstackgerritKeigo Noha proposed openstack/nova master: Change default num_retries for glance to 3  https://review.opendev.org/74038901:34
*** Liang__ has joined #openstack-nova01:35
*** markvoelker has joined #openstack-nova01:36
*** markvoelker has quit IRC01:41
*** brinzhang has joined #openstack-nova01:47
*** brinzhang_ has joined #openstack-nova01:50
openstackgerritBrin Zhang proposed openstack/nova master: Cyborg shelve/unshelve support  https://review.opendev.org/72956301:52
*** brinzhang has quit IRC01:54
*** brinzhang0 has joined #openstack-nova02:17
*** brinzhang_ has quit IRC02:20
*** sapd1 has joined #openstack-nova02:34
*** Liang__ has quit IRC02:54
*** Liang__ has joined #openstack-nova02:55
openstackgerritTony Su proposed openstack/nova master: Provider Config File: YAML file loading and schema validation  https://review.opendev.org/67334103:01
openstackgerritTony Su proposed openstack/nova master: Provider Config File: Function to further validate and retrieve configs  https://review.opendev.org/67602903:01
openstackgerritTony Su proposed openstack/nova master: Provider Config File: Functions to merge provider configs to provider tree  https://review.opendev.org/67652203:01
openstackgerritTony Su proposed openstack/nova master: Provider Config File: Enable loading and merging of provider configs  https://review.opendev.org/69346003:01
*** tetsuro has quit IRC03:02
*** rcernin has quit IRC03:02
*** Liang__ has quit IRC03:06
*** yaawang has quit IRC03:08
*** yaawang has joined #openstack-nova03:08
*** dave-mccowan has joined #openstack-nova03:11
*** Liang__ has joined #openstack-nova03:18
*** rcernin has joined #openstack-nova03:21
*** rcernin has quit IRC03:25
*** rcernin has joined #openstack-nova03:26
*** brinzhang_ has joined #openstack-nova03:46
*** brinzhang0 has quit IRC03:50
*** yaawang has quit IRC03:54
*** yaawang has joined #openstack-nova03:55
*** tetsuro has joined #openstack-nova03:57
*** dave-mccowan has quit IRC03:57
*** ociuhandu has joined #openstack-nova04:28
*** ociuhandu has quit IRC04:33
*** ratailor has joined #openstack-nova04:39
*** vishalmanchanda has joined #openstack-nova04:51
*** sapd1 has quit IRC05:17
openstackgerritKevin Zhao proposed openstack/nova master: [WIP] CI: add tempest-integrated-compute-aarch64 job  https://review.opendev.org/71443905:30
*** links has joined #openstack-nova05:44
*** udesale has joined #openstack-nova05:48
*** brinzhang0 has joined #openstack-nova05:56
*** brinzhang_ has quit IRC05:59
*** brinzhang_ has joined #openstack-nova06:06
*** brinzhang0 has quit IRC06:09
*** jsuchome has joined #openstack-nova06:14
*** tetsuro has quit IRC06:34
bauzasgood morning Nova06:35
*** tetsuro has joined #openstack-nova06:36
brinzhang_morning bauzas^06:39
*** yaawang has quit IRC06:41
*** yaawang has joined #openstack-nova06:42
*** maciejjozefczyk has joined #openstack-nova06:43
*** nightmare_unreal has joined #openstack-nova06:57
gibigood morning folks07:00
*** Luzi has joined #openstack-nova07:02
brinzhang_gibi: morning ^07:03
brinzhang_The nova cyborg interaction patches will be deadline in the runway slot, do we need to ask help from efried?07:04
*** tesseract has joined #openstack-nova07:05
gibiefried is gone so I don't think he will help07:07
brinzhang_emm..I know, but...07:08
*** sapd1 has joined #openstack-nova07:11
*** ociuhandu has joined #openstack-nova07:12
gibithe last two weeks of mine was busier than I anticipated so I did not have time to review all the things in the slots. I'm wondering that maybe 3 slots is more than what I can cover... maybe more than what we can cover07:13
*** tetsuro_ has joined #openstack-nova07:15
*** tetsuro has quit IRC07:19
*** rcernin has quit IRC07:24
*** yaawang has quit IRC07:27
*** dtantsur|afk is now known as dtantsur07:27
*** yaawang has joined #openstack-nova07:28
openstackgerritElod Illes proposed openstack/nova stable/queens: Check cherry-pick hashes in pep8 tox target  https://review.opendev.org/74188307:33
*** tosky has joined #openstack-nova07:34
tony_sugibi: hello gibi, could you spare some time reviewing provider-config-file patches as I saw its expire date is July 23. Today, I rebased them against master tip. I checked zuul failure and it's NOT related to us.07:37
*** rcernin has joined #openstack-nova07:38
*** derekh has joined #openstack-nova07:39
gibitony_su: I will try but there is simply too much on my plate07:39
gibisorry07:39
*** xek_ has joined #openstack-nova07:39
nightmare_unrealhello, what's the command for zuul to rebuild the patch. Something we put in patch comments and zuul perform it's checks again07:40
tony_sugibi: got it, Thanks. I will ask alex_xu to make a round of review if he has some bandwidth. https://review.opendev.org/#/q/status:open+project:openstack/nova+branch:master+topic:bp/provider-config-file07:41
*** tetsuro_ has quit IRC07:41
*** tetsuro has joined #openstack-nova07:41
*** rcernin has quit IRC07:44
gibitony_su: ack07:45
openstackgerritElod Illes proposed openstack/nova stable/rocky: fix scsi disk unit number of the attaching volume when cdrom bus is scsi  https://review.opendev.org/73031207:47
nightmare_unrealfound it, it was recheck07:48
*** martinkennelly has joined #openstack-nova07:51
*** yaawang has quit IRC07:52
*** yaawang has joined #openstack-nova07:52
alex_xutony_su: willl begin to review tomorrow or tonight if possible07:54
*** ralonsoh has joined #openstack-nova07:55
*** xiaolin has joined #openstack-nova07:55
*** alex_xu has quit IRC08:03
*** markvoelker has joined #openstack-nova08:04
*** markvoelker has quit IRC08:09
*** fumesover3 has quit IRC08:12
*** markvoelker has joined #openstack-nova08:14
*** links has quit IRC08:19
*** markvoelker has quit IRC08:24
*** markvoelker has joined #openstack-nova08:27
*** yaawang has quit IRC08:29
*** yaawang has joined #openstack-nova08:30
*** markvoelker has quit IRC08:32
*** derekh has quit IRC08:34
*** ralonsoh_ has joined #openstack-nova08:53
*** sapd1 has quit IRC08:54
*** ralonsoh has quit IRC08:55
*** sapd1 has joined #openstack-nova09:01
*** k_mouza has joined #openstack-nova09:01
*** links has joined #openstack-nova09:02
*** rcernin has joined #openstack-nova09:10
*** tetsuro has quit IRC09:12
*** rcernin has quit IRC09:15
*** yaawang has quit IRC09:33
*** yaawang has joined #openstack-nova09:34
*** alex_xu has joined #openstack-nova09:38
*** ratailor has quit IRC09:58
*** ratailor_ has joined #openstack-nova09:58
*** sapd1 has quit IRC10:01
*** fumesover3 has joined #openstack-nova10:02
*** sapd1 has joined #openstack-nova10:05
*** ociuhandu has quit IRC10:05
*** ociuhandu has joined #openstack-nova10:08
*** yaawang has quit IRC10:08
*** k_mouza has quit IRC10:08
*** tosky has quit IRC10:08
*** Luzi has quit IRC10:08
*** irclogbot_3 has quit IRC10:08
*** johnthetubaguy has quit IRC10:08
*** vesper11 has quit IRC10:08
*** jdillaman has quit IRC10:09
*** yaawang has joined #openstack-nova10:10
*** k_mouza has joined #openstack-nova10:10
*** tosky has joined #openstack-nova10:10
*** Luzi has joined #openstack-nova10:10
*** irclogbot_3 has joined #openstack-nova10:10
*** johnthetubaguy has joined #openstack-nova10:10
*** vesper11 has joined #openstack-nova10:10
*** jdillaman has joined #openstack-nova10:10
*** ociuhandu has quit IRC10:13
*** amodi has quit IRC10:14
gibibrinzhang_: hi! I left comments in https://review.opendev.org/#/c/71618610:14
*** derekh has joined #openstack-nova10:20
*** tkajinam has quit IRC10:29
*** ociuhandu has joined #openstack-nova10:32
*** trident has quit IRC10:35
*** trident has joined #openstack-nova10:36
*** Liang__ has quit IRC10:40
*** brinzhang0 has joined #openstack-nova10:56
*** brinzhang_ has quit IRC10:59
*** brinzhang0 has quit IRC11:04
*** brinzhang_ has joined #openstack-nova11:04
*** jangutter_ is now known as jangutter11:12
openstackgerritStephen Finucane proposed openstack/nova master: libvirt: Remove two-level loop  https://review.opendev.org/74192011:16
sean-k-mooneyjust an fyi i wont be around as much this week11:22
*** brinzhang_ has quit IRC11:31
*** brinzhang_ has joined #openstack-nova11:32
*** sapd1 has quit IRC11:38
*** raildo has joined #openstack-nova11:38
*** jraju__ has joined #openstack-nova11:56
*** links has quit IRC11:57
*** brinzhang0 has joined #openstack-nova11:58
gibisean-k-mooney: ack11:58
*** xek_ has quit IRC12:00
*** brinzhang_ has quit IRC12:01
gibidoes somebody knows about the failing novnc tempest test https://c50701f61b79b2f93421-0d0b23b7f62d479822a1b52b605369e3.ssl.cf2.rackcdn.com/673341/50/check/nova-next/5f7dd59/testr_results.html ?12:12
gibinovnc proxy has a stacktrace so I have something to start with12:21
gibihttps://zuul.opendev.org/t/openstack/build/1180d2313ade4d5480550313bcd20b79/log/controller/logs/screen-n-novnc-cell1.txt#2912:21
sean-k-mooney that looks like an interaction issue with eventlet? maybe12:25
sean-k-mooneywas there a new releae of either eventlet or websockify12:25
sean-k-mooneynot websockify anyway12:26
sean-k-mooneythat last release last august12:26
sean-k-mooneyeventlet was april12:27
openstackgerritMerged openstack/nova stable/pike: Clean up allocation if unshelve fails due to neutron  https://review.opendev.org/74063612:28
openstackgerritMerged openstack/nova stable/rocky: fix scsi disk unit number of the attaching volume when cdrom bus is scsi  https://review.opendev.org/73031212:28
openstackgerritMerged openstack/nova stable/train: libvirt: Mark e1000e VIF as supported  https://review.opendev.org/73696212:29
sean-k-mooneynot obvious changes in the reqruiement repo either12:30
*** brinzhang_ has joined #openstack-nova12:32
gibiif I understand correclty the error coming from the ssl c lib so I cannot really debug it12:35
*** brinzhang0 has quit IRC12:35
sean-k-mooneyright so eventlet is monkeypatch the ssl socket12:41
sean-k-mooneybut it looks like the version of eventlet we are using is not handelign the wrapping correctly12:42
openstackgerritMerged openstack/nova stable/train: Correct reported system memory  https://review.opendev.org/74066712:42
sean-k-mooneymaybe they broke 3.6 compatiablity but there has not been a releae of eventlet in a few months12:42
sean-k-mooneyso im not sure why we would only see it now12:43
*** JamesBenson has joined #openstack-nova12:43
sean-k-mooneythey have had this issue in the past https://github.com/eventlet/eventlet/issues/20112:43
sean-k-mooneyand it happened again https://github.com/eventlet/eventlet/issues/31112:45
gibithose fixes talk about urllib2 but as far as I see we install urllib312:49
sean-k-mooneyyes my point was more they have missed monkeypatching the context correctly in the past12:49
*** jangutter_ has joined #openstack-nova12:55
*** jangutter has quit IRC12:58
sean-k-mooneyill open an eventlet bug and see what they think13:01
gibisean-k-mooney: thanks13:01
gibiI'm still trying to figure out what changed now that break us as all the sw releases are couple months old13:02
sean-k-mooneymaybe the host openssl version13:02
sean-k-mooneyor a recent python 3.6.z release13:03
gibilooking...13:03
*** udesale_ has joined #openstack-nova13:03
*** jangutter has joined #openstack-nova13:04
*** udesale has quit IRC13:06
*** jangutter_ has quit IRC13:07
*** rcernin has joined #openstack-nova13:07
gibi1.1.1-1ubuntu2.1~18.04.6 was used for the last passing test as well as the failing test13:08
gibiI mean  openssl is already the newest version (1.1.1-1ubuntu2.1~18.04.6).13:08
*** mvkr has joined #openstack-nova13:09
sean-k-mooneyand the python version?13:09
*** k_mouza has quit IRC13:09
gibithat is also the same python3.6 is already the newest version (3.6.9-1~18.04ubuntu1).13:10
gibiI'm comparing the nova-next run from here (passed) https://review.opendev.org/#/c/739209/ with here (failed) https://review.opendev.org/#/c/728481/13:11
*** rcernin has quit IRC13:12
gibilibssl-dev is also the same13:12
*** nweinber has joined #openstack-nova13:14
*** lbragstad has joined #openstack-nova13:16
*** artom has joined #openstack-nova13:18
*** ratailor_ has quit IRC13:19
*** eharney has joined #openstack-nova13:26
*** xiaolin has quit IRC13:26
gibithere was some image rebuild during the weekend but it only affected 3.7 and 3.8 images as far as I see https://review.opendev.org/#/c/741789/113:27
sean-k-mooneyhttps://github.com/eventlet/eventlet/issues/63213:28
*** ociuhandu_ has joined #openstack-nova13:29
sean-k-mooneyya but most of the relevent packages seam to be the same between the passing and failing build13:29
*** k_mouza has joined #openstack-nova13:29
*** ociuhandu has quit IRC13:31
efriedbrinzhang_: Yeah, sorry, I wouldn't be able to make enough time for something the size and scope of nova/cyborg.13:32
*** bnemec has joined #openstack-nova13:32
gibisean-k-mooney: thanks for reporting that to eventlet13:33
sean-k-mooneythere isnt much for them to go on really but maybe that can suggest something13:33
sean-k-mooneygibi: is this repetable by the way13:34
*** k_mouza has quit IRC13:34
*** derekh has quit IRC13:34
gibisean-k-mooney: I saw it in two independent job run13:35
*** efried has quit IRC13:36
gibisean-k-mooney: here is another reporduction https://zuul.opendev.org/t/openstack/build/5f7dd591c62843ce9a114276b933a932/log/controller/logs/screen-n-novnc-cell1.txt#6813:36
gibifrom https://review.opendev.org/#/c/673341/5013:36
*** efried has joined #openstack-nova13:36
*** k_mouza has joined #openstack-nova13:37
*** k_mouza has quit IRC13:41
gibiI don't have SSL/TLS configured for vnc proxy in my devstack13:43
*** brinzhang0 has joined #openstack-nova13:44
*** jangutter has quit IRC13:47
*** k_mouza has joined #openstack-nova13:47
*** brinzhang_ has quit IRC13:48
*** jangutter has joined #openstack-nova13:49
*** k_mouza has quit IRC13:51
*** sapd1 has joined #openstack-nova13:51
*** brinzhang_ has joined #openstack-nova14:01
*** k_mouza has joined #openstack-nova14:03
*** brinzhang0 has quit IRC14:04
gibiso I'm restacking right now14:05
*** ociuhandu_ has quit IRC14:05
*** ociuhandu has joined #openstack-nova14:06
*** derekh has joined #openstack-nova14:06
*** k_mouza has quit IRC14:06
sean-k-mooney im not seeign it in logstash so i can see how often its happens14:10
gibilogstash indexing is behind with about 120 hours14:13
sean-k-mooneyok so 5 days14:14
sean-k-mooneyand the failures were 3-4 days ago14:14
sean-k-mooneywell in the previous 30 days there are no hits14:15
gibithe last patch merged ~3 days ago14:15
gibisean-k-mooney: here is the nova-next job history https://zuul.opendev.org/t/openstack/builds?job_name=nova-next&project=openstack%2Fnova&branch=master14:22
sean-k-mooney yep but if it was in logstash i could have added an elasticrecheck query for it14:23
*** sapd1 has quit IRC14:24
sean-k-mooneyi would expect this to fail on other jobs too14:25
*** xek_ has joined #openstack-nova14:26
gibiI'm not sure where we run the tempest test test_novnc14:27
*** k_mouza has joined #openstack-nova14:27
gibihm tempest-integrated-compute also runs that test14:27
gibiand that is green14:28
gibibut that runs without SSL setupo14:28
openstackgerritGhanshyam Mann proposed openstack/nova master: DNM: tesging nova greande job  https://review.opendev.org/74195614:28
gibihttps://zuul.opendev.org/t/openstack/build/675ff69ce6624ddb83b7ff7fec1cef9b/log/controller/logs/screen-n-novnc-cell1.txt#714:28
gmanngibi: ^^ testing the grenade jobs failure fix ^^14:28
sean-k-mooneygibi: ah right we might not have tls or run it in other jobs14:28
gibigmann: thanks for looking into grenade!14:29
gmannlegacy jobs were left for glance stabdalone thing14:29
gmannstandalone14:29
* artom stabs Dalone.14:30
* sean-k-mooney artom is grumpy today14:31
artomAm I?14:32
sean-k-mooney^14:32
artomMaybe Dalone is just a prick.14:32
artomAnd deserves to be stabbed?14:32
sean-k-mooneymaybe14:32
* artom checks - OK, no one with "dalone" as their nick here14:32
dansmithartom: always check *after* the threat14:34
dansmithit's funnier (for the rest of us) that way14:34
artomYou mean my usual method of 1. open mouth 2. say dumb thing 3. check 4. regret14:34
*** brinzhang0 has joined #openstack-nova14:35
sean-k-mooneygmann: is standalone glance just without the registry service?14:35
sean-k-mooneyor something else14:35
*** brinzhang0 has quit IRC14:35
*** brinzhang0 has joined #openstack-nova14:36
gmannsean-k-mooney:  not running  glance on uwsgi and rest all service on uwsgi. register service was removed much earlier14:37
*** brinzhang_ has quit IRC14:38
gmannsean-k-mooney:  this one https://review.opendev.org/#/c/740322/1014:38
sean-k-mooneyoh ok14:38
sean-k-mooneythis is for the multi store stuff so the glance import pipelene stuff works14:38
gmanndansmith: setting standalone as false for legacy grenade jobs too - https://review.opendev.org/#/c/741955/14:38
dansmithgmann: I saw, thanks14:39
dansmithgmann: lots of tentacles...14:39
gmannyeah, legacy jobs fixes are always 'missed-and-fixed-later' :)14:40
*** mgariepy has quit IRC14:41
* gibi is still trying to build a devstack with USE_SSL=True ...14:43
*** mlavalle has joined #openstack-nova14:43
dansmithgmann: hah14:43
*** brinzhang0 has quit IRC14:48
*** brinzhang0 has joined #openstack-nova14:49
*** xek_ has quit IRC14:50
*** mgariepy has joined #openstack-nova14:51
openstackgerritElod Illes proposed openstack/nova stable/queens: fix scsi disk unit number of the attaching volume when cdrom bus is scsi  https://review.opendev.org/73031314:53
gibisean-k-mooney: opened a nova bug for the novnc breakage https://bugs.launchpad.net/nova/+bug/188823714:55
openstackLaunchpad bug 1888237 in OpenStack Compute (nova) "novan-next job fails as novnc service fails with TypeError: _wrap_socket() argument 1 must be _socket.socket, not GreenSSLSocket" [Critical,New]14:55
*** xek_ has joined #openstack-nova14:56
sean-k-mooneylyarwood: you might be interested in ^ by the way since we were discussing the need to test new version of it last week14:57
lyarwoodsean-k-mooney: yup ack, I'll try to look this evening14:58
sean-k-mooneyin this case there has not been a new release but its now failing with tls enabled14:58
sean-k-mooneyactully14:59
*** dklyle has joined #openstack-nova14:59
sean-k-mooneythere was a new tag of novnc 6 days ago14:59
sean-k-mooneyv1.2.014:59
lyarwoodyeah that did drop a few days ago14:59
lyarwooddoesn't nova-next use NOVNC from source?15:00
sean-k-mooneypossibly that is what is failing15:00
sean-k-mooneybut the error seams to be in websockify15:00
gibiwe are pulling 1.1.0 https://zuul.opendev.org/t/openstack/build/1180d2313ade4d5480550313bcd20b79/log/job-output.txt#215415:00
sean-k-mooneyor eventlet not sure15:00
lyarwoodkk then it's not that15:00
* lyarwood jumps on a call15:00
openstackgerritSergey Galas' proposed openstack/nova stable/train: fixes broken shared live migration of a vm with a vif  https://review.opendev.org/74196315:04
*** xek_ has quit IRC15:07
*** Luzi has quit IRC15:09
*** ralonsoh_ is now known as ralonsoh15:10
*** hamalq has joined #openstack-nova15:15
jgwentworthlyarwood: you had tried to get nova-next to work to install novnc from source in the past but didn't finish https://review.opendev.org/67149015:18
*** jgwentworth is now known as melwitt15:19
melwittI think there were some issues but I don't remember now15:19
*** spatel has joined #openstack-nova15:29
*** artom has quit IRC15:31
*** artom has joined #openstack-nova15:31
*** xiaolin has joined #openstack-nova15:32
gibisean-k-mooney, lyarwood: I was able to reproduce https://bugs.launchpad.net/nova/+bug/1888237 in a local devstack. I've added the needed local.conf config15:34
openstackLaunchpad bug 1888237 in OpenStack Compute (nova) "nova-next job fails as novnc service fails with TypeError: _wrap_socket() argument 1 must be _socket.socket, not GreenSSLSocket" [Critical,New]15:34
gibiI mean I added the local.conf info to the bug15:34
sean-k-mooneyok so enabligh the tls proxy or is there more15:35
gibitls proxy and forcing devstack to configure ssl for the novncproxy15:35
sean-k-mooneyoh you need NOVA_CONSOLE_PROXY_COMPUTE_TLS=True as well15:35
gibiUSE_SSL=True15:35
gibiENABLED_SERVICES+=,tls-proxy15:35
gibiGLANCE_STANDALONE=False15:35
gibiNOVA_CONSOLE_PROXY_COMPUTE_TLS=True15:35
sean-k-mooneyyep just read the bug15:35
sean-k-mooneygiven you have repoduced it i thnk its safe to move to confirmed15:37
sean-k-mooneycritical instead of high is because its a gate blocker right15:38
gibiright, it is a gate blocker, so I set it to critical15:39
gibiand now it is in confirmed state too15:40
* gibi going backward in the nova repo trying to find a commit where novncproxy still works15:46
*** gyee has joined #openstack-nova15:51
melwittgibi: saw your latest comment on the lp bug, fwiw urllib* is mentioned on this old eventlet issue (linked by sean-k-mooney) too. so I wonder if patching it/not patching it is related15:54
melwitthttps://github.com/eventlet/eventlet/issues/20115:54
gibimelwitt: I think something changed the import order and now urllib3 in imported before monkey_patch15:54
melwittahh15:55
melwittI might be remembering it wrong but I want to say that was done intentionally a long time ago.. /me looks15:55
sean-k-mooneydont we have a log message for that15:56
gibicould be that I'm mixing that up15:56
gibisean-k-mooney: yes, and that is what I find as a difference between the failing and the passing job15:56
sean-k-mooneyah ok15:56
melwittoh wait sorry I misunderstood the original change15:57
openstackgerritLee Yarwood proposed openstack/nova master: libvirt: Handle VIR_ERR_DEVICE_MISSING when detaching devices  https://review.opendev.org/74156115:57
lyarwoodmelwitt: ^ losing my mind with this, hopefully it's slightly better this time around.15:57
melwittthis is what I was looking at the guts of the log message https://github.com/openstack/nova/blob/b7161fe9b92f0045e97c300a80e58d32b6f49be1/nova/monkey_patch.py#L3915:57
melwittthis made me think we import urllib3 before patching intentionally?15:58
gibithis is the current import order http://paste.openstack.org/show/796133/16:00
melwittlyarwood: haha ok. sorry about that, not trying to cause headaches but if all 4 errors are fair game for both live/inactive detach failures then it seemed to me like all possible combos should have test cases. that's why I was asking are we holding to what the code comments say, only some errors are for live and only some are for inactive16:00
*** hamalq has quit IRC16:01
gibiit seems eventlet imports greendns that imports requests that imports urllib316:01
*** hamalq has joined #openstack-nova16:01
melwittI see, I was about to say I didn't see us importing urllib3 directly anywhere16:02
lyarwoodmelwitt: yeah sorry wasn't a complaint about your review, just a rant :)16:02
melwittlyarwood: heh ok :)16:02
gibithe passing job uses dnspython===1.16.0 the failing one uses dnspython===2.0.016:03
gibiand the dns package is in the import trace aboce16:03
gibiI'm getting close...16:04
gibiyepp pulling dnspython back to 1.16.0 resolves the novnc issue locally16:06
gibithis was the commit bumping that version during the weekned https://review.opendev.org/#/c/741800/16:07
*** hamalq_ has joined #openstack-nova16:07
melwittgibi: ahhh nice find! I don't think I ever would have figured it out 😆16:08
gibimelwitt: it was a shoot in the dark based on the import trace16:10
*** hamalq has quit IRC16:11
gibishould I propose a pin in the global upper-constraints to fix it?16:11
sean-k-mooneyah that was one of the packages that did change between the runs16:11
*** yoctozepto has quit IRC16:12
*** yoctozepto has joined #openstack-nova16:13
melwittyeah I'm not sure what else we could do, either pin globally or pin nova. I'd ask the requirements peeps: prometheanfire or smcginnis16:13
sean-k-mooneypaste.openstack.org/show/796119/16:13
prometheanfirewat16:13
sean-k-mooneyso we went form 1.16.0 to 2.0.016:13
gibiprometheanfire: dnspython 1.16.0 -> 2.0.0 bump broke nova novncproxy16:14
melwittthe new dnspython 2.0.0 broke our novnc ssl stuff16:14
melwittin a way that seems to not be our fault16:14
melwitthttps://bugs.launchpad.net/nova/+bug/188823716:14
openstackLaunchpad bug 1888237 in OpenStack Compute (nova) "nova-next job fails as novnc service fails with TypeError: _wrap_socket() argument 1 must be _socket.socket, not GreenSSLSocket" [Critical,Triaged]16:14
prometheanfireupstream bug in novnc?16:14
gibiprometheanfire: more like a change in behavior (import order) in dnspython16:14
gibiprometheanfire: that breaks eventlet monkey patching for nova16:14
*** k_mouza has quit IRC16:15
prometheanfireok, upstream bug in dnspython?16:15
prometheanfireI just want to make sure that if/when we mask this we refrence the right bug :D16:15
gibiprometheanfire: I have no idea if dnspython intenationally change things in 2.0.0 that does not work with eventlet16:16
gibithere is a related eventlet issue is https://github.com/eventlet/eventlet/issues/61916:16
gibieventlet is incompatible with dnspython 2.0.0rc116:16
prometheanfiremy main requirement (hehe, get it) is that we refrence when we can remove or relax the masking16:17
gibiprometheanfire: OK, I will reference the above eventlet issue16:17
prometheanfirethe eventlet bug looks promising16:17
gibiprometheanfire: so, can I pin dnspython to 1.16.0 in the requirements repo by referencing the eventlet bug?16:18
prometheanfireya, eventlet deps on dnspython, so the bug is in eventlet16:19
prometheanfiregibi: and mask dnspython 2.0.016:19
gibiprometheanfire: ack, will propose a patch soon16:20
*** xiaolin has quit IRC16:21
*** xiaolin has joined #openstack-nova16:22
openstackgerritGhanshyam Mann proposed openstack/nova-specs master: DNM: testing focal  https://review.opendev.org/74198416:23
sean-k-mooneygibi do you want to update https://github.com/eventlet/eventlet/issues/63216:24
gibisean-k-mooney: I will do16:24
gibisean-k-mooney: I duplicate it to 61916:24
openstackgerritLee Yarwood proposed openstack/nova master: libvirt: Track blockjob status in DEBUG when polling for completion  https://review.opendev.org/72959516:25
openstackgerritLee Yarwood proposed openstack/nova master: libvirt: Remove blockjob.end == 0 workaround resolved in libvirt v2.3.0  https://review.opendev.org/72959616:25
gibisean-k-mooney: I think I have no rights to mark it as duplicate but I left a comment stating it16:26
sean-k-mooneyi can close it16:26
gibigo ahead16:27
sean-k-mooneyjust commenting on the other bug is enough to cross link them so ya its closed16:27
gibicool, thanks16:27
*** udesale_ has quit IRC16:28
*** dtantsur is now known as dtantsur|afk16:29
*** ociuhandu_ has joined #openstack-nova16:30
gibisean-k-mooney, prometheanfire, melwitt, lyarwood: pushed https://review.opendev.org/#/c/741986/16:32
sean-k-mooney27 secods ago16:32
sean-k-mooneyoh you pushed16:32
sean-k-mooneyi missread16:33
sean-k-mooneyi thougt you ment lyarwood pushed it and you noticed and were telling us16:33
gibisorry :)16:33
*** ociuhandu has quit IRC16:33
*** ociuhandu_ has quit IRC16:34
openstackgerritHuaqiang Wang proposed openstack/nova master: compute: bump nova-compute version and check in API  https://review.opendev.org/74093316:39
openstackgerritHuaqiang Wang proposed openstack/nova master: hardware: Enable 'hw:cpu_dedicated_mask' for creating a mixed instance  https://review.opendev.org/71470616:39
openstackgerritHuaqiang Wang proposed openstack/nova master: hardware: create 'mixed' instance for realtime CPUs  https://review.opendev.org/72848516:39
openstackgerritHuaqiang Wang proposed openstack/nova master: metadata: export the vCPU IDs that are pinning on the host CPUs  https://review.opendev.org/68893616:39
openstackgerritHuaqiang Wang proposed openstack/nova master: release note: add new features introduces by use-pcpu-and-vcpu-in-one-instance  https://review.opendev.org/73699716:39
openstackgerritHuaqiang Wang proposed openstack/nova master: test: add some notes and new tests for mixed instance  https://review.opendev.org/74139516:39
*** xiaolin has quit IRC16:40
gibiOK, I'm ending my workday see you tomorrow16:44
melwitto/16:47
*** jraju__ has quit IRC16:49
*** jangutter has quit IRC16:58
*** derekh has quit IRC17:00
*** nweinber has quit IRC17:02
*** jangutter has joined #openstack-nova17:03
*** nweinber has joined #openstack-nova17:03
*** brinzhang_ has joined #openstack-nova17:12
*** brinzhang0 has quit IRC17:15
*** dklyle has quit IRC17:21
*** dklyle has joined #openstack-nova17:21
*** jsuchome has quit IRC17:32
*** ociuhandu has joined #openstack-nova17:34
*** brinzhang0 has joined #openstack-nova17:37
*** ociuhandu has quit IRC17:38
*** brinzhang_ has quit IRC17:40
openstackgerritStephen Finucane proposed openstack/nova master: Add support for resize and cold migration of emulated TPM files  https://review.opendev.org/63993417:42
openstackgerritStephen Finucane proposed openstack/nova master: Don't unset Instance.old_flavor, new_flavor until necessary  https://review.opendev.org/74199517:42
*** brinzhang_ has joined #openstack-nova17:43
stephenfinhuaqiang: Have +2d those two patches. However, we're having issues with the gate so I wouldn't recheck yet17:45
* stephenfin -> 🏃17:45
*** brinzhang0 has quit IRC17:46
*** ralonsoh has quit IRC17:48
*** nightmare_unreal has quit IRC17:56
*** brinzhang0 has joined #openstack-nova17:56
*** tesseract has quit IRC17:58
*** brinzhang_ has quit IRC17:59
*** brinzhang_ has joined #openstack-nova18:00
*** brinzhang0 has quit IRC18:03
*** brinzhang0 has joined #openstack-nova18:26
*** brinzhang has joined #openstack-nova18:28
*** brinzhang_ has quit IRC18:29
*** brinzhang0 has quit IRC18:31
*** artom has quit IRC18:38
*** spatel has quit IRC18:42
*** artom has joined #openstack-nova18:52
*** tosky has quit IRC18:57
*** xek_ has joined #openstack-nova19:02
*** maciejjozefczyk has quit IRC19:33
*** ociuhandu has joined #openstack-nova19:36
*** vishalmanchanda has quit IRC19:41
*** brinzhang_ has joined #openstack-nova19:59
*** brinzhang has quit IRC20:02
*** ociuhandu has quit IRC20:06
*** dave-mccowan has joined #openstack-nova20:47
*** bnemec has quit IRC20:55
*** markvoelker has joined #openstack-nova20:58
*** markvoelker has quit IRC21:02
*** nweinber has quit IRC21:04
*** bnemec has joined #openstack-nova21:10
*** nweinber has joined #openstack-nova21:11
*** raildo has quit IRC21:25
*** nweinber has quit IRC21:31
*** brinzhang_ has quit IRC21:33
*** brinzhang0 has joined #openstack-nova21:33
openstackgerritGhanshyam Mann proposed openstack/nova master: Add new default roles in FIP policies  https://review.opendev.org/74203021:34
*** jdillaman has quit IRC21:35
*** xek_ has quit IRC21:40
openstackgerritGhanshyam Mann proposed openstack/nova master: Add new default roles in baremetal nodes policies  https://review.opendev.org/74171621:43
openstackgerritGhanshyam Mann proposed openstack/nova master: Pass the actual target in baremetal nodes policy  https://review.opendev.org/74171721:43
openstackgerritGhanshyam Mann proposed openstack/nova master: Correct the check_str and pass actual target in FIP pools policy  https://review.opendev.org/74179721:53
*** brinzhang_ has joined #openstack-nova22:00
*** brinzhang0 has quit IRC22:03
*** brinzhang0 has joined #openstack-nova22:08
*** brinzhang_ has quit IRC22:11
*** martinkennelly has quit IRC22:32
*** fumesover32 has joined #openstack-nova22:39
*** fumesover3 has quit IRC22:39
*** tkajinam has joined #openstack-nova22:53
*** markvoelker has joined #openstack-nova22:53
*** rcernin has joined #openstack-nova22:58
*** mlavalle has quit IRC22:59
*** rcernin has quit IRC22:59
*** rcernin has joined #openstack-nova22:59
*** markvoelker has quit IRC23:03
*** artom has quit IRC23:05
*** artom has joined #openstack-nova23:05
*** bnemec has quit IRC23:20

Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!