Monday, 2020-07-20

*** ryohayakawa has joined #openstack-infra00:00
*** irclogbot_1 has quit IRC00:16
*** rlandy has joined #openstack-infra00:18
*** rlandy has quit IRC00:18
*** tetsuro has joined #openstack-infra00:19
*** irclogbot_3 has joined #openstack-infra00:19
*** ociuhandu has joined #openstack-infra00:25
*** ryohayakawa has quit IRC00:28
*** ryohayakawa has joined #openstack-infra00:29
*** ociuhandu has quit IRC00:30
*** rfolco has quit IRC00:33
*** tetsuro has quit IRC00:39
*** tetsuro has joined #openstack-infra00:41
*** armax has quit IRC01:15
*** markvoelker has joined #openstack-infra01:36
*** markvoelker has quit IRC01:41
*** ricolin has joined #openstack-infra01:43
*** tetsuro has quit IRC03:02
*** rcernin has quit IRC03:02
*** dave-mccowan has joined #openstack-infra03:11
*** rcernin has joined #openstack-infra03:21
*** rcernin has quit IRC03:25
*** rcernin has joined #openstack-infra03:26
*** tetsuro has joined #openstack-infra03:57
*** dave-mccowan has quit IRC03:57
*** armax has joined #openstack-infra04:18
*** ociuhandu has joined #openstack-infra04:28
*** ociuhandu has quit IRC04:33
*** Lucas_Gray has quit IRC04:33
*** ykarel has joined #openstack-infra04:43
*** vishalmanchanda has joined #openstack-infra04:51
*** ysandeep|away is now known as ysandeep04:59
*** ysandeep is now known as ysandeep|rover04:59
*** matt_kosut has joined #openstack-infra05:44
*** udesale has joined #openstack-infra05:48
*** lmiccini has joined #openstack-infra05:55
*** soniya29 has joined #openstack-infra06:09
*** marios has joined #openstack-infra06:28
*** eolivare has joined #openstack-infra06:28
*** ykarel_ has joined #openstack-infra06:31
*** ykarel has quit IRC06:33
*** tetsuro has quit IRC06:34
*** tetsuro has joined #openstack-infra06:36
openstackgerritMerged zuul/zuul-jobs master: Remove copy paste from upload-logs-swift  https://review.opendev.org/74184006:41
*** nightmare_unreal has joined #openstack-infra06:57
*** ykarel_ is now known as ykarel07:08
*** ociuhandu has joined #openstack-infra07:12
*** tetsuro_ has joined #openstack-infra07:15
*** jcapitao has joined #openstack-infra07:16
*** tetsuro has quit IRC07:19
*** rcernin has quit IRC07:24
*** pkopec has joined #openstack-infra07:25
*** dtantsur|afk is now known as dtantsur07:27
*** yolanda has joined #openstack-infra07:31
*** tosky has joined #openstack-infra07:34
*** rcernin has joined #openstack-infra07:38
*** derekh has joined #openstack-infra07:39
*** xek_ has joined #openstack-infra07:39
*** tetsuro_ has quit IRC07:41
*** tetsuro has joined #openstack-infra07:41
*** rcernin has quit IRC07:44
*** jpena|off is now known as jpena07:46
*** rcernin has joined #openstack-infra07:48
*** soniya29 has quit IRC07:49
*** soniya29 has joined #openstack-infra07:52
*** rcernin has quit IRC07:52
*** ralonsoh has joined #openstack-infra07:55
*** andrewbonney has joined #openstack-infra07:58
*** lucasagomes has joined #openstack-infra07:58
*** markvoelker has joined #openstack-infra08:04
*** markvoelker has quit IRC08:09
*** fumesover3 has quit IRC08:12
*** markvoelker has joined #openstack-infra08:14
*** markvoelker has quit IRC08:24
*** markvoelker has joined #openstack-infra08:27
*** markvoelker has quit IRC08:32
*** derekh has quit IRC08:34
*** dchen has quit IRC08:38
*** ysandeep|rover is now known as ysandeep|lunch08:48
*** Lucas_Gray has joined #openstack-infra08:51
*** ralonsoh_ has joined #openstack-infra08:53
*** ralonsoh has quit IRC08:55
*** rcernin has joined #openstack-infra09:10
*** tetsuro has quit IRC09:12
*** rcernin has quit IRC09:15
*** ricolin has quit IRC09:17
openstackgerritSorin Sbarnea (zbr) proposed zuul/zuul-jobs master: Bump ansible-lint to speed it up  https://review.opendev.org/74189709:27
*** sshnaidm|off is now known as sshnaidm09:27
*** ysandeep|lunch is now known as ysandeep09:30
*** ysandeep is now known as ysandeep|rover09:31
*** fumesover3 has joined #openstack-infra10:02
*** ociuhandu has quit IRC10:05
*** ociuhandu has joined #openstack-infra10:08
*** Lucas_Gray has quit IRC10:08
*** tosky has quit IRC10:08
*** irclogbot_3 has quit IRC10:08
*** bdodd_ has quit IRC10:08
*** vesper11 has quit IRC10:08
*** fdegir has quit IRC10:09
*** mmethot has quit IRC10:09
*** stevebaker has quit IRC10:09
*** Lucas_Gray has joined #openstack-infra10:10
*** tosky has joined #openstack-infra10:10
*** irclogbot_3 has joined #openstack-infra10:10
*** bdodd_ has joined #openstack-infra10:10
*** vesper11 has joined #openstack-infra10:10
*** fdegir has joined #openstack-infra10:10
*** mmethot has joined #openstack-infra10:10
*** stevebaker has joined #openstack-infra10:10
*** Lucas_Gray has quit IRC10:10
*** Lucas_Gray has joined #openstack-infra10:11
*** ociuhandu has quit IRC10:13
*** derekh has joined #openstack-infra10:20
*** tkajinam has quit IRC10:29
*** ociuhandu has joined #openstack-infra10:32
*** trident has quit IRC10:35
*** trident has joined #openstack-infra10:36
*** ysandeep|rover is now known as ysandeep|afk11:04
*** jcapitao is now known as jcapitao_lunch11:13
*** ysandeep|afk is now known as ysandeep|rover11:31
*** weshay_pto is now known as weshay_11:34
*** jpena is now known as jpena|lunch11:45
*** rfolco has joined #openstack-infra11:59
*** xek_ has quit IRC12:00
*** yolanda has quit IRC12:14
*** yolanda has joined #openstack-infra12:15
*** Goneri has joined #openstack-infra12:17
*** ryohayakawa has quit IRC12:23
*** jcapitao_lunch is now known as jcapitao12:31
*** jpena|lunch is now known as jpena12:32
*** ricolin has joined #openstack-infra12:37
*** rlandy has joined #openstack-infra12:52
*** rlandy is now known as rlandy|ruck12:54
*** __ministry1 has joined #openstack-infra13:01
*** udesale_ has joined #openstack-infra13:03
*** udesale has quit IRC13:06
*** rcernin has joined #openstack-infra13:07
*** ricolin_ has joined #openstack-infra13:08
*** ricolin has quit IRC13:10
*** rcernin has quit IRC13:12
*** ricolin has joined #openstack-infra13:16
*** lbragstad has joined #openstack-infra13:16
*** ricolin_ has quit IRC13:17
*** dave-mccowan has joined #openstack-infra13:17
*** artom has joined #openstack-infra13:18
*** eharney has joined #openstack-infra13:26
*** ociuhandu_ has joined #openstack-infra13:29
*** ociuhandu has quit IRC13:31
*** bnemec has joined #openstack-infra13:32
*** derekh has quit IRC13:34
*** d34dh0r53 has joined #openstack-infra13:55
artomfungi, o/ continuing our multi-numa/SMT conversation from a while ago (in the context of OpenEdge being down)13:58
artomIf we wanted to get other providers on board with those kinds of features (the ones that already provide nested-virt), what would be the best way forward?13:59
donnydAs soon as the heat wave breaks here OE will be back online13:59
artomI was told it was possible to use image metadata and configure the properties we need in nodepool directly13:59
donnydI think they just need to create a flavor that is setup correctly14:00
clarkbartom: I'm not sure that is true since numa requires specific flavor support I thought14:00
clarkbdonnyd: yes that is my undersatnding because you have to be scheduler aware (hypervisors only have so many numa slots)14:00
artomclarkb, as long as the flavor doesn't explicitly forbid it (ie, sets hw:numa_nodes=1), we can use image properties to override it (ie hw_numa_nodes=2)14:01
clarkbartom: what happens if you get scheduled to a node where that isn't possible?14:01
clarkbdoes it soft fail?14:01
artomclarkb, but that feels impolite, essentially going behind providers's backs14:02
clarkbartom: also its likely to fail because the scheduler can't help ensure it will work14:02
artomDoing it in the flavor would be more transparent and honest14:02
artomclarkb, Nova takes image properties into account when scheduling14:02
clarkbgotcha14:03
artomSo I could just try to hack image meta in project-config and push that as a WIP/DNM to get the ball rolling14:03
clarkbusually what we do is ask the cloud providers directly to see what is possible or feasible. mnaser logan- and artom may have specific thoughts on numa (I believe all 3 have clouds that enabled nested virt already)14:03
clarkbI imagine this won't work on xen or where nested virt is disabled (which I think rules out rax and inap but I would have to check on inap)14:04
artomclarkb, yeah, this would only work on providers that already have a nested-virt- flavor14:04
artomAnd doesn't actually require anything else. The guest NUMA topology is completely independent of the host's14:04
clarkbalso you may want to work wtih ericcson and citynetwork to see if we can do anything to make that cloud more reliable?14:05
*** ociuhandu_ has quit IRC14:05
clarkbthings were happy before because donnyd stepped in to fill the gap they left, but that left the problem existing on that side I Think14:05
*** ociuhandu has joined #openstack-infra14:06
*** derekh has joined #openstack-infra14:06
fungiyes, the scheduling issue in citycloud has continued14:06
clarkboh I see, if you pin guests to numa node on the hypervisor you can increase performance, but then within that you can present a different virtualized numa topology in the guest itself?14:09
artomclarkb, they're essentially independent things14:09
artom(And I had the wrong image property, it's hw_cpu_sockets)14:10
*** aedc has joined #openstack-infra14:10
clarkbartom: ya, I've always seen it in the context of the hyper visor not the guest, but that leads me to wondering what that image property controls14:10
artomThe guest topology can be whatever the image properties/flavor extra specs say14:10
clarkbis it controling the hypervisor side configuration or the virtual topology in the guest?14:10
clarkbalso I don't think a cpu socket is necessarily a numa node?14:10
artomTrue14:11
clarkbWe need a Sean Mooney signal as I'm sure Sean could clarify all of this for us14:11
artomHeh, it was talking to him that I first hatched this thought14:12
artomBut now we've confused ourselves14:12
clarkbI'm not sure I was ever unconfused :) but ya I think we need to be clear what the actual needs are: is it performance and we need/want the hypervisor to be smart of scheduling the VM workloads? is it testing of nova's NUMA code paths within a guest and we are happy with a "virtual" numa topology? then depending on which of those things it is do we actually get what we want via image properties?14:14
artomclarkb, oh, I'm clear on what I want14:14
clarkbartom: fwiw it was sean that set up the NUMA stuff with donnyd on openedge and the flavor was used there not image properties14:14
clarkbartom: its not clear to me14:14
artomThe second - testing the Nova NUMA codepaths14:14
clarkbthanks14:14
artomSo virtual topology is perfectly fine14:14
artomMy confusion is can we actually achieve that with only image properties14:15
artomSean was saying "yes"14:15
artomBut now that I look at it (the docs, at least), I'm not so sure14:15
artomWith pure libvirt it's possible in the XML14:15
artomBut with Nova it looks like we need flavor and compute host hardware support (in the shape of multiple NUMA nodes)14:16
clarkbya I'm looking at flavor docs and hw_cpu_sockets isn't quite what we want14:17
clarkband it seems to make this work we need the provider to have configured things to avoid errors14:17
artomRight - well, in any case, the thing to do seems to be "have a discussion with the providers" and not "do stuff behind their back with images in nodepool"14:21
artomSo short-term we can use OpenEdge in non-voting job for the stuff we need (multi-numa and SMT)14:21
artomWhile we work on expanding that support with other providers14:21
donnydWhat is the timeline for getting these jobs done?14:21
donnydIts possible I could get something fired up, but I won't be able to leave it on until I get a replacement air handler14:22
artomdonnyd, take your time, this isn't life or death :)14:22
donnydkk14:22
artomOpenStack will live on even if NUMA isn't tested in the gate ;)14:22
openstackgerritGhanshyam Mann proposed openstack/devstack-gate master: Always assume glance is uwsgi mode for grenade  https://review.opendev.org/74195514:23
artomAnd as always, what you do is very appreciated :)14:23
artomWe just need to actually be able to land patches14:23
artom(not in a Nova itself, the NUMA testing is done in a tempest plugin)14:23
artomAnd for that, we need to move off openedge while it's cooling ;)14:23
*** xek_ has joined #openstack-infra14:26
*** zxiiro has joined #openstack-infra14:38
*** ysandeep|rover is now known as ysandeep|away14:47
*** ykarel is now known as ykarel|away14:50
*** xek_ has quit IRC14:50
*** ysandeep|away is now known as ysandeep14:54
*** rcernin has joined #openstack-infra14:55
*** kaiokmo has joined #openstack-infra14:56
*** xek_ has joined #openstack-infra14:56
kaiokmohello gents, does anyone know a way to hide openstack openrc variables from an ansible execution in debug mode?14:58
*** dklyle has joined #openstack-infra14:59
*** rcernin has quit IRC15:00
fungikaiokmo: you're probably looking for the #openstack or #openstack-ansible channels... you've found the channel used by the testing and collaboration tools sig15:00
kaiokmofungi: I see... tyvm15:00
fungiyou're welcome!15:00
*** Tengu has quit IRC15:04
*** Tengu has joined #openstack-infra15:05
*** xek_ has quit IRC15:07
*** pkopec has quit IRC15:09
*** ralonsoh_ is now known as ralonsoh15:10
*** ykarel|away has quit IRC15:11
*** weshay_ is now known as weshay|ruck15:12
*** zbr|ruck is now known as zbr|rover15:13
*** hamalq has joined #openstack-infra15:15
*** ysandeep is now known as ysandeep|away15:17
*** jgwentworth is now known as melwitt15:19
*** artom has quit IRC15:31
*** artom has joined #openstack-infra15:31
*** lmiccini has quit IRC15:47
*** marios is now known as marios|out15:48
*** marios|out has quit IRC15:49
openstackgerritElod Illes proposed openstack/openstack-zuul-jobs master: Enable neutron-grenade again for Stein, Rocky and Queens  https://review.opendev.org/74127415:50
*** gyee has joined #openstack-infra15:51
openstackgerritMerged zuul/zuul-jobs master: add-build-sshkey: Ensure .ssh exists, enable admin authorized_keys  https://review.opendev.org/74035015:57
*** sshnaidm is now known as sshnaidm|afk15:57
*** lucasagomes has quit IRC16:00
*** hamalq has quit IRC16:01
*** hamalq has joined #openstack-infra16:01
*** hamalq_ has joined #openstack-infra16:07
*** hamalq has quit IRC16:11
*** yoctozepto has quit IRC16:12
*** yoctozepto has joined #openstack-infra16:13
*** ricolin has quit IRC16:14
jokkeAnyone aware of releases over the weekend in our dependencies. Looks like glance functional jobs are failing by the api rpocess not starting but no clear indication why16:17
fungijokke: check the discuss ml, there was a change in devstack which make glance no longer work without tls-proxy enabled16:19
fungier, which made16:19
jokkethe functional jobs are not running against devstack, so that should not be it, but thanks will have a look on that still16:19
fungii think the solution has been to start turning on tls-proxy anywhere it wasn't already used, and to consider making it on by default in devstack16:19
fungiahh, not sure what would have impacted non-devstack functional jobs then. have a link to a build which failed?16:20
fungibut yeah, the reason my mind jumped to tls-proxy is that the symptom was g-api not starting16:20
jokkefungi: https://1673d900d285d3364834-2ea81b3730bb9953c88a89ad16b24dc1.ssl.cf5.rackcdn.com/741321/1/gate/openstack-tox-functional-py38/81597a5/16:24
jokkeWe saw that on our periodic tips jobs and just realized that it wasn't just tips failing16:25
fungijokke: thanks, i'm in the middle of a meeting but will pull it up and see if i can spot anything16:25
clarkbpymysql dropped context manager support in some place that hit tooz16:25
jokkefungi: amazing, thanks16:25
jokkeclarkb: thanks, will look into that as possibility16:26
*** udesale_ has quit IRC16:28
*** dtantsur is now known as dtantsur|afk16:29
*** ociuhandu_ has joined #openstack-infra16:30
*** jcapitao has quit IRC16:31
*** ociuhandu has quit IRC16:33
*** ociuhandu_ has quit IRC16:34
*** jpena is now known as jpena|off16:45
openstackgerritGhanshyam Mann proposed openstack/devstack-gate master: Always assume glance is uwsgi mode for grenade  https://review.opendev.org/74195516:53
*** rcernin has joined #openstack-infra16:56
*** derekh has quit IRC17:00
*** rcernin has quit IRC17:01
*** eolivare has quit IRC17:05
*** andrewbonney has quit IRC17:06
jokkehttp://paste.openstack.org/show/796135/ our potential culprits. That's pip3 freeze between working functional venv and broken one17:15
*** dklyle has quit IRC17:21
*** dklyle has joined #openstack-infra17:21
*** __ministry1 has quit IRC17:33
*** ociuhandu has joined #openstack-infra17:34
*** ociuhandu has quit IRC17:38
*** Lucas_Gray has quit IRC17:45
*** ralonsoh has quit IRC17:48
*** nightmare_unreal has quit IRC17:56
fungii guess we don't collect the api log for that job?18:04
*** artom has quit IRC18:38
*** soniya29 has quit IRC18:50
*** artom has joined #openstack-infra18:52
fungiprobably the next step is to attempt to reproduce that result locally and see if the api log indicates what caused the service not to start (i expect there will be a python traceback)18:57
*** rcernin has joined #openstack-infra18:57
*** tosky has quit IRC18:57
*** rcernin has quit IRC19:01
*** xek_ has joined #openstack-infra19:02
AJaegerconfig-core, please review https://review.opendev.org/737578 and https://review.opendev.org/74127419:06
gmannAJaeger: clarkb fungi can you review this to unblock legacy grenade jobs -https://review.opendev.org/#/c/741955/19:16
jokkefungi: so we've been hammmering this with abhishekk for few hours now. He just found out that if we bypass stestr the tests executes clean, but looking my broken venv and non-broken one the stestr version has not changed so kind of in a dead end again19:20
fungijokke: when it breaks, is there an api log left behind by the service which might indicate why it didn't start (traceback, anything)?19:21
jokkefungi: no there is message:19:22
jokkeCaptured api:19:22
jokke~~~~~~~~~~~~ /tmp/tmpcg9lcwrq/multiple_backend_api.log for api did not exist19:22
*** ociuhandu has joined #openstack-infra19:36
AJaegergmann: I'm not a devstack-gate core19:38
gmannohk19:38
gmannmordred: corvus  if you are around, this is to unblock legacy grenade jobs  https://review.opendev.org/#/c/741955/19:39
corvusgmann: +2 clarkb ^19:40
gmanncorvus: thanks19:40
*** vishalmanchanda has quit IRC19:41
clarkbcorvus: gmann looking19:42
fungii was trying to digest the related discussion in #openstack-tc first19:47
fungibut yeah, it seems to jive19:47
clarkbfungi: we're forcing unsupported glance config in grenade because supported glance config support was never added to devsatck/grenade19:52
clarkbits quite fun19:52
fungiyep19:56
fungithat's an accurate summary19:57
*** ociuhandu has quit IRC20:06
jokkefungi: looks like it's dnspython ... once that's capped we are working again20:08
jokkeneed to see what's changed and hunt down where do we use that20:08
fungithat library used to introduce a lot of breakage every time they released, but it's been a while now since the last time i remember it causing an issue20:12
fungimaybe they took a few years off ;)20:12
jokkewell they released 2.0.0 so this time they have at least versioned it correctly for breaking20:14
fungioh good20:18
*** dave-mccowan has quit IRC20:44
*** dave-mccowan has joined #openstack-infra20:47
openstackgerritMerged zuul/zuul-jobs master: Enable tls-proxy in ensure-devstack  https://review.opendev.org/74182020:49
*** bnemec has quit IRC20:55
*** rcernin has joined #openstack-infra20:58
*** markvoelker has joined #openstack-infra20:58
*** rcernin has quit IRC21:02
*** markvoelker has quit IRC21:02
*** bnemec has joined #openstack-infra21:10
*** aedc has quit IRC21:14
*** aedc has joined #openstack-infra21:15
openstackgerritMerged openstack/devstack-gate master: Always assume glance is uwsgi mode for grenade  https://review.opendev.org/74195521:18
*** aedc has quit IRC21:20
*** matt_kosut has quit IRC21:29
*** matt_kosut has joined #openstack-infra21:29
*** matt_kosut has quit IRC21:33
*** xek_ has quit IRC21:40
*** kaiokmo has quit IRC22:36
*** fumesover32 has joined #openstack-infra22:39
*** fumesover3 has quit IRC22:39
*** tkajinam has joined #openstack-infra22:53
*** markvoelker has joined #openstack-infra22:53
*** Lucas_Gray has joined #openstack-infra22:55
*** rcernin has joined #openstack-infra22:58
*** rcernin has quit IRC22:59
*** rcernin has joined #openstack-infra22:59
*** markvoelker has quit IRC23:03
*** artom has quit IRC23:05
*** artom has joined #openstack-infra23:05
*** bnemec has quit IRC23:20
*** armax has quit IRC23:23
*** dchen has joined #openstack-infra23:29
*** rlandy|ruck has quit IRC23:33
*** armax has joined #openstack-infra23:36

Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!