Thursday, 2019-04-25

ccstoneThanks mriedem, we were able to correct the transport_url in the DB using that and that's what finally made the error stop.00:00
openstackgerritMatt Riedemann proposed openstack/nova master: Make nova.compute.rpcapi.ComputeAPI.router a singleton  https://review.opendev.org/64919700:01
mriedemccstone: "using that" => nova-manage cell_v2 update_cell?00:01
*** cooper6581 has joined #openstack-nova00:05
*** dave-mccowan has quit IRC00:05
*** lbragstad has quit IRC00:05
cooper6581mriedem:  (with ccstone and eandersson being trained).  You are right that it should have just returned the URL as-is, but I'm assuming CONF.transport_url ended up not being None00:08
cooper6581https://ghosthub.corp.blizzard.net/openstack/nova/blob/import/rocky/nova/objects/cell_mapping.py#L14500:08
cooper6581oops, wrong link00:08
ccstonemriedem: Yeah - we had updated nova.conf with a new rabbit password and were still getting the "unmatched '{'" error using nova-manage. Updating the transport_url in the DB via update_cell and the link you provided was what ultimately fixed it for us. Seems like simple_cell_setup allowed us to provide a bad transport_url without URL encoding.00:08
mriedemok00:09
cooper6581https://github.com/openstack/nova/blob/master/nova/objects/cell_mapping.py#L14500:09
mriedemi guess if there is anything bug worthy to follow up with this to help someone else from falling in the same trap that would be good to report, maybe it could just be docs00:11
*** slaweq has joined #openstack-nova00:11
*** gibi has quit IRC00:16
*** gibi has joined #openstack-nova00:18
mriedemha this actually ran tests https://review.opendev.org/#/c/655222/00:18
mriedemwas sure that would explode in devstack setup00:18
mriedemdansmith: ^00:19
mriedemlooks like the only thing that shit the bed was cold migration testing since tempest doesn't have a toggle for that00:19
*** dikonoor has joined #openstack-nova00:21
*** slaweq has quit IRC00:24
*** gyee has quit IRC00:24
*** lbragstad has joined #openstack-nova00:24
mriedemi see why it's failing there, i'm saying resize to same host is ok b/c i only have 1 compute per cell, but you can't cold migrate to the same host with the libvirt driver, that will be fixed with cross-cell resize ftw00:32
*** itlinux has joined #openstack-nova00:34
*** yedongcan has joined #openstack-nova00:38
*** igordc has quit IRC00:40
openstackgerritMerged openstack/os-vif master: Refactor functional base test classes  https://review.opendev.org/64310100:46
*** itlinux has quit IRC00:58
*** brinzhang has joined #openstack-nova00:58
*** gibi has quit IRC00:58
*** whoami-rajat has joined #openstack-nova01:02
openstackgerritMatt Riedemann proposed openstack/nova master: WIP: Add nova-multi-cell job  https://review.opendev.org/65522201:03
*** ricolin has joined #openstack-nova01:05
mriedemefried: just fyi i've removed the cross-cell-resize series from the runway slot (time expired) and added https://blueprints.launchpad.net/nova/+spec/add-host-and-hypervisor-hostname-flag-to-create-server01:09
mriedemboxiang: ^01:09
mriedemhttps://etherpad.openstack.org/p/nova-runways-train01:09
*** mriedem has quit IRC01:11
*** slaweq has joined #openstack-nova01:13
*** gibi has joined #openstack-nova01:16
*** alex_xu has quit IRC01:17
*** slaweq has quit IRC01:24
*** tetsuro has joined #openstack-nova01:25
*** markvoelker has quit IRC01:34
*** cooper6581 has quit IRC01:45
*** tetsuro has quit IRC01:51
melwittlyarwood, mriedem: ceph dashboard fix attempt https://review.opendev.org/65559101:52
*** tetsuro has joined #openstack-nova02:06
*** slaweq has joined #openstack-nova02:11
*** tetsuro has quit IRC02:18
*** tetsuro has joined #openstack-nova02:23
*** slaweq has quit IRC02:24
*** dikonoor has quit IRC02:25
*** alex_xu has joined #openstack-nova02:29
*** itlinux has joined #openstack-nova02:32
*** lbragstad has quit IRC02:36
*** JamesBenson has joined #openstack-nova02:41
*** alex_xu has quit IRC02:45
*** ricolin has quit IRC02:46
*** hongbin has joined #openstack-nova02:51
*** smcginnis has quit IRC03:06
openstackgerritMerged openstack/nova stable/stein: Fix {min|max}_version in ironic Adapter setup  https://review.opendev.org/65542903:08
openstackgerritMerged openstack/nova stable/rocky: Fix regression in glance client call  https://review.opendev.org/65516703:08
*** ivve has quit IRC03:10
*** dikonoor has joined #openstack-nova03:11
*** hongbin has quit IRC03:12
*** slaweq has joined #openstack-nova03:16
*** slaweq has quit IRC03:24
*** brinzhang has quit IRC03:25
*** brinzhang has joined #openstack-nova03:26
*** markvoelker has joined #openstack-nova03:35
*** psachin has joined #openstack-nova03:52
*** udesale has joined #openstack-nova03:55
*** ricolin has joined #openstack-nova04:10
*** pcaruana has joined #openstack-nova04:11
*** slaweq has joined #openstack-nova04:13
*** JamesBenson has quit IRC04:18
*** ileixe has quit IRC04:22
*** slaweq has quit IRC04:24
*** ivve has joined #openstack-nova04:25
*** ileixe has joined #openstack-nova04:25
*** ileixe has quit IRC04:26
*** ileixe has joined #openstack-nova04:27
*** pcaruana has quit IRC04:43
*** tetsuro has quit IRC04:51
*** ivve has quit IRC05:06
*** kukacz has quit IRC05:06
*** kukacz has joined #openstack-nova05:08
*** slaweq has joined #openstack-nova05:11
*** ivve has joined #openstack-nova05:18
*** slaweq has quit IRC05:21
*** Luzi has joined #openstack-nova05:27
*** slaweq has joined #openstack-nova05:27
*** itlinux has quit IRC05:30
*** slaweq has quit IRC05:41
*** jiaopengju_2 has joined #openstack-nova05:49
*** jiaopengju_1 has quit IRC05:52
*** slaweq has joined #openstack-nova05:56
*** slaweq has quit IRC06:02
*** igordc has joined #openstack-nova06:08
*** phasespace has quit IRC06:09
*** shilpasd has joined #openstack-nova06:10
*** slaweq has joined #openstack-nova06:11
*** slaweq has quit IRC06:16
*** JamesBenson has joined #openstack-nova06:18
*** pcaruana has joined #openstack-nova06:20
*** JamesBenson has quit IRC06:22
*** jiaopengju_1 has joined #openstack-nova06:27
*** jiaopengju_2 has quit IRC06:30
*** Luzi has quit IRC06:33
*** Luzi has joined #openstack-nova06:34
*** slaweq has joined #openstack-nova06:44
*** luksky has joined #openstack-nova06:46
*** phasespace has joined #openstack-nova06:48
*** jiaopengju_1 has quit IRC06:49
*** jiaopengju_1 has joined #openstack-nova06:49
*** phasespace has quit IRC06:52
*** igordc has quit IRC06:53
*** ricolin has quit IRC06:53
*** evrardjp has quit IRC06:55
*** evrardjp has joined #openstack-nova06:55
*** evrardjp has quit IRC06:58
*** evrardjp has joined #openstack-nova06:58
*** ccamacho has joined #openstack-nova07:08
*** ccamacho has quit IRC07:09
*** ccamacho has joined #openstack-nova07:10
*** dpawlik has quit IRC07:14
*** zbr has joined #openstack-nova07:25
*** ricolin has joined #openstack-nova07:28
*** slaweq has quit IRC07:31
*** slaweq has joined #openstack-nova07:32
*** slaweq has quit IRC07:34
*** slaweq has joined #openstack-nova07:34
*** slaweq has quit IRC07:35
*** slaweq has joined #openstack-nova07:36
*** slaweq has quit IRC07:36
*** dpawlik has joined #openstack-nova07:36
*** slaweq has joined #openstack-nova07:37
*** slaweq has quit IRC07:38
*** slaweq has joined #openstack-nova07:38
*** ttsiouts has joined #openstack-nova07:42
kashyapefried: Hehe, I learnt a new expression, "one-armed-wallpaperer" (or "one-armed paper hanger", I see).07:44
*** rpittau|afk is now known as rpittau07:58
*** phasespace has joined #openstack-nova08:02
*** tetsuro has joined #openstack-nova08:05
*** ralonsoh has joined #openstack-nova08:05
*** tssurya has joined #openstack-nova08:05
*** ttsiouts_ has joined #openstack-nova08:05
*** tetsuro has quit IRC08:06
*** ttsiouts has quit IRC08:07
*** derekh has joined #openstack-nova08:14
*** zbr is now known as zbr|rover08:18
*** ttsiouts has joined #openstack-nova08:19
*** ttsiouts_ has quit IRC08:20
kashyapefried: sean-k-mooney: Just looked at at the video models thing: yes, it makes sense to add the me, FWIW.  However, I'd appreciate a one-line summary of each of the three models being added in the BP08:23
kashyapefried: sean-k-mooney E.g. "The 'none' video model type disables the automatic addition of a video device to domains with 'graphics' specified in their XML.  This can be useful with GPU mediated devices which can serve as the only rendering devices within the guest."08:23
openstackgerritRodolfo Alonso Hernandez proposed openstack/os-vif master: Prevent "qbr" Linux Bridge from replying to ARP messages  https://review.opendev.org/65533208:23
*** zigo has quit IRC08:27
*** gmann has quit IRC08:28
*** ttsiouts_ has joined #openstack-nova08:32
*** ttsiouts has quit IRC08:35
*** ttsiouts has joined #openstack-nova08:49
*** ttsiouts_ has quit IRC08:50
*** tkajinam has quit IRC08:54
*** rha has joined #openstack-nova08:57
*** rcernin has quit IRC08:58
gibiefried, mriedem: regarding the length of the PTG slot :1330-1415: Let's plan the next steps of the bandwidth support feature (gibi)09:00
gibiefried, mriedem: I think we can speed things up, so we can be done in ~15-20 mins09:01
*** tobias-urdin has quit IRC09:01
*** tobias-urdin has joined #openstack-nova09:17
*** dims has quit IRC09:20
*** luksky has quit IRC09:21
aspierscheck jobs seem pretty flakey at the moment09:25
aspiersit's not normal to need 2 rechecks, right?09:25
*** dims has joined #openstack-nova09:26
*** tobias-urdin has quit IRC09:27
aspierskashyap: any idea? https://review.opendev.org/#/c/655268/2 keeps getting V-1 for no good reason AFAICS09:30
kashyapaspiers: Sometimes is completely "normal" to require more than 2 rechecks :-(09:31
aspiers:-(09:31
aspiersThis is an awesome technique for dealing with troublesome CI https://cfp.all-systems-go.io/ASG2018/talk/207/09:31
aspiersI think until OpenStack has something like that, flakey gates will always remain common09:31
kashyapI don't have any bright ideas (besides "recheck and pray to the JuJu at the bottom of the sea").  I've experienced the same in the past.09:31
aspiersUsing ML means the "flakes" can be clustered automatically (unsupervised) which then means it's crystal clear which flakes cause the most failures and require the most human attention to fix09:32
aspiersThat way we could get to the long tail of weird corner cases quicker09:33
aspierse.g. one flake might cause 30% of failures, and another 0.3%09:33
*** dims has quit IRC09:33
*** dtantsur|afk is now known as dtantsur09:33
aspiersbut until you gather stats on that, it's hard to know where to put effort09:33
*** mdbooth has quit IRC09:34
*** dims has joined #openstack-nova09:34
aspierselastic recheck aims for that of course, but it requires manual maintenance09:34
kashyapaspiers: Yeah, suggest it on the 'openstack-discuss' list, if you have a minute to write the email09:35
*** tobias-urdin has joined #openstack-nova09:40
*** zbr|rover has quit IRC09:42
*** mdbooth has joined #openstack-nova09:43
*** zbr has joined #openstack-nova09:44
*** jbernard has quit IRC09:48
*** dklyle has quit IRC09:50
*** dklyle has joined #openstack-nova09:50
*** luksky has joined #openstack-nova09:59
aspierskashyap: might be better to find mtreinish and the infra folks in Denver next week first10:03
aspiersI've already reached out to the cockpit project10:03
aspiersno reply yet10:03
*** jbernard has joined #openstack-nova10:04
aspierskashyap: when you get a moment, reviews on https://review.opendev.org/#/c/655268/ and https://review.opendev.org/#/c/633855/ would be much appreciated ;-)10:04
kashyapaspiers: Will do.  Currently doing something I promised efried that I'll address (in my own spec, though) today.  Trying to keep the word. :D10:05
aspierssure, good luck with it :)10:05
*** ccamacho has quit IRC10:11
*** mvkr has joined #openstack-nova10:13
*** ttsiouts_ has joined #openstack-nova10:24
openstackgerritStephen Finucane proposed openstack/nova-specs master: Standardize CPU resource tracking  https://review.opendev.org/55508110:26
*** ttsiouts has quit IRC10:29
*** ttsiouts_ has quit IRC10:29
*** ttsiouts has joined #openstack-nova10:31
openstackgerritAdam Spiers proposed openstack/os-resource-classes master: Add MEM_ENCRYPTION_CONTEXT resource class  https://review.opendev.org/65566510:33
*** lajoskatona has joined #openstack-nova10:37
*** tbachman has quit IRC10:43
openstackgerritStephen Finucane proposed openstack/nova-specs master: Standardize CPU resource tracking  https://review.opendev.org/55508110:46
stephenfinsean-k-mooney: You're mostly happy with ^ at this point, I assume?10:46
sean-k-mooneyya mostly10:47
stephenfinI've added more words and fixed the typos to assuage efried's concerns (I hope)10:47
sean-k-mooneyi was going to suggest using a different threath the HW_CPU_HYPERTHREADING10:47
sean-k-mooneybut its already in os traits10:47
sean-k-mooneye.g. HW_CPU_SMT10:47
stephenfinsean-k-mooney: We could just _not_ use that?10:48
sean-k-mooneythen we have another dead trait in plamcent10:48
stephenfinOr even remove it if no-one else is using it yet10:48
sean-k-mooney*os-traits10:48
sean-k-mooneystephenfin: we cant remove traits from os-traits10:48
sean-k-mooneyits an add only model10:49
stephenfinif no one is using it anywhere though?10:49
sean-k-mooneyto simplfy db updates10:49
stephenfinand never has10:49
stephenfinOh, there's a DB involved10:49
sean-k-mooneyeven then10:49
sean-k-mooneyya10:49
aspiersstephenfin: you're repeating an earlier discussion ;-)10:49
aspiersthis is now mentioned in http://specs.openstack.org/openstack/nova-specs/specs/train/approved/amd-sev-libvirt-support.html10:49
aspiersI asked the exact same question a few days ago10:49
sean-k-mooneyyep10:50
stephenfinFair. I'd also be ok with a dead trait, tbh, but I also don't care that much10:50
sean-k-mooneyim fine with hte hyperthreaing one10:50
aspiersI think dead traits can be hidden to the point that they're only visible in the os-traits code where hardly anyone looks10:50
sean-k-mooneypeople use it as a generic term even if it is a threadmark10:50
aspierswell I suppose they're visible via introspection too10:50
stephenfinfwiw, I tried to use SMT in the documentation e.g. https://docs.openstack.org/nova/pike/admin/cpu-topologies.html10:50
sean-k-mooneyyes i know and you gave a talk on the topic10:51
stephenfinindeed :)10:51
sean-k-mooneyso ya i was in the middel of reading v2710:51
*** dpawlik has quit IRC10:52
sean-k-mooneyill skim through v2810:52
stephenfinYeah, spotted an issue with my description of that trait, actually :)10:52
stephenfinI must admit, I really don't like Gerrit for doc reviews :-\10:52
sean-k-mooneywhy10:53
stephenfinIt's too hard to keep track of conversations10:53
sean-k-mooneyyou jut have to wait a bit and collect the changes10:53
aspiersstephenfin: that's because we're using an ancient Gerrit10:53
aspiersGerrit 2.16 is awesome10:53
stephenfinaspiers: Oh yeah? Anything specific?10:53
sean-k-mooneywe are using 2.1310:53
stephenfinThe polymer UI?10:53
aspiersstephenfin: much more than that10:53
aspiershttps://review.gerrithub.io/q/status:open10:54
sean-k-mooneywhats so different about that10:55
aspiershttps://youtu.be/WsPhoPGUsss?t=9m47s10:55
aspiershttps://youtu.be/WsPhoPGUsss?t=19m10s10:55
sean-k-mooneythat is just the polomer ui + a theme10:55
aspiersnope10:55
stephenfinaspiers: Oh, nice10:56
aspiersthis is just one of many specific improvements10:56
aspiersI can't list them all for you10:56
stephenfinResolvable comments are indeed one of the things I'd like10:56
stephenfinI like that in gdocs10:56
aspiersalso bear in mind those videos are 1.5 years old now10:58
stephenfinyeah, so things will have moved on a lot since then10:58
aspiersOpenStack really REALLY needs a Gerrit upgrade10:58
stephenfinI imagine the infra folks are snowed under with the opendev migration but this sure would be a nice next step10:58
sean-k-mooneyaspiers: we used to have some downstream only changes10:58
sean-k-mooneybecause we need to fix bug that had not merged upstream10:59
aspiersstephenfin: yeah. it will happen10:59
stephenfinbig changes in 3.0 too, from the looks of things https://www.gerritcodereview.com/3.0.html10:59
sean-k-mooneyim not sure what is left to get upstream at this point10:59
*** ttsiouts has quit IRC10:59
aspierssean-k-mooney: if a patch to 2.13 hasn't landed by now, it never will10:59
sean-k-mooneyaspiers: yes which may mean we never update10:59
aspierssean-k-mooney: highly doubt that10:59
sean-k-mooneythere were one or two that we need to work at our scalse in the past  the got push back which stop openstack updating11:00
aspiersclarkb / corvus / fungi started working on it a long time ago11:00
sean-k-mooneybut i think most of that has gone upstream now11:00
*** ttsiouts has joined #openstack-nova11:00
aspierssean-k-mooney: yeah, we're not the biggest Gerrit user11:00
aspiersthere are several other huge scale installs11:01
aspiersI met them all at the conference11:01
sean-k-mooneyi think we were 6 years ago  when i started on openstack11:01
aspiersand the others are all head in terms of HA too11:01
aspiersahead11:01
aspiersGoogle is heavily investing in Gerrit11:01
aspiersthey have a team of 20 or so IIRC11:01
aspiersstephenfin: 3.0 is more about ditching the dead stuff which everyone will have migrated off of by 2.1611:02
stephenfinlooks like it, yeah11:02
stephenfinI'd imagine we'll be getting 2.16 first, in that case11:02
aspiersfor sure11:03
*** ttsiouts_ has joined #openstack-nova11:05
*** ttsiouts has quit IRC11:06
openstackgerritAdam Spiers proposed openstack/os-traits master: Update SEV trait docs to avoid misleading people  https://review.opendev.org/65567111:12
openstackgerritAdam Spiers proposed openstack/os-traits master: Document policy of never removing traits  https://review.opendev.org/65567311:19
openstackgerritAdam Spiers proposed openstack/os-traits master: Document policy of never removing traits  https://review.opendev.org/65567311:23
*** dpawlik has joined #openstack-nova11:25
*** zbr has quit IRC11:31
*** udesale has quit IRC11:33
*** zbr has joined #openstack-nova11:34
openstackgerritRodolfo Alonso Hernandez proposed openstack/os-vif stable/stein: Prevent "qbr" Linux Bridge from replying to ARP messages  https://review.opendev.org/65567811:36
openstackgerritKashyap Chamarthy proposed openstack/nova-specs master: Add "CPU selection with hypervisor consideration" spec  https://review.opendev.org/64581411:36
kashyapefried: As promised  ^^11:36
*** boxiang has quit IRC11:37
*** boxiang has joined #openstack-nova11:37
jaypipesefried: morning. sorry, just saw your question from yesterday evening.11:38
jaypipesefried: I'll be leaving around 1:30-2pm on the Friday. I'll just make do with side conversations etc at the summit about the various specs I'm concerned about (and continue to add reviews to said specs)11:39
sean-k-mooneyjaypipes: i think efried has tried to front load most of the topic that are in your usuall area. not sure they all fit11:41
sean-k-mooneyone thing i think will come up in the feeback session is the ptg and summit should over lap again11:41
sean-k-mooneywe will see but the room was pretty split at the last ptg on that topic11:42
jaypipesack11:43
*** jiaopengju_1 has quit IRC11:44
*** jiaopengju_1 has joined #openstack-nova11:44
*** dpawlik has quit IRC11:52
*** zigo has joined #openstack-nova11:53
*** panda is now known as panda|lunch11:59
openstackgerritMerged openstack/nova stable/queens: Fix regression in glance client call  https://review.opendev.org/65518612:05
*** tbachman has joined #openstack-nova12:07
openstackgerritMerged openstack/nova stable/queens: libvirt: disconnect volume when encryption fails  https://review.opendev.org/65303312:10
*** markvoelker has quit IRC12:14
*** markvoelker has joined #openstack-nova12:15
*** dpawlik has joined #openstack-nova12:15
fungiaspiers: stephenfin: sean-k-mooney: yes, current plan i think is going straight from 2.13 to 2.16 we just need to re-test the upgrade process on our staging server. the transition from mysql to notedb is going to be the biggest challenge as it will likely require a day-long service outage to complete12:23
*** derekh has quit IRC12:23
fungiwe didn't want to conflate that work with the opendev switch for gerrit/git, but this ptg will be the time to make plans12:24
sean-k-mooneyis there an infra meetup at the ptg or just people will be about so it will come up12:24
fungiinfra is meeting thursday and friday all day (we share a room with qa)12:25
*** tosky has joined #openstack-nova12:25
sean-k-mooneyah ok12:25
sean-k-mooneyi kind of want to get a bit more involved with the infra side of things but on the other hand was not sure how to go about that so i was just going to say hi12:26
*** vabada has joined #openstack-nova12:28
*** altlogbot_1 has quit IRC12:32
*** ttsiouts has joined #openstack-nova12:37
*** nicolasbock has joined #openstack-nova12:38
*** ttsiouts_ has quit IRC12:38
*** altlogbot_0 has joined #openstack-nova12:39
*** ttsiouts_ has joined #openstack-nova12:40
*** ttsiouts has quit IRC12:42
*** ttsiouts has joined #openstack-nova12:43
*** ttsiouts_ has quit IRC12:43
*** derekh has joined #openstack-nova12:44
*** altlogbot_0 has quit IRC12:47
*** altlogbot_0 has joined #openstack-nova12:49
kashyapfungi: What is "notedb"?  /me looks up12:54
kashyapOkay, it is this: "NoteDb is the next generation of Gerrit storage backend, which replaces the traditional SQL backend ..."12:54
kashyaphttps://gerrit-review.googlesource.com/Documentation/note-db.html12:54
fungikashyap: yep, basically we'll be etl'ing gigabytes of data from an rdbms into git repositories12:55
funginot a speedy process12:55
*** udesale has joined #openstack-nova12:56
fungiat one point the upgrade process was necessary to be performed stepwise first moving some of the data into an h2 database and then moving it from there into git in a subsequent gerrit version, but we hear they've added the ability to migrate directly from sql to notedb now so it may go a lot more quickly, we just need to test12:57
*** eharney has quit IRC13:00
stephenfinfungi: Thanks for the context. That's all very good to hear :)13:05
*** mchlumsky has joined #openstack-nova13:05
*** dikonoor has quit IRC13:06
kashyapfungi: Yeah, noted13:07
fungialso part of the upgrade process involved moving to a new server on a newer ubuntu release so we would have a new enough version of java to work with newer gerrit. we got that done last year, which was a nontrivial undertaking13:07
sean-k-mooneyfungi: out of interst shicne the data will now be sotred in git13:08
sean-k-mooneycould we use the notedb data in tools like gertty13:08
sean-k-mooneyor will that content not be downloadable by others13:08
fungiwell, gertty already has access to the relevant data (modulo access controls allowing your account to access it) through gerrit's rest api13:09
sean-k-mooneygerrty effectivly buildis its own sqlite db replicting the gerrit one if that was not required it could be a nice improvemnt13:09
fungioh, i see what you're asking... i think the data in notedb is aggregated in ways which wouldn't make that useful for per-user data tracking13:10
fungiwhat it does enable though is distribution of state, an there's a plugin recently contributed upstream to gerrit by the gerrithub folks which does multi-master replication between notedbs13:11
*** smcginnis has joined #openstack-nova13:11
fungiso we could finally have a decent cross-provider high availability solution for it13:11
sean-k-mooneyya that would be cool13:11
fungii know the wmf folks have just upgraded to 2.16 and are looking at leveraging that plugin soon13:12
sean-k-mooneywmf?13:12
sean-k-mooneywiki media foundaiton?13:12
sean-k-mooneyactully looking at https://gitenterprise.me/tag/notedb/ it looks like it will allow you to work offline13:15
stephenfinffs https://review.opendev.org/65129713:15
*** tetsuro has joined #openstack-nova13:15
* stephenfin rechecks _again_13:15
stephenfinthere is definitely something buggy with one of our lower-constraint packages to say I keep seeing these StringException errors http://logs.openstack.org/97/651297/4/gate/openstack-tox-lower-constraints/758a74d/testr_results.html.gz13:16
stephenfinBut who knows what13:16
*** tetsuro has quit IRC13:18
*** tetsuro has joined #openstack-nova13:19
efriedstephenfin: That's bug 182543513:19
openstackbug 1825435 in OpenStack Compute (nova) "TestRPC unit tests intermittently fail with "'>' not supported between instances of 'NoneType' and 'datetime.datetime'" - maybe due to "Fatal Python error: Cannot recover from stack overflow."" [High,Confirmed] https://launchpad.net/bugs/182543513:19
sean-k-mooneydid we find where that was intoduced yet13:20
*** lajoskatona has quit IRC13:20
efriedit seems almost exclusively isolated to py36, so it hits lower-constraints (which runs on py36) and the py36 UT.13:20
efriedsean-k-mooney: No. mriedem has been trying to debug it (https://review.opendev.org/#/c/654468/ and its dep) but we can't get it to reproduce with those patches <fume>13:20
sean-k-mooneyapparently it started to show up first in stephenfin cells v1 removal patches but that could also be a coincidnces13:20
efriedsean-k-mooney: If you can update the video models bp description as requested by kashyap, I'll approve it.13:21
sean-k-mooneyam sure ill go do that now.13:21
efriedjaypipes: I was specifically asking if those two topics were ones you wanted to be around for. If so, I can try to shuffle things around.13:21
efriedthanks sean-k-mooney13:21
*** ricolin has quit IRC13:21
kashyapsean-k-mooney: Thank you!13:21
*** lbragstad has joined #openstack-nova13:22
efriedaspiers: If you're around and want to fix that one typo in https://review.opendev.org/#/c/655673/ I'll fast approve.13:22
*** mriedem has joined #openstack-nova13:22
sean-k-mooneyi have it more or less in the relse note https://review.opendev.org/#/c/647733/3/releasenotes/notes/extend-libvirt-video-model-support-d630b99ef5039f51.yaml13:22
efriedIf you're not around, I'll fix it.13:22
*** tetsuro has quit IRC13:22
efriedsean-k-mooney: copy paste, baby13:22
efriedkashyap: You okay with the wording there --^ ?13:23
* kashyap looks13:23
sean-k-mooneyi need to change it slightly but to make sense in a blueprint13:23
sean-k-mooneymainly the tense but ill use it as teh basis13:23
stephenfinefried: I don't think it is, actually13:23
mriedemlyarwood: seems like an easy bug to fix, but you should look https://bugs.launchpad.net/nova/+bug/182588213:23
openstackLaunchpad bug 1825882 in OpenStack Compute (nova) "Virsh disk attach errors silently ignored" [Undecided,New]13:23
stephenfinhttp://logs.openstack.org/97/651297/4/gate/openstack-tox-lower-constraints/758a74d/job-output.txt.gz13:23
*** altlogbot_0 has quit IRC13:23
kashyapefried: I'll add a comment there.  But the 'none' model should be described.  You can copy/paste the phrase I posted earlier13:24
stephenfinIt looks like the test never finishes. If you grep for 'test_get_transport_url' the status is inprogress13:24
* kashyap adds a comment in the change13:24
stephenfinthough I could be misreading that13:24
efriedmriedem: Is that not the same bug? http://logs.openstack.org/97/651297/4/gate/openstack-tox-lower-constraints/758a74d/job-output.txt.gz#_2019-04-25_07_44_46_91929513:25
stephenfinI also missed that you'd already rechecked. Oops13:25
efriedno takesie backsies13:26
kashyapsean-k-mooney: efried I added the phrasing in the change.13:26
mriedemefried: yes http://logs.openstack.org/97/651297/4/gate/openstack-tox-lower-constraints/758a74d/job-output.txt.gz#_2019-04-25_07_42_59_70975613:26
efriedstephenfin: ^13:27
stephenfinjaypipes: If you've time, mind glancing over the cpu-resources spec to make sure I haven't cannibalized it too much? https://review.opendev.org/#/c/55508113:27
*** artom has quit IRC13:27
openstackgerritKashyap Chamarthy proposed openstack/nova-specs master: Add "CPU selection with hypervisor consideration" spec  https://review.opendev.org/64581413:28
openstackgerritEric Fried proposed openstack/os-traits master: Document policy of never removing traits  https://review.opendev.org/65567313:28
efriedaspiers: FTFY and fast approved ^13:29
stephenfinefried, mriedem: I saw that but I didn't see the link to the failing tests. I'd have expected to have seen the name of the failing test reported before stacktrace13:29
kashyapefried: Meanwhile, in your copious free time ^  As best as I can addressed remarks from mriedem, sean-k-mooney et al.  If you want to fast-approve.13:29
efriedstephenfin: It varies which test fails13:29
kashyap(Also I will add / update the spec as I learn other important details, docs while implenenting.)13:30
openstackgerritSurya Seetharaman proposed openstack/nova master: Microversion 2.73: Support adding the reason behind a server lock  https://review.opendev.org/64866213:30
efriedjeebuz, with that big a delta, I doubt I'll be able to fast approve13:30
efriedbut thanks for updating.13:30
kashyapefried: Nothing fundamental has changed.  I largely added clarifying text13:30
kashyapefried: Or removed unnecessary bit.  The main work mentioned in the Work Items is still in tact13:31
openstackgerritBalazs Gibizer proposed openstack/nova-specs master: Server move operations with ports having resource request  https://review.opendev.org/65260813:31
lyarwoodmriedem: ack I'll take a look now13:32
mriedemefried: want to continue the backport train? https://review.opendev.org/#/c/655429/13:33
jaypipesstephenfin: ack. will do that right now.13:33
*** altlogbot_0 has joined #openstack-nova13:33
efriedkashyap: Okay, you've said nova will validate, but I'm still not seeing the part where you say: "...and will cause {spawn to fail | nova-compute to refuse to start}, with a helpful log message indicating which combination of thingies was bad"13:33
jaypipesefried: no, not a huge prio for me.13:33
stephenfinta13:33
efriedjaypipes: ack, thx13:33
efriedmriedem: Yes, need to do it manually, will get to it asap.13:33
kashyapefried: Okay, I used the word "validate" too loseely then.  Let me add that phrase in.13:33
efriedright, I (as the reader) don't know what that means.13:34
kashyap(The word "validate" was supposed to capture that.  And I know I will uncover more details as I mull on the implementation.)13:34
openstackgerritRodolfo Alonso Hernandez proposed openstack/os-vif stable/rocky: Prevent "qbr" Linux Bridge from replying to ARP messages  https://review.opendev.org/65569213:35
gibiefried: I saw the discussion about the lenght of the bandwith PTG slot13:35
gibiefried: I think we can shrink that to 15-20 minutes13:35
efriedgibi: ack, I saw your response, will shrink.13:35
efriedthanks.13:35
gibiefried: thanks13:36
*** tetsuro has joined #openstack-nova13:36
gibiefried: I also saw that you prepared the commit for the onboarding session. Is there anything I can still help with?13:36
openstackgerritEric Fried proposed openstack/nova stable/rocky: Fix {min|max}_version in ironic Adapter setup  https://review.opendev.org/65569313:38
openstackgerritRodolfo Alonso Hernandez proposed openstack/os-vif stable/queens: Prevent "qbr" Linux Bridge from replying to ARP messages  https://review.opendev.org/65569413:39
efriedmriedem, melwitt: ^13:39
efriedgibi: I'm not completely sure how I'm going to present that. What I would like to do is have my actual IDE running on the projector. Fallback is showing everything in browser.13:40
efriedI also don't know if I can fill 40 minutes with that demo :P13:40
efriedif you have suggestions, I'm all ears.13:40
efriedIn any case, I would love to have you around for moral support.13:40
*** eharney has joined #openstack-nova13:41
gibiefried: If we run out of things and there are no questions then we can pull out the berlin presentation and talk about some part of it13:41
gibiefried: I will definitly in the room to help13:41
efriedcool, thank you.13:41
gibiefried: especially the last slide https://docs.google.com/presentation/d/1T6_CZvDf2eFgbtTecIfbrMBkPiX9-yYRXXKakZjlyXs/edit?usp=sharing13:42
*** altlogbot_0 has quit IRC13:42
sean-k-mooneyefried: kashyap i think https://blueprints.launchpad.net/nova/+spec/libvirt-video-device-models should now be fine?13:42
kashyapsean-k-mooney: I'd use capitalization correctly, but that's a nit :D.  Also there is an unwanted line break for the word "graphic"13:44
openstackgerritRodolfo Alonso Hernandez proposed openstack/os-vif master: Remove IP proxy methods  https://review.opendev.org/65569513:44
*** jiaopengju_2 has joined #openstack-nova13:44
*** jiaopengju_2 has quit IRC13:44
sean-k-mooneyupdated13:45
*** jiaopengju_2 has joined #openstack-nova13:45
sean-k-mooneythe content is there and anyone can edit so ill leave it to other to fix futher if it bothers them13:45
*** brinzhang has quit IRC13:46
efriedsean-k-mooney, kashyap: Approved, thanks for the follow through.13:46
*** tetsuro has quit IRC13:46
*** jiaopengju_1 has quit IRC13:46
kashyapefried: Nod13:47
openstackgerritKashyap Chamarthy proposed openstack/nova-specs master: Add "CPU selection with hypervisor consideration" spec  https://review.opendev.org/64581413:47
kashyapefried: ^ Added the sentence on error logging.  Does it look any better?13:47
openstackgerritMerged openstack/os-traits master: Document policy of never removing traits  https://review.opendev.org/65567313:48
efriedkashyap: Yes, thank you, except for the typo serice13:48
sean-k-mooneyby the way i treied to have https://review.opendev.org/#/c/647733/ more or less be mergable in the first revision but i did wonder is there a place in our docs where the values of hw_video_model would/should be documented13:48
kashyapgibi: You were also +2 on the above.  If you want to re-{ACK,NACK} in your copious free time13:48
kashyapDamn13:48
sean-k-mooneystephenfin: ^ any idea13:48
kashyapefried: I corrected it, didn't commit it :-(13:48
gibikashyap: ack, I will try to go back there today13:48
mriedemadrianc: can you or hamdy (or moshe) start a backport for https://review.opendev.org/#/c/649345/ to stable/stein so we have it lined up?13:49
*** yedongcan has left #openstack-nova13:49
openstackgerritKashyap Chamarthy proposed openstack/nova-specs master: Add "CPU selection with hypervisor consideration" spec  https://review.opendev.org/64581413:49
mriedemadrianc: oh nvm wrong patch13:49
kashyapgibi: Thanks.  It's largely clarification text added based on the feedback13:49
mriedemadrianc: oh i guess that was right, i was looking at the neutron spec for some reason13:50
efriedmriedem: Do you want a final look at ^ ?  It had multiple +2s and I just asked kashyap to add detail on what "validate" means (it means refuse to start nova-compute, with a helpful log) so I was going to fast approve it.13:50
openstackgerritLee Yarwood proposed openstack/nova master: libvirt: Stop ignoring unknown libvirtError exceptions during volume attach  https://review.opendev.org/65569613:50
mriedemif you and gibi and alex_xu and everyone else are fine with it then no i don't need to spend time re-reviewing13:50
efriedack, thx13:51
mriedemlyarwood: thanks13:51
kashyapmriedem: Your comments were useful.  I spent 3+ hours thinking and writing a response to it this morning.  Thanks for taking time!13:52
stephenfinsean-k-mooney: Hmm, not sure. We have https://docs.openstack.org/nova/latest/admin/flavors.html but no image equivalent13:52
*** dikonoor has joined #openstack-nova13:52
mriedemstephenfin: it's in the glance docs13:53
mriedemoh, that's not the extra spec one13:53
aspiersefried: nova onboarding? maybe I can help with that13:53
sean-k-mooneystephenfin: ya i quickly looked before and didnt fined one so didnt add any13:53
aspiersefried: as a newbie to nova but not newbie to git/gerrit etc.13:54
efriedaspiers: Love it. What did you have in mind?13:54
sean-k-mooneyi am updating the glance metadef however to track the new values13:54
aspiersefried: nothing, I only just saw your reference to onboarding in the scrollback ;)13:54
efriedaspiers: Good perspective: what were the most important things that helped you onboard into nova, being already familiar with openstack and its toolage?13:54
sean-k-mooneystephenfin: we also have ovos for the image properties to that also track the allowed values.13:54
mriedemstephenfin: sean-k-mooney: if you're looking for image metadata docs they are here https://docs.openstack.org/glance/latest/admin/useful-image-properties.html13:54
sean-k-mooneyah they are in glance13:55
mriedemmanaging images via the command line would be something i'd expect to find in the glance admin docs13:55
efriedaspiers: i.e. what would you tell someone to do if they were in a similar position and looking to get onboarded13:55
*** Luzi has quit IRC13:55
aspiersefried: hrm, I guess "read the docs and ask a lot of questions on IRC"13:55
sean-k-mooneyill update them when i update the image metadefs13:55
mriedemhttps://docs.openstack.org/glance/latest/user/glanceclient.html13:55
efriedaspiers: If you want to give a brief off-the-cuff/anecdotal talk in the session, that would be most welcome.13:55
aspiersefried: that might be possible. when is it?13:55
efried...13:55
sean-k-mooney actuly i think these might be auto generated form teh metadefs13:55
mriedemstephenfin: https://docs.openstack.org/glance/latest/admin/manage-images.html13:56
*** tetsuro has joined #openstack-nova13:56
mriedemsean-k-mooney: they are not13:56
mriedemthe useful-image-properties page is hand-written13:56
efriedaspiers: https://www.openstack.org/summit/denver-2019/summit-schedule/events/23620/nova-project-onboarding13:56
sean-k-mooneyoh ok ill update them so. thanks for the pointer13:56
kashyapedleafe: On that traits patch, please see if I've answered your question13:56
mriedemlyarwood: if you fix the alignment here https://review.opendev.org/#/c/655696/1/nova/virt/libvirt/driver.py i'm +213:56
kashyapedleafe: Ah, you've actually responded; I just needed to refresh it13:57
*** altlogbot_1 has joined #openstack-nova13:57
edleafekashyap: damn, I'm fast!13:57
*** tetsuro has quit IRC13:57
kashyapDamn, what I thought was a drive-by is becoming a shaving a farm of yaks :D13:58
efriedkashyap: I was going to respond on that one too, with a summary. TLDR, what I said in my last response, do that split and "deprecate" the HW_CPU_AMD_SEV in favor of HW_CPU_X86_AMD_SEV13:58
kashyapefried: By "deprecate", you mean just add a note that it's bogus, right?13:59
efriedyes13:59
sean-k-mooneyefried: just re reviews kashyap cpu selection spec and im fine with the new verions. thanks kashyap13:59
kashyapsean-k-mooney: Thanks!13:59
edleafeYeah, since the trait won't ever be removed. More of a "we screwed up; don't use this one" note14:00
kashyapsean-k-mooney: We'll get to debate fun details in the implementation :D  I have a WIP post patch to introduce the infra, before it gets lost14:00
*** hongbin has joined #openstack-nova14:00
efriedkashyap: re edleafe's comment on the NO_SSB thing, I thought that wasn't so much a negative trait as advertising that "this CPU has had the horrible thing removed".14:00
*** tetsuro has joined #openstack-nova14:00
efriediow, absent that trait, you can't tell whether it's there or not14:00
efriedso we want that trait so the consumer can be sure the horrible thing is gone.14:01
kashyapExactly14:01
edleafeefried: if that's the case, then rename it to a positive assertion of a capability14:01
kashyapThat's why I wanted to mention it.  Let me read Ed's comment14:01
efriededleafe: like SSB_REMOVED?14:01
edleafesure, something along those lines14:01
efriededleafe: Would prefer to retain symmetry with the CPU traits themselves, though.14:01
*** panda|lunch is now known as panda14:02
kashyap"NO_SSB" == No more SSB vulnerability on this host CPU -- isn't that clear enough?14:02
sean-k-mooneykashyap: i would like to see two thing. 1 on sucessful validation print the cpu configurtion  the guest will see as an info long. on failure pritn the requested combination, whant the host supports and what could not be supproted as an error log.14:02
efriedwith the names from the architecture, that is14:02
*** tetsuro has quit IRC14:02
efriedkashyap: IMO it is, and is preferable to renaming it to make it sound more positive14:02
efriedjust need to get edleafe on board with that14:02
kashyapOkay, I'll go re-read the comments.  These flags have some dizzying acronyms14:02
sean-k-mooneykashyap: im not sure we should be tracking vulnerblities as traits14:03
kashyapsean-k-mooney: Huh, we're not14:03
edleafeefried: not having something isn't a capabilty. Being clean of some defect is.14:03
stephenfinsean-k-mooney: What happens if this isn't true? https://review.opendev.org/#/c/629589/27/nova/virt/libvirt/driver.py@816114:03
efriededleafe: These are going to be consumed by people who are familiar with the arch and are looking for the equivalent trait, not people who are just casting around and trying to understand a trait based on its name.14:03
sean-k-mooneyNO_SSB kind of is14:03
kashyapsean-k-mooney: It's a CPU flag.  Please ask before you assume :-)14:03
sean-k-mooneyi know what it is14:04
sean-k-mooneyi was not assuming14:04
kashyapWell, it's a CPU flag.  We need to track it as a "trait", if you want to report that a given Compute host is no more vulnerable to it14:04
kashyapIs something wrong with t?14:04
kashyaps/t/it/14:04
edleafekashyap: is it generally referred to as NO_SSB in the relevant documentation (asking because I have no idea)14:04
sean-k-mooneyif we have NO_SSB i can do traits:no_SSB=forbiden and then deploy a workload that expliot it to a host that has the vulnerablity14:04
kashyapedleafe: Exactly what efried said.  These are for admins who read their documentation (and they will be sufficiently rewarded)14:05
efriededleafe: https://git.qemu.org/?p=qemu.git;a=blob;f=docs/qemu-cpu-models.texi#l31314:05
mriedemtssurya: fyi in case you haven't seen this https://review.opendev.org/#/c/636947/14:06
*** dpawlik has quit IRC14:06
efriededleafe: This set of traits is being mapped directly from that document14:06
mriedembauzas: ^ is a nice little perf improvement when listing AZs in a large cloud14:06
edleafeefried: ok, I understand a bit more.14:06
bauzasmriedem: okay14:06
efriededleafe: sections important_cpu_features_intel_x86 and important_cpu_features_amd_x8614:06
tssuryamriedem: yeah I have it open14:06
bauzasmriedem: FWIW, I'll be traveling starting tonight14:06
tssuryathanks for doing that14:07
edleafeThat doc also defines things such as mutually exclusive traits, which os-traits and placement doesn't enforce14:07
sean-k-mooneyefried: edleafe i still think we need to consider the securtiy impact of having NO_SSB as a trait before we add it14:07
efriededleafe: true story, but not really relevant; the driver is going to detect what's on the CPU and report that.14:07
mriedemtssurya: thank avolkov14:07
kashyapedleafe: Only 'amd-no-ssb14:07
kashyapedleafe: Only 'amd-no-ssb' is the mutually exclusive trait.14:08
kashyapAgain, what efried said ^ (on driver detecting)14:08
edleafesean-k-mooney: what would the potential impact be? A CPU reporting it that is actually vulnerable to it?14:08
sean-k-mooneyyes14:08
tssuryaI tweaked a downstream patch when we opened that bug, never got around to doing it upstream14:08
sean-k-mooneyand then a user uploading an image that require a cpu with the vulnerablity14:09
efriededleafe: sean-k-mooney is saying a malicious user could specify NO_SSB as a forbidden trait to deliberately land on a vulnerable host.14:09
kashyapsean-k-mooney: Also what is "forbidden" in this context?14:09
efriedkashyap: We have the ability to specify a request with "DON'T land on a host if it has this trait"14:09
sean-k-mooneykashyap: we support requireing traitns or selecting host that do not have a trait14:09
kashyapefried: Ah-ha14:09
efriedkashyap: The placement request would look like required=!HW_CPU_X86_AMD_NO_SSB14:09
sean-k-mooneyso if we report either the presence or absence of ssbd14:09
efriedkashyap: From a flavor extra spec like trait:HW_CPU_X86_AMD_NO_SSB=forbidden IIRC14:10
sean-k-mooneyi can exploit that to target a  host with the volnerablity with my payload14:10
efriedYeah, I don't know if this is a problem though sean-k-mooney14:10
kashyapYeah, noted.14:10
edleafesean-k-mooney: so without that trait, attackers could just spin up random VMs and see if they are on a vulnerable host. Having the trait just makes those hosts easier to find. Is that it?14:10
kashyapefried: Yeah, that's what I was wondering.  The positives of having it outweigh some theoretical threat14:10
efriedThe admin is responsible for putting together the flavors. Why would the admin do that?14:10
sean-k-mooneyedleafe: yes14:10
sean-k-mooneyi would like the openstack security team to weigh in on this personally14:11
efriedthere's an email address for that, right?14:11
mriedemefried: gibi: trying to recreate this TestRPC infinite recursion is getting frustrating :) https://review.opendev.org/#/c/654468/ - i'm thinking maybe push another change to oslo.config (or nova?) that sets the max recursion depth and see if that blows things up?14:11
sean-k-mooneyefried: users can upload image with traits request14:11
sean-k-mooneyso its not admin im worried about14:11
kashyapefried: Yes, very good point on admin's responsibility14:12
efriedhttp://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-security14:12
gibimriedem: I also tried to reproduce it locally without success. Do you assume that if you set the recursion limit to small enough then the stack trace will not be truncated?14:12
efriedugh, do you have to be subscribed to post to that?14:12
*** phasespace has quit IRC14:12
kashyapefried: Not necessarily14:13
sean-k-mooneyefried: there is but we can also talk to them in person next week14:13
kashyapefried: Mailman moderator can approve you14:13
sean-k-mooneywe will all be in the same place14:13
kashyapsean-k-mooney: In-person, things get too long-winded, and they get lost.14:13
efriedsean-k-mooney: Okay, so maybe we need to have an explicit check for such traits and disallow them from image meta?14:13
gibimriedem: anyhow sounds like a good idea14:13
mriedemgibi: that's the idea14:13
*** dikonoor has quit IRC14:13
sean-k-mooneyefried: maybe14:13
mriedemalternatively, and this is probably easier, just change my oslo.config change from logging a traceback whe it hits a recursion of 3 to just raise14:13
mriedemi'll do that first14:13
efriedsean-k-mooney, kashyap: So is one of you going to compose an email? Please copy me. (I'm not subscribed to that list.)14:14
*** mmethot has quit IRC14:14
stephenfinralonsoh: It's a nit, but any chance you can fix this before we merge and backport? https://review.opendev.org/#/c/655332/4/vif_plug_linux_bridge/linux_net.py14:14
*** mlavalle has joined #openstack-nova14:14
kashyapefried: I'm not subscribed either.  But will do, if I write.14:14
efriedthanks.14:14
sean-k-mooneyok as i said im sure we can grab 10 mins with them next week at the ptg/fourm too14:15
kashyapAlso, "exploiting" here is all hand-wavy, and requires some serious competency, and timing to take advantage of such things.14:15
sean-k-mooneyefried: and yes we could have a blacklist of traits that cant be in the image14:15
sean-k-mooneykashyap: sure but not allo fo them do14:16
efriedsean-k-mooney: Security people will tell you it should be a whitelist :(14:16
*** tetsuro has joined #openstack-nova14:16
kashyapsean-k-mooney: The thing is, I'll forget it in 5 seconds.  My memory is worse than a bumblebee's (less than 8 seconds)14:16
sean-k-mooneywell the sepc said that too if i recall14:16
efriedor a goldfish14:16
efrieddid we have this conversation?14:16
sean-k-mooneybut we never implemented it14:16
kashyapefried: Heh, I guess so14:16
efriedand you forgot14:16
efriedtypical14:16
efried(My memory is pretty unpredictable. Some things I remember forever, some I need hammered in a thousand times and still can't hold onto.)14:17
kashyapefried: Yeah.  On the other hand, we (I) should work towards getting the rest of the "uncontroversial" triats in, as that helps admins _today_14:17
aspiersefried, kashyap: what's the current thinking on how to handle the SEV trait? keep as HW_CPU_AMD_SEV, or move to HW_CPU_X86_AMD_SEV?14:18
kashyapefried: Yeah, I hear ya...something similar here.14:18
efriedkashyap, sean-k-mooney: I'm going to say it's fine to add the traits to os-traits regardless; the vulnerability occurs when we expose them via compute.14:18
ralonsohstephenfin, well it makes sense. Because both parameters should be different, the current implementation is valid. But yes, I'll change it14:18
aspiersefried, kashyap: I have ongoing reviews which kinda rely on a decision on this ...14:18
sean-k-mooneyefried: im wondering if we should namespace them14:18
efriedaspiers: "deprecate" HW_CPU_AMD_SEV, "add" HW_CPU_X86_AMD_SEV14:18
sean-k-mooneyor otherwise mark them up in os-traits14:19
aspiersefried: do we need a traits deprecation mechanism? other than just documenting?14:19
efriedsean-k-mooney: Namespace them as HW_CPU_X86_POTENTIAL_SECURITY_VULNERABILITY_NO_SSB??14:19
*** tetsuro_ has joined #openstack-nova14:19
efriedaspiers: It's been discussed before, but we decided not to do it. So just documenting/commenting.14:19
aspiersefried: decided not to do it ever, or just not any time soon?14:19
kashyapaspiers: I saw your reorg comment, still have to look at it.  (I'm still a bit unwell, so, slow today as well)14:19
aspierskashyap: sorry to hear that :-( no pressure14:20
efriedwhen we first split os-traits out of placement (in nova) there was a proposal (from jaypipes iirc) to do it with a more complex architecture that would allow for deprecation and removal.14:20
efriedbut we decided on the super-simple approach instead.14:20
*** JamesBenson has joined #openstack-nova14:20
sean-k-mooneyefried: not quite but maybe have _NO_IMG_ or something14:20
efriedaspiers: No plans in the forseeable future.14:20
aspiersefried: just wondering if https://review.opendev.org/#/c/655673/ should have explained that too14:20
sean-k-mooneythat or add them to a seperate dict in os-traits14:21
efriedsean-k-mooney: Yeah, no. We don't want to try to predict what those would be, and that's not the responsibility of os-traits IMHO.14:21
sean-k-mooneyor list14:21
kashyapaspiers: Ah, nice.  It's actually merged; efried++14:21
*** tetsuro has quit IRC14:21
efriedaspiers: Sure, could do, looks like mriedem felt the same. It's docs, easy to add more.14:21
sean-k-mooneywell we should ask ourselve is this a security issue when adding new traits in general. anway i need to go finsih packing14:22
openstackgerritRodolfo Alonso Hernandez proposed openstack/os-vif master: Prevent "qbr" Linux Bridge from replying to ARP messages  https://review.opendev.org/65533214:22
kashyapefried: Thanks; good point -- so nothing is blocking from adding the trait.  Good14:23
jaypipesefried: you mean os-resource-classes, not os-traits, but yeah.14:23
efriedsean-k-mooney: If we do the whitelist thing, then it forces us to think about it for every trait. But in the context of nova, not os-traits.14:23
aspiersefried: Well, if adding a deprecation mechanism to the code isn't completely ruled out, then probably no change is needed14:23
efriedjaypipes: oh, was it os-resource-classes, okay.14:23
jaypipesefried: yeah. my proposal was here: https://github.com/jaypipes/os-resource-classes14:23
efriedaspiers: FYI ^14:23
sean-k-mooneyefried: we could do it in nova i guess instead14:23
aspiersefried: In general I don't think it makes sense to document "we have no plan to do $X any time soon", since $X could be any number of things :)14:23
kashyapefried: (The "good point" being your "vuln. occurs when we expose them [traits] via compute")14:23
jaypipesefried: with a version history: https://github.com/jaypipes/os-resource-classes/blob/master/os_resource_classes/__init__.py#L2114:23
sean-k-mooneyjust have a list of traits in nova that lists trats that cant be in an image or put it in a config14:24
jaypipeswe ended up not going with that and instead simply listing constants in a library.14:24
efriedaspiers: Yes, that's true, but I think we're talking more about like in the SEV case: "We misnamed this one when we introduced it - please use $Y instead."14:24
sean-k-mooneythen as we add usage in nova we can extend the defualt of the config option14:24
aspiersefried: no I was talking specifically about https://docs.openstack.org/os-traits/latest/contributor/index.html14:24
aspiersefried: but I was also about to ask whether I should submit a follow up to https://review.opendev.org/#/c/655671/ to "rename" the SEV trait (deprecating the original) ?14:25
aspiersefried: or whether I should instead amend that review14:25
*** tetsuro_ has quit IRC14:25
aspiersI guess amending makes more sense14:25
*** Sundar has joined #openstack-nova14:25
efriedaspiers: I think kashyap is going to do that as part of his patch, because he's going to be splitting x86 into amd and intel14:25
efriedIt could be done as separate patches, but meh.14:25
*** gmann has joined #openstack-nova14:26
aspierskashyap: when do you expect your reorg to land?14:26
kashyapaspiers: As soon as I  address the feedback :-)  We can work it out14:27
openstackgerritSurya Seetharaman proposed openstack/python-novaclient master: Microversion 2.73: Support adding the reason behind a server lock  https://review.opendev.org/64865914:27
efriedkashyap: CPU with hypervisors spec is merging and bp is approved, striking from PTG agenda, cool?14:28
mriedembtw, with trait names like HW_CPU_X86_POTENTIAL_SECURITY_VULNERABILITY_NO_SSB i look forward to the day when we blast the url limit for GET query params14:28
openstackgerritMatt Riedemann proposed openstack/nova master: DNM: debug config opts infinite recursion  https://review.opendev.org/65446814:29
efriedI was being facetious with that.14:29
sean-k-mooneymriedem: im sure we already to if we try to use everything we currently support14:29
efriedmriedem: By then we'll surely have json payload for requests.14:29
jaypipesmriedem: coupled with NET_BW_EGR_KILOBIT_PER_SEC, that would be awesome.14:29
aspierskashyap: just trying to work out the order of work, since a lot of these changes depend on each other14:29
jaypipesused to be even longer than that :)14:29
*** tssurya has quit IRC14:29
sean-k-mooneyefried: if we do have a json payload we will also need to use a post not a get14:29
efriedyou keep saying that :P14:30
openstackgerritMatt Riedemann proposed openstack/nova master: Don't run tempest/devstack jobs on nova/test.py only changes  https://review.opendev.org/65512114:30
openstackgerritMatt Riedemann proposed openstack/nova master: DNM: debug config opts infinite recursion  https://review.opendev.org/65446814:30
sean-k-mooneyi do14:30
efriedI'm sure you must be right. I don't really care what http verb it is.14:30
aspiersdo we have any mechanism for plotting dependency graphs between commits in Gerrit?14:30
*** artom has joined #openstack-nova14:30
sean-k-mooneybecause i keep seeing peole propose apis with get bodies14:30
efriedaspiers: You mean visually?14:30
mriedemefried: gibi: ^ so that series now does 2 things: 1. the nova change sets a low recursion limit (probably won't help) and 2. the oslo.config change doesn't print, it just counts recursion and raises if we call _get() more than 5 times on the same option14:31
efriedmriedem: ack14:31
kashyapaspiers: Will address that as quickly as I can.14:31
gibimriedem: ack, fingers crossed14:31
efriedaspiers: And do you mean dependencies based on line conflicts, or based on commits in series and Depends-On?14:31
openstackgerritMerged openstack/nova-specs master: Add "CPU selection with hypervisor consideration" spec  https://review.opendev.org/64581414:32
efriedThe former is displayed in gerrit, top right corner, "Conflicts With"14:32
efried^ kashyap \o/14:32
aspiersefried: commits in series and Depends-On14:32
kashyapefried: In your "copious free time", want to respond to Ed here on "NO_{trait_name}", since your "Traits muscle" is better trained? -- https://review.opendev.org/#/c/655193/2/os_traits/hw/cpu/amd.py@1914:32
aspiersefried: well, IMNSHO https://github.com/aspiers/git-deps does a nicer job of the former ;-)14:33
efriedaspiers: I know of no graphical tool, but that definitely doesn't mean there isn't one. Might ask the infra guys14:33
kashyapefried: Thank you!  IIRC, it is "encouraged" to update the spec file as "new information arises" (I know jaypipes likes to use Labowski's phrase here :D)?14:33
aspiersefried: yeah I was thinking of something like the way launchpad shows a dep graph for blueprints14:34
efriedkashyap: Yes, definitely. Common practice to propose spec amendments e.g. once the impl lands to make it reflect reality.14:34
kashyapAlright, I've seen a few amendments in the specs repo.14:34
aspiersefried: speaking of which, I noticed one of the SEV work items might need changing but I'm not sure14:34
mriedemgibi: i see you have https://blueprints.launchpad.net/nova/+spec/support-server-move-operations-with-ports-having-resource-request which your move spec is tracked against, and you also have https://blueprints.launchpad.net/nova/+spec/enhance-support-for-ports-having-resource-request14:34
efriedkashyap: I can do that (respond to edleafe in the review) or you could just point to the eavesdrop of the conversation we had above.14:34
mriedemgibi: does the former supersede the latter?14:35
kashyapefried: Okay, I'll do the latter; it'll save you time.14:35
efriedkashyap: thanks14:35
mriedemgibi: or is the latter the reason for ptg discussion?14:35
gibimriedem: https://blueprints.launchpad.net/nova/+spec/enhance-support-for-ports-having-resource-request is a collection of further enhancements for bandwidth14:35
aspiersefried: work item #7 http://specs.openstack.org/openstack/nova-specs/specs/train/approved/amd-sev-libvirt-support.html14:35
efriedmriedem: As I understand it, enhance-support is an overarching longer-term bp that excludes support-server-move14:35
mriedemok14:35
gibimriedem: on the ptg I would like to get decision about 1) the microversion for move 2) the priority of the items in https://blueprints.launchpad.net/nova/+spec/enhance-support-for-ports-having-resource-request14:36
aspiersefried: should it be checking the trait or the resource class? I guess only the trait is SEV-specific and so is the required XML, so maybe this is still correct14:36
mriedembtw, we have a working multi-cell ci job https://review.opendev.org/#/c/655222/14:36
efriedaspiers: I would check the allocation. Remember, the request won't necessarily include the trait.14:36
efriedaspiers: So yeah, that should be amended at some point.14:37
dansmithis this a known thing? http://logs.openstack.org/37/655137/2/check/openstack-tox-lower-constraints/6363387/testr_results.html.gz14:37
efriedYou may want to open a change set with that, but leave it -W to collect such deltas as they arise.14:37
dansmithtestrpc fail14:37
kashyapaspiers: BTW, to quickly answer on the SEV trait -- I'd suggest to keep it as-is: HW_CPU_AMD_SEV (in amd.py)14:37
dansmithon a lower constraints job no less :)14:37
aspiersefried: I was thinking the request filter could add not only resources:MEM_ENCRYPTION_CONTEXT=1 but also traits:HW_CPU_X86_AMD_SEV=required14:37
efrieddansmith: yes, that's bug...14:37
kashyapaspiers: Because we'll be consolidating all AMD-related stuff in one place, as it should be.14:38
efrieddansmith: bug 182543514:38
openstackbug 1825435 in OpenStack Compute (nova) "TestRPC unit tests intermittently fail with "'>' not supported between instances of 'NoneType' and 'datetime.datetime'" - maybe due to "Fatal Python error: Cannot recover from stack overflow."" [High,Confirmed] https://launchpad.net/bugs/182543514:38
dansmithack14:38
aspierskashyap: so not under HW_CPU_X86_AMD?14:38
aspiersisn't that the opposite of what efried wrote 20 mins ago?14:38
kashyapaspiers: Wait, disregard me.  I'll re-read the chat and slow-respond later.14:39
aspierskashyap: OK :)14:39
efriedaspiers: We talked about this. The request filter shouldn't do that. The admin (or whoever authors the flavor/image-meta) should request SEV via the trait if they want SEV specifically. Otherwise you could theoretically land anywhere that supports mem enc.14:39
aspiersefried: but at some point in the workflow the code needs figure out that SEV is the technology which will be used to implement it. Where should that happen?14:40
aspiersI guess your point is that doing it in a request filter is too early14:40
sean-k-mooneyaspiers: in the virt dirver14:40
efriedaspiers: Right; at spawn time, libvirt will use the same mechanism it used to decide to expose that trait in the first place.14:40
efriedaspiers: ...or possibly just look for the trait itself as already exposed, not sure if you have access to the provider tree in spawn14:41
sean-k-mooneyplacment will land the vm on a host that support mem encryption and then the virt driver can figure out which tech implementes it14:41
efried^14:41
aspiersyeah makes sense14:41
mriedemdansmith: yes14:42
openstackgerritMatt Riedemann proposed openstack/nova master: Add nova-multi-cell job  https://review.opendev.org/65522214:42
openstackgerritMatt Riedemann proposed openstack/nova master: Enable n-novnc in nova-multi-cell job  https://review.opendev.org/65571114:42
openstackgerritLee Yarwood proposed openstack/nova master: libvirt: Stop ignoring unknown libvirtError exceptions during volume attach  https://review.opendev.org/65569614:42
openstackgerritLee Yarwood proposed openstack/nova master: libvirt: Always disconnect volumes after libvirtError exceptions  https://review.opendev.org/65571214:42
mriedemhttp://status.openstack.org/elastic-recheck/#182543514:42
aspiersefried, sean-k-mooney: so other than updating the commit message and maybe the trait name, I think https://review.opendev.org/#/c/633855/ can remain the same14:42
mriedemhttp://lists.openstack.org/pipermail/openstack-discuss/2019-April/005392.html14:43
dansmithmriedem: I didn't get an e-r report on it14:43
mriedemdansmith: e-r bot has been off for weeks as far as i know14:43
openstackgerritStephen Finucane proposed openstack/os-vif stable/stein: Prevent "qbr" Linux Bridge from replying to ARP messages  https://review.opendev.org/65567814:43
aspiersefried, sean-k-mooney: and I'll add a separate review which adds the new config option and inventory of the new resource class from it14:43
mriedemdansmith: i have a couple of debug patches up to try and recreate to figure out what's causing the recursion stack dump14:43
dansmithmriedem: consider me shamed14:43
*** mvkr has quit IRC14:43
mriedemdansmith: https://review.opendev.org/#/q/topic:bug/1825435+status:open14:43
mriedemif you have ideas on that it'd be cool because so far my recreate attempts are failing14:44
mriedemit seems to have started somewhere along the lines of the cells v1 removal series14:44
mriedembut looking through those changes i don't see anything obvious14:44
mriedemit's in both py36 (upper-constraints) and lower-constraints so it's not some specific oslo.config version either14:44
efriedaspiers: I haven't reviewed that patch in forever, sorry.14:45
mriedemi also didn't see any python3.6 changes recently in the bionic UCA changelog14:45
dansmithmriedem: ack, I heard about the recursion thing, but this is a single output message (not a very long stack trace) so I didn't connect the two14:45
aspiersefried: it's drastically simpler now14:45
efriedaspiers: I probably won't get a chance to review patches-that-aren't-specs until after the PTG dust settles.14:45
mriedemdansmith: yeah the TestRPC ... inprogress seems to be a result of the stack dump earlier in the logs14:46
aspiersefried: because kashyap persuaded me to split out most of it into https://review.opendev.org/#/c/655268/14:46
mriedemthe former doesn't always show up when this hits, but the latter does14:46
efriedaspiers: sight unseen, split is good.14:46
aspiersefried: ok14:46
kashyapYeah, that split is useful -- because that split-up patch, which introduces getDomainCapabilities API, can be used by many other things14:47
kashyapSo it can be merged as a separate piece once we iron out any remaining wrinkles.14:47
kashyapThanks for doing the split, aspiers.14:48
* kashyap --> "low tea"14:48
openstackgerritMatt Riedemann proposed openstack/nova master: Enable n-novnc in nova-multi-cell job  https://review.opendev.org/65571114:50
*** jiaopengju_2 has quit IRC14:50
*** jiaopengju_2 has joined #openstack-nova14:50
mriedemdansmith: btw https://review.opendev.org/#/c/655222/ is passing which is pretty exciting14:50
dansmithmriedem: that's cool14:51
dansmithmriedem: is that intended to be voting?14:52
mriedemonce that lands, i can throw a patch at the end of my cross-cell-resize series to enable resize and cold migrate across the cells14:52
mriedemdansmith: yes14:52
dansmithare you ready for it to go in?14:52
mriedemwe could recheck a few times if we care and then turn it on next week14:53
mriedemi can also send a heads up to the ML14:53
openstackgerritgaryk proposed openstack/nova master: VMware: populate datastore refs at init  https://review.opendev.org/57468814:53
mriedemif we want, we can land it as non-voting check only for now14:53
mriedembut i'd really like it to be voting asap since we should have had this several releases ago14:53
mriedemi'll post to the ML first14:54
openstackgerritgaryk proposed openstack/nova master: Improve metadata performance  https://review.opendev.org/61543514:55
*** ttsiouts has quit IRC14:56
*** ttsiouts has joined #openstack-nova14:57
*** mmethot has joined #openstack-nova14:57
efriedmriedem: Do we start -2ing pike patches now, unless they're security or something?14:59
mriedemno14:59
efried(and by "we" I mean stable folk)14:59
mriedemthat's pre-em process15:00
mriedembecause there was an eol15:00
openstackgerritAdam Spiers proposed openstack/nova-specs master: Update SEV work item to new approach based on MEM_ENCRYPTION_CONTEXT  https://review.opendev.org/65571715:00
mriedemwith em we've done away with eol unless we can't manage the branch anymore (or don't want to)15:00
aspiersefried, sean-k-mooney: ^^^ minor tweak based on discussion just now15:00
efriedso what does em actually mean?15:00
*** _erlon_ has joined #openstack-nova15:00
efriedhow is it different from just continuing to backport changes?15:00
*** ttsiouts has quit IRC15:01
mriedemefried: https://governance.openstack.org/tc/resolutions/20180301-stable-branch-eol.html15:01
mriedemor https://superuser.openstack.org/articles/extended-maintenance-openstack/ if you like bloggy15:02
sean-k-mooneyefried: extended mainatince15:02
*** ivve has quit IRC15:03
efriedaspiers: not quite, commented.15:03
efriedmriedem: ugh, I'm totally not going to read that much text right now. I'll satisfy my curiosity later.15:03
mriedemmission accomplished15:04
aspiersefried: OK thanks. I was thinking it could rely on the translation from hw:mem_encryption to resource class in the request filter, but checking allocations dict also sounds fine15:04
efriedaspiers: Remember that translation isn't being persisted, so the flavor spawn gets to look at won't have it.15:05
aspiersefried: ohhhh, you mean that only happens in the scheduler, and the resulting RequestSpec doesn't get passed through to the nova-compute service?15:06
efriedmriedem: I've updated the nova mtg agenda15:06
efriedaspiers: Yes, iiuc. I could be missing something though15:06
aspiersefried: that makes sense - this stuff isn't obvious to me yet since I'm still very unfamiliar with the nova codebase, so thanks for the pointers15:07
efriedaspiers: But IMO looking at the allocation is a cleaner solution regardless.15:07
aspiersefried: yup, sounds fine to me15:07
mtreinishaspiers: unfortunately I won't be at Denver. But I save that talk to watch later, at least based on the abstract it looks like they had a similar idea to what andreaf and I were starting to do in CIML (at a high level)15:08
efriedaspiers: If you wanted to use hw:mem_encryption=true, you would have to check both the flavor and the image. So yeah, use the allocation.15:08
mtreinishbut they approched it with different ML techniques (and ones that might be better suited for the problem)15:08
aspiersmtreinish: cool! very interested to hear your take on the talk. BTW I noticed earlier that elastic recheck / crm114 work seems to have made some v nice progress since last time I looked15:09
aspiersefried: changing the spec tweak now15:09
aspiersmtreinish: sorry to miss you this time though. we'll have to sort out those stestr issues online instead ;-)15:10
mtreinishaspiers: yeah it's too bad. Luckily for you though masayukig will be there and he can help sort through those in person :P15:11
aspiersmtreinish: true :)15:12
mtreinishaspiers: hmm, I wasn't aware anyone had done anything with the crm114 stuff in a while. But I hadn't been watching it super closely15:12
aspiersmtreinish: well it's a *long* time since I last more than glanced at http://status.openstack.org/elastic-recheck/15:13
*** itlinux has joined #openstack-nova15:13
aspiersmtreinish: maybe I just never noticed some of the nice stuff before15:14
*** itlinux has quit IRC15:14
*** igordc has joined #openstack-nova15:19
bauzasefried: I wasn't originally thinking to discuss about NUMA affinity for vGPUs in the PTG https://review.opendev.org/#/c/650963/ but given gibi provided some good concerns, I wondered when we could have time for eventually discussing it at the PTG15:20
bauzasthis isn't technically a placement-related spec, but I'd love to see it discussed then15:21
openstackgerritAdam Spiers proposed openstack/nova-specs master: Update SEV work item to new approach based on MEM_ENCRYPTION_CONTEXT  https://review.opendev.org/65571715:21
*** luksky has quit IRC15:22
bauzasefried: so I proposed it for the x-proj session between nova/placement if you are okay with15:22
*** itlinux has joined #openstack-nova15:25
*** lpetrut has joined #openstack-nova15:25
efriedbauzas: Yup, wfm. cdent is going to be organizing that chunk of time.15:25
bauzask15:25
dansmithmriedem: what's your feeling on adding (i.e. requiring) tests for the virt drivers I'm adding image type flags for?15:26
dansmithif there's nothing dynamic, then it's just a test to assert that two dicts match15:26
mriedemthus not a very useful test, i don't have strong feelings about that15:27
mriedemi'd say make sure it's mentioned in the commit message though...15:27
dansmithokay15:28
openstackgerritBalazs Gibizer proposed openstack/nova master: handle port allocation during migration  https://review.opendev.org/65511215:28
openstackgerritBalazs Gibizer proposed openstack/nova master: func test for migrate server with ports having resource request  https://review.opendev.org/65511315:28
openstackgerritBalazs Gibizer proposed openstack/nova master: Extend NeutronFixture to handle migrations  https://review.opendev.org/65511415:28
openstackgerritBalazs Gibizer proposed openstack/nova master: Add request_spec to server move RPC calls  https://review.opendev.org/65572115:28
efriedkashyap: secure boot topic, okay if I move it to Thursday 1400?15:29
jaypipesstephenfin: +215:30
kashyapefried: Thursday is the first day, right?15:30
stephenfinjaypipes: \o/15:30
efriedyes15:30
stephenfinjaypipes: Thanks for taking the time to look through that15:30
kashyapefried: I might need to be elsewhere.  But we've discussed it also in Dublin, and people agreed it is something we want.15:30
kashyapefried: So even if I miss it, it should be OK?15:30
efriedkashyap: If Saturday is better, I'll leave it there. Just had an opening.15:31
jaypipesstephenfin: np, that *you* for having the energy to keep it going.15:31
efriedkashyap: But even better, if it's something we don't need to discuss, I can just strike it.15:31
kashyapefried: You know what, maybe just put it on Thurs 14:00.  I'll work it out to be there15:31
efriedokay15:32
stephenfinjaypipes: Also, fwiw, I'd love to deprecate those options and would like to do so at some point in the future. Just need to figure out how to deprecate these things that are essentially unversioned APIs15:32
stephenfinBaby steps15:32
kashyapefried: It should be a quick thing.  No elaborate discussion needed15:32
* kashyap bbiab15:32
mriedemstephenfin: do you know what might have changed in our tox setup such that if you specify a target that doesn't exist, it used to fail fast but now it doesn't15:34
mriedemi.e. tox -e foo15:34
mriedempretty sure i've fat-fingered some target to see it pass when the actual thing would have failed a test15:35
mriedemyeah heh i ran tox -e fast715:36
mriedemand tox -e api-samles15:36
stephenfinmriedem: Hmm15:36
stephenfinmriedem: I've spotted that and I don't know, unfortunately.15:36
stephenfin$ tox -e fast715:36
stephenfinERROR: unknown environment 'fast7'15:36
stephenfin^ I still see that on my machine (TM)15:36
mriedemwhich version of tox do you have?15:36
stephenfin$ tox --version15:37
stephenfin3.4.015:37
*** liuyulong has quit IRC15:37
openstackgerritAdam Spiers proposed openstack/nova master: Add new "supports_amd_sev" capability to libvirt driver  https://review.opendev.org/63868015:37
mriedem$ tox --version15:40
mriedem3.7.0 imported from /usr/local/lib/python2.7/dist-packages/tox/__init__.pyc15:40
stephenfinmriedem: https://tox.readthedocs.io/en/latest/changelog.html#v3-8-0-2019-03-2715:40
stephenfin"Fix missing error for tox -e unknown when tox.ini declares envlist. - by @medmunds #1160"15:40
stephenfinLooks like a bug in 3.7.015:40
mriedemah https://github.com/tox-dev/tox/issues/116015:40
mriedemSuccessfully installed tox-3.8.015:41
mriedemosboxes@osboxes:~/git/nova$ tox -e foo15:41
mriedemERROR: unknown environment 'foo'15:41
mriedemhot diggity damn15:41
mriedemthanks stephenfin15:41
stephenfinno problemo15:41
*** efried has quit IRC15:43
*** efried has joined #openstack-nova15:43
efriedwow15:44
*** dave-mccowan has joined #openstack-nova15:44
efriednote to self: Ctrl+Alt+C does *not* open the calculator15:44
melwittmriedem: grafana shenanigans proposed here https://review.opendev.org/655591 wondering if I should add stable branch panels to it too15:44
aspiersefried: lol15:45
mriedemmelwitt: ah it was the per-project thing that was busted, maybe that's what i talked about with corvus before15:46
mriedemthanks for fixing that15:46
*** lajoskatona has joined #openstack-nova15:47
*** boxiang has quit IRC15:48
*** dave-mccowan has quit IRC15:49
*** boxiang has joined #openstack-nova15:49
lajoskatonamriedem: Hi, regardig https://review.opendev.org/640600 & https://review.opendev.org/640601 (tempest schema validation for nova v2.70 & 2.71) if you need help to push them I am happy to help you out15:50
mriedemlyarwood: fyi https://graphite.opendev.org/render/?width=1231&height=593&_salt=1556207646.49&from=-8weeks&target=stats_counts.zuul.tenant.openstack.pipeline.check.project.opendev_org.openstack_nova.master.job.tempest-full-py3.FAILURE&target=stats_counts.zuul.tenant.openstack.pipeline.check.project.opendev_org.openstack_nova.master.job.devstack-plugin-ceph-tempest.FAILURE15:54
mriedemyou can get the graph from graphite15:54
mriedemlajoskatona: go ahead, i'm not working on those15:55
mriedemmelwitt: nice catch on the tempest-full-py3 vs devstack-plugin-ceph-tempest (not py3) in the nova case, we should probably change nova to run against the devstack-plugin-ceph-tempest-py3 job instead15:55
lajoskatonamriedem: ok, do I need any special thing to test in devstack?15:55
melwittmriedem: yeah, I was wondering that too15:56
mriedemmelwitt: that will throw the graph data off, but we could switch now, make your graph change depend on that and then revisit in a couple of weeks to compare stability15:56
mriedemor just do that in a follow up, whatever15:57
mriedemlajoskatona: i'm not sure i understand the question15:57
* melwitt nods15:57
*** weshay|rover is now known as weshay15:59
lajoskatonamriedem: never mind, I try to to push, and come back if I have questions :-)16:00
mriedemlajoskatona: ack and thanks16:00
lajoskatonamriedem: :-)16:01
*** lajoskatona has quit IRC16:01
*** boxiang has quit IRC16:05
*** rpittau is now known as rpittau|afk16:05
*** boxiang has joined #openstack-nova16:05
*** wwriverrat has quit IRC16:10
openstackgerritDan Smith proposed openstack/nova master: Add ironic driver image type capabilities  https://review.opendev.org/65572916:11
openstackgerritDan Smith proposed openstack/nova master: Add vmware driver image type capabilities  https://review.opendev.org/65573016:11
openstackgerritDan Smith proposed openstack/nova master: Add xenapi driver image type capabilities  https://review.opendev.org/65573116:11
openstackgerritDan Smith proposed openstack/nova master: Add zvm driver image type capabilities  https://review.opendev.org/65573216:11
dansmithI dunno who to tag on the zvm one for review16:12
dansmithbut I'm pretty sure they only support raw16:12
melwittmriedem: I dunno if you saw, they did hit the eventlet monkey patching issue in ubuntu, last comment https://review.opendev.org/64731016:13
*** dklyle has quit IRC16:15
*** dklyle has joined #openstack-nova16:15
openstackgerritMatt Riedemann proposed openstack/nova master: Always pass HostAPI to get_availability_zones  https://review.opendev.org/65558316:16
*** pcaruana has quit IRC16:19
*** igordc has quit IRC16:21
*** igordc has joined #openstack-nova16:23
*** cooper6581 has joined #openstack-nova16:26
*** tbachman has quit IRC16:27
*** ivve has joined #openstack-nova16:30
*** wwriverrat has joined #openstack-nova16:30
*** Sundar has quit IRC16:30
mriedemmelwitt: yeah i saw16:35
mriedemcoreycb didn't +1 it though...16:35
mriedemdansmith: i suppose add jichenjc16:35
*** zbr is now known as zbr|rover16:37
melwittI guess I took his comment on the review to be in support. but yeah, coreycb if you could +1 https://review.opendev.org/647310 if you are in support of backporting it16:38
*** alex_xu has joined #openstack-nova16:40
*** pcaruana has joined #openstack-nova16:40
imacdonnsomewhat of a tangent, I suppose ... but I think we need to figure out what to do about the eventlet monkey_patching vs. WSGI issue ... it's blocking me from upgrading to Stein16:40
melwittafaik there's not a clear thing we should do to address the problem, or did I miss it? eventlet monkey patching is not something I understand well16:42
imacdonnsame here16:44
*** gyee has joined #openstack-nova16:44
imacdonnthe nova docs rather strongly suggest moving to WSGI, but maybe my only option is to go back to running nova-api the old way :/16:45
imacdonnalternative interim may be to hack the monkey-patching out, but I don't understand the implications of that well enough ...16:46
dansmithyeah I don't think you can just remove it16:46
imacdonnI don't have it in my Rocky deployments, though16:47
dansmithdon't have what? the problem or the monkeypatching?16:47
imacdonnboth ;)16:47
dansmithhrm, I'm trying to think about what the api service does that might continue to work without being patched16:48
dansmithif we use all the regular primitives, then it might just work with regular threading, but I expect there are some subtle behavior assumptions16:48
imacdonnit was added here: https://github.com/openstack/nova/commit/23ba1c690652832c655d57476630f02c268c87ae16:48
dansmithlocks we don't hold16:48
melwittI have a patch series that is going on the route to removing eventlet dependency for nova-api https://review.opendev.org/650172 but it needs a follow up patch that makes the executor type configurable (or outright changes it away from green threads)16:48
dansmithimacdonn: right, but it was added because we didn't realize that we weren't patching when the wsgi stuff was split out from service originally, IIRC16:49
melwittI'm not sure if there are other reasons aside from scatter-gather that eventlet is a dependency in nova-api16:49
imacdonndansmith: right, and presumably someone decided that it was, infact, needed, with WSGI16:50
*** udesale has quit IRC16:50
imacdonndansmith: there's mention of multiple cells (which I don't have (other than cell0)) ... but dunno if there are other cases where it's "needed"16:50
dansmithmelwitt: that'd be the big one, but any other spawn we do in some layer that can be called from api and other services16:50
dansmithimacdonn: right but even with cell0 you're affected by that16:51
coreycbmelwitt: mriedem: added my +116:51
*** igordc has quit IRC16:52
dansmithI've been a bit disconnected.. is the problem that seems to be introduced the amq heartbeat thing?16:52
*** ivve has quit IRC16:52
mriedemimacdonn: i thought https://review.opendev.org/#/c/647310/ was in response to https://github.com/openstack/nova/commit/23ba1c690652832c655d57476630f02c268c87ae in a way, but you said that fix doesn't help you right?16:53
imacdonnmriedem: that just changes where monkey-patching gets inserted ... I have my problem either way16:54
mriedemi guess you could try exporting OS_NOVA_DISABLE_EVENTLET_PATCHING=1 but that's likely not something you want to run with16:57
imacdonnyeah16:59
mriedemdansmith: there are only 4 places we use the long_rpc_timeout heartbeat stuff and they'd all be calls from either conductor or compute neither of which are wsgi so i'm not sure how that would be involved, unless it was the moving of the monkey patch location in https://github.com/openstack/nova/commit/23ba1c690652832c655d57476630f02c268c87ae16:59
dansmithmriedem: I think the long rpc stuff all uses regular threading primitives so it should work regardless of if you're patched or not right?17:00
mriedemidk, i've never looked at the oslo.messaging implementation17:00
dansmithmriedem: do you have reason to believe the the long rpc stuff is related, or are you just looking for related things?17:00
mriedemimacdonn: have you tried putting melwitt's series https://review.opendev.org/#/q/topic:cell-scatter-gather-futurist+(status:open+OR+status:merged) into rocky and see if that fixes your issue?17:01
dansmithmriedem: that patch doesn't actually change anything yet, IIUC17:01
mriedemdansmith: just in response to "(11:52:08 AM) dansmith: I've been a bit disconnected.. is the problem that seems to be introduced the amq heartbeat thing?"17:01
dansmithit just wraps use of the eventlet stuff so we can do it differentyly17:01
melwittthat's still using eventlet, you'd have to change the futurist executor to the native one17:01
dansmithyeah ^17:01
dansmithmriedem: I think you're confusing heartbeats no?17:02
dansmithmriedem: the heartbeat that long_rpc added is layer 7, unrelated to AMQ heartbeats, which I thought was the subject of that thread17:02
mriedemoh i guess i am then17:03
mriedemi'll step out of this mess (this is why i'm not involved)17:03
dansmithmriedem: yeah, re-reading, they're getting a connection timeout because AMQP heartbeats aren't being sent (a different thing)17:04
dansmiththe theory is that eventlet patching prevents some thread from running that sends those17:04
dansmithtotes unrelated to long_rpc17:04
imacdonnyes, that's how it appears17:04
*** derekh has quit IRC17:08
*** Sundar has joined #openstack-nova17:10
openstackgerritmelanie witt proposed openstack/nova master: nova-manage db archive_deleted_rows is not multi-cell aware  https://review.opendev.org/50748617:13
*** dtantsur is now known as dtantsur|afk17:14
*** ricolin has joined #openstack-nova17:20
*** ralonsoh has quit IRC17:20
openstackgerritAdam Spiers proposed openstack/os-traits master: Update SEV trait docs to avoid misleading people  https://review.opendev.org/65567117:21
*** luksky has joined #openstack-nova17:27
*** psachin has quit IRC17:28
*** lpetrut has quit IRC17:34
*** igordc has joined #openstack-nova17:35
dansmithmelwitt: the Future object makes it sound like cancel() might not always be supported.. have you confirmed?17:38
*** itlinux has quit IRC17:39
melwittdansmith: yeah, was just writing that on the review. I think it only does something for eventlet GreenThreadPoolExecutor. been looking around in the source to confirm because it doesn't seem like the docs are explicit about which types support cancel()17:39
dansmithmelwitt: "If the call is currently being executed and cannot be cancelled then the method will return"17:39
dansmithI think since it's really just a wrapper around a queue, it's just going to cancel if the thing hasn't been dequeued yet17:40
*** itlinux has joined #openstack-nova17:40
dansmithmight not even kill a thread in process, I dunno17:40
melwittyeah, cancel() returns True/False and for native threads I expect it just always returns False17:40
melwittright17:40
melwittyeah, I don't see futurist doing anything different for cancel() other than passing it through to the primitives17:42
*** jangutter has quit IRC17:42
dansmithyeah17:43
melwittso it would be a matter of what does the Future primitive do for cancel() for the different types of executors17:43
melwittnot really seeing it documented on https://docs.python.org/3/library/concurrent.futures.html#future-objects17:43
dansmithmelwitt: just to be clear, when I say the primitives, I'm talking about the threading library not the concurrent stuff17:43
melwittoh, ok. my bad17:43
dansmiththe latter is the high-level implementation, which may very well not be in py217:43
*** igordc has quit IRC17:43
dansmithconcurrent (and c.futures) just use the low-level primitives in threading, I imagine17:44
melwittyeah, it's not but it's all included in a separate futures package containing backports https://pypi.org/project/futures/17:44
dansmithwhich is why I said what I did17:44
dansmithaye17:44
melwittyeah, I imagine the same, using same low-level primitives underneath17:44
*** igordc has joined #openstack-nova17:45
*** itlinux has quit IRC17:46
*** itlinux has joined #openstack-nova17:47
openstackgerritArtom Lifshitz proposed openstack/nova master: Run revert resize tests in nova-live-migration  https://review.opendev.org/65349817:51
openstackgerritArtom Lifshitz proposed openstack/nova master: Revert "Wait for network-vif-plugged on resize revert"  https://review.opendev.org/63939617:51
openstackgerritArtom Lifshitz proposed openstack/nova master: Revert resize: wait for external events in compute manager  https://review.opendev.org/64488117:51
openstackgerritMerged openstack/os-vif master: Remove IP proxy methods  https://review.opendev.org/65569517:51
*** nicolasbock has quit IRC17:53
*** tbachman has joined #openstack-nova17:55
mriedemthe cancel() thing came up in kevin's changes for canceling queued live migrations,17:57
mriedemthere is probably something in the compute manager code around that17:57
*** itlinux has quit IRC17:58
mriedemoh right cancel() just returns false if it's already executing and you can't stop it17:58
melwittyeah, so far I'm not seeing in the futurist source that it event does a green thread kill() when cancel() is called with the GreenThreadPoolExecutor17:59
melwitt*even17:59
melwittI can't tell that cancel() ever does anything17:59
*** igordc has quit IRC18:00
*** itlinux has joined #openstack-nova18:00
*** nicolasbock has joined #openstack-nova18:02
dansmith...because it's just a wrapper around a work queue :)18:03
*** itlinux has quit IRC18:04
*** igordc has joined #openstack-nova18:06
*** tbachman has quit IRC18:07
melwitthere's what cancel() does in concurrent.futures https://github.com/python/cpython/blob/d7befad328ad1a6d1f812be2bf154c1cd1e01fbc/Lib/concurrent/futures/_base.py#L35218:17
dansmith"I'm guessing that this cancel is actually a condition variable"18:18
dansmithsome dude said ^18:18
*** lpetrut has joined #openstack-nova18:18
*** tosky has quit IRC18:20
*** lpetrut has quit IRC18:23
melwittdansmith: you.... win!18:31
melwittI like futures, this is fun18:36
openstackgerritmelanie witt proposed openstack/nova master: Use futurist.ThreadPoolExecutor in scatter_gather_cells  https://review.opendev.org/65017218:40
*** itlinux has joined #openstack-nova18:43
cooper6581mriedem:  Quick follow up for the { in the password for the transport_url yesterday during our upgrade.  I don't know enough if this is a bug or a documentation issue, but I wrote a quick test that reproduces the issue - https://gist.github.com/cooper6581/467f982a0a44494ca32d0c2f755112bc18:44
efriedgibi: FYI I copied the berlin onboarding deck and updated it for denver/train https://docs.google.com/presentation/d/1W81DsZGG_bqns2JEdSKwqPCLdAQlR5gTtss-kiGT7Pk/edit?usp=sharing18:45
cooper6581Specifically, I don't know if we should have ever gotten into a state where transport_url was set, and the config had a non templated url.  Maybe it could have been an error on our side during the upgrade v0v18:45
melwittlyarwood, mriedem: it's alive... http://grafana.openstack.org/d/-iKINcImz/ceph-failure-rate?orgId=1 (and not tracking tempest-full as nicely as we would hope)18:46
mriedemcooper6581: hmm, i'm not sure, that would require some brain power for sorting out the templated stuff which i don't have loaded up right now, dansmith might know off hand though18:47
mriedemi have to re-learn how the templating code each time i look at it b/c i'm slow18:47
mriedem*code works18:47
mriedemmelwitt: that's just 6 hours though18:48
melwittI'm gonna propose a new change to make the graphs cover more time (make look more like http://grafana.openstack.org/d/Hj5IHcSmz/neutron-failure-rate?orgId=1) and add more projects and stable branches18:48
mriedemhttp://grafana.openstack.org/d/-iKINcImz/ceph-failure-rate?orgId=1&fullscreen&panelId=2&from=now-30d&to=now18:48
mriedemmelwitt: you can toggle that in the dashboard18:48
melwittoh cool, I didn't know that18:48
mriedemtop right18:48
melwittoh I see it now18:48
melwittniiiiiice18:48
mriedemlooks like the neutron one is default 7 days18:48
mriedemi'm more interested in why the nova graph doesn't go beyond .... oh18:49
mriedemopendev rename18:49
mriedemthe graph won't go beyond 4/2018:49
mriedemwhen infra smoked a bowl and renamed everything18:49
melwittyeah, bummer18:49
melwitthaha18:49
mriedembut the 7 day trend seems normalish18:50
mriedemhttp://grafana.openstack.org/d/-iKINcImz/ceph-failure-rate?orgId=1&fullscreen&panelId=2&from=now-7d&to=now18:50
melwittthe cinder graph is whack18:50
mriedemhttp://grafana.openstack.org/d/-iKINcImz/ceph-failure-rate?orgId=1&fullscreen&panelId=3&from=now-7d&to=now18:50
melwittyeah 7 day looks reasonable18:50
mriedemyeah...18:50
mriedemnot sure what's going on there18:50
mriedemmaybe ask eharney if he knows?18:50
mriedemi don't really know why these would be different between the projects18:51
dansmithcooper6581: not sure I understand the test.. "...if the base URLs are set" -- what does that mean?18:51
melwittyeah, me neither, that's what I'm stuck on18:51
*** itlinux has quit IRC18:52
cooper6581I just copied the comment from the test test_non_formatted_url_with_no_base :p - I'm assuming what is referred to as base URL ends up being CONF.transport_url (I'm totally keyboard dog right now though) https://github.com/openstack/nova/blob/master/nova/objects/cell_mapping.py#L14518:53
cooper6581this is the test a copied https://github.com/openstack/nova/blob/ca6c32f279cf62915a11b32339cbac8128a8656e/nova/tests/unit/objects/test_cell_mapping.py#L25218:53
dansmithcooper6581: those assertions are just to handle a case where you have templated a url in the database but don't have anything to parse from the config to format those things.. I'm not sure what that has to do with your case18:55
*** itlinux has joined #openstack-nova18:58
*** ivve has joined #openstack-nova18:58
cooper6581Ahh, I see.  I understand better now.  Thanks!19:00
melwittmriedem: hm, I'm seeing the snapshot tests failing in cinder land http://logs.openstack.org/83/651183/6/check/devstack-plugin-ceph-tempest/9ed0779/testr_results.html.gz which looks like our glance issue we just fixed on stable recently on the surface19:02
melwitthttp://logs.openstack.org/83/651183/6/check/devstack-plugin-ceph-tempest/9ed0779/controller/logs/screen-n-cpu.txt.gz?level=TRACE#_Apr_16_18_32_53_33546019:02
melwittit's the same thing, but how if it's been fixed on master for a long time19:03
melwittoh that's stable/rocky19:03
melwittbad example on my part19:04
dansmithcooper6581: would it be better (than what we have) if we catch ValueError and explicitly log "yo dawg, this looks like a template because it has templating characters in it, but it failed to work with python's format() so you might want to fix it" ?19:06
melwittI looked at a couple of failures on master and they looked random19:06
dansmithcooper6581: we could also treat a ValueError as "probably not a template", but then someone trying to use templates loses the checking until they realize that nova is trying to connect to rabbit as "worker-{user"19:07
*** lpetrut has joined #openstack-nova19:14
*** igordc has quit IRC19:17
*** eharney has quit IRC19:25
*** rchurch_ has quit IRC19:28
openstackgerritmelanie witt proposed openstack/nova master: Warn for duplicate host mappings during discover_hosts  https://review.opendev.org/65194719:35
*** itlinux has quit IRC19:35
*** itlinux has joined #openstack-nova19:37
*** rchurch has joined #openstack-nova19:41
*** samueldmq has joined #openstack-nova19:44
*** igordc has joined #openstack-nova19:49
*** boxiang has quit IRC19:55
*** boxiang has joined #openstack-nova19:56
efriedmelwitt: I made a grokkable tinyurl and put it on the master. lmk if you hate it.19:57
cooper6581dansmith:  I still need to double check with ccstone, but I think the code is totally fine.  I think the problem was when we stepped to newton, that logic wasn't in there, so it got inserted into the DB20:01
mriedemweeee https://bugs.launchpad.net/nova/+bug/182638220:03
openstackLaunchpad bug 1826382 in OpenStack Compute (nova) "Updates to placement api fail if placement endpoint changes" [Undecided,Triaged] - Assigned to Liam Young (gnuoy)20:03
mriedemwe're too smart for our own good20:03
mriedemjaypipes: ^ thoughts on how to handle and auto-detect/correct that?20:03
mriedemtl;dr if don't configure nova to talk to placement with a specific endpoint url, we look it up via the service catalog and ksa; if the endpoint changes and we're using a stale client adapter, all requests fail (http to https)20:04
mriedemyou have to restart all the computes to fix it20:04
efriedmriedem: There was some attempt to address that via safe_connect20:05
mriedemthat probably gets a bit confusing though b/c we raise a nova-specific exception,20:06
efriedmriedem: One thing we could do is rebuild the adapter on SIGHUP20:06
mriedemin this case ResourceProviderRetrievalFailed20:06
mriedemheh, i mentioned that in the bug,20:06
mriedembut if sighup were fixed20:06
efriedmriedem: Well, there was *one* path through @safe_connect that swallowed the error and rebuilt the adapter and retried. But I don't think it was the right one.20:06
mriedemstill, let's say you have 1000 computes and you change the placement endpoint, do you want to sighup 1k computes? maybe that's normal20:06
*** mgariepy has quit IRC20:07
mriedemright that's the EndpointNotFound case20:07
efriedyeah, why tf are you changing the endpoint20:07
mriedemksa doesn't raise an exception here, we just get a 400 response back20:07
efriedyou deserve to have to sighup 1000 computes20:07
efriedwhoah20:07
mriedemtrue, i can't really decide if this is low priority or what20:07
*** Sundar has quit IRC20:07
efriedBecause the endpoint is there, just no longer accepting http? Is that what you meant by (http to https)?20:07
artomYeah, I'd have thunk if you change the endpoint you put a redirect in place first20:08
*** mchlumsky has quit IRC20:08
mriedem"In my deployment this occurred when the placement end point switched from http to https after the nova-compute node had started."20:08
efriedbecause if the old endpoint is gone gone, that should be a connect failure, which ksa converts.20:08
efriedYeah, that makes... some sense I guess?20:08
efriedbut yeah, a redirect would be nice20:08
*** mvkr has joined #openstack-nova20:09
*** _erlon_ has quit IRC20:09
artomI don't really think it's reasonable to expect Nova to pull the endpoint list form the catalog for every single request.20:09
efriedmriedem: btw, I got ultra screwed on the school runs, won't be back until even later than originally anticipated.20:09
efriedartom: ++20:09
artomWhat do we do for Neutron and Cinder?20:10
mriedemartom: i don't think anyone is suggesting that20:10
efriedmriedem: actually, the fact that we get a 400 from the API is the kicker here.20:10
mriedemright20:10
efriedIf it were any kind of 500, I would say we might should do something about it.20:10
mriedemwe could put a decorator on the get/put/post/delete methods to sniff the response for a 400 and if "Reason: You're speaking plain HTTP to an SSL-enabled server port." is in the text, but that's super hacky20:10
efriedbut the fact that there's still something at that endpoint, just ain't working anymore, is not our fault.20:10
efriedyeah, that's bs20:10
jrollI've never seen an https web server return a response when you try to talk http to it20:11
artomWait, they switched to HTTPS *on the same port*?20:11
artomIs that a thing?20:11
jrollthe client just fails to negotiate a tls connection20:11
fungii *think* you can do starttls that way, but it's more common to see it on the assigned plaintext port equivalent20:11
efriedartom: Suspect portless therefore using defaults (80/443)20:11
efriedgotta run20:12
fungialso more common for other protocols than http20:12
jrollhuh, looking at the bug and apparently it's a thing, wtf20:12
artomfungi, ah, true20:12
fungie.g., you can initiate a tls negotiation blind on the smtps port, or on the smtp port you can issue a starttls command (if its advertised on ehlo) and then start tls negotiation20:13
fungihttps://en.wikipedia.org/wiki/Opportunistic_TLS20:14
fungileads to https://en.wikipedia.org/wiki/HTTP/1.1_Upgrade_header20:14
melwittefried: tinyurl for what now?20:15
fungiso possible, and has a standard, but... not common in the wild20:15
artomfungi, yeah, though in this case it's almost inverted: the server responds with a 400 because the request *isn't* HTTPS20:18
*** eharney has joined #openstack-nova20:18
*** itlinux has quit IRC20:18
* artom has to run (bike, to be exact) as well20:18
fungioh, so protocol downgrade? that's generally just considered a security vulnerability20:18
*** itlinux has joined #openstack-nova20:19
fungiahh, or client trying to make plaintext connection on tls port? i think the phrase for that is "broken configuration"20:19
jrollfungi: right, I'm surprised that apache actually responds to that, rather than the tls negotiation failing20:22
*** dave-mccowan has joined #openstack-nova20:23
*** artom has quit IRC20:24
fungii'd have come up with a more humorous term, but my comedy circuits are currently overridden by the fact that my mower just kicked up a clam shell, and i'm certain there's a joke in there somewhere if i can just shuck it out20:26
mriedemi see what you did there20:26
*** panda is now known as panda|off20:30
*** cooper6581 has quit IRC20:33
*** igordc has quit IRC20:36
*** cooper6581 has joined #openstack-nova20:38
melwittmriedem: I was just reading into the ML thread and looking at your results here https://review.opendev.org/654468 and wondered, don't you need LIBS_FROM_GIT=oslo.config to avoid it using the installed version from pypi? I see it checking out the commit you Depends-On in the job output but then see oslo.config installed later? http://logs.openstack.org/68/654468/6/check/openstack-tox-py27/05facc9/job-output.txt.gz#_2019-04-25_20_12_50_2720:38
melwitt027520:38
melwitthttp://logs.openstack.org/68/654468/6/check/openstack-tox-py27/05facc9/job-output.txt.gz#_2019-04-25_20_12_50_27027520:38
mriedemi was under the impression zuulv3 handled that for us20:41
mriedembut maybe that's only if it's in required_projects in zuul.yaml20:42
mriedempip freeze is showing a tag rather than a git hash though http://logs.openstack.org/68/654468/6/check/openstack-tox-py36/6c5ead4/tox/py36-4.log20:43
mriedemoslo.config==6.8.120:43
melwittyeah that was my concern20:44
mriedemnot sure what this is about "cannot set the recursion limit to 10 at the recursion depth 55: the limit is too low"20:45
mriedemthe python docs don't say anything about a lower bound20:45
*** eharney has quit IRC20:46
melwittwhere are you seeing that?20:46
mriedemhttp://logs.openstack.org/68/654468/6/check/openstack-tox-py36/6c5ead4/testr_results.html.gz20:46
melwittoh different job20:47
*** pcaruana has quit IRC20:48
mriedemyeah the bug only hits on py36 for some reason20:48
fungimriedem: melwitt: yes, a repository will only be checked out alongside if it's in the required-projects list20:49
mriedemack, ok, that's easily fixable in this hack patch20:49
*** ricolin has quit IRC20:49
fungiotherwise depends-on means "don't merge until this other change merges"20:49
fungi(which also applies to a required project, but an rp will also be tested in conjunction with your change)20:50
*** takashin has joined #openstack-nova20:50
melwittmriedem: I guess the doc does say "Changed in version 3.5.1: A RecursionError exception is now raised if the new limit is too low at the current recursion depth." https://docs.python.org/3/library/sys.html#sys.setrecursionlimit20:51
melwittso it has to be at least 55?20:51
melwittbased on the error message it put20:51
mriedemhmm20:52
mriedemb'RecursionError: cannot set the recursion limit to 56 at the recursion depth 55: the limit is too low'20:52
*** ttsiouts has joined #openstack-nova20:52
melwittlol, what are we supposed to20:52
melwittdo20:52
melwittI guess that means at that moment, the depth is 5520:53
mriedemyeah20:54
mriedemi bumped to 100 and then it passed20:54
melwittso the limit has to be some headroom above that to allow for more20:54
melwittok, cool20:54
mriedemit was 1000 in my local py3.620:54
mriedemi mean the default or whatever20:54
mriedemwhich is a bionic node20:54
melwittok, I see20:54
*** cooper6581 has quit IRC20:59
openstackgerritMatt Riedemann proposed openstack/nova master: DNM: debug config opts infinite recursion  https://review.opendev.org/65446820:59
mriedemfungi: melwitt: ^ hopefully that does it20:59
mriedemnova meeting21:00
melwittgood call excluding all the other jobs too21:00
*** ttsiouts_ has joined #openstack-nova21:00
*** gmann is now known as gmann_afk21:01
*** ttsiouts has quit IRC21:02
*** itlinux has quit IRC21:03
*** ivve has quit IRC21:11
*** alex_xu has quit IRC21:12
*** sean-k-mooney has quit IRC21:19
openstackgerritMatt Riedemann proposed openstack/nova stable/pike: Move legacy-grenade-dsvm-neutron-multinode-live-migration in-tree  https://review.opendev.org/64020721:27
openstackgerritTakashi NATSUME proposed openstack/nova master: Remove deprecated 'default_flavor' config option  https://review.opendev.org/64547621:28
openstackgerritTakashi NATSUME proposed openstack/nova master: Fix cleaning up console tokens  https://review.opendev.org/63771621:29
*** itlinux has joined #openstack-nova21:31
openstackgerritMatt Riedemann proposed openstack/nova stable/queens: Fix nova-grenade-live-migration run book for opendev migration  https://review.opendev.org/65580021:31
efriedmriedem: thanks for running that.21:38
mriedemnp21:38
*** eharney has joined #openstack-nova21:44
*** tbachman has joined #openstack-nova21:46
efriedmelwitt: tinyurl pointing to the charts for the project update21:47
efriedmelwitt: since there's all those links on 'em21:47
melwittoh, the slides. yeah, that's fine21:48
mriedemdansmith: i think the multi-cell job has yielded it's first victim21:49
dansmithin a good way?21:50
mriedemif latent bugs are a good thing21:50
mriedemapparently in this addFloatingIP code, we fetch the instance from the cell at the top and pass down the targeted context for that instance,21:51
mriedemand if the floating IP is associated with another instance we fetch that https://github.com/openstack/nova/blob/a991980863f056323c1ee9fd6a46dbc4cb899eca/nova/network/neutronv2/api.py#L239021:51
mriedembut if that instance is in another cell, bingo bango you've got yourself a 404 mister21:51
mriedemshits the bed right here https://github.com/openstack/nova/blob/a991980863f056323c1ee9fd6a46dbc4cb899eca/nova/network/neutronv2/api.py#L240021:52
*** itlinux has quit IRC21:52
*** JamesBenson has quit IRC21:52
dansmithmriedem: pretty sure you're over your bingo bango quota for the week21:52
mriedemumm, actually one of the cells was down and now i have no quota limit21:53
mriedemyou get to experience that kind of dry wit and more in person next week21:53
dansmithlooking forward to the "and more" there21:54
* dansmith packs his pepper spray21:54
mriedemyou'll have to pay for the full riderman experience21:54
dansmithso...many...nws...things....to...say21:55
*** slaweq has quit IRC21:57
efriedmriedem, melwitt: As soon as I can figure out how to fixture the sdk (see https://storyboard.openstack.org/#!/story/2005475) we could actually do clouds.yaml immediately.21:57
efriedBeyond that, yes, the long-term benefit is cutting the python-*client packages out of the picture, removing that tech debt. That relationship is brittle and causes us to either fear changing it or break things when we do.21:57
*** ccamacho has joined #openstack-nova21:59
efried...immediately. Because the sdk proxies are Adapter subclasses, so we can just swap them seamlessly for the ksa adapter's we're hacking up today. Then cut the actual client libs out at our leisure.21:59
mriedembut then we'll have (1) old style config for cinder (2) normalized ksa config for all the other non-sdk clients we use and (3) sdk clouds.yaml thing21:59
efrieddeprecation period for oslo.config stuff obviously.21:59
mriedemseems like a configuration mess until everyting is cut over21:59
efriedAnd operators already have clouds.yaml for the most part, so they'll have very little to do.22:00
efried(by "already" I mean, if they're running stein+)22:00
mriedem"That relationship is brittle and causes us to either fear changing it or break things when we do." as the recent ironicclient stuff can tell i suppose,22:01
efriedexactly22:01
dansmithefried: which is like nobody right?22:01
mriedembut i imagine we'll hit a new class of weird bugs once we start using the sdk22:01
efriedthough I didn't want to bring that up22:01
efriedbecause I broke it by introducing the ksa config...22:01
mriedemi'm not sure how much damage we get from glance/neutron/cinder client22:01
mriedemwhat's being used for nova/cyborg in that series? just ksa?22:02
*** imacdonn has quit IRC22:02
mriedemor the sdk?22:02
efrieddansmith: Right, this is setting up for a future time when people are actually using the releases we're working on and we've got new jobs and don't care anymore.22:02
efriedI've been pushing Sundar to use sdk22:02
*** imacdonn has joined #openstack-nova22:02
efriedI gave him the placement code for the cyborg side of his demo and set it up to use sdk22:03
mriedemso if we did the sdk thing, cyborg integration could be the guinea pig, and then i'd think cinder since we haven't ksa'd our cinder stuff yet22:03
efriedI was going to do placement first, because it would be a transparent swap under the covers - we don't have a placementclient, so we would just set up SchedulerReportClient._client as the Connection.placement proxy (which is an Adapter subclass) instead of the raw Adapter.22:04
efriedget/put/etc primitives come along for free.22:04
efriedI PoC'd this already and it works seamlessly in tempest. It just blows up all the unit and functional tests until I figure out fixtures (see above)22:04
mriedemok22:05
efried...and then ironicclient, because we have lots of support for that from the ironic team, and dtantsur|afk is a sdk core who can help to some extent22:05
efriedand I have a guy coming on board (hi dustinc) who's trying to be involved in ironic who has started looking at it.22:06
efriedbut yeah, could look into tackling cinder early, I see the benefit since it got left behind with ksa.22:07
efriednever could figure that f'in thing out.22:07
*** ttsiouts_ has quit IRC22:07
*** ttsiouts has joined #openstack-nova22:07
*** slaweq has joined #openstack-nova22:11
*** ttsiouts has quit IRC22:12
openstackgerritEric Fried proposed openstack/nova master: WIP: Correct signature on fake_get_availability_zones  https://review.opendev.org/65580622:15
*** alex_xu has joined #openstack-nova22:19
mriedemdansmith: so i'm going to rev this multi-cell job patch to disable the failing test here until there is a bug fix - while doing that, what do you think about just making the job non-voting in the check queue for a bit until we're happy with it through the ptg?22:21
dansmithmriedem: I'm good with whatever, but it seems like non-voting to collect data is what we would normally do22:21
*** lpetrut has quit IRC22:26
mriedemack22:28
mriedemdone22:28
openstackgerritMatt Riedemann proposed openstack/nova master: Add nova-multi-cell job  https://review.opendev.org/65522222:28
openstackgerritMatt Riedemann proposed openstack/nova master: Enable n-novnc in nova-multi-cell job  https://review.opendev.org/65571122:28
*** slaweq has quit IRC22:37
*** whoami-rajat has quit IRC22:51
*** cooper6581 has joined #openstack-nova22:52
openstackgerritmelanie witt proposed openstack/nova master: Use futurist.ThreadPoolExecutor in scatter_gather_cells  https://review.opendev.org/65017222:52
*** cooper6581 has quit IRC22:56
*** luksky has quit IRC22:58
*** tkajinam has joined #openstack-nova23:01
*** rcernin has joined #openstack-nova23:03
efriedo/ see y'all tomorrow23:06
*** slaweq has joined #openstack-nova23:16
*** slaweq has quit IRC23:24
*** artom has joined #openstack-nova23:39
artom23:41
rm_workso, is it possible to run nova unit tests with tox on OSX? anyone manage this? or is it just impossible (seems that pyinotify is not compatible with osx so the env build just fails)23:56
*** gyee has quit IRC23:57
*** hongbin has quit IRC23:57
*** cooper6581 has joined #openstack-nova23:58

Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!