Thursday, 2019-05-16

*** hongbin has quit IRC00:01
*** ttsiouts has joined #openstack-meeting00:08
*** artom has joined #openstack-meeting00:15
*** yamamoto has quit IRC00:19
*** gyee has quit IRC00:36
*** ttsiouts has quit IRC00:41
*** armax has quit IRC00:45
*** diablo_rojo has quit IRC00:45
*** ricolin has joined #openstack-meeting00:50
*** lseki has quit IRC00:54
*** d34dh0r53 has joined #openstack-meeting01:01
*** armax has joined #openstack-meeting01:02
*** bbowen has quit IRC01:27
*** baojg has joined #openstack-meeting01:29
*** rbudden has joined #openstack-meeting01:31
*** ykatabam has quit IRC01:32
*** ykatabam has joined #openstack-meeting01:33
*** armax has quit IRC01:37
*** lbragstad has quit IRC01:42
*** lbragstad has joined #openstack-meeting01:51
*** ijw has quit IRC01:55
*** apetrich has quit IRC01:57
*** whoami-rajat has joined #openstack-meeting02:06
*** dklyle has joined #openstack-meeting02:14
*** dhellmann has quit IRC02:23
*** dhellmann has joined #openstack-meeting02:23
*** ttsiouts has joined #openstack-meeting02:38
*** lbragstad has quit IRC02:40
*** lbragstad has joined #openstack-meeting02:43
*** armax has joined #openstack-meeting02:45
*** yamamoto has joined #openstack-meeting02:48
*** ykatabam has quit IRC02:51
*** armax has quit IRC03:05
*** ttsiouts has quit IRC03:11
*** hongbin has joined #openstack-meeting03:14
*** rbudden has quit IRC03:14
*** rbudden has joined #openstack-meeting03:18
*** ykatabam has joined #openstack-meeting03:25
*** ykatabam has quit IRC03:31
*** ykatabam has joined #openstack-meeting03:31
*** yamamoto has quit IRC03:50
*** radeks_ has joined #openstack-meeting03:50
*** yamamoto has joined #openstack-meeting03:51
*** hongbin has quit IRC03:55
*** radeks_ has quit IRC04:40
*** janki has joined #openstack-meeting04:43
*** _d34dh0r53_ has joined #openstack-meeting04:55
*** lbragstad has quit IRC04:59
*** _d34dh0r53_ has quit IRC05:08
*** ttsiouts has joined #openstack-meeting05:09
*** _d34dh0r53_ has joined #openstack-meeting05:11
*** rbudden has quit IRC05:17
*** rbudden has joined #openstack-meeting05:17
*** rbudden has quit IRC05:17
*** rbudden has joined #openstack-meeting05:18
*** rbudden has quit IRC05:18
*** rbudden has joined #openstack-meeting05:19
*** rbudden has quit IRC05:19
*** rbudden has joined #openstack-meeting05:19
*** rbudden has joined #openstack-meeting05:20
*** rbudden has joined #openstack-meeting05:21
*** rbudden has quit IRC05:21
*** rbudden has joined #openstack-meeting05:22
*** rbudden has joined #openstack-meeting05:22
*** rbudden has joined #openstack-meeting05:23
*** rbudden has quit IRC05:24
*** rbudden has joined #openstack-meeting05:24
*** rbudden has quit IRC05:24
*** rbudden has joined #openstack-meeting05:25
*** rbudden has quit IRC05:25
*** rbudden has joined #openstack-meeting05:26
*** rbudden has quit IRC05:26
*** rbudden has joined #openstack-meeting05:26
*** Luzi has joined #openstack-meeting05:40
*** ttsiouts has quit IRC05:41
*** jbadiapa has joined #openstack-meeting05:49
*** e0ne has joined #openstack-meeting05:53
*** ykatabam has quit IRC06:00
*** ykatabam has joined #openstack-meeting06:03
*** radeks_ has joined #openstack-meeting06:03
*** radeks_ has quit IRC06:04
*** radeks_ has joined #openstack-meeting06:04
*** e0ne has quit IRC06:36
*** ykatabam has quit IRC06:56
*** apetrich has joined #openstack-meeting06:56
*** rcernin has quit IRC07:01
*** tesseract has joined #openstack-meeting07:05
*** pcaruana has joined #openstack-meeting07:13
*** kopecmartin|off is now known as kopecmartin07:18
*** ralonsoh has joined #openstack-meeting07:23
*** iyamahat has quit IRC07:31
*** tssurya has joined #openstack-meeting07:31
*** yamamoto has quit IRC07:33
*** yamamoto has joined #openstack-meeting07:39
*** ttsiouts has joined #openstack-meeting07:39
*** slaweq has joined #openstack-meeting07:40
*** hyunsikyang has joined #openstack-meeting07:46
*** hyunsikyang__ has quit IRC07:50
*** tesseract has quit IRC07:52
*** tesseract has joined #openstack-meeting07:52
*** ttsiouts has quit IRC07:57
*** e0ne has joined #openstack-meeting08:08
*** ttsiouts has joined #openstack-meeting08:16
*** sridharg has joined #openstack-meeting08:53
*** baojg has quit IRC09:14
*** panda is now known as panda|rover09:17
*** ttsiouts has quit IRC09:20
*** helenafm has joined #openstack-meeting09:35
*** yamamoto has quit IRC10:09
*** yamamoto has joined #openstack-meeting10:20
*** yamamoto has quit IRC10:25
*** pcaruana has quit IRC10:27
*** mmethot has quit IRC10:29
*** ricolin has quit IRC10:39
*** panda|rover is now known as panda|rover|eat11:04
*** njohnston has joined #openstack-meeting11:06
*** bbowen has joined #openstack-meeting11:11
*** ttsiouts has joined #openstack-meeting11:17
*** carloss has joined #openstack-meeting11:19
*** mmethot has joined #openstack-meeting11:19
*** pcaruana has joined #openstack-meeting11:37
*** yamamoto has joined #openstack-meeting11:44
*** yamamoto has quit IRC11:45
*** yamamoto has joined #openstack-meeting11:45
*** ttsiouts has quit IRC11:49
*** ttsiouts has joined #openstack-meeting11:49
*** rsimai_away is now known as rsimai11:51
*** yamamoto has quit IRC12:03
*** ttsiouts has quit IRC12:06
*** panda|rover|eat is now known as panda12:06
*** yamamoto has joined #openstack-meeting12:10
*** panda is now known as panda|rover12:11
*** kashyap has joined #openstack-meeting12:11
*** yamamoto has quit IRC12:19
*** rbudden has joined #openstack-meeting12:20
*** baojg has joined #openstack-meeting12:20
*** janki has quit IRC12:28
*** raildo has joined #openstack-meeting12:37
*** ttsiouts has joined #openstack-meeting12:38
*** tetsuro has joined #openstack-meeting12:41
*** tetsuro has quit IRC12:41
*** ttsiouts has quit IRC12:43
*** lseki has joined #openstack-meeting13:01
*** baojg has quit IRC13:04
*** lbragstad has joined #openstack-meeting13:09
*** ttsiouts has joined #openstack-meeting13:12
*** mriedem has joined #openstack-meeting13:15
*** yamamoto has joined #openstack-meeting13:15
*** artom has quit IRC13:18
*** ttsiouts has quit IRC13:19
*** ttsiouts has joined #openstack-meeting13:20
*** wwriverrat has quit IRC13:22
*** ianychoi has joined #openstack-meeting13:26
*** Luzi has quit IRC13:45
*** takashin has joined #openstack-meeting13:50
*** cdent has joined #openstack-meeting13:53
*** jangutter has joined #openstack-meeting13:56
*** efried has left #openstack-meeting13:56
*** pcaruana has quit IRC13:56
*** efried has joined #openstack-meeting13:56
*** artom has joined #openstack-meeting13:57
* mriedem looks at the clock14:00
efried#startmeeting nova14:00
openstackMeeting started Thu May 16 14:00:30 2019 UTC and is due to finish in 60 minutes.  The chair is efried. Information about MeetBot at
openstackUseful Commands: #action #agreed #help #info #idea #link #topic #startvote.14:00
*** openstack changes topic to " (Meeting topic: nova)"14:00
openstackThe meeting name has been set to 'nova'14:00
artom(my name is)14:00
mriedemslim shady14:00
artom*scratching noises*14:01
* johnthetubaguy lurks until he has to run to a doctors appointment 14:01
artomCell service in the subway ftw14:01
efried#link agenda
efried#topic Last meeting14:02
efried#link Minutes from last meeting:
*** openstack changes topic to "Last meeting (Meeting topic: nova)"14:02
efriedA few items from last time to follow up on...14:02
*** aarents has joined #openstack-meeting14:02
efriedfup efried action to track down owner of review status page
efriedI have gotten as far as finding out where the source is (openstack/reviewday project) but haven't dug in yet.14:02
mriedem             Page refreshed at 2019-05-09 06:38:29 UTC                   466 active reviews14:03
efriedAnyone wants to hack around, knock yourself out. Lemme know what you find.14:03
mriedemmight ping infra (fungi) to see if it's busted14:03
efriedwhy, is that wrong?14:03
mriedemhe's been pinged14:03
efriedoh, yeah, looks like there's a bit over 700 actually open14:04
cdentwhat does that page even mean?14:04
fungiyeah, when our current fires are extinguished hopefully someone can check on status.o.o and find out if there's any error from the cron or whatever that regenerates that content14:04
efriedthat's what we'd like to figure out.14:05
efried^ to cdent14:05
mriedemcdent: we're not sure how the scoring is calculated14:05
efried...and figure out how we can use it.14:05
mriedembut otherwise it's just a place with all the open nova reviews, sortable14:05
efriedsortable by some criteria we don't understand.14:05
mriedemi think part of the heat factor is age14:05
fungiyou may have to dig into the reviewstats source code, but i think the scoring has to do with launchpad bug priority14:05
fungiand age14:05
mriedemand maybe lp heat value, idk14:06
mriedembut that would make sense14:06
efried#link cycle themes are still up for review
efriedThis has a couple +2s and a number of +1s. I'm tempted to say I'll merge it in a week if no objections from this point.14:06
efrieddoes that work?14:07
johnthetubaguy+1 doing the merge14:07
mriedemi'll take a look after this meeting14:08
efriedand last fup, couple of patches were highlighted for review last week.14:08
artomGot dragged away for downstream bugfixes/backports, didn't get a chance to look :(14:08
efriedIt looked like sean-k-mooney was into them as well, but was on vacation last week; I'll poke.14:08
efried#topic Release News14:09
*** openstack changes topic to "Release News (Meeting topic: nova)"14:09
efried#topic Bugs (stuck/critical)14:10
efriedNo Critical bugs14:10
efried#link 78 new untriaged bugs (up 2 since the last meeting):
efried#link 10 untagged untriaged bugs (no change since the last meeting):*&field.status%3Alist=NEW14:10
*** openstack changes topic to "Bugs (stuck/critical) (Meeting topic: nova)"14:10
efriedFrom last week, bug 182708314:11
openstackbug 1827083 in OpenStack-Gate "ERROR: Could not install packages due to an EnvironmentError: HTTPSConnectionPool(host='', port=443): Max retries exceeded with url: /cgit/openstack/requirements/plain/upper-constraints.txt (Caused by NewConnectionError('<pip._vendor.urllib3.connection.VerifiedHTTPSConnection object at 0x7febbf6ae630>: Failed to establish a new connection: [Errno -3] Temporary failure in name resolution',)) in vexxhost-sjc1" [Undecided,Confirmed]
efriedlooks like this has at least been worked around by making vexxhost not use ipv614:11
efriedand then some conditional ipv6ing14:11
efriedbut the bug isn't closed. mriedem, whassa deal, yo?14:11
mriedemmnaser is in china14:12
mriedemso the workaround is forced ipv414:12
mriedemnext steps are up to infra, not me14:12
efriedflatline since those fixes merged, so that's good.14:12
efriedstill open because more permanent solution pending?14:12
mriedemwell we want ipv6 testing in the gate per one of the release goals i believe,14:12
mriedemand that region was all ipv6 until this week i think14:13
mnaserah yes, that.  IPv6 works till it doesn’t and I don’t know why only that hits it. Anyways, I’ll try to dig deeper soon with Indra hopefully.14:13
mriedemso yeah i'm sure people (again, infra) will be working on it14:13
efriedcool. Meantime mitigated, so \o/14:13
mriedemyes thank clarkb14:13
artomBut... the underlying setup can be done with IPv4 even if we then test IPv6 in the tenant networks, no?14:13
mriedemyes, we've had ipv6 testing in the gate with tempest for a long time14:14
clarkbya the switch here was to use external dns via ipv4 instead of ipv614:15
clarkbthe tests themselves can still.use ipv6 internally14:15
clarkbit was external connectivity we struggled with14:15
artomYeah, we're still testing IPv6 correctly14:15
efriedotherwise gate looks pretty healthy (keinehorah, ptoo-ptoo-ptoo)14:16
efried3rd party CI14:16
efried#link 3rd party CI status
efriedironic-tempest-ipa-wholedisk-bios-agent_ipmitool-tinyipa looks bad - anyone know anything about this?14:16
mriedemdtantsur was asking about some ironic job n-cpu logs the other day, but for a stable branch (rocky) i think14:17
mriedemnot sure if that would be related14:17
mriedemlooks like the job is f'ed14:18
efriedalso this
efriednot sure if that's related14:18
mriedemdie 1865 'Timed out waiting for Nova to track 1 nodes'14:18
efriedwhen did that job stop voting?14:18
mriedemlooks like they are waiting for some CUSTOM_GOLD trait to show up14:18
mriedemi don't think it ever was voting14:18
efriedhmph, okay.14:18
mriedemit used to timeout all the time (years ago)14:18
mriedemTheJulia: ^ known issue?14:19
mriedem++ /opt/stack/ironic/devstack/lib/ironic:wait_for_nova_resources:1865 :   die 1865 'Timed out waiting for Nova to track 1 nodes'14:19
mriedemanyway we can sort that out and track it outside of the meeting14:19
efriedAnything else on bugs, gate, CI, etc?14:20
cdentI've raised a flag internally (again) on the lack of health from vmware ci. there was some enthusiam last week about "move everything to zuul v3" but that's dependent on locating some "lost" hardware14:20
mriedemstarlingx has reported what is for them a critical bug
openstackLaunchpad bug 1829062 in StarlingX "nova placement api non-responsive due to eventlet error" [Critical,In progress] - Assigned to Gerry Kopec (gerry-kopec)14:20
mriedemrelated to the eventlet wsgi stuff14:20
mriedemit sounds like the ultimate fix is melwitt's series to drop eventlet usage from the api14:21
efriedwho's qualified to deep-review ^ ?14:21
mriedemi haven't been paying much attention to it, but i know mdbooth has,14:21
*** sean-k-mooney has joined #openstack-meeting14:21
cdentI continue to think (as I said on the review) that we should only scatter gather when there are >2 cells14:21
mriedemit sounds like the open nagging issue is not knowing if a thread is hung or something?14:21
cdentthat's an aspect yes, but mdbooth thinks that shouldn't be a "real" problem14:22
mriedemthere was some talk about down cells behavior with that change and i dropped my testing guide patch for down cells if people want to test that out with melwitt's patches applied14:22
sean-k-mooneymriedem: the concern was if you had several tread hang waiting for a respoce then you could exaust the thread pool14:22
mriedembut we don't actually know if we have a case for hung threads right?14:23
mriedemthis is just conjecture?14:23
*** baojg has joined #openstack-meeting14:23
sean-k-mooneyyes more or less14:23
mriedemwe can find out if a down cell breaks this by testing it with devstack, it's pretty easy14:23
sean-k-mooneyi mean it could happen14:23
mriedemanything can happen...14:23
mriedemso we know eventlet + wsgi is bad14:23
mriedemwe're not sure what can happen with mel's changes, but we can get more info by testing it with a down cell14:24
sean-k-mooneyit was raised by gibi and dan on the review which is why we are giving it credence14:24
mriedemok gibi is out for a bit14:24
mriedemi'm not sure what dansmith's current thoughts are on it14:24
* dansmith is on a call14:24
mriedemsounds like next step is testing her patches with down cells?14:24
sean-k-mooneyfor a donw cell it should not be an issue14:24
dansmithit needs to be multiple down cells with lots of api traffic,14:25
sean-k-mooneythe edgecase was if the requst hang after the connection to the cell has started14:25
dansmithbut I also kinda don't see the point of doing this tbh, and I thought there were a couple things we could do to get the monkey patching in order to fix the acute problem14:25
dansmithI'm super wary of having two threading models in code that doesn't have a strong separation... asking for trouble, IMHO,14:25
dansmithbut I don't really have time to dig deep on this14:26
sean-k-mooneywell the issue is the api was not monkey patched before when runing under wsgi and wsgi + eventlest has issues14:26
mriedemok i don't know what the alternatives are that dan's referring to, like i said i'm not heavily involved in this one14:26
mriedemwe could punt and only scatter/gather if there are >2 cells, but that just punts the problem to someone like cern to hit it when they get to stein14:27
mriedemso i'm not in love with that option personally14:27
mriedemanyway i guess we can move on14:28
mriedemseems like by now people would have figured out problems with wsgi and multi-threading in python?14:28
* mriedem re-writes nova-api in EJBs!14:28
artomWell, eventlet isn't real multithreading...14:28
cdentyes, several ideas are discussed on the review, but nothing has congealed out of the goo14:28
mriedemartom: i mean without eventlet14:28
edleafeEJB does sound promising!14:28
mriedemif there are dangers with wsgi + python std lib concurrency stuff14:29
sean-k-mooneyby the way the work around for people untill we fix this is to go back to running the api via the console scipt command14:29
efriededleafe is going to rewrite nova with graph databases14:29
artomIt'd immediately solve NUMA in placement ;)14:29
edleafeefried: s/graph/distributed14:29
mriedemi was also going to kill the nova-api eventlet stuff about a year ago...14:29
sean-k-mooneythat is a performacne hit but it works14:29
mriedemgood thing i got busy14:29
efriedmoving on.14:29
efried#topic Reminders14:29
efriedSummit, Forum, and PTG happened14:29
efried#link PTG summary emails (searching for ".*[nova].*[ptg] Summary" will get most of them)
*** openstack changes topic to "Reminders (Meeting topic: nova)"14:29
efriedAny other reminders?14:30
efried#topic Stable branch status14:31
efried#link Stein regressions:
efriedNo change since last week (one bug still open, bug 1824435, no great solution yet)14:31
*** openstack changes topic to "Stable branch status (Meeting topic: nova)"14:31
openstackbug 1824435 in OpenStack Compute (nova) stein "fill_virtual_interface_list migration fails on second attempt" [High,Triaged]
efried#link stable/stein:
efried#link stable/rocky:
efried#link stable/queens:
efried#link stable/pike:
efriedThere was a question from cdent a few days ago about backporting something to ocata. It sounded like it was de-confusing a message?14:31
cdentmriedem and I worked out it wasn't worth doing14:32
efriedokay, cool.14:32
cdentas it wouldn't be a backport as pike changed it14:32
cdents/it/it alot/14:32
efriedso it would be an ocata-only change, and noncritical, so punt?14:32
*** yamamoto has quit IRC14:33
efriedAnything else stable-related?14:33
cdent(yes on the punt)14:33
*** yamamoto has joined #openstack-meeting14:33
*** yamamoto has quit IRC14:33
efried#topic Sub/related team Highlights14:34
*** openstack changes topic to "Sub/related team Highlights (Meeting topic: nova)"14:34
efriedcdent was traveling, but we had a brief meeting on Monday without him14:34
efried#link placement meeting log
efriedThe main nova-related things were...14:34
*** yamamoto has joined #openstack-meeting14:34
efried#link WIP spec for nested magic
efried#link spec for rg/rp mapping
efriedIt would be nice if nova folks could have a look at those ^ and make sure they're going to satisfy nova use cases14:34
* artom adds the nested magic one to this queue14:35
efriedalso look with an eye for how we could simplify ^ and still meet the use cases :) (especially the nested magic one)14:35
artomWill get to it when all the downstream fires have been put out. So, next year :P14:35
efriedartom: it's a scintillating read, I promise you.14:36
efriedcdent: anything else placement-that-affects-nova you want to go over?14:36
*** d34dh0r53 has quit IRC14:36
cdentno sir14:36
efriedanyone else?14:36
efriedAPI (gmann)14:36
efriedno notes in the agenda, no gmann in the channel. Anyone have anything here?14:36
*** d34dh0r53 has joined #openstack-meeting14:37
*** armax has joined #openstack-meeting14:37
*** pcaruana has joined #openstack-meeting14:37
*** Lucas_Gray has joined #openstack-meeting14:37
efried#topic Stuck Reviews14:38
*** openstack changes topic to "Stuck Reviews (Meeting topic: nova)"14:38
efriednothing on the agenda. Anyone?14:38
efried#topic Review status page14:38
*** openstack changes topic to "Review status page (Meeting topic: nova)"14:38
*** yamamoto has quit IRC14:39
efriedwe talked about this above. fup with infra to make sure it's working. fup hacking the repo to see wtf it's doing. fup brainstorm on whether/how to use it to make the world a better place.14:39
efried#topic Open discussion14:39
*** openstack changes topic to "Open discussion (Meeting topic: nova)"14:39
efriedin the spirit of being good little community citizens, I have started14:39
efried#link WIP TC Vision Reflection
efried#help with this, please.14:39
efriedAny other opens?14:40
jangutterSorry for asking a question that might already have been answered: has anyone managed to dig up a mirror to the train ptg etherpad somewhere?14:40
efriedjangutter: Yeah, sean-k-mooney sent a copy (undecorated, unfortunately) to the ML14:41
efried#link nova train ptg etherpad backup
*** d34dh0r53 is now known as Guest2890414:41
jangutterefried: thanks!14:41
aspiers#link alternative etherpad backup from infra team
efriedthanks aspiers.14:42
efriedunfortunately, same lack of formatting, but content is there.14:42
efriedwe had lost the authorship colors in the various transitions anyway, so the main loss is just the strikethroughs14:43
efriedand we had struck through everything pretty much anyway, so...14:43
efriedOkay, anything else before we wrap?14:43
artomIs there going to be an open discussion thing?14:43
cdentyou're in it14:43
efriedartom: your mic14:43
artomWanted to quickly ask about stable branch same company approvals - for instance, who would be able to +W
efriedo right14:44
*** gagehugo has joined #openstack-meeting14:44
efriedmy knee-jerk reaction is that same-company approvals don't really apply to backports.14:44
artom(Are we writing down the master branch policy anywhere? Might be good to add the stable branch policy as well)14:44
efriedstable decisions are based on suitability for backporting; the technical decisions were already made on the master patch.14:44
sean-k-mooneyartom: we decided to not write it down14:45
efried(except for the long email thread that's written down)14:45
artomsean-k-mooney, aha, keep it in the cloud ;)14:45
mriedemas a stable core i would be able to +2 it, if i don't -1 it first14:45
efriedartom: in case you missed it:14:45
efried#link same-company approvals ML thread
artommriedem, right, but you're not RH14:45
sean-k-mooneymriedem: :)14:45
artomI was more wondering if melwitt, for example, could come along and +W it14:46
mriedemshe should not IMO14:46
artomefried, yeah, I followed that14:46
mriedemespecially given this change is arguably a feature14:46
sean-k-mooneyartom: am she could if a non redhater was the first +214:46
artomSo stable is also 2 +2s?14:46
mriedemnot necessarily14:46
efriedassume we count the author of the master patch, not the proposer of the backport, as the author of record for purposes of same-company approval decisions?14:47
mriedembackport from a stable core is generally considered a proxy +2 if it's a clean backport14:47
mriedemefried: the original author of this change is RH14:47
efriedright, I'm confirming that the original author, not the backport proposer, is who we care about when talking about same-company.14:47
artommriedem, ah, so Lee backports a thing, that's a +2 in the bag if it's clean14:47
mriedemartom: for some things yes14:48
sean-k-mooneyefried: yes it is14:48
mriedemnot really for this b/c it's big as hell14:48
artomDammit, nothing it black and white!14:48
mriedemand feature-y14:48
mriedemsorry, i'll get right to work on that law degree14:48
efriedartom: That's why we don't want to write it down.14:48
artomefried, fair enough.14:48
artomAnyways, don't want to take up too much time. My takeaway is, use your judgment and don't piss off mriedem  :D14:48
sean-k-mooneyany way i think people are awre of this patch and can review it now14:49
efriedso the answer to the general question is "case by case". And sounds like the answer for this specific patch is "let's not allow same-company".14:49
artomsean-k-mooney, for the record, I was using the patch as an example because I got asked that question about that patch14:49
mriedemis there a day where you guys downstream aren't talking about this same company approval thing?14:50
artommriedem, no, we all dream about it14:50
sean-k-mooneyya well for that patch the aser is mriedem johnthetubaguy or claudiu can +w14:50
mriedemjust start forking the code and be done with it14:50
mriedemclaudiu isn't really upstream anymore14:50
mriedemanyway, can we move on?14:51
artomDammit dude, the reason I'm asking here is because I actually care about upstream :)14:51
mriedemlet's hug14:51
artomBring it in brah14:51
mriedemi know you do, but i shoot the messenger14:51
efriedcan we end on another rap tune?14:51
efriedyou don't14:51
efriedwanna f with mriedem14:51
efriedcause mriedem14:51
efriedwill f'in hug you14:51
artomGot 99 problems but upstream ain't one?14:52
*** openstack changes topic to "OpenStack Meetings ||"14:52
openstackMeeting ended Thu May 16 14:52:05 2019 UTC.  Information about MeetBot at . (v 0.1.4)14:52
openstackMinutes (text):
*** sean-k-mooney has left #openstack-meeting14:52
*** jangutter has left #openstack-meeting14:52
*** helenafm has left #openstack-meeting14:53
gagehugo#startmeeting security15:00
openstackMeeting started Thu May 16 15:00:43 2019 UTC and is due to finish in 60 minutes.  The chair is gagehugo. Information about MeetBot at
openstackUseful Commands: #action #agreed #help #info #idea #link #topic #startvote.15:00
*** openstack changes topic to " (Meeting topic: security)"15:00
openstackThe meeting name has been set to 'security'15:00
gagehugo#link agenda15:01
efried(kashap, interested?)15:01
*** takashin has left #openstack-meeting15:01
efriedkashyap: typo ^15:01
gagehugo#topic spam pinging15:03
*** openstack changes topic to "spam pinging (Meeting topic: security)"15:03
* gagehugo lost the link he was saving15:03
*** artom has quit IRC15:03
fungifwiw, i got a highlight on this buffer when you did the startmeeting, so no need to ping me .;)15:04
efriedif someone can figure out how to do highlight regexes in thunderbird IRC client, let me know please.15:04
fungi(also the remind utility lets me know when the meeting is due to start)15:04
gagehugothe ptl tips and tricks session in Denver, it was mentioned that spamming people is bad IRC etiquette, so I'll start sending a reminder in openstack-security prior to the meeting15:05
gagehugoand less spamming15:05
* gagehugo needs to setup highlight for startmeeting15:05
gagehugo#topic On reporting CPU flags that provide mitiation15:06
*** openstack changes topic to "On reporting CPU flags that provide mitiation (Meeting topic: security)"15:06
gagehugofungi: was this you?15:06
fungiwell, i added it to the agenda because folks discussing on that ml thread wanted feedback from security-minded members of our community15:07
*** baojg has quit IRC15:08
efriedI'll represent, since kashyap and sean-k-mooney aren't around.15:08
efriedI'll summarize the issue in terms of the bit we need security SIG's advice on.15:09
efriedCPUs that are vulnerable to spectre/meltdown (or other things) have ways you can find out about that if you're on the host. CPU flags, queries in sysfs, etc.15:10
efriedNova has the ability to decorate resource providers with traits and then users can cause instances to be scheduled to (or avoid) hosts that have those traits.15:10
efriedSo the security question is:15:11
efrieddoes it pose a security problem for nova to expose traits like "IS VULNERABLE" or "HAS FIX FOR VULNERABILITY" thus allowing users to say "Schedule me to a host that IS VULNERABLE" ?15:11
efriedMy take on this was that it's not a whole lot worse than if you just have that host in your cloud and the user could land on it randomly.15:11
efriedi.e. if there's a vulnerability, there's a vulnerability; you're not making it better by making it slightly harder to exploit.15:12
fungithe only reason i can see for surfacing those cpu flags to users is operating systems/kernels which are making behavioral decisions based on their absence/presence. the linux kernel for example has specific optimizations it can perform when it sees the cpu flags which indicate patched microcode for some speculative execution flaws15:13
fungii don't think they're relevant to scheduling decisions15:13
efriedYes. Or I might actually want to land on a host that has one of these flags because a) I trust my VM, and b) it gets me better performance.15:13
efriedfurther, if we expose the trait, security-auditing agents (outside of nova) would be able to key on it to e.g. completely disable vulnerable hosts15:14
gagehugoIf we want to ensure non-vulnerable hosts, then being able to specify would be useful15:15
*** artom has joined #openstack-meeting15:15
fungimaybe i'm misunderstanding, but as a user if given the option to schedule my workload to a vulnerable or non-vulnerable host, i'll always just choose the latter15:16
fungiunless the idea is that providers might charge more money for non-vulnerable service15:16
gagehugoor you have the auditing agents that can check/disable IS VULNERABLE ones15:16
Tenguor maybe wants to get better perfs for some non-important tasks, since meltdown correction slows down the perfs15:16
cdentwhat Tengu said is the main thing15:17
fungior that for a private deployment i might select some portion of my workloads which aren't at risk of whatever a given vulnerability is and use that to distribute better15:17
fungibut in my opinion, none of those are security-relevant choices, they're choices which just happen to be related to security-oriented information15:18
efriedpoint is, there are reasons to want to schedule to a "non-vulnerable" host, and there are reasons to want to schedule to a "vulnerable" host even if you're not a hacker.15:18
efriedAnd if you're a hacker, not having these traits exposed doesn't prevent you from being scheduled to a vulnerable host. Something else has to do that.15:18
efried(whoah, that was a lot of negatives)15:19
gagehugoare there any downsides to exposing those then?15:19
fungimy only real concern in this is that we expose the appropriate flags in the guest instances so that kernels know how to operate. scheduling decisions aren't anything which i care about either way. if users want to choose to run workloads on unpatched servers and providers want to make that an option then those are the operators and users you want to hear from15:20
gagehugowe don't want "hackers" to be able to target vulnerable systems, but they can still get scheduled to vulnerable systems anyway as is15:25
gagehugoimo I would assume operators would typically want to use NOT VULNERABLE for most cases (outside of non-risk workloads for performance gains)15:27
fungiyeah, if a provider doesn't want anyone scheduled to vulnerable systems, disable/patch those systems. scheduler roulette is not a security measure15:28
*** samP has quit IRC15:28
fungii hear plenty of stories from providers of malicious users spinning up batches of servers, often with some anti-affinity, to attempt to land on particular hosts15:29
*** samP has joined #openstack-meeting15:29
efriedCool, sounds like we have agreement.15:30
efriedfungi: would you like to be the one to address this on the thread, or would you like me to do so?15:31
fungii can add replying to my to do pile, but it may not happen until tomorrow15:31
efriedI've already got a response started for other pieces, I can add this if you like.15:31
fungioh, in that case feel free and i'll just mee-too yours ;)15:32
efried(should have led with that, but forgot I hadn't already sent it :)15:32
efriedight, will do. Thanks y'all.15:32
kashyapefried: Sorry, I was (and am) stuck in conflicting meetings :-(15:34
fungikashyap: no worries, me too!15:34
efriedkashyap: no worries, we figured it out, stand by for ML response.15:34
kashyapefried: Yeah, noted.  Most appreciated.  I will pay attention to the list15:35
*** ttsiouts has quit IRC15:35
kashyap(Better to sort out on the e-mail these things.)  Thanks for all the responses, everyone.15:35
*** cdent has left #openstack-meeting15:36
gagehugocool, I can chime in as well if needed15:37
gagehugo#topic Summit BoF Session15:38
*** openstack changes topic to "Summit BoF Session (Meeting topic: security)"15:38
*** _d34dh0r53_ is now known as d34dh0r5315:38
*** e0ne has quit IRC15:38
gagehugothe security SIG has a BoF session at the summit, the notes from that were mailed out, but can also be found here15:38
fungithat week was such a blur... did i show up to the bof?15:38
gagehugoyes haha15:38
*** ricolin has joined #openstack-meeting15:38
gagehugoI was writing on the tiny whiteboard15:39
*** macza has joined #openstack-meeting15:39
gagehugo#topic open discussion15:39
*** openstack changes topic to "open discussion (Meeting topic: security)"15:39
gagehugoI have pangolin stickers (Security mascot) if anyone wants some, will need to figure out mailing15:40
gagehugoreach out/ping me if you want some15:40
gagehugootherwise that's all I got15:40
gagehugofungi: anything else?15:40
funginope, not from me at least15:41
gagehugothanks everyone!15:41
*** openstack changes topic to "OpenStack Meetings ||"15:41
openstackMeeting ended Thu May 16 15:41:31 2019 UTC.  Information about MeetBot at . (v 0.1.4)15:41
fungithanks gagehugo!15:41
openstackMinutes (text):
gagehugoTengu yes15:41
TenguSTICKERS :D15:41
fungileft over from when there was a security team instead of a sig15:41
Tenguany way to ship to Switzerland, or maybe push one in Brno office since I'm going there in June?15:42
gagehugopossibly, I can look into it15:43
*** pcaruana has quit IRC15:47
*** yamamoto has joined #openstack-meeting15:50
*** ttsiouts has joined #openstack-meeting15:51
*** wwriverrat has joined #openstack-meeting15:54
*** ttsiouts has quit IRC15:55
*** yamamoto has quit IRC15:58
*** kopecmartin is now known as kopecmartin|off16:01
*** sridharg has quit IRC16:02
*** dims has quit IRC16:05
*** tesseract has quit IRC16:05
*** mattw4 has joined #openstack-meeting16:28
*** ricolin has quit IRC16:44
*** ttsiouts has joined #openstack-meeting16:50
*** bauzas has quit IRC16:54
*** bauzas has joined #openstack-meeting16:57
*** dims has joined #openstack-meeting17:03
*** lbragstad has quit IRC17:05
*** tssurya has quit IRC17:10
*** dims has quit IRC17:12
*** lbragstad has joined #openstack-meeting17:13
*** panda|rover is now known as panda|rover|off17:13
*** dims has joined #openstack-meeting17:13
*** Lucas_Gray has quit IRC17:18
*** ttsiouts has quit IRC17:24
*** _alastor_ has quit IRC17:29
*** _alastor_ has joined #openstack-meeting17:31
*** dklyle has quit IRC17:34
*** electrofelix has quit IRC17:38
*** pcaruana has joined #openstack-meeting17:39
*** rubasov has quit IRC18:04
*** ianychoi has quit IRC18:15
*** ralonsoh has quit IRC18:20
*** rbudden has quit IRC18:31
*** armstrong has joined #openstack-meeting18:32
*** trident has quit IRC18:51
*** trident has joined #openstack-meeting18:52
*** iyamahat has joined #openstack-meeting18:53
*** rbudden has joined #openstack-meeting18:57
*** rbudden has quit IRC18:59
*** ttsiouts has joined #openstack-meeting19:22
*** yamahata has joined #openstack-meeting19:31
*** gagehugo has left #openstack-meeting19:34
*** pcaruana has quit IRC19:52
*** ttsiouts has quit IRC19:54
*** radeks_ has quit IRC19:58
*** armstrong has quit IRC20:03
*** whoami-rajat has quit IRC20:05
*** e0ne has joined #openstack-meeting20:10
*** e0ne has quit IRC20:12
*** Lucas_Gray has joined #openstack-meeting20:19
*** ijw has joined #openstack-meeting20:46
*** raildo has quit IRC20:48
*** diablo_rojo has joined #openstack-meeting20:48
*** armax has quit IRC20:55
*** artom has quit IRC20:57
*** trident has quit IRC20:59
*** trident has joined #openstack-meeting21:00
*** ttsiouts has joined #openstack-meeting21:01
*** enriquetaso has quit IRC21:17
*** slaweq has quit IRC21:18
*** enriquetaso has joined #openstack-meeting21:21
*** hongbin has joined #openstack-meeting21:31
*** ttsiouts has quit IRC21:33
*** mriedem has quit IRC21:53
*** armax has joined #openstack-meeting21:55
*** hongbin has quit IRC22:39
*** ijw has quit IRC22:52
*** macza has quit IRC22:57
*** rcernin has joined #openstack-meeting23:04
*** slaweq has joined #openstack-meeting23:05
*** _erlon_ has quit IRC23:05
*** ttsiouts has joined #openstack-meeting23:06
*** slaweq has quit IRC23:10
*** ttsiouts has quit IRC23:11
*** slaweq has joined #openstack-meeting23:11
*** Lucas_Gray has quit IRC23:14
*** Lucas_Gray has joined #openstack-meeting23:15
*** slaweq has quit IRC23:16
*** dklyle has joined #openstack-meeting23:21
*** diablo_rojo has quit IRC23:24
*** dklyle has quit IRC23:35
*** Lucas_Gray has quit IRC23:43
*** artom has joined #openstack-meeting23:45
*** yamamoto has joined #openstack-meeting23:49
*** diablo_rojo has joined #openstack-meeting23:52
*** yamamoto has quit IRC23:53

Generated by 2.15.3 by Marius Gedminas - find it at!