Tuesday, 2017-04-18

« Monday, 2017-04-17 Index Wednesday, 2017-04-19 »
johnsomOk, I am going to be away for a while too.  Maybe tomorrow I will setup an official coverage gate on infra and leave it at that.00:04
*** mdurrant_ has quit IRC00:11
*** mdurrant_ has joined #openstack-lbaas00:12
rm_workhmm k00:26
*** JudeC has quit IRC00:34
*** reedip_ has joined #openstack-lbaas00:36
rm_workcool, that worked: http://logs.openstack.org/93/457393/1/check/gate-octavia-v1-dsvm-scenario-kvm-centos.7-ubuntu-xenial-nv/326f231/console.html#_2017-04-18_00_00_50_02923800:37
rm_workand is merging00:37
rm_workin that, at least we get the right exception message :/00:37
rm_worknot that the test suite passes00:37
rm_workstill waiting on the py3x fix but it is looking promising00:38
rm_workas the test is still running00:38
rm_workugh new problem00:40
openstackgerritAdam Harwell proposed openstack/neutron-lbaas master: Trying a different strategy for fixing py3 errors in testing  https://review.openstack.org/45739400:43
rm_workk easy enough00:43
openstackgerritMerged openstack/octavia master: Scenario base uses the wrong exception lib  https://review.openstack.org/45739300:52
*** JudeC has joined #openstack-lbaas01:03
*** strigazi has quit IRC02:30
*** strigazi has joined #openstack-lbaas02:31
*** JudeC has quit IRC02:42
*** reedip_ has quit IRC02:42
*** sanfern has joined #openstack-lbaas02:52
*** cody-somerville has joined #openstack-lbaas03:16
*** amotoki has joined #openstack-lbaas03:22
*** bzhao has quit IRC03:36
*** links has joined #openstack-lbaas03:40
*** dlundquist has quit IRC03:41
*** dlundquist has joined #openstack-lbaas03:45
*** dlundquist has quit IRC04:32
*** bzhao_ has joined #openstack-lbaas04:39
*** gcheresh has joined #openstack-lbaas04:46
*** kobis has joined #openstack-lbaas04:46
*** kobis has quit IRC04:59
*** cody-somerville has quit IRC05:26
*** chandankumar has quit IRC05:44
*** kobis has joined #openstack-lbaas05:45
openstackgerritAdam Harwell proposed openstack/neutron-lbaas master: Trying a different strategy for fixing py3 errors in testing  https://review.openstack.org/45739405:57
*** sanfern has quit IRC06:04
*** rcernin has joined #openstack-lbaas06:06
*** reedip_afk has quit IRC06:19
*** reedip_afk has joined #openstack-lbaas06:31
*** JudeC has joined #openstack-lbaas06:35
*** tesseract has joined #openstack-lbaas06:40
*** bzhao_ has quit IRC06:49
*** voelzmo has joined #openstack-lbaas06:59
*** pcaruana has joined #openstack-lbaas06:59
*** voelzmo has quit IRC07:08
*** aojea has joined #openstack-lbaas07:29
*** aojea_ has joined #openstack-lbaas07:30
*** aojea has quit IRC07:33
*** cody-somerville has joined #openstack-lbaas08:03
*** cody-somerville has quit IRC08:03
*** cody-somerville has joined #openstack-lbaas08:03
*** dmk0202 has joined #openstack-lbaas08:22
*** JudeC has quit IRC08:29
*** openstackgerrit has quit IRC08:33
*** JudeC has joined #openstack-lbaas08:48
*** JudeC has quit IRC08:54
*** JudeC has joined #openstack-lbaas08:55
*** cody-somerville has quit IRC08:57
*** derekjhyang has quit IRC09:03
*** JudeC has quit IRC09:14
*** gongysh has joined #openstack-lbaas09:37
*** krypto has joined #openstack-lbaas09:55
*** gongysh has quit IRC10:31
*** reedip_ has joined #openstack-lbaas11:02
*** openstackgerrit has joined #openstack-lbaas11:17
openstackgerritNir Magnezi proposed openstack/octavia master: Auto detect haproxy user_group  https://review.openstack.org/42939811:17
*** med_ has quit IRC11:28
*** Guest45512 has joined #openstack-lbaas11:31
*** voelzmo has joined #openstack-lbaas11:37
*** chlong has joined #openstack-lbaas12:13
*** gongysh has joined #openstack-lbaas12:37
*** gongysh has quit IRC12:38
*** gongysh has joined #openstack-lbaas12:39
*** gongysh has quit IRC12:40
*** aojea has joined #openstack-lbaas13:08
*** aojea_ has quit IRC13:11
*** sanfern has joined #openstack-lbaas13:12
*** pcaruana has quit IRC13:27
*** KeithMnemonic has joined #openstack-lbaas13:30
*** aojea_ has joined #openstack-lbaas13:30
*** aojea has quit IRC13:34
*** links has quit IRC13:35
beaglesin case it hasn't already been brought up, a db initialization problem showed up in puppet CI https://bugs.launchpad.net/puppet-octavia/+bug/168362813:57
openstackLaunchpad bug 1683628 in puppet-octavia "Ci is break" [Undecided,In progress] - Assigned to zhongshengping (chdzsp)13:57
*** Guest45512 is now known as med_14:05
*** med_ has quit IRC14:05
*** med_ has joined #openstack-lbaas14:05
*** reedip_ has quit IRC14:07
*** reedip has joined #openstack-lbaas14:08
*** pcaruana has joined #openstack-lbaas14:10
*** fnaval has joined #openstack-lbaas14:29
*** fnaval has quit IRC14:35
*** aojea_ has quit IRC14:40
*** fnaval has joined #openstack-lbaas14:49
*** belharar has joined #openstack-lbaas14:56
*** ianychoi has quit IRC15:00
*** rcernin has quit IRC15:09
*** gcheresh_ has joined #openstack-lbaas15:13
*** gcheresh has quit IRC15:13
*** belharar has quit IRC15:17
johnsombeagles This is really strange, our gates are running the migrations just fine15:21
*** catintheroof has joined #openstack-lbaas15:22
johnsomThat is very odd, I am not seeing a problem in our master branch15:31
*** kobis has quit IRC15:37
*** pcaruana has quit IRC15:41
*** fnaval has quit IRC15:49
*** fnaval has joined #openstack-lbaas15:53
*** JudeC has joined #openstack-lbaas15:53
*** fnaval_ has joined #openstack-lbaas15:54
*** fnaval has quit IRC15:57
*** aojea has joined #openstack-lbaas15:59
*** kobis has joined #openstack-lbaas15:59
*** voelzmo has quit IRC16:05
*** voelzmo has joined #openstack-lbaas16:06
*** voelzmo has quit IRC16:10
rm_workjohnsom: looks like I finally got it: https://review.openstack.org/#/c/457394/16:32
johnsomNice16:33
johnsom+2'd16:33
johnsomI've been messing with this strange migration thing.  It seems to be centos related16:33
rm_workhmm16:34
johnsomThough I'm running into trouble getting alembic to run right on centos16:35
*** aojea has quit IRC16:36
beaglesis nmagnezi around? I think he got a heads up on it as well through a different - I don't know if he has some other insight into the weird migration thing or not16:37
*** aojea has joined #openstack-lbaas16:37
johnsomYeah, he might have more info.  I created a centos vm, but alembic isn't finding the mysql plugin so, I'm missing something16:38
*** strigazi is now known as strigazi_AFK16:39
*** strigazi_AFK is now known as strigazi16:41
*** aojea has quit IRC16:41
*** gcheresh_ has quit IRC16:46
johnsomOk, got alembic going, but the "upgrade head" succeeded....16:47
johnsomNo idea why the puppet gate is having a problem16:47
rm_workhmm16:49
rm_workour centos gate has been failing too, but for a different reason16:49
rm_worknot sure why it's failing to connect yet16:49
johnsomIt passed in our gate, passed their gate ubuntu, passes local both ubuntu and centos 7, and history shows a good chain16:49
beaglesweird16:50
*** harlowja_ has joined #openstack-lbaas16:50
*** harlowja has quit IRC16:52
*** dmk0202 has quit IRC16:52
rm_workyeah so in puppet it seems like it might be creating the DB twice?16:52
johnsomYeah, that is my guess as well.16:52
johnsomIt's odd that it wouldn't fail earlier in the chain though16:53
rm_workor rather, just the one16:53
rm_workfac584114642 -> 034b2dc2f3e0, modernize_l7policy_fields16:53
rm_worklike it didn't record that it was at the latest16:53
rm_workbut thinks it is one back16:53
johnsomYeah, that is a really strange migration16:54
*** amotoki has quit IRC16:57
*** cody-somerville has joined #openstack-lbaas17:28
*** cody-somerville has quit IRC17:28
*** cody-somerville has joined #openstack-lbaas17:28
*** sanfern has quit IRC17:30
openstackgerritJude Cross proposed openstack/octavia master: Fix health_manager to exit without waiting  https://review.openstack.org/45642017:35
*** krypto has quit IRC17:42
rm_workhmm, busy trying to melt my laptop running tests --until-failure lol17:44
johnsomYeah, those really spike the CPU...17:44
rm_workugh restarted Chrome and the log I was looking at for the HM failure was from the Sonar gate >_<17:46
rm_workso that's gone17:46
johnsomYeah I looked into that....17:46
johnsomI am going to put together an official infra coverage gate today17:47
*** dlundquist has joined #openstack-lbaas17:58
johnsomI have that log still open, want me to post it in an pastebin?18:01
johnsomhttp://paste.openstack.org/show/607036/18:02
*** blogan_ has joined #openstack-lbaas18:04
*** blogan has quit IRC18:05
*** blogan_ is now known as blogan18:06
rm_workthanks18:06
rm_workjohnsom: i've been running this for about 30 minutes and no failures :/ how long did it take you18:08
johnsomAbout that long.  However, if I tried to limit the tests run, I never got it to fail18:08
*** gcheresh_ has joined #openstack-lbaas18:11
johnsomI also saw two different failures, one with the updated_at and one with18:13
johnsomhttps://www.irccloud.com/pastebin/gAJc8tNO/18:13
rm_worklol ok yeah18:16
rm_worki was limiting to only octavia.tests.functional.api.v2.test_health_monitor18:17
rm_workthat is dumb18:17
johnsomYeah, never got it to fail, but I have NO idea why18:17
rm_workgoing to let me CPU cool down for a minute and then i'll try again with the whole suite18:17
*** kobis has quit IRC18:19
*** kobis has joined #openstack-lbaas18:19
*** kobis has quit IRC18:20
*** kobis has joined #openstack-lbaas18:20
*** kobis has quit IRC18:20
*** kobis has joined #openstack-lbaas18:21
*** kobis has quit IRC18:21
*** kobis has joined #openstack-lbaas18:22
*** kobis has quit IRC18:22
*** tesseract has quit IRC18:22
rm_workjohnsom: i bet it has to do with this stupid handler mock workaround thing18:39
rm_workbut i have no idea why that is necessary18:39
rm_workonly for HM, L7P and Pools18:40
rm_workeverything else works fine with the regular handler mock18:40
rm_workI don't understand what is different18:40
rm_workone thing they all have in common is using a Pool? :/18:43
rm_workbut nm, so does Listener and it works fine18:46
rm_workfff18:46
*** kobis has joined #openstack-lbaas18:54
rm_workwhat happened to gerritbot18:59
johnsomI don't know, did something not pop up?19:00
rm_workwe had several merges go through...19:01
rm_worki didn't see them19:01
rm_workdid you?19:01
rm_workoh nm19:01
rm_workthey didn't actually go through >_<19:02
rm_workhttp://logs.openstack.org/94/457394/4/gate/gate-neutron-lbaasv2-dsvm-api-ubuntu-xenial/128ff89/console.html19:02
rm_workkeystone failing19:02
johnsomYeah, was just looking at that.19:03
rm_workboth octavia and n-lbaas19:03
rm_workdid keystone merge something today19:03
rm_workor release19:03
johnsomI think it is about time to run out and grab lunch....19:03
rm_worklol19:03
johnsomMust have been19:03
JudeClol19:03
rm_workdoes that test use release versions or master19:05
johnsomIt should be released versions19:05
rm_workerr19:05
rm_worki don't see it being installed19:06
rm_workdid it like19:06
rm_workget removed from some default list19:06
johnsomMaybe keystone is a special snowflake since it is so early in devstack19:06
johnsomYeah, it looks like keystone is trying the proxy pass idea that was floating on the mailing list.19:12
rm_workwait how is keystone even packaged19:13
rm_worki can't find it on pypi19:13
rm_workah so you figured it out?19:13
johnsomThis changed from last time I looked at it: http://logs.openstack.org/50/456950/1/gate/gate-neutron-lbaasv2-dsvm-api-ubuntu-xenial/1d76b95/logs/apache_config/keystone-wsgi-public.conf.txt.gz19:13
rm_workis that the cause?19:13
johnsomThis could be a devstack change19:13
rm_workah19:14
rm_workhttps://review.openstack.org/#/c/456344/19:15
rm_workis keystone HTTPS now maybe?19:16
rm_workwe're hitting HTTP19:16
johnsomI think the port is gone19:16
rm_workhmm19:16
*** kobis has quit IRC19:23
*** fnaval_ has quit IRC19:25
*** fnaval has joined #openstack-lbaas19:29
openstackgerritSean Dague proposed openstack/neutron-lbaas master: Use OS_AUTH_URL instead of baked in defaults  https://review.openstack.org/45778119:30
rm_workjohnsom: ^^19:34
rm_workalso: got the error19:34
rm_workin HM19:34
rm_workit is what I thought19:34
rm_workjust trying to figure out how the heck it's happening19:35
*** fnaval has quit IRC19:35
rm_workthis should be impossible19:41
rm_workbut I have proof it's happening19:41
rm_workI feel like when that happens, you should win a prize19:41
*** aojea has joined #openstack-lbaas19:49
rm_workok so this problem will go away if we can figure out why this stupid handler_mock bug exists19:50
rm_workand just fix it so we can use the normal handler_mock19:50
rm_workinstead of this stupid bug workaround thing19:50
johnsomSee, my plan of going to lunch and it will be fixed worked..  Grin19:55
rm_workok so now help me with this19:56
rm_workthis is SOOOOO fucking weird19:56
rm_workJust replace your entire test_pool.py with this:19:57
rm_workor, actually, sec19:57
rm_workhttp://paste.openstack.org/show/607046/20:01
rm_workjust make a new file20:01
rm_workput that in it20:01
rm_workrun it20:01
rm_workthat's a replication of the stupid bug20:02
rm_workthe pecan api handler thing isn't resetting between tests20:05
rm_workthe way it is supposed to20:05
rm_workbut only for specific test files???20:05
rm_workunless it's doing it for the rest too20:05
rm_workand they just aren't catching it20:05
rm_workyep fff i bet that's it20:06
rm_workyep20:11
rm_workffff20:11
rm_worknone of them work20:11
rm_workit just happened that we didn't *have* those tests for the other types20:11
openstackgerritMichael Johnson proposed openstack/octavia master: Update keystone URL to be consistent  https://review.openstack.org/45778820:21
openstackgerritAdam Harwell proposed openstack/neutron-lbaas master: Use OS_AUTH_URL instead of baked in defaults  https://review.openstack.org/45778120:22
rm_workjohnsom: note that it doesn't work like that20:23
rm_workneeds to be versioned20:23
rm_workI just pushed up the fix20:23
rm_workoh, though hold on, what does KEYSTONE_AUTH_URI come from20:23
johnsomHa, ok20:23
johnsomI just asked Sean about that in the qa channel20:24
rm_workyeah he'd mentioned that possibly being an issue earlier20:24
rm_workand i said i'd track it20:24
rm_workand it was20:24
rm_workthough i am about to throw my laptop at a brick wall20:24
rm_workthe issue is that pecan doesn't actually reload completely20:31
*** openstackgerrit has quit IRC20:33
johnsomBTW, octavia has the version auto-detect stuff so the version isn't needed in the url20:35
rm_workah, cool20:37
*** mdurrant__ has joined #openstack-lbaas20:45
*** mdurrant_ has quit IRC20:48
rm_workjohnsom: ok wtf20:48
rm_workI found the "issue"20:48
rm_workwhen you create a new RootController object20:49
rm_workit somehow always reuses the same subobject???20:49
rm_workthat shouldn't happen, right? why wouldn't it be making new ones?20:49
rm_workthis isn't a singleton pattern is it?20:49
rm_workoh god it is20:49
johnsomI still have some to learn about how this API/pecan stuff was setup20:52
rm_worki'm fixing it to not be a freaking singleton20:52
rm_workI don't think it is supposed to be20:53
*** csomerville has joined #openstack-lbaas20:56
*** cody-somerville has quit IRC20:59
johnsomYou are talking about the app, 'root' right?20:59
*** gcheresh_ has quit IRC21:09
*** dmk0202 has joined #openstack-lbaas21:11
nmagnezibeagles, were you looking for me?21:16
johnsomnmagnezi the puppet-octavia project is having a DB migration problem, but only on centos 7.  I think beagles was wondering if you had run into something similar.21:19
johnsomI think it's just a puppet script problem myself.  I loaded up centos 7 and couldn't reproduce the migration problem21:19
nmagnezijohnsom, thanks Michael :-)21:20
nmagneziI don't recall seeing anything related to migration issues that were specific to centos21:21
nmagnezibeagles, ^^21:21
nmagnezibeagles, how can I reproduce it? I'll take a look21:21
*** openstackgerrit has joined #openstack-lbaas21:22
openstackgerritNir Magnezi proposed openstack/octavia master: Auto detect haproxy user_group  https://review.openstack.org/42939821:22
nmagnezirm_work, hey! I just got back to work today. your suggestion to use StringIO worked ^^ . Tested Okay on Ubuntu amp, now I need to try Centos AMP21:23
rm_workcool21:31
rm_workjohnsom: https://review.openstack.org/#/c/457788/ didn't work?21:31
johnsomYeah, it worked for the octavia gates21:32
johnsomYour patch fixes the lbaas gates21:32
rm_workerr21:35
rm_workoh so we need to merge mine first21:36
rm_workto unblock octavia gates21:36
johnsomYep21:36
rm_workerg21:36
rm_workit failed though21:36
rm_workhttps://review.openstack.org/#/c/457781/21:36
johnsomhttps://www.irccloud.com/pastebin/zpdhN56q/21:39
*** aojea has quit IRC21:39
rm_workerr21:39
rm_workwut21:39
rm_workAUTH_URL=${AUTH_URL:-$OS_AUTH_URL/v3}21:40
rm_workthat works21:40
rm_workI tested it21:40
rm_workwtf21:40
rm_workhttp://paste.openstack.org/show/607057/21:42
rm_workis it not yet set at that point?21:42
rm_workI guess not21:42
rm_workfff k21:42
*** aojea has joined #openstack-lbaas21:44
*** aojea has quit IRC21:45
openstackgerritAdam Harwell proposed openstack/neutron-lbaas master: Use OS_AUTH_URL instead of baked in defaults  https://review.openstack.org/45778121:45
rm_workjohnsom: ^^ what do you think21:45
johnsomYeah, that will probably work21:46
rm_workok fixed this stupid bug21:51
rm_workno more workaround mock21:51
rm_workand no more accidental singleton api-controllers21:51
*** dmk0202 has quit IRC21:59
*** catintheroof has quit IRC22:01
*** amotoki has joined #openstack-lbaas22:13
rm_workwoop woop found a bug lol22:21
johnsomYay!  Just one!22:22
johnsomGrin22:22
rm_workyeah listener controller wasn't setting the LB back to ACTIVE when the listener create failed22:22
rm_workso could get stuck in pending22:23
openstackgerritAdam Harwell proposed openstack/octavia master: Remove singleton pattern from API controllers  https://review.openstack.org/45781322:25
*** aojea has joined #openstack-lbaas22:27
openstackgerritMichael Johnson proposed openstack/octavia master: Adjust tox cover to check for 80% coverage  https://review.openstack.org/45781422:30
openstackgerritAdam Harwell proposed openstack/octavia master: Introduce Octavia v2 API for Health Monitor  https://review.openstack.org/40625622:30
rm_workOK22:30
rm_workThat should be good now22:31
rm_workpending all these gate fixes...22:31
rm_workugh mine failed again, wut22:31
rm_workhttp://logs.openstack.org/81/457781/3/check/gate-neutron-lbaasv2-dsvm-scenario-ubuntu-xenial-nv/efc960b/console.html#_2017-04-18_22_09_34_96318322:32
*** aojea has quit IRC22:33
johnsomWell, it filled in this time: auth_url = http://10.35.17.62/identity_admin/v322:33
johnsomMaybe it's the "[keystone_authtoken]" section, it doesn't have a version22:35
rm_workhmm yeah22:40
rm_workwhy not22:40
rm_workumm22:41
rm_workwe don't even touch that?22:41
johnsomYeah, not sure22:42
rm_workit SHOULD be like22:42
rm_work... the same thing22:42
rm_workhttp://10.35.17.62/identity_admin/v322:42
rm_workyou said [service_auth] `auth_url` doesn't need to be versioned?22:44
johnsomI did?22:46
johnsomIt doesn't in octavia22:46
rm_workyeah we're talking about octavia right?22:46
johnsomIt wasn't before22:46
johnsomhere is an older config file: http://logs.openstack.org/73/454873/8/check/gate-neutron-lbaasv2-dsvm-api-ubuntu-xenial/fb71d24/logs/etc/octavia/octavia.conf.txt.gz22:47
rm_workso for keystone_authtoken we just run `configure_auth_token_middleware`22:47
rm_workwhich comes from ... devstack?22:47
rm_workwtf22:52
rm_workthey're the same?22:52
rm_workI am not seeing the issue22:52
johnsomOctavia should be fine, it's neutron-lbaas that is borked22:52
rm_workhmmm22:52
rm_workI don't know why this would be happenin22:59
rm_workmaybe neutron-lbaas can't v3 auth properly?23:00
rm_workit was still on v2, I "casually" switched it to v3 because that's what we were changing the thing to...23:00
rm_workbut I can switch the whole thing back to v2.023:00
rm_workyeah gonna do that and see23:01
openstackgerritAdam Harwell proposed openstack/neutron-lbaas master: Use OS_AUTH_URL instead of baked in defaults  https://review.openstack.org/45778123:01
rm_workjohnsom: how much longer are you on?23:10
johnsomAn hour or so23:10
rm_workI don't want to have to self-merge that gatefix, but I will if you're gone by then23:10
johnsomJust ping me, I will just be prepping dinner.23:10
rm_workwant to get this mess cleaned up and everything flowing smoothly before tomorrow23:11
johnsomMe too23:11
*** ianychoi has joined #openstack-lbaas23:14
*** JudeC has quit IRC23:18
johnsomOk, once those two patches merge we should have a coverage gate with similar reporting to our old sonar gate.23:19
johnsomWe just won't have the sonar stuff.23:19
*** JudeC has joined #openstack-lbaas23:28
rm_work:(23:31
*** aojea has joined #openstack-lbaas23:33
*** fnaval has joined #openstack-lbaas23:36
*** amotoki has quit IRC23:37
*** aojea has quit IRC23:38
johnsomUgh, we have a lot of stuff basically ready.  Tomorrow could be one of those big merge days I hate23:44
rm_workhmm23:44
rm_worknot if we merge it all tonight! :P23:44
rm_workyeah ok it passed23:45
rm_workjust throw a +A on it23:45
rm_workI'll +A and recheck the rest on the octavia side23:45
*** amotoki has joined #openstack-lbaas23:50
*** reedip has quit IRC23:55
« Monday, 2017-04-17 Index Wednesday, 2017-04-19 »

Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!