Friday, 2019-11-22

*** goldyfruit_ has quit IRC00:09
johnsomrm_work Ok, paid my penance. I have reviewed the whole AZ chain.00:18
johnsomThe gerrit bot is broken BTW, so we won't see patches announced00:21
*** armax has quit IRC00:22
*** spatel has joined #openstack-lbaas00:41
*** sapd1 has quit IRC00:47
*** abaindur has joined #openstack-lbaas01:02
*** abaindur has quit IRC01:03
*** openstackstatus has joined #openstack-lbaas01:03
*** ChanServ sets mode: +v openstackstatus01:03
*** abaindur has joined #openstack-lbaas01:03
*** abaindur has quit IRC01:19
*** abaindur has joined #openstack-lbaas01:48
*** abaindur has quit IRC01:48
*** abaindur has joined #openstack-lbaas01:49
*** abaindur has quit IRC02:13
*** spatel has quit IRC02:37
*** ricolin has joined #openstack-lbaas02:50
*** armax has joined #openstack-lbaas03:14
*** Trevor_V has quit IRC03:18
rm_workAh lol k04:13
*** ramishra has joined #openstack-lbaas05:49
rm_worknow just need cgoncalves to get back to reviewing so we can get stuff moving :D06:05
rm_workugh still have that bug or whatever where on py37 the test_driver_agent functional tests take FOREEEVER06:07
rm_worki forget exactly what caused it, but annoying06:08
*** openstackgerrit has joined #openstack-lbaas06:09
openstackgerritAdam Harwell proposed openstack/octavia master: Availability Zone admin API
openstackgerritAdam Harwell proposed openstack/octavia master: Availability Zone admin API
openstackgerritAdam Harwell proposed openstack/octavia master: Support creating a LB in a specified availability-zone
openstackgerritAdam Harwell proposed openstack/octavia master: Support creating an LB in a specified availability-zone
rm_workcgoncalves: :D06:40
rm_workdayou: ^^ :P06:41
openstackgerritMerged openstack/octavia-lib master: Availability zone / metadata validation
rm_workwooo thanks dayou :D07:13
dayourm_work, my pleasure :D07:13
openstackgerritAdam Harwell proposed openstack/octavia master: Support creating an LB in a specified availability-zone
*** ccamposr__ has joined #openstack-lbaas08:17
*** ccamposr has quit IRC08:20
*** pcaruana has joined #openstack-lbaas08:21
*** tesseract has joined #openstack-lbaas08:35
*** rpittau|afk is now known as rpittau08:58
*** gcheresh has joined #openstack-lbaas09:07
*** ricolin has quit IRC09:43
*** gcheresh has quit IRC09:47
*** luksky has joined #openstack-lbaas11:34
*** rcernin has quit IRC12:01
*** goldyfruit_ has joined #openstack-lbaas13:20
openstackgerritMerged openstack/octavia-dashboard master: Imported Translations from Zanata
*** TrevorV has joined #openstack-lbaas15:00
openstackgerritGregory Thiemonge proposed openstack/octavia-tempest-plugin master: Add UDP test scenario
*** armax has joined #openstack-lbaas15:42
*** luksky has quit IRC16:04
rm_workjohnsom: re-review? ignore the post-failure and grenade being dumb16:13
rm_workI think we could merge :D16:13
johnsomrm_work That file comment was that you are missing the equivalent to
gthiemongejohnsom: thanks16:18
rm_workah would that go in the same controller?16:19
rm_workin i mean16:19
johnsomgthiemonge Sorry those patches fell off the radar and are still open. We were not sure if Zhao was going to finish them or not but at this point it is pretty clear that he is no longer working on them.16:19
johnsomYeah, I think so16:19
rm_workyeah ok i see16:20
rm_workotherwise good?16:20
johnsomI think so. I did a pretty in depth review yesterday. I will scan your changes now but I don't expect any issues16:21
johnsomI missed one extra indent, but otherwise, yeah, I would +2 it with the capabilities API in place. You fixed all of the other stuff I found.16:26
*** ricolin has joined #openstack-lbaas16:32
colin-is ComputeWaitTimeoutException in octavia-worker something i can override in config?16:46
colin-seeing output related to it that i don't remember before stein just want to understand it better16:47
rm_workjohnsom: uhh is this wrong?
johnsomcolin- The retries and timeout are CONF.controller_worker.amp_active_retries and CONF.controller_worker.amp_active_wait_sec16:48
johnsomcolin- We added a new timeout that is connectivity to help clarify the problem and to help with failovers working better when both amps of a act/stdby pair failed at the same time. Maybe that is what is different?16:49
colin-perhaps, will take a look thx16:50
colin-that is plenty to go on16:50
johnsomrm_work Yes, that is wrong. Lucky it is just for documentation and not actual policy enforcement....  Though the functional test would have caught that.16:51
rm_workwould have?16:51
rm_workwhy didn't it?16:51
rm_workdo we not have functional tests? O_o16:51
johnsomBecause that line is only for docs, not functionality16:51
rm_workah k16:51
rm_worki'll just fix it in this patch16:51
johnsomThanks. Yeah, that dict in the list is only used for the "sample file generation" which we don't really use, but an operator could run.16:52
johnsomrm_work This is the test that covers the functionality:
openstackgerritMerged openstack/octavia-lib master: Switch to Ussuri jobs
johnsomhaleyb Would you mind adding Ajay as a co-author on to resolve the issue. Technically his patch was first.16:56
*** rpittau is now known as rpittau|afk17:01
TrevorVHey johnsom , when you get a few seconds, could you answer a quick question for me?  In Triple-O, do you know if any of the different octavia service require being run on a compute node?17:14
johnsomTrevorV No, we have to code that runs on the compute nodes directly. We of course spin up service VMs inside the compute nodes, but nothing goes on the node itself.17:17
TrevorVthanks for confirming that, I thought as much from a conversation I had with rm_work a while ago.  Could you humor me with an example or two of what service VMs you mean?17:19
johnsomThe amphora17:19
*** ricolin has quit IRC17:21
TrevorVjohnsom alright, thanks.  So amphora are created and run on the compute nodes, is that worded properly?17:21
johnsomThey are not run on the compute nodes, they are run inside the compute environment, i.e. inside nova.17:22
johnsomThere is zero Octavia code installed on compute nodes.17:22
TrevorVOh okay, so, after the amphora image is uploaded to glance, octavia just spawns nova servers with that image as the amphora?17:23
TrevorVwhich of course, lives on the compute nodes.  Got it!  thanks a bunch, that helps me immensely17:23
johnsomPretty much your quote is "There is zero Octavia code installed on the compute nodes"17:23
rm_workthink of octavia as just another cloud-user17:23
rm_workwe can be installed off to the side17:23
johnsomA really spiffy cloud user. lol17:24
rm_workand we do need admin access, but otherwise we just use standard APIs17:24
johnsomNo, we do not need admin access.17:24
johnsomThat is not true.17:24
rm_workwe di17:24
rm_work*we do17:24
rm_workto create ports on user networks17:24
rm_workand plug them to VMs in different projects17:24
rm_workthat requires admin17:24
johnsomNo, you can run Octavia under non-admin projects/users. And you should17:24
rm_workwe definitely need admin to neutron17:24
johnsomIt would require RBAC rules, but not admin level17:25
rm_worklast i checked it was literally an admin check to do the stuff we do with cross-binding ports17:25
johnsomNot even full "service" project level, but that is a good compromise17:25
johnsomNope. This is a common mis-conception. We do not need to run as full "admin" in the cloud.17:25
johnsomMany people do it simply because it's "easy"17:26
johnsomBut it is definitely not the right answer17:26
rm_worki actually wrote one of the neutron patches to make one of those things work... hold on17:27
johnsomIt has been done in previous roles/products I was a part of....17:28
johnsomReally at some point I should take a week and write that up as another chapter in our install guide, but....  time happens17:28
rm_workit was for my L3 driver T_T17:28
rm_workliterally "is_admin" check lol17:29
johnsomI would argue that is implemented wrong.....17:29
johnsomIf fact, is_admin is deprecated17:29
rm_worklol whelp17:29
rm_worki mean that was 2.5 years ago17:29
rm_workso was it deprecated THEN? :D17:29
johnsomWell, it was still wrong then at least17:30
rm_workkevin benton helped me with it :D17:30
rm_workand kevin and miguel +2'd heh17:31
rm_workoh hey and brian reviewed too lol17:31
rm_workjohnsom calling out ALL the cores :D17:31
rm_workand two PTLs17:31
johnsomYes, yes I am17:32
rm_workI don't disagree with you I guess :D just funny no one else seemed to care17:32
johnsomIt's putting policy logic in a strange spot in the code and it probably should be the context_is_advsvc role that neutron uses17:34
rm_workyeah that sounds like it'd be more right17:34
rm_worki wonder if this line is even still there17:35
johnsomIt is17:35
*** tesseract has quit IRC17:35
johnsomBut they still haven't met the policy-in-code goal either, so maybe it will get cleaned up when they finish that work.17:35
johnsomAnyway, Octavia *should* not require full cloud ADMIN rights to run.17:36
johnsomAside from your hack, I don't think it does today17:36
openstackLaunchpad bug 1853637 in neutron "Assign floating IP to port owned by another tenant is not override-able with RBAC policy" [Undecided,New]17:41
johnsomSee something, say something. lol17:42
rm_workhmm i think i'm done with this patch17:49
openstackgerritAdam Harwell proposed openstack/octavia master: Availability Zone admin API
johnsomYeah, assuming the tests pass, that version looks good to me18:02
*** irclogbot_1 has quit IRC18:27
*** irclogbot_3 has joined #openstack-lbaas18:30
*** beisner has quit IRC18:35
*** beisner has joined #openstack-lbaas18:36
rm_workneed to update the tempest side18:43
rm_worktrying to cut down the number of outstanding patches I have waiting to merge18:43
rm_workalso i promised that feature to someone internally >_<18:44
rm_workok, updated tempest tests to include new tests for that API19:09
openstackgerritAdam Harwell proposed openstack/octavia-tempest-plugin master: [WIP] Test the Availability Zone [Profile] admin API
rm_worki really really hate our grenade right now19:09
rm_worki hate it so so much19:10
johnsomYeah, annoying. I mentioned some of the neutron issues I have seen to the neutron team. Not sure if the nova stuff is really neutron related or not19:14
rm_workthe last one i saw seemed to fail on keystone upgrade? :/19:14
johnsomYeah, I think I have seen one of those too19:14
rm_worksame this time19:15
rm_work2019-11-22 19:00:47.992 |   Found existing installation: PyYAML 3.1219:16
rm_work2019-11-22 19:00:48.412 | ERROR: Cannot uninstall 'PyYAML'. It is a distutils installed project and thus we cannot accurately determine which files belong to it which would lead to only a partial uninstall.19:16
rm_workanother one of those19:16
johnsomHmm, I thought all of those were fixed a long time ago....19:16
johnsomWe found one issue in the nova scripts for grenade yesterday. It is using the py2 OSC when devstack is py3, but that shouldn't impact us yet as we are still running that under py219:20
johnsomMy guess is it is fallout from this:
johnsompurely a guess though19:31
*** ccamposr has joined #openstack-lbaas19:32
johnsomMight be one of those packages keystone is installing for tests also raised their minimum19:33
johnsomIt is definitely installed in the OS however: ii  python3-yaml                          3.12-1build2                           amd64        YAML parser and emitter for Python319:34
*** ccamposr__ has quit IRC19:34
*** gcheresh has joined #openstack-lbaas19:48
rm_workeugh well ... i guess our gates are broken until this is resolved? joy19:59
rm_worki won't be able to track this today19:59
openstackgerritGregory Thiemonge proposed openstack/octavia-tempest-plugin master: Use VM as client in traffic operation scenario tests
*** abaindur has joined #openstack-lbaas20:23
*** abaindur has quit IRC20:28
*** luksky has joined #openstack-lbaas20:32
johnsomI can try the disable n-vnc advice infra gave.20:35
johnsomFunny enough, I already have a fixgrenade folder. I guess it is a recurring problem20:36
johnsomWe certainly have zero need for that vnc stuff20:37
*** ccamposr__ has joined #openstack-lbaas20:38
*** ccamposr has quit IRC20:40
*** gcheresh has quit IRC20:44
*** rcernin has joined #openstack-lbaas20:49
openstackgerritMichael Johnson proposed openstack/octavia master: Disable nova console services in grenade job
*** abaindur has joined #openstack-lbaas21:11
*** servagem has quit IRC21:26
*** spatel has joined #openstack-lbaas21:38
*** spatel has quit IRC21:39
*** abaindur has quit IRC22:05
*** spatel has joined #openstack-lbaas22:08
*** spatel has quit IRC22:13
*** spatel has joined #openstack-lbaas22:15
*** rcernin has quit IRC22:24
*** KeithMnemonic has quit IRC22:44
*** spatel has quit IRC22:58
*** luksky has quit IRC23:04
*** TrevorV has quit IRC23:30
openstackgerritBrian Haley proposed openstack/octavia-lib master: Stop testing python 2
openstackgerritMichael Johnson proposed openstack/octavia master: Disable nova console services in grenade job

Generated by 2.15.3 by Marius Gedminas - find it at!