| *** jmlowe has quit IRC | 00:03 | |
| *** wolverineav has quit IRC | 00:04 | |
| *** wolverineav has joined #openstack-neutron | 00:05 | |
| *** wolverineav has quit IRC | 00:10 | |
| *** wolverineav has joined #openstack-neutron | 00:14 | |
| *** lemko has quit IRC | 00:20 | |
| *** macza has joined #openstack-neutron | 00:29 | |
| *** macza has quit IRC | 00:34 | |
| *** wolverineav has quit IRC | 00:37 | |
| *** wolverineav has joined #openstack-neutron | 00:39 | |
| *** iyamahat has joined #openstack-neutron | 00:42 | |
| *** wolverineav has quit IRC | 00:43 | |
| *** wolverineav has joined #openstack-neutron | 00:44 | |
| *** yamamoto has joined #openstack-neutron | 00:51 | |
| *** yamamoto has quit IRC | 00:56 | |
| *** dougwig has joined #openstack-neutron | 00:59 | |
| *** mnasiadka has joined #openstack-neutron | 00:59 | |
| *** sweston has joined #openstack-neutron | 00:59 | |
| *** mrhillsman has joined #openstack-neutron | 00:59 | |
| *** TheJulia has joined #openstack-neutron | 00:59 | |
| *** coreycb has joined #openstack-neutron | 00:59 | |
| *** preece has joined #openstack-neutron | 00:59 | |
| *** moshele has joined #openstack-neutron | 01:07 | |
| *** macza has joined #openstack-neutron | 01:08 | |
| *** wolverineav has quit IRC | 01:09 | |
| *** wolverineav has joined #openstack-neutron | 01:09 | |
| *** ijw has joined #openstack-neutron | 01:12 | |
| *** macza has quit IRC | 01:13 | |
| *** mriedem has quit IRC | 01:23 | |
| *** moshele has quit IRC | 01:25 | |
| *** yedongcan has joined #openstack-neutron | 01:25 | |
| *** markvoelker has quit IRC | 01:30 | |
| *** markvoelker has joined #openstack-neutron | 01:31 | |
| *** markvoelker has quit IRC | 01:35 | |
| *** _fragatina has quit IRC | 01:37 | |
| openstackgerrit | Reedip proposed openstack/networking-ovn master: [DNM]Add ovn octavia CI jobs https://review.openstack.org/589712 | 01:39 |
|---|---|---|
| *** wolverineav has quit IRC | 01:49 | |
| *** Dinesh_Bhor has joined #openstack-neutron | 01:49 | |
| *** wolverineav has joined #openstack-neutron | 01:52 | |
| *** yamamoto has joined #openstack-neutron | 01:53 | |
| *** wolverineav has quit IRC | 01:56 | |
| *** ijw has quit IRC | 02:07 | |
| *** macza has joined #openstack-neutron | 02:22 | |
| *** macza has quit IRC | 02:27 | |
| *** macza has joined #openstack-neutron | 02:27 | |
| *** hongbin has joined #openstack-neutron | 02:29 | |
| *** macza has quit IRC | 02:31 | |
| *** markvoelker has joined #openstack-neutron | 02:31 | |
| *** ihar_ has quit IRC | 02:34 | |
| *** iyamahat has quit IRC | 02:35 | |
| *** tidwellr has quit IRC | 02:44 | |
| *** sapd1 has joined #openstack-neutron | 02:47 | |
| *** yamamoto has quit IRC | 02:48 | |
| *** zhanglong has joined #openstack-neutron | 02:55 | |
| *** markvoelker has quit IRC | 03:05 | |
| *** armax has quit IRC | 03:12 | |
| *** yamamoto has joined #openstack-neutron | 03:14 | |
| *** armax has joined #openstack-neutron | 03:34 | |
| *** jmlowe has joined #openstack-neutron | 03:43 | |
| *** sapd1 has quit IRC | 03:59 | |
| *** sapd1 has joined #openstack-neutron | 03:59 | |
| *** markvoelker has joined #openstack-neutron | 04:02 | |
| *** ramishra has joined #openstack-neutron | 04:06 | |
| *** armax has quit IRC | 04:28 | |
| *** janki has joined #openstack-neutron | 04:30 | |
| *** armax has joined #openstack-neutron | 04:30 | |
| *** markvoelker has quit IRC | 04:34 | |
| *** manjeets has quit IRC | 04:41 | |
| *** jmlowe has quit IRC | 04:50 | |
| *** armax has quit IRC | 04:59 | |
| openstackgerrit | Chengqian Liu proposed openstack/neutron-fwaas master: Fix the verification method before creating and updating the firewall rule https://review.openstack.org/618485 | 05:01 |
| *** ratailor has joined #openstack-neutron | 05:08 | |
| *** hongbin has quit IRC | 05:30 | |
| *** ArchiFleKs has quit IRC | 05:31 | |
| *** markvoelker has joined #openstack-neutron | 05:32 | |
| *** ArchiFleKs has joined #openstack-neutron | 05:37 | |
| *** lajoskatona has joined #openstack-neutron | 05:45 | |
| *** gvrangan has joined #openstack-neutron | 05:46 | |
| *** wolverineav has joined #openstack-neutron | 05:55 | |
| openstackgerrit | Yang Youseok proposed openstack/neutron-specs master: Registering RouterInfo by L3 extention API https://review.openstack.org/625647 | 06:05 |
| *** markvoelker has quit IRC | 06:05 | |
| *** Dinesh_Bhor has quit IRC | 06:05 | |
| *** Dinesh_Bhor has joined #openstack-neutron | 06:09 | |
| *** gcheresh has joined #openstack-neutron | 06:17 | |
| *** sridharg has joined #openstack-neutron | 06:20 | |
| *** lemko has joined #openstack-neutron | 06:30 | |
| *** wolverineav has quit IRC | 06:38 | |
| *** zufar has quit IRC | 06:38 | |
| *** Luzi has joined #openstack-neutron | 06:53 | |
| *** mosulica has joined #openstack-neutron | 06:53 | |
| *** ralonsoh has joined #openstack-neutron | 06:59 | |
| *** ksambor has joined #openstack-neutron | 07:00 | |
| *** NewBruce has joined #openstack-neutron | 07:01 | |
| *** markvoelker has joined #openstack-neutron | 07:02 | |
| *** NewBruce has quit IRC | 07:03 | |
| *** ccamposr has joined #openstack-neutron | 07:04 | |
| *** moshele has joined #openstack-neutron | 07:12 | |
| *** NewBruce has joined #openstack-neutron | 07:23 | |
| *** snapiri has joined #openstack-neutron | 07:24 | |
| *** hamdyk has joined #openstack-neutron | 07:30 | |
| *** markvoelker has quit IRC | 07:35 | |
| *** ramishra_ has joined #openstack-neutron | 07:37 | |
| *** ramishra has quit IRC | 07:38 | |
| *** slaweq has joined #openstack-neutron | 07:44 | |
| *** ksambor has quit IRC | 07:50 | |
| *** pckizer has quit IRC | 07:51 | |
| *** aojea has joined #openstack-neutron | 07:52 | |
| *** ramishra_ is now known as ramishra | 07:56 | |
| *** gvrangan has quit IRC | 07:57 | |
| *** ksambor has joined #openstack-neutron | 07:59 | |
| *** awalende has joined #openstack-neutron | 08:06 | |
| *** velizarx has joined #openstack-neutron | 08:07 | |
| *** gkadam has joined #openstack-neutron | 08:08 | |
| *** tesseract has joined #openstack-neutron | 08:09 | |
| *** awalende has quit IRC | 08:09 | |
| *** rpittau has joined #openstack-neutron | 08:13 | |
| *** gkadam_ has joined #openstack-neutron | 08:15 | |
| *** gkadam has quit IRC | 08:17 | |
| *** gildub has joined #openstack-neutron | 08:18 | |
| *** awalende has joined #openstack-neutron | 08:20 | |
| *** tkajinam has quit IRC | 08:29 | |
| *** markvoelker has joined #openstack-neutron | 08:32 | |
| openstackgerrit | Merged openstack/neutron master: Fix the way how upgrade checks are loaded https://review.openstack.org/635588 | 08:34 |
| *** jaewook_oh has joined #openstack-neutron | 08:35 | |
| sapd1 | hi numans dalvarez | 08:35 |
| dalvarez | hey sapd1 | 08:36 |
| sapd1 | Is this bug solved ? - https://bugs.launchpad.net/networking-ovn/+bug/1689880 | 08:37 |
| openstack | Launchpad bug 1689880 in networking-ovn "The "neutron_sync_mode = repair" option breaks the whole cloud!" [High,New] | 08:37 |
| *** velizarx has quit IRC | 08:38 | |
| sapd1 | dalvarez: Do you remember my issue which I posted yesterday? | 08:41 |
| openstackgerrit | Merged openstack/neutron master: Remove redundant event listening logic from neutron/db/api https://review.openstack.org/635978 | 08:42 |
| dalvarez | sapd1: not sure | 08:42 |
| sapd1 | after upgrade networking-ovn to latest of branch stable/rocky. All VMs with FIPs now loss 50% packets | 08:43 |
| dalvarez | sapd1: https://bugs.launchpad.net/networking-ovn/+bug/1689880 this should be more than fixed, i use the repair option all the time | 08:43 |
| openstack | Launchpad bug 1689880 in networking-ovn "The "neutron_sync_mode = repair" option breaks the whole cloud!" [High,New] | 08:43 |
| sapd1 | dalvarez: http://paste.openstack.org/show/744908/ | 08:43 |
| sapd1 | Do you get this problem before? | 08:44 |
| ajo | oh, nice, all my networking-ovn patches related to migration merged | 08:44 |
| ajo | I'll backport to rocky then :D | 08:44 |
| dalvarez | sapd1: every time you ping you lose 50% of the packets? | 08:45 |
| sapd1 | dalvarez: yes. | 08:45 |
| dalvarez | sapd1: even though when the destination mac has been learnt? | 08:45 |
| sapd1 | I have rebooted all VM then all compute/controller node. | 08:45 |
| sapd1 | dalvarez: yes | 08:45 |
| sapd1 | I don't get this error when use provider network. | 08:46 |
| dalvarez | sapd1: no idea, never seen that before | 08:46 |
| dalvarez | sapd1: geneve tenant networks? | 08:46 |
| sapd1 | dalvarez: yes. | 08:46 |
| dalvarez | maybe numans knows but i havent seen it | 08:47 |
| numans | dalvarez, no neither me | 08:47 |
| numans | sapd1, that's strange | 08:47 |
| sapd1 | does anyone have time to access my setup? | 08:48 |
| sapd1 | numans: dalvarez | 08:48 |
| openstackgerrit | Merged openstack/neutron master: doc: replace nova security_group_api option with use_neutron https://review.openstack.org/634729 | 08:50 |
| *** jpena|off is now known as jpena | 08:53 | |
| sapd1 | 2019-02-12T08:53:56.754Z|00009|ovn_northd|WARN|Bad configuration: multiple ports with redirect-chassis on same logical router neutron-b6079606-2e45-4ca0-a261-f67c18053830 | 08:54 |
| sapd1 | 08:54 | |
| *** iyamahat has joined #openstack-neutron | 08:54 | |
| numans | sapd1, may be the sync stuff is broken | 08:56 |
| numans | sapd1, my suggestion would be .. take a backup of ovnnb_db.db. delete it and run the sync util again | 08:57 |
| numans | sapd1, and see if the issue is still seen | 08:57 |
| numans | sapd1, i can take a look a bit later.. after lunch. | 08:58 |
| numans | sapd1, can you plz try once. this :). plz take a back up though :). stop the ovn nb db server, delete the ovn nb db , start ovn nb db server and run sync util | 08:58 |
| sapd1 | numans: I will try now | 09:03 |
| numans | sapd1, cool. let me know how it goes | 09:03 |
| *** panda|off is now known as panda | 09:04 | |
| *** velizarx has joined #openstack-neutron | 09:04 | |
| *** markvoelker has quit IRC | 09:05 | |
| *** jistr_ is now known as jistr | 09:09 | |
| openstackgerrit | Miguel Angel Ajo proposed openstack/networking-ovn master: Make networking-ovn-migration-mtu output less confusing https://review.openstack.org/635042 | 09:15 |
| *** tssurya has joined #openstack-neutron | 09:17 | |
| openstackgerrit | Miguel Angel Ajo proposed openstack/networking-ovn master: Add extra validation and update the migration guide https://review.openstack.org/619624 | 09:18 |
| *** iyamahat has quit IRC | 09:19 | |
| openstackgerrit | Chengqian Liu proposed openstack/neutron-fwaas master: Fix the verification method before creating and updating the firewall rule https://review.openstack.org/618485 | 09:21 |
| *** ramishra has quit IRC | 09:22 | |
| sapd1 | numans: dalvarez solved | 09:26 |
| sapd1 | remove nb-db and resync db | 09:26 |
| dalvarez | sapd1: how? | 09:26 |
| dalvarez | ah | 09:26 |
| dalvarez | that's bad if we have a bug there | 09:26 |
| sapd1 | :( | 09:27 |
| *** rubasov has joined #openstack-neutron | 09:27 | |
| sapd1 | anyway | 09:27 |
| sapd1 | we have a way to fix it | 09:27 |
| openstackgerrit | Lucas Alvares Gomes proposed openstack/networking-ovn master: Increase nb_cfg when needed only (reducing the # of transactions) https://review.openstack.org/634912 | 09:28 |
| *** ramishra has joined #openstack-neutron | 09:29 | |
| openstackgerrit | Lucas Alvares Gomes proposed openstack/networking-ovn master: Make networking-ovn-migration-mtu output less confusing https://review.openstack.org/635042 | 09:39 |
| numans | sapd1, dalvarez so the issue is seen when you did this right - after upgrade networking-ovn to latest of branch stable/rocky. All VMs with FIPs now loss 50% packets ? | 09:41 |
| numans | dalvarez, we need to test this out man | 09:42 |
| sapd1 | I think I need re-proceduce. | 09:42 |
| sapd1 | downgrade networking-ovn and then upgrade | 09:42 |
| *** jlibosva has joined #openstack-neutron | 09:47 | |
| sapd1 | numans: I got it. | 09:47 |
| numans | sapd1, ok. thanks. may be a launchpad bug :) | 09:50 |
| numans | lucasagomes, dalvarez ^ | 09:50 |
| numans | sapd1++ | 09:50 |
| dalvarez | right! | 09:50 |
| dalvarez | sapd1: pls report it with reproduction steps :) | 09:50 |
| *** NewBruce has quit IRC | 09:54 | |
| sapd1 | dalvarez: numans Yes. I'm writing now. | 09:55 |
| *** NewBruce has joined #openstack-neutron | 09:59 | |
| *** markvoelker has joined #openstack-neutron | 10:02 | |
| *** NewBruce has quit IRC | 10:03 | |
| *** zhanglong has quit IRC | 10:05 | |
| openstackgerrit | Rodolfo Alonso Hernandez proposed openstack/neutron master: Use pyroute2 to check vlan/vxlan in use https://review.openstack.org/636296 | 10:07 |
| *** moshele has quit IRC | 10:08 | |
| sapd1 | https://bugs.launchpad.net/networking-ovn/+bug/1815574 | 10:11 |
| openstack | Launchpad bug 1815574 in networking-ovn "loss 50% packets when upgrade from 5.0.0 to latest of stable/rocky" [Undecided,New] | 10:11 |
| *** davidsha has joined #openstack-neutron | 10:18 | |
| *** gildub_ has joined #openstack-neutron | 10:19 | |
| *** NewBruce has joined #openstack-neutron | 10:19 | |
| *** gildub has quit IRC | 10:20 | |
| *** Dinesh_Bhor has quit IRC | 10:32 | |
| *** markvoelker has quit IRC | 10:35 | |
| *** moshele has joined #openstack-neutron | 10:39 | |
| *** salmankhan has joined #openstack-neutron | 10:41 | |
| *** NewBruce has quit IRC | 10:42 | |
| openstackgerrit | Slawek Kaplonski proposed openstack/neutron master: Migrate neutron-functional job to zuul v3 syntax https://review.openstack.org/619742 | 10:43 |
| *** sapd1 has quit IRC | 10:45 | |
| openstackgerrit | Akihiro Motoki proposed openstack/neutron master: doc: Use DocumentedRuleDefault https://review.openstack.org/630050 | 10:47 |
| *** NewBruce has joined #openstack-neutron | 10:49 | |
| openstackgerrit | Slawek Kaplonski proposed openstack/neutron master: [DNM] Migrate neutron-tempest-dvr-ha-multinode-full job to zuulv3 https://review.openstack.org/633979 | 10:49 |
| openstackgerrit | Slawek Kaplonski proposed openstack/neutron master: [DNM] Migrate neutron-tempest-dvr-ha-multinode-full job to zuulv3 https://review.openstack.org/633979 | 10:51 |
| openstackgerrit | Slawek Kaplonski proposed openstack/neutron master: [DNM] Migrate neutron-tempest-dvr-ha-multinode-full job to zuulv3 https://review.openstack.org/633979 | 10:53 |
| *** NewBruce has quit IRC | 10:54 | |
| openstackgerrit | Slawek Kaplonski proposed openstack/neutron master: Migrate neutron-tempest-dvr-ha-multinode-full job to zuulv3 https://review.openstack.org/633979 | 10:56 |
| *** NewBruce has joined #openstack-neutron | 11:04 | |
| *** NewBruce has quit IRC | 11:06 | |
| *** salmankhan1 has joined #openstack-neutron | 11:08 | |
| *** salmankhan has quit IRC | 11:09 | |
| *** salmankhan1 is now known as salmankhan | 11:09 | |
| *** obondarev has joined #openstack-neutron | 11:11 | |
| *** lpetrut has joined #openstack-neutron | 11:15 | |
| *** farelysnow has joined #openstack-neutron | 11:29 | |
| *** NewBruce has joined #openstack-neutron | 11:31 | |
| *** markvoelker has joined #openstack-neutron | 11:32 | |
| *** velizarx has quit IRC | 11:34 | |
| *** farelysnow has quit IRC | 11:37 | |
| *** gildub_ has quit IRC | 11:42 | |
| openstackgerrit | Lucas Alvares Gomes proposed openstack/networking-ovn master: Maintenance: Re-create the metadata port https://review.openstack.org/636312 | 11:43 |
| *** mvkr has quit IRC | 11:44 | |
| *** gildub_ has joined #openstack-neutron | 11:46 | |
| *** NewBruce has quit IRC | 11:47 | |
| *** NewBruce has joined #openstack-neutron | 11:50 | |
| openstackgerrit | Lucas Alvares Gomes proposed openstack/networking-ovn master: Maintenance: Re-create the metadata port https://review.openstack.org/636312 | 11:50 |
| *** _fragatina has joined #openstack-neutron | 11:56 | |
| openstackgerrit | Bence Romsics proposed openstack/neutron master: rally-jobs: Add FloatingIP workloads: assoc/dissoc https://review.openstack.org/624038 | 11:56 |
| *** _fragatina has quit IRC | 11:59 | |
| *** _fragatina has joined #openstack-neutron | 12:00 | |
| openstackgerrit | Merged openstack/networking-ovn master: Fix instances of events as strings instead of tuples https://review.openstack.org/635572 | 12:01 |
| *** markvoelker has quit IRC | 12:05 | |
| *** rkukura has quit IRC | 12:18 | |
| *** rkukura has joined #openstack-neutron | 12:22 | |
| *** velizarx has joined #openstack-neutron | 12:26 | |
| *** jpena is now known as jpena|lunch | 12:26 | |
| *** tbachman has quit IRC | 12:29 | |
| *** velizarx has quit IRC | 12:30 | |
| *** panda is now known as panda|lunch | 12:32 | |
| *** ratailor has quit IRC | 12:34 | |
| *** coreycb has quit IRC | 12:36 | |
| *** coreycb has joined #openstack-neutron | 12:37 | |
| *** velizarx has joined #openstack-neutron | 12:38 | |
| *** lpetrut has quit IRC | 12:38 | |
| *** lpetrut has joined #openstack-neutron | 12:39 | |
| openstackgerrit | Rodolfo Alonso Hernandez proposed openstack/neutron master: Add TC filter functions implemented with pyroute2 https://review.openstack.org/625685 | 12:43 |
| *** janki has quit IRC | 12:44 | |
| *** janki has joined #openstack-neutron | 12:44 | |
| *** sapd1 has joined #openstack-neutron | 12:44 | |
| *** mvkr has joined #openstack-neutron | 12:46 | |
| openstackgerrit | Rodolfo Alonso Hernandez proposed openstack/neutron-lib master: Add traffic control constants https://review.openstack.org/633951 | 12:47 |
| openstackgerrit | Rodolfo Alonso Hernandez proposed openstack/neutron master: Use pyroute2 to check vlan/vxlan in use https://review.openstack.org/636296 | 12:47 |
| *** _fragatina has quit IRC | 12:56 | |
| *** jaewook_oh has quit IRC | 12:56 | |
| *** moshele has quit IRC | 13:00 | |
| *** markvoelker has joined #openstack-neutron | 13:02 | |
| *** yamamoto has quit IRC | 13:03 | |
| *** mriedem has joined #openstack-neutron | 13:04 | |
| *** Dinesh_Bhor has joined #openstack-neutron | 13:04 | |
| *** tbachman has joined #openstack-neutron | 13:04 | |
| *** mvkr has quit IRC | 13:08 | |
| *** yamamoto has joined #openstack-neutron | 13:08 | |
| *** panda|lunch is now known as panda | 13:13 | |
| *** yamamoto has quit IRC | 13:15 | |
| *** tbachman has quit IRC | 13:22 | |
| *** tbachman has joined #openstack-neutron | 13:23 | |
| *** ociuhandu has joined #openstack-neutron | 13:26 | |
| *** NewBruce has quit IRC | 13:29 | |
| openstackgerrit | Andreas Karis proposed openstack/neutron master: Improve invalid port ranges error message https://review.openstack.org/636174 | 13:30 |
| openstackgerrit | David Shaughnessy proposed openstack/neutron-lib master: (PoC)Classification field in Neutron DSCP https://review.openstack.org/636330 | 13:30 |
| *** jpena|lunch is now known as jpena | 13:30 | |
| *** NewBruce has joined #openstack-neutron | 13:30 | |
| *** velizarx has quit IRC | 13:32 | |
| openstackgerrit | Slawek Kaplonski proposed openstack/neutron master: Migrate neutron-functional job to zuul v3 syntax https://review.openstack.org/619742 | 13:32 |
| *** markvoelker has quit IRC | 13:35 | |
| *** tbachman has quit IRC | 13:35 | |
| *** velizarx has joined #openstack-neutron | 13:37 | |
| *** openstackgerrit has quit IRC | 13:37 | |
| *** mvkr has joined #openstack-neutron | 13:40 | |
| *** yedongcan has left #openstack-neutron | 13:41 | |
| *** yamamoto has joined #openstack-neutron | 13:46 | |
| *** gildub_ has quit IRC | 13:48 | |
| *** gildub has joined #openstack-neutron | 13:53 | |
| *** boden has joined #openstack-neutron | 13:55 | |
| *** yamamoto has quit IRC | 13:56 | |
| *** openstackgerrit has joined #openstack-neutron | 13:56 | |
| openstackgerrit | David Shaughnessy proposed openstack/neutron master: (PoC)Classification field in Neutron DSCP https://review.openstack.org/636333 | 13:56 |
| *** kklimonda_ has joined #openstack-neutron | 13:58 | |
| openstackgerrit | David Shaughnessy proposed openstack/neutron-lib master: (PoC)Classification field in Neutron DSCP https://review.openstack.org/636330 | 13:59 |
| *** mlavalle has joined #openstack-neutron | 13:59 | |
| *** ihar_ has joined #openstack-neutron | 14:03 | |
| *** yamamoto has joined #openstack-neutron | 14:06 | |
| *** yamamoto has quit IRC | 14:06 | |
| openstackgerrit | Édouard Thuleau proposed openstack/neutron-fwaas master: Add service checks before trying to configure it https://review.openstack.org/636340 | 14:08 |
| *** mriedem is now known as mriedem_snow | 14:10 | |
| frickler | slaweq: (or others) is there a multinode test running two dhcp agents? I'm chasing some issues we are seeing locally and I'm wondering whether this scenario is being tested at all | 14:12 |
| *** NewBruce has quit IRC | 14:13 | |
| openstackgerrit | Merged openstack/neutron master: Change OVS agent to log message after failure https://review.openstack.org/633079 | 14:14 |
| *** liuyulong has joined #openstack-neutron | 14:15 | |
| *** sapd1 has quit IRC | 14:15 | |
| *** Dinesh_Bhor has quit IRC | 14:18 | |
| *** NewBruce has joined #openstack-neutron | 14:18 | |
| openstackgerrit | Brian Haley proposed openstack/neutron master: Change SR-IOV agent to log message after failure https://review.openstack.org/633080 | 14:19 |
| slaweq | frickler: I think that in multinode scenario dhcp agent is only on controller node | 14:20 |
| frickler | slaweq: that's exactly what I've found so far. what I'd need is a test moving a network from one agent to another and also a test with two agents active in parallel | 14:22 |
| frickler | slaweq: guess I'll have to build that myself ... | 14:22 |
| slaweq | frickler: we are testing something like that in fullstack job IIRC | 14:23 |
| slaweq | https://github.com/openstack/neutron/blob/master/neutron/tests/fullstack/test_dhcp_agent.py | 14:23 |
| *** obondarev has quit IRC | 14:25 | |
| frickler | slaweq: ah, thx, I'll dig into those a bit more | 14:30 |
| slaweq | frickler: yw :) | 14:31 |
| *** markvoelker has joined #openstack-neutron | 14:32 | |
| *** ArchiFleKs has quit IRC | 14:34 | |
| *** NewBruce has quit IRC | 14:34 | |
| *** ironfoot_ is now known as ironfoot | 14:35 | |
| *** rkukura has quit IRC | 14:36 | |
| *** rkukura has joined #openstack-neutron | 14:40 | |
| *** janki has quit IRC | 14:47 | |
| *** hongbin has joined #openstack-neutron | 14:49 | |
| *** tbachman has joined #openstack-neutron | 14:49 | |
| *** ArchiFleKs has joined #openstack-neutron | 14:49 | |
| *** NewBruce has joined #openstack-neutron | 14:51 | |
| *** gkadam_ has quit IRC | 14:54 | |
| *** gkadam has joined #openstack-neutron | 14:54 | |
| *** snapiri has quit IRC | 14:56 | |
| *** awalende has quit IRC | 14:58 | |
| *** maciejjozefczyk has quit IRC | 14:58 | |
| *** awalende has joined #openstack-neutron | 14:58 | |
| *** sapd1 has joined #openstack-neutron | 15:02 | |
| *** awalende has quit IRC | 15:02 | |
| mlavalle | slaweq: ping | 15:03 |
| *** moshele has joined #openstack-neutron | 15:04 | |
| slaweq | mlavalle: hi | 15:04 |
| mlavalle | slaweq: I won't be able to attend today the CI meeting. Only for today, I have a conflict with an inernal meeting | 15:04 |
| slaweq | mlavalle: ok, thx for info | 15:05 |
| slaweq | I will try to run it but I don't know how many people will be :) | 15:05 |
| *** markvoelker has quit IRC | 15:05 | |
| mlavalle | slaweq: but I wanted to update you on https://bugs.launchpad.net/neutron/+bug/1795870 | 15:05 |
| openstack | Launchpad bug 1795870 in neutron "Trunk scenario test test_trunk_subport_lifecycle fails from time to time" [High,In progress] - Assigned to Miguel Lavalle (minsel) | 15:05 |
| slaweq | mlavalle: sure | 15:07 |
| mlavalle | slaweq: problem is with the test migrating router from HA to dvr/legacy. That migration implies deleting the existing router state_change_monitor | 15:08 |
| mlavalle | slaweq: so we try to execute 'kill', '-15', 'pid'' | 15:09 |
| mlavalle | and that kill fails | 15:09 |
| *** hongbin has quit IRC | 15:09 | |
| mlavalle | once we do that failure occurs, the agent in question becomes unresponsive and the server stops scheduling router to it | 15:10 |
| *** hamdyk has quit IRC | 15:10 | |
| slaweq | mlavalle: interesting | 15:10 |
| slaweq | can You reproduce it locally? | 15:10 |
| mlavalle | slaweq: please look here: http://paste.openstack.org/show/744001/ | 15:10 |
| mlavalle | slaweq: with the job change, my infra is in "flux". But I am getting to a point where I think I will be able to try it locally | 15:11 |
| *** annp has joined #openstack-neutron | 15:11 | |
| *** hongbin has joined #openstack-neutron | 15:12 | |
| slaweq | mlavalle: ok, great progress :) | 15:13 |
| slaweq | mlavalle: thx for update | 15:13 |
| mlavalle | slaweq: but I wanted to ask you: do you think we might be missing a filter here: https://github.com/openstack/neutron/blob/master/etc/neutron/rootwrap.d/l3.filters? | 15:14 |
| slaweq | mlavalle: You mean KillFilter for neutron-keepalived-state-change? | 15:15 |
| slaweq | mlavalle: possibly | 15:15 |
| *** yamamoto has joined #openstack-neutron | 15:16 | |
| mlavalle | slaweq: yeah, that's what I mean | 15:17 |
| slaweq | mlavalle: but how it worked before? :) | 15:17 |
| mlavalle | slaweq: I have the same question | 15:17 |
| slaweq | mlavalle: You can try to add this rule and push patch | 15:17 |
| slaweq | we will see if that will help :) | 15:18 |
| mlavalle | slaweq: yeap. anyway, that is my update for today's meeting | 15:18 |
| slaweq | ok, thx | 15:18 |
| slaweq | great progress on this | 15:18 |
| rubasov | boden: hello | 15:21 |
| rubasov | boden: what do you think about cutting a new neutron-lib release? | 15:21 |
| rubasov | boden: I'd like to get this change released to allow the neutron side of it to progress: https://review.openstack.org/635160 | 15:22 |
| rubasov | boden: I'd happily prepare the change in openstack/releases | 15:23 |
| boden | rubasov hi, it's fine with me, though ideally I'd like to land and include https://review.openstack.org/#/c/635209/. maybe mlavalle can help land it quick :) | 15:24 |
| rubasov | boden: sure I can wait for that to merge | 15:25 |
| *** yamamoto has quit IRC | 15:25 | |
| rubasov | boden: is there anything else I should look out to include? | 15:26 |
| mlavalle | boden: taking a look over the next few minutes, as soon as I finish the QoS meeting | 15:26 |
| *** liuyulong has quit IRC | 15:26 | |
| rubasov | boden: or just propose a release for master at 635209 when that gets merged? | 15:27 |
| *** gildub_ has joined #openstack-neutron | 15:28 | |
| *** moshele has quit IRC | 15:28 | |
| mlavalle | boden: done | 15:29 |
| boden | rubasov I'm not aware of anything else that is needed for a release, at least no one has mentioned anything and we just did a release last week so probably not much new... IMHO its fine to propose a release with 635209 | 15:29 |
| rubasov | boden: okay, then I'll propose the release change when the gate merges 635209 | 15:30 |
| rubasov | boden: and invite you guys in gerrit | 15:30 |
| boden | rubasov thanks... it should land soon, just got the +W vote | 15:31 |
| rubasov | boden: yep, mlavalle was fast :-) | 15:31 |
| rubasov | boden, mlavalle: thanks a lot | 15:31 |
| *** gildub has quit IRC | 15:31 | |
| mlavalle | rubasov: on the contrary. Thanks to you, lajoskatona and gibi | 15:31 |
| *** sapd1 has quit IRC | 15:32 | |
| *** njohnston_ has quit IRC | 15:32 | |
| openstackgerrit | Bernard Cafarelli proposed openstack/neutron master: Switch grenade jobs to python3 https://review.openstack.org/636356 | 15:32 |
| gibi | :) | 15:34 |
| *** NewBruce has quit IRC | 15:39 | |
| *** armax has joined #openstack-neutron | 15:40 | |
| *** liuyulong has joined #openstack-neutron | 15:41 | |
| *** Luzi has quit IRC | 15:41 | |
| *** macza has joined #openstack-neutron | 15:42 | |
| *** maciejjozefczyk has joined #openstack-neutron | 15:44 | |
| *** moshele has joined #openstack-neutron | 15:47 | |
| *** gcheresh has quit IRC | 15:50 | |
| *** mriedem_snow is now known as mriedem | 15:52 | |
| *** binh_ has quit IRC | 15:53 | |
| *** mosulica has quit IRC | 15:55 | |
| *** gildub__ has joined #openstack-neutron | 16:00 | |
| *** maciejjozefczyk has quit IRC | 16:00 | |
| openstackgerrit | Doug Wiegley proposed openstack/neutron master: Modify api and rpc default number of workers https://review.openstack.org/636363 | 16:01 |
| *** NewBruce has joined #openstack-neutron | 16:02 | |
| *** gildub_ has quit IRC | 16:02 | |
| *** markvoelker has joined #openstack-neutron | 16:03 | |
| *** NewBruce has quit IRC | 16:03 | |
| *** lpetrut has quit IRC | 16:03 | |
| *** NewBruce has joined #openstack-neutron | 16:04 | |
| *** wwriverrat has quit IRC | 16:07 | |
| *** gkadam has quit IRC | 16:09 | |
| *** NewBruce has quit IRC | 16:13 | |
| *** gkadam has joined #openstack-neutron | 16:14 | |
| *** wwriverrat has joined #openstack-neutron | 16:14 | |
| *** moshele has quit IRC | 16:15 | |
| *** wwriverrat has quit IRC | 16:18 | |
| *** wwriverrat has joined #openstack-neutron | 16:18 | |
| *** jackivanov has joined #openstack-neutron | 16:20 | |
| jackivanov | Hi everybody! Does anyone have any idea of what's going on here? https://bugs.launchpad.net/neutron/+bug/1814209 | 16:20 |
| openstack | Launchpad bug 1814209 in neutron "Messages for security rules are being sent to a wrong MQ topic. Security rules are out of sync" [Undecided,New] | 16:20 |
| *** manjeets has joined #openstack-neutron | 16:27 | |
| *** tbachman has quit IRC | 16:33 | |
| *** markvoelker has quit IRC | 16:35 | |
| openstackgerrit | Merged openstack/neutron master: Remove redundant if condition check https://review.openstack.org/635691 | 16:36 |
| *** annp has quit IRC | 16:38 | |
| *** ccamposr has quit IRC | 16:42 | |
| *** jmlowe has joined #openstack-neutron | 16:44 | |
| *** ccamposr has joined #openstack-neutron | 16:44 | |
| *** mvkr has quit IRC | 16:44 | |
| openstackgerrit | David Shaughnessy proposed openstack/neutron master: (PoC)Classification field in Neutron DSCP https://review.openstack.org/636333 | 16:44 |
| *** ccamposr__ has joined #openstack-neutron | 16:45 | |
| *** tbachman has joined #openstack-neutron | 16:45 | |
| *** ccamposr has quit IRC | 16:48 | |
| slaweq | otherwiseguy: hi | 16:49 |
| slaweq | otherwiseguy: we again hit few times issue with ovsdb timeouts https://bugs.launchpad.net/neutron/+bug/1815142 | 16:49 |
| openstack | Launchpad bug 1815142 in neutron "ovsdbapp.exceptions.TimeoutException in functional tests" [High,Confirmed] | 16:49 |
| slaweq | otherwiseguy: maybe You can take a look if You would have some time :) | 16:50 |
| *** moshele has joined #openstack-neutron | 16:51 | |
| *** ccamposr__ has quit IRC | 16:53 | |
| *** moshele has quit IRC | 16:54 | |
| otherwiseguy | slaweq: I'll take a look. Though sometimes I wonder if it is just that sometimes servers get overloaded and slow. | 16:54 |
| slaweq | otherwiseguy: I just saw exactly the same on CI meeting :) | 16:54 |
| slaweq | otherwiseguy: it looks like that for me TBH | 16:54 |
| otherwiseguy | They seem to come and go. I remember one of the other bugs on it, it'd just not show up on kibana for like a week, then hit a bunch of times, then disappear again. | 16:55 |
| slaweq | otherwiseguy: we have some issue with mysql migration tests and iops performance sometimes | 16:56 |
| slaweq | otherwiseguy: but there is one difference between those 2 issues - mysql tests are failing only on limestone-regionone and one of ovh regions | 16:57 |
| slaweq | ovsdb timeouts are on many different providers | 16:57 |
| *** ramishra has quit IRC | 16:57 | |
| *** sridharg has quit IRC | 16:58 | |
| *** tbachman has quit IRC | 16:58 | |
| *** rossella_s has quit IRC | 16:58 | |
| otherwiseguy | slaweq: I think we could improve things if I could find a good way to use unix sockets instead of TCP, but the permissions issues were...complex. Anyway, I'll dig for a bit. :) | 16:58 |
| slaweq | otherwiseguy: thx a lot :) | 16:58 |
| *** hongbin has quit IRC | 16:59 | |
| *** tbachman has joined #openstack-neutron | 17:01 | |
| *** sapd1 has joined #openstack-neutron | 17:04 | |
| *** hamzy has quit IRC | 17:06 | |
| *** rpittau has quit IRC | 17:08 | |
| *** tesseract has quit IRC | 17:09 | |
| *** liuyulong has quit IRC | 17:09 | |
| *** velizarx has quit IRC | 17:11 | |
| *** moshele has joined #openstack-neutron | 17:11 | |
| *** aojea has quit IRC | 17:14 | |
| *** sapd1 has quit IRC | 17:14 | |
| trident | mlavalle: A friend with an Queens OSA on CentOS installation probably bumped into exactly that missing KillFilter for neutron-keepalived-state-change just today. "unauthorized command: kill -15 7647 (no filter matched)" - really strange as I have never seen that on any other installation between Kilo and Rocky.... | 17:15 |
| *** mvkr has joined #openstack-neutron | 17:16 | |
| openstackgerrit | Terry Wilson proposed openstack/ovsdbapp master: Break out match_fn from matches https://review.openstack.org/636392 | 17:16 |
| *** jlibosva has quit IRC | 17:18 | |
| *** gkadam has quit IRC | 17:19 | |
| *** jmlowe has quit IRC | 17:21 | |
| otherwiseguy | lucasagomes: I notice we have a non-voting ooo job in the networking-ovn gating queue. Does that make sense? | 17:24 |
| *** rossella_s has joined #openstack-neutron | 17:27 | |
| openstackgerrit | Doug Wiegley proposed openstack/neutron master: Modify api and rpc default number of workers https://review.openstack.org/636363 | 17:29 |
| openstackgerrit | Doug Wiegley proposed openstack/neutron master: Modify api and rpc default number of workers https://review.openstack.org/636363 | 17:30 |
| *** markvoelker has joined #openstack-neutron | 17:32 | |
| *** armax has quit IRC | 17:32 | |
| *** armax has joined #openstack-neutron | 17:33 | |
| openstackgerrit | boden proposed openstack/neutron master: use callback payloads for ROUTER_CONTROLLER events https://review.openstack.org/636394 | 17:35 |
| *** moshele has quit IRC | 17:36 | |
| openstackgerrit | boden proposed openstack/networking-odl master: use callback payloads for ROUTER_CONTROLLER events https://review.openstack.org/636395 | 17:36 |
| *** spsurya has quit IRC | 17:40 | |
| mlavalle | trident: ahhh, that is insteresting, because in the rror you report, you can actually see the cause of the kill failure (no filter matched). With the tests I am debugging in master, the additional complication is that I cannot see the cause of the kill failure. Thanks. This is useful | 17:41 |
| mlavalle | slaweq: ^^^^ | 17:41 |
| *** idlemind has joined #openstack-neutron | 17:42 | |
| *** yamamoto has joined #openstack-neutron | 17:44 | |
| *** mriedem is now known as mriedem_afk | 17:45 | |
| *** yamamoto has quit IRC | 17:48 | |
| mlavalle | trident: do you think your firend could send me a traceback? | 17:50 |
| *** davidsha has quit IRC | 17:51 | |
| trident | mlavalle: I'll ask him! | 17:51 |
| mlavalle | trident: thanks. Very useful information anyways | 17:51 |
| *** armax has quit IRC | 17:53 | |
| *** tbachman has quit IRC | 17:56 | |
| openstackgerrit | Merged openstack/neutron-lib master: rehome trunk related callback resource names https://review.openstack.org/635209 | 17:57 |
| boden | rubasov ^ | 17:58 |
| slaweq | mlavalle: I see | 17:59 |
| slaweq | mlavalle: maybe it's different issue on python2.7 and python3.6 | 17:59 |
| *** salmankhan has quit IRC | 18:03 | |
| *** markvoelker has quit IRC | 18:06 | |
| *** panda is now known as panda|off | 18:06 | |
| openstackgerrit | Merged openstack/networking-ovn master: Target proper Chassis events for agent liveness https://review.openstack.org/635354 | 18:08 |
| openstackgerrit | Merged openstack/networking-ovn master: stop using common db mixin https://review.openstack.org/636164 | 18:08 |
| *** wwriverrat has quit IRC | 18:09 | |
| *** tssurya has quit IRC | 18:09 | |
| *** wwriverrat has joined #openstack-neutron | 18:10 | |
| *** yamamoto has joined #openstack-neutron | 18:11 | |
| *** tbachman has joined #openstack-neutron | 18:13 | |
| *** mosulica has joined #openstack-neutron | 18:18 | |
| rubasov | boden: thank you | 18:22 |
| rubasov | boden: here's the proposed 1.24.0: https://review.openstack.org/636407 | 18:23 |
| boden | rubasov ack, I'll take a look in a bit | 18:23 |
| boden | thx | 18:23 |
| *** jpena is now known as jpena|off | 18:23 | |
| *** imacdonn has quit IRC | 18:24 | |
| *** imacdonn has joined #openstack-neutron | 18:25 | |
| *** armax has joined #openstack-neutron | 18:31 | |
| *** panda|off has quit IRC | 18:34 | |
| *** ralonsoh has quit IRC | 18:37 | |
| *** gcerami_ has joined #openstack-neutron | 18:37 | |
| *** mosulica has quit IRC | 18:39 | |
| *** mosulica has joined #openstack-neutron | 18:41 | |
| *** mosulica has quit IRC | 18:43 | |
| *** ociuhandu_ has joined #openstack-neutron | 18:43 | |
| *** hamzy has joined #openstack-neutron | 18:43 | |
| *** wolverineav has joined #openstack-neutron | 18:45 | |
| *** ociuhandu has quit IRC | 18:47 | |
| *** ociuhandu_ has quit IRC | 18:48 | |
| *** _fragatina has joined #openstack-neutron | 18:55 | |
| *** ociuhandu has joined #openstack-neutron | 18:55 | |
| *** gildub__ has quit IRC | 18:56 | |
| *** slaweq has quit IRC | 18:59 | |
| *** ociuhandu has quit IRC | 18:59 | |
| *** ociuhandu has joined #openstack-neutron | 19:00 | |
| *** wolverineav has quit IRC | 19:00 | |
| *** binh_ has joined #openstack-neutron | 19:01 | |
| *** boden_ has joined #openstack-neutron | 19:01 | |
| *** markvoelker has joined #openstack-neutron | 19:03 | |
| *** jiapei_ has joined #openstack-neutron | 19:03 | |
| *** lemko_ has joined #openstack-neutron | 19:03 | |
| *** wolverineav has joined #openstack-neutron | 19:04 | |
| *** ociuhandu has quit IRC | 19:04 | |
| *** adrianreza has quit IRC | 19:04 | |
| *** lemko has quit IRC | 19:04 | |
| *** boden has quit IRC | 19:04 | |
| *** sm806 has quit IRC | 19:04 | |
| *** jiapei has quit IRC | 19:04 | |
| *** mugsie has quit IRC | 19:04 | |
| *** yamahata__ has quit IRC | 19:04 | |
| *** dr_gogeta86 has quit IRC | 19:04 | |
| *** jiapei_ is now known as jiapei | 19:04 | |
| *** lemko_ is now known as lemko | 19:04 | |
| openstackgerrit | boden proposed openstack/neutron master: use callback payloads for AGENT OVS_RESTARTED events https://review.openstack.org/636421 | 19:04 |
| *** yamahata__ has joined #openstack-neutron | 19:04 | |
| *** sm806 has joined #openstack-neutron | 19:05 | |
| *** mugsie has joined #openstack-neutron | 19:05 | |
| *** dr_gogeta86 has joined #openstack-neutron | 19:05 | |
| *** boden_ has quit IRC | 19:05 | |
| *** boden has joined #openstack-neutron | 19:06 | |
| *** ociuhandu has joined #openstack-neutron | 19:06 | |
| openstackgerrit | boden proposed openstack/neutron master: use callback payloads for AGENT OVS_RESTARTED events https://review.openstack.org/636421 | 19:08 |
| mlavalle | boden: I +1ed https://review.openstack.org/#/c/636407 but asked to wait for your +1 | 19:09 |
| *** lemko has quit IRC | 19:09 | |
| openstackgerrit | boden proposed openstack/networking-bagpipe master: use callback payloads for AGENT OVS_RESTARTED events https://review.openstack.org/636422 | 19:09 |
| *** gcerami_ is now known as panda | 19:14 | |
| trident | mlavalle: Got this from him: http://paste.openstack.org/show/744971/ | 19:15 |
| mlavalle | trident: thank you very much. Very useful! | 19:19 |
| *** tidwellr has joined #openstack-neutron | 19:20 | |
| trident | mlavalle: We'll probably play around a bit with adding filters tomorrow and see if we can get a solution. Will get back to you with what we find. | 19:21 |
| mlavalle | trident: cool | 19:22 |
| trident | mlavalle: If you find something else on the issue, please tag me here. | 19:22 |
| mlavalle | trident: will do | 19:22 |
| trident | mlavalle: Thanks! | 19:23 |
| openstackgerrit | Ryan Tidwell proposed openstack/neutron master: Enable adoption of subnets into a subnet pool https://review.openstack.org/348080 | 19:24 |
| *** abaindur has joined #openstack-neutron | 19:24 | |
| *** markvoelker has quit IRC | 19:36 | |
| openstackgerrit | Sai Sindhur Malleni proposed openstack/neutron master: Enhance rally-job for sec group rules to creates multiple rules https://review.openstack.org/636431 | 19:42 |
| *** wolverineav has quit IRC | 19:43 | |
| *** wolverineav has joined #openstack-neutron | 19:44 | |
| *** wolverineav has quit IRC | 19:46 | |
| *** wolverineav has joined #openstack-neutron | 19:47 | |
| *** wolverineav has quit IRC | 19:49 | |
| *** wolverineav has joined #openstack-neutron | 19:50 | |
| openstackgerrit | Merged openstack/neutron master: Change SR-IOV agent to log message after failure https://review.openstack.org/633080 | 19:53 |
| openstackgerrit | Brian Haley proposed openstack/neutron master: Change common agent code to log message after failure https://review.openstack.org/633081 | 19:54 |
| *** wolverineav has quit IRC | 19:55 | |
| *** wolverineav has joined #openstack-neutron | 19:57 | |
| *** mriedem_afk is now known as mriedem | 20:05 | |
| openstackgerrit | Doug Wiegley proposed openstack/neutron master: Modify api and rpc default number of workers https://review.openstack.org/636363 | 20:12 |
| *** moshele has joined #openstack-neutron | 20:14 | |
| *** slaweq has joined #openstack-neutron | 20:19 | |
| *** wolverineav has quit IRC | 20:20 | |
| *** wolverineav has joined #openstack-neutron | 20:21 | |
| *** wolverineav has quit IRC | 20:26 | |
| *** moshele has quit IRC | 20:27 | |
| *** penick has joined #openstack-neutron | 20:32 | |
| *** markvoelker has joined #openstack-neutron | 20:33 | |
| *** moshele has joined #openstack-neutron | 20:34 | |
| openstackgerrit | boden proposed openstack/neutron master: remove neutron.db.api references https://review.openstack.org/633283 | 20:49 |
| openstackgerrit | Doug Wiegley proposed openstack/neutron master: Make security group API reads and deletes faster. https://review.openstack.org/634500 | 20:49 |
| openstackgerrit | Carlos Goncalves proposed openstack/networking-ovn master: [DNM] Add OVN Octavia CI jobs https://review.openstack.org/589712 | 20:49 |
| *** jmlowe has joined #openstack-neutron | 20:55 | |
| openstackgerrit | Carlos Goncalves proposed openstack/networking-ovn master: [DNM] Add OVN Octavia CI jobs https://review.openstack.org/589712 | 20:57 |
| *** wolverineav has joined #openstack-neutron | 21:00 | |
| *** slaweq_ has joined #openstack-neutron | 21:03 | |
| *** boden_ has joined #openstack-neutron | 21:04 | |
| *** slaweq has quit IRC | 21:06 | |
| *** mugsie has quit IRC | 21:06 | |
| *** markvoelker has quit IRC | 21:06 | |
| *** boden has quit IRC | 21:06 | |
| *** mugsie has joined #openstack-neutron | 21:06 | |
| *** moshele has quit IRC | 21:18 | |
| *** moshele has joined #openstack-neutron | 21:19 | |
| openstackgerrit | Carlos Goncalves proposed openstack/networking-ovn master: [DNM] Add OVN Octavia CI jobs https://review.openstack.org/589712 | 21:20 |
| *** dsneddon has quit IRC | 21:22 | |
| *** wolverineav has quit IRC | 21:30 | |
| slaweq_ | mlavalle: are You around? | 21:32 |
| mlavalle | slaweq_: yes | 21:32 |
| *** slaweq_ is now known as slaweq | 21:32 | |
| slaweq | mlavalle: I was just debugging why job neutron-tempest-dvr-ha-multinode-full migrated to zuulv3 is failing | 21:33 |
| slaweq | results of this job are in http://logs.openstack.org/79/633979/7/check/neutron-tempest-dvr-ha-multinode-full/2ed9446/job-output.txt.gz | 21:33 |
| slaweq | I asked frickler to hold me those nodes and I was now debugging on it | 21:33 |
| slaweq | it looks that it is indeed issue with this missing rootwrap filter | 21:34 |
| *** wolverineav has joined #openstack-neutron | 21:34 | |
| slaweq | and it looks that it is exactly the same issue like You have in neutron-tempest-plugin-scenario job with dvr | 21:34 |
| slaweq | I reproduced it with steps: | 21:34 |
| slaweq | 1. spawn vm and associate FIP to it - all works fine | 21:34 |
| slaweq | 2. remove vm | 21:35 |
| slaweq | 3. spawn new vm on other host than vm from p.1 | 21:35 |
| slaweq | 4. associate same FIP to this new vm | 21:35 |
| slaweq | 5. it is not working at all, and L3 agent is hang | 21:35 |
| slaweq | after kill l3 agent and start it again, FIP is again pinging | 21:36 |
| mlavalle | slaweq: yeah when you get that kill command failure, the agent just hangs | 21:36 |
| slaweq | yep | 21:36 |
| slaweq | I had to kill it with -9 | 21:36 |
| mlavalle | so I think we have gotten to the root cause of many problems | 21:36 |
| slaweq | at least in multinode dvr jobs :) | 21:36 |
| mlavalle | earlier today, trident gave me a traceback from a Queens deployment | 21:37 |
| slaweq | please send patch with this missing filter, and I will then rebase my patch https://review.openstack.org/#/c/633979/ on top of it to check | 21:37 |
| mlavalle | and there you can clearly see that the kill fails for lack of correct filter | 21:37 |
| mlavalle | slaweq: yes, Ill do it later today | 21:37 |
| mlavalle | slaweq: see here: http://paste.openstack.org/show/744971/ | 21:38 |
| mlavalle | this is from Queens^^^^ | 21:38 |
| slaweq | yep, and I think that maybe there is some other issue in L3 agent or oslo rootwrap that cause this hang when running on python 36 | 21:39 |
| mlavalle | The other thing is that we have a problem with oslo_rootwrap | 21:39 |
| slaweq | I think we are thinking about the same thing :) | 21:39 |
| mlavalle | which is scrambling completely this kind of errors and makes it difficult to debug | 21:39 |
| *** ArchiFleKs has quit IRC | 21:40 | |
| slaweq | yep, but that is on python3, on python 2.7 there wasn't this issue but there was clear message as trident reported today :) | 21:40 |
| slaweq | in neutron-tempest-dvr-ha-multinode-full job I also noticed that when I tried to switch to python 3 | 21:40 |
| slaweq | it is running with python 2.7 currently | 21:41 |
| *** wolverineav has quit IRC | 21:41 | |
| *** wolverineav has joined #openstack-neutron | 21:41 | |
| slaweq | mlavalle: but what is strange is fact that in logs for neutron-tempest-dvr-ha-multinode-full job on python27 I don't see any errors related to this missing rootwrap filter | 21:43 |
| mlavalle | slaweq: currently, in check queue, is neutron-tempest-plugin-dvr-multinode-scenario running in Python3? | 21:46 |
| *** hamzy has quit IRC | 21:46 | |
| slaweq | mlavalle: yes, since some time it's running python 3.6 | 21:46 |
| mlavalle | slaweq: how long ago? | 21:46 |
| slaweq | as all neutron-tempest-plugin jobs | 21:46 |
| slaweq | https://review.openstack.org/#/c/621401/ | 21:47 |
| slaweq | since December 4th | 21:47 |
| mlavalle | ok, something to keep in mind | 21:48 |
| mlavalle | but it fits my recollection of the problem | 21:48 |
| slaweq | and neutron-tempest-dvr-ha-multinode-full is very similar job but is still running on python 2.7 | 21:48 |
| mlavalle | the trunk test in neutron-tempest-plugin-dvr-multinode-scenario was failing due to something else | 21:48 |
| mlavalle | and then the nature of the failure changed | 21:49 |
| slaweq | probably yes, but other tests should works (I hope) with this fix | 21:49 |
| mlavalle | yes | 21:49 |
| slaweq | ok, lets try with this fix and we will see :) | 21:49 |
| mlavalle | yes, one thiong at a time | 21:49 |
| mlavalle | LOL | 21:49 |
| slaweq | yes :) | 21:49 |
| slaweq | ok, thx | 21:49 |
| slaweq | I just wanted to share with You my findings as it was related :) | 21:50 |
| slaweq | now I can get back nodes to infra | 21:50 |
| *** ArchiFleKs has joined #openstack-neutron | 21:52 | |
| *** moshele has quit IRC | 21:52 | |
| *** moshele has joined #openstack-neutron | 21:53 | |
| *** mchlumsky has quit IRC | 21:56 | |
| *** markvoelker has joined #openstack-neutron | 22:03 | |
| openstackgerrit | Sai Sindhur Malleni proposed openstack/neutron master: Remove the rally-jobs plugins https://review.openstack.org/636448 | 22:08 |
| *** moshele has quit IRC | 22:08 | |
| *** maciejjozefczyk has joined #openstack-neutron | 22:09 | |
| *** maciejjozefczyk has quit IRC | 22:13 | |
| openstackgerrit | Sai Sindhur Malleni proposed openstack/neutron master: Remove trunk rally scenario from plugins https://review.openstack.org/636450 | 22:14 |
| openstackgerrit | Merged openstack/neutron-interconnection master: Define default policies in code https://review.openstack.org/633534 | 22:23 |
| openstackgerrit | Slawek Kaplonski proposed openstack/neutron-lib master: Add policy module to neutron-lib https://review.openstack.org/636453 | 22:24 |
| *** boden_ has quit IRC | 22:31 | |
| *** slaweq has quit IRC | 22:34 | |
| *** markvoelker has quit IRC | 22:36 | |
| *** slaweq has joined #openstack-neutron | 22:50 | |
| *** armax has quit IRC | 22:53 | |
| *** wolverineav has quit IRC | 22:54 | |
| *** armax has joined #openstack-neutron | 22:54 | |
| *** armax has quit IRC | 22:54 | |
| *** slaweq has quit IRC | 22:54 | |
| *** wolverineav has joined #openstack-neutron | 22:55 | |
| *** wolverineav has quit IRC | 22:58 | |
| *** wolverineav has joined #openstack-neutron | 22:59 | |
| openstackgerrit | Ryan Tidwell proposed openstack/neutron-dynamic-routing master: Bump neutron-lib to 1.23.0 https://review.openstack.org/636465 | 23:02 |
| openstackgerrit | Merged openstack/networking-ovn master: Make networking-ovn-migration-mtu output less confusing https://review.openstack.org/635042 | 23:10 |
| *** tkajinam has joined #openstack-neutron | 23:16 | |
| openstackgerrit | LIU Yulong proposed openstack/neutron master: Dynamically increase l3 router process queue green pool size https://review.openstack.org/633869 | 23:20 |
| openstackgerrit | LIU Yulong proposed openstack/neutron master: Check dvr local router is up during port creation https://review.openstack.org/633871 | 23:22 |
| *** ociuhandu has quit IRC | 23:30 | |
| *** ociuhandu has joined #openstack-neutron | 23:30 | |
| *** markvoelker has joined #openstack-neutron | 23:33 | |
| trident | mlavalle: https://github.com/openstack/neutron/commit/f046031456c3718f506ff22b4195d36246cf722f#diff-0598a550e4aed25caabb3c320a93b652 | 23:36 |
| trident | mlavalle: That's probably the commit breaking my friends installation as the process will show up as: /openstack/venvs/neutron-17.1.6/bin/python /usr/local/bin/neutron-keepalived-state-change | 23:38 |
| trident | mlavalle: Not sure if related to your issue though. | 23:38 |
| openstackgerrit | Swaminathan Vasudevan proposed openstack/neutron master: Packets getting lost during SNAT with too many connections https://review.openstack.org/636473 | 23:41 |
| *** macza has quit IRC | 23:42 | |
| *** macza has joined #openstack-neutron | 23:42 | |
| *** zhanglong has joined #openstack-neutron | 23:45 | |
| *** wolverineav has quit IRC | 23:49 | |
| *** wolverineav has joined #openstack-neutron | 23:50 | |
| *** rkukura has quit IRC | 23:52 | |
| *** wolverineav has quit IRC | 23:56 | |
| trident | mlavalle: Or hmm, that should not have been committed in queens, right? I have asked my friend to double check how his current l3.filters look. We'll see. | 23:59 |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!