Thursday, 2020-02-20

*** errr has joined #openstack-ansible00:01
*** errr has quit IRC00:02
*** errr has joined #openstack-ansible00:09
*** errr has quit IRC00:10
*** errr has joined #openstack-ansible00:11
*** errr has joined #openstack-ansible00:12
*** errr has quit IRC00:13
*** errr has joined #openstack-ansible00:14
*** errr has quit IRC00:15
*** errr has joined #openstack-ansible00:17
*** errr has joined #openstack-ansible00:19
*** errr has joined #openstack-ansible00:26
*** errr has quit IRC00:27
*** errr has joined #openstack-ansible00:28
*** errr has quit IRC00:29
*** errr has joined #openstack-ansible00:31
*** errr has quit IRC00:32
*** errr has joined #openstack-ansible00:33
*** errr has quit IRC00:34
*** errr has joined #openstack-ansible00:35
*** errr has quit IRC00:36
*** errr has joined #openstack-ansible00:38
*** errr has quit IRC00:38
*** errr has joined #openstack-ansible00:40
*** errr has quit IRC00:41
*** errr has joined #openstack-ansible00:42
*** errr has quit IRC00:43
*** errr has joined #openstack-ansible00:45
*** errr has quit IRC00:45
*** errr has joined #openstack-ansible00:47
*** gyee has quit IRC00:48
*** errr has quit IRC00:48
*** errr has joined #openstack-ansible00:50
*** errr has quit IRC00:50
*** errr has joined #openstack-ansible00:52
*** errr has quit IRC00:53
*** errr has joined #openstack-ansible00:54
*** errr has joined #openstack-ansible00:56
*** errr has quit IRC00:57
*** errr has joined #openstack-ansible00:58
*** errr has quit IRC00:59
*** gillesMo has quit IRC01:00
*** errr has joined #openstack-ansible01:01
*** errr has quit IRC01:01
*** errr has joined #openstack-ansible01:03
*** errr has joined #openstack-ansible01:05
*** errr has quit IRC01:07
*** errr has joined #openstack-ansible01:16
*** errr has quit IRC01:17
*** errr has joined #openstack-ansible01:18
*** errr has quit IRC01:19
*** errr has joined #openstack-ansible01:21
*** errr has quit IRC01:22
*** errr has joined #openstack-ansible01:23
*** errr has joined #openstack-ansible01:25
*** errr has quit IRC01:26
*** errr has joined #openstack-ansible01:28
*** errr has joined #openstack-ansible01:29
*** errr has quit IRC01:29
*** errr has joined #openstack-ansible01:31
*** errr has quit IRC01:32
*** errr has joined #openstack-ansible01:33
*** errr has quit IRC01:35
*** errr has joined #openstack-ansible01:35
*** errr has quit IRC01:37
*** errr has joined #openstack-ansible01:38
*** errr has quit IRC01:40
*** errr has joined #openstack-ansible01:41
*** errr has quit IRC01:42
*** errr has joined #openstack-ansible01:43
*** errr has quit IRC01:44
*** errr has joined #openstack-ansible01:46
*** errr has quit IRC01:47
*** errr has joined #openstack-ansible01:48
*** errr has quit IRC01:49
*** errr has joined #openstack-ansible01:50
*** errr has quit IRC01:51
*** errr has joined #openstack-ansible01:53
*** errr has quit IRC01:54
*** errr has joined #openstack-ansible01:56
*** errr has quit IRC01:57
*** errr has joined #openstack-ansible01:58
*** errr has quit IRC01:59
*** errr has joined #openstack-ansible02:00
*** errr has quit IRC02:01
*** errr has joined #openstack-ansible02:03
*** errr has quit IRC02:04
*** errr has joined #openstack-ansible02:05
*** errr has quit IRC02:06
*** errr has joined #openstack-ansible02:07
*** errr has quit IRC02:08
*** errr has joined #openstack-ansible02:10
*** errr has quit IRC02:10
*** errr has joined #openstack-ansible02:12
*** errr has quit IRC02:14
*** errr has joined #openstack-ansible02:16
*** errr has quit IRC02:17
*** errr has joined #openstack-ansible02:18
*** errr has quit IRC02:20
*** errr has joined #openstack-ansible02:21
*** errr has quit IRC02:22
*** errr has joined #openstack-ansible02:23
*** errr has quit IRC02:24
*** errr has joined #openstack-ansible02:25
*** errr has joined #openstack-ansible02:28
*** hwoarang has quit IRC02:29
*** errr has joined #openstack-ansible02:30
*** hwoarang has joined #openstack-ansible02:31
*** errr has quit IRC02:31
*** errr has joined #openstack-ansible02:33
*** errr has quit IRC02:33
*** errr has joined #openstack-ansible02:34
*** errr has quit IRC02:34
*** errr has joined #openstack-ansible02:36
*** errr has quit IRC02:37
*** errr has joined #openstack-ansible02:39
*** errr has quit IRC02:40
*** errr has joined #openstack-ansible02:41
*** errr has quit IRC02:42
*** errr has joined #openstack-ansible02:43
*** errr has joined #openstack-ansible02:46
*** errr has joined #openstack-ansible02:47
*** errr has quit IRC02:48
*** errr has joined #openstack-ansible02:49
*** errr has joined #openstack-ansible02:50
*** errr has quit IRC02:52
*** openstackstatus has joined #openstack-ansible02:55
*** ChanServ sets mode: +v openstackstatus02:55
*** errr has joined #openstack-ansible03:09
*** errr has quit IRC03:12
*** errr has joined #openstack-ansible03:15
*** errr has quit IRC03:17
*** errr has joined #openstack-ansible03:21
*** errr has quit IRC03:22
*** errr has joined #openstack-ansible03:24
*** errr has quit IRC03:25
*** errr has joined #openstack-ansible03:26
*** errr has quit IRC03:27
*** errr has joined #openstack-ansible03:28
*** errr has quit IRC03:29
*** errr has joined #openstack-ansible03:31
*** errr has quit IRC03:31
*** errr has joined #openstack-ansible03:33
*** errr has quit IRC03:34
*** errr has joined #openstack-ansible03:35
*** errr has quit IRC03:36
*** errr has joined #openstack-ansible03:38
*** errr has quit IRC03:39
*** errr has joined #openstack-ansible03:40
*** evrardjp has quit IRC03:42
*** errr has joined #openstack-ansible03:43
*** errr has quit IRC03:44
*** evrardjp has joined #openstack-ansible03:45
*** errr has joined #openstack-ansible03:45
*** errr has quit IRC03:46
*** errr has joined #openstack-ansible03:47
*** errr has joined #openstack-ansible03:48
*** errr has joined #openstack-ansible03:49
*** errr has quit IRC03:50
*** errr has joined #openstack-ansible03:52
*** errr has quit IRC03:53
*** errr has joined #openstack-ansible03:55
*** errr has quit IRC03:55
*** errr has joined #openstack-ansible03:57
*** errr has quit IRC03:58
*** errr has joined #openstack-ansible03:59
*** errr has quit IRC04:00
*** hwoarang has quit IRC04:01
*** errr has joined #openstack-ansible04:01
*** errr has quit IRC04:02
*** errr has joined #openstack-ansible04:04
*** errr has joined #openstack-ansible04:07
*** errr has quit IRC04:07
*** errr has joined #openstack-ansible04:09
*** errr has quit IRC04:10
*** errr has joined #openstack-ansible04:12
*** errr has quit IRC04:13
*** errr has joined #openstack-ansible04:14
*** errr has joined #openstack-ansible04:19
*** errr has quit IRC04:20
*** udesale has joined #openstack-ansible04:20
*** nicolasbock has quit IRC04:21
*** errr has joined #openstack-ansible04:22
*** errr has quit IRC04:23
*** errr has joined #openstack-ansible04:24
*** errr has joined #openstack-ansible04:26
*** errr has quit IRC04:27
*** errr has joined #openstack-ansible04:29
*** errr has joined #openstack-ansible04:30
*** errr has quit IRC04:30
*** errr has joined #openstack-ansible04:32
*** errr has quit IRC04:33
*** errr has joined #openstack-ansible04:34
*** errr has joined #openstack-ansible04:37
*** errr has quit IRC04:37
*** errr has joined #openstack-ansible04:39
*** errr has quit IRC04:40
*** errr has joined #openstack-ansible04:41
*** errr has quit IRC04:42
*** dave-mccowan has joined #openstack-ansible04:43
*** errr has joined #openstack-ansible04:43
*** errr has quit IRC04:45
*** errr has joined #openstack-ansible04:46
*** errr has quit IRC04:47
*** errr has joined #openstack-ansible04:48
*** errr has quit IRC04:49
*** errr has joined #openstack-ansible04:50
*** errr has quit IRC04:52
*** errr has joined #openstack-ansible04:53
*** errr has joined #openstack-ansible04:54
*** errr has quit IRC04:55
*** errr has joined #openstack-ansible04:56
*** errr has quit IRC04:57
*** errr has joined #openstack-ansible04:58
*** errr has joined #openstack-ansible05:04
*** errr has quit IRC05:05
*** errr has joined #openstack-ansible05:06
*** errr has quit IRC05:07
*** errr has joined #openstack-ansible05:08
*** errr has quit IRC05:11
*** errr has joined #openstack-ansible05:13
*** errr has quit IRC05:14
*** errr has joined #openstack-ansible05:17
*** errr has quit IRC05:19
*** evrardjp has quit IRC05:34
*** evrardjp has joined #openstack-ansible05:34
*** dave-mccowan has quit IRC05:47
*** errr has joined #openstack-ansible05:54
*** errr has quit IRC05:55
*** errr has joined #openstack-ansible05:56
*** errr has quit IRC05:56
*** errr has joined #openstack-ansible05:57
*** errr has joined #openstack-ansible05:59
*** errr has quit IRC06:00
*** errr has joined #openstack-ansible06:02
*** errr has quit IRC06:03
*** udesale_ has joined #openstack-ansible06:03
*** errr has joined #openstack-ansible06:04
*** errr has quit IRC06:05
*** udesale has quit IRC06:06
*** errr has joined #openstack-ansible06:06
*** errr has quit IRC06:07
*** elenalindq has joined #openstack-ansible06:08
*** errr has joined #openstack-ansible06:09
*** elenalindq has quit IRC06:09
*** elenalindq has joined #openstack-ansible06:09
*** errr has quit IRC06:10
*** errr has joined #openstack-ansible06:11
*** errr has joined #openstack-ansible06:14
*** errr has quit IRC06:15
*** errr has joined #openstack-ansible06:16
*** errr has quit IRC06:17
*** errr has joined #openstack-ansible06:19
*** errr has quit IRC06:19
*** errr has joined #openstack-ansible06:21
*** yolanda has joined #openstack-ansible06:22
*** errr has joined #openstack-ansible06:23
*** errr has quit IRC06:24
*** errr has joined #openstack-ansible06:25
*** errr has quit IRC06:26
*** errr has joined #openstack-ansible06:28
*** errr has quit IRC06:29
*** errr has joined #openstack-ansible06:30
*** errr has quit IRC06:31
*** ahosam has joined #openstack-ansible06:33
*** errr has joined #openstack-ansible06:36
*** errr has joined #openstack-ansible06:38
*** errr has quit IRC06:39
*** errr has joined #openstack-ansible06:40
*** bhyrted has quit IRC06:40
*** bhyrted has joined #openstack-ansible06:40
*** errr has quit IRC06:40
*** errr has joined #openstack-ansible06:41
*** errr has quit IRC06:42
*** errr has joined #openstack-ansible06:43
*** errr has joined #openstack-ansible06:44
*** errr has quit IRC06:45
*** cshen has joined #openstack-ansible06:58
*** errr has joined #openstack-ansible07:05
*** errr has quit IRC07:06
*** errr has joined #openstack-ansible07:07
*** errr has quit IRC07:08
*** cshen has quit IRC07:08
*** errr has joined #openstack-ansible07:09
*** errr has quit IRC07:10
*** errr has joined #openstack-ansible07:12
*** miloa has joined #openstack-ansible07:12
*** errr has quit IRC07:12
*** errr has joined #openstack-ansible07:14
*** errr has quit IRC07:15
*** errr has joined #openstack-ansible07:16
*** errr has quit IRC07:17
*** errr has joined #openstack-ansible07:19
*** ivve has quit IRC07:19
*** errr has quit IRC07:20
*** errr has joined #openstack-ansible07:21
*** errr has quit IRC07:22
*** errr has joined #openstack-ansible07:23
*** errr has quit IRC07:24
*** alvinstarr has quit IRC07:35
*** errr has joined #openstack-ansible07:36
*** errr has quit IRC07:37
*** errr has joined #openstack-ansible07:38
*** errr has quit IRC07:39
*** errr has joined #openstack-ansible07:40
*** errr has quit IRC07:41
*** errr has joined #openstack-ansible07:43
*** errr has quit IRC07:43
*** rgogunskiy has joined #openstack-ansible07:44
*** errr has joined #openstack-ansible07:45
*** errr has quit IRC07:45
*** errr has joined #openstack-ansible07:46
*** errr has quit IRC07:46
*** errr has joined #openstack-ansible07:48
*** errr has quit IRC07:49
*** errr has joined #openstack-ansible07:50
*** elenalindq has quit IRC07:51
*** errr has quit IRC07:51
*** jbadiapa has joined #openstack-ansible07:52
*** errr has joined #openstack-ansible07:53
*** errr has quit IRC07:54
*** errr has joined #openstack-ansible07:55
*** errr has quit IRC07:56
*** leo8a has joined #openstack-ansible07:57
*** elenalindq has joined #openstack-ansible07:58
*** errr has joined #openstack-ansible07:58
*** errr has joined #openstack-ansible08:00
*** errr has quit IRC08:01
*** errr has joined #openstack-ansible08:03
*** leo8a has quit IRC08:03
*** errr has quit IRC08:04
*** errr has joined #openstack-ansible08:05
*** cshen has joined #openstack-ansible08:06
*** errr has quit IRC08:06
*** leo8a has joined #openstack-ansible08:07
*** errr has joined #openstack-ansible08:08
*** errr has quit IRC08:09
leo8aHi there guys, good morning! I have installed Train in Ubuntu 18 (bionic) using openstack-ansible (AIO) and all ok. I have ping to VMs attached to public network (flat) created by tempest (172.29.248.0/22), but the problem is when i try to create a VLAN network and then try to ping VMs attached to that new VLAN network (pls bear in mind this is not a VXLAN network). Any help here is highly appreciated! I can also provide more details if needed...08:11
*** DanyC has joined #openstack-ansible08:12
*** errr has joined #openstack-ansible08:16
*** errr has quit IRC08:17
*** DanyC has quit IRC08:17
*** DanyC has joined #openstack-ansible08:17
*** errr has joined #openstack-ansible08:18
*** errr has quit IRC08:19
*** errr has joined #openstack-ansible08:20
*** leo8a has quit IRC08:21
*** errr has quit IRC08:21
*** DanyC has quit IRC08:22
*** errr has joined #openstack-ansible08:23
*** leo8a has joined #openstack-ansible08:23
*** errr has quit IRC08:24
*** jcosmao has joined #openstack-ansible08:24
*** DanyC has joined #openstack-ansible08:24
*** DanyC has quit IRC08:25
*** errr has joined #openstack-ansible08:25
*** rpittau|afk is now known as rpittau08:26
*** errr has quit IRC08:26
*** errr has joined #openstack-ansible08:27
*** errr has quit IRC08:28
*** tosky has joined #openstack-ansible08:29
*** errr has joined #openstack-ansible08:30
*** errr has quit IRC08:31
*** errr has joined #openstack-ansible08:32
*** errr has quit IRC08:32
*** errr has joined #openstack-ansible08:37
*** errr has quit IRC08:37
*** errr has joined #openstack-ansible08:43
*** errr has quit IRC08:43
*** errr has joined #openstack-ansible08:45
*** errr has quit IRC08:46
*** errr has joined #openstack-ansible08:47
*** errr has quit IRC08:48
*** errr has joined #openstack-ansible08:50
*** errr has quit IRC08:50
*** errr has joined #openstack-ansible08:52
*** ivve has joined #openstack-ansible08:53
*** errr has quit IRC08:53
*** errr has joined #openstack-ansible08:54
*** errr has quit IRC08:55
*** errr has joined #openstack-ansible08:57
*** errr has quit IRC08:57
*** errr has joined #openstack-ansible08:59
*** errr has quit IRC09:00
*** errr has joined #openstack-ansible09:02
*** errr has quit IRC09:03
*** errr has joined #openstack-ansible09:04
leo8aHello! Anyones to ask (the specifics) of how br-vlan is conected inside an AIO node?09:04
*** errr has quit IRC09:05
*** errr has joined #openstack-ansible09:06
*** errr has quit IRC09:07
*** errr has joined #openstack-ansible09:09
*** errr has quit IRC09:10
*** errr has joined #openstack-ansible09:11
*** errr has quit IRC09:12
*** errr has joined #openstack-ansible09:13
*** errr has quit IRC09:14
*** errr has joined #openstack-ansible09:14
*** errr has joined #openstack-ansible09:15
*** errr has quit IRC09:16
*** maharg101 has joined #openstack-ansible09:17
*** errr has joined #openstack-ansible09:18
*** errr has quit IRC09:18
*** errr has joined #openstack-ansible09:20
*** errr has quit IRC09:21
*** errr has joined #openstack-ansible09:22
*** errr has quit IRC09:23
*** errr has joined #openstack-ansible09:25
*** errr has quit IRC09:25
jrosserleo8a: you should read this about how the host networking is set up for the AIO https://github.com/openstack/openstack-ansible/blob/master/tests/roles/bootstrap-host/tasks/prepare_networking.yml09:26
*** errr has joined #openstack-ansible09:27
*** hwoarang has joined #openstack-ansible09:27
jrosserleo8a: br-vlan is defined here https://github.com/openstack/openstack-ansible/blob/master/tests/roles/bootstrap-host/tasks/prepare_networking.yml#L143-L15409:29
*** errr has quit IRC09:29
*** leo8a has quit IRC09:29
*** leo8a has joined #openstack-ansible09:30
*** errr has joined #openstack-ansible09:30
jrosserso it has an IP on it so that the neutron flat network eth12 can NAT out of the host (br-vlan -> eth12 veth pair)09:30
*** errr has quit IRC09:31
leo8aHi jrosser, that's correct. The server has an IP09:31
*** errr has joined #openstack-ansible09:32
jrosserbear in mind that the AIO networking is designed as a test setup so might not be representative of a useful setup spanning more than one node09:32
*** errr has quit IRC09:33
*** errr has joined #openstack-ansible09:34
*** errr has quit IRC09:35
*** errr has joined #openstack-ansible09:37
*** errr has joined #openstack-ansible09:38
*** errr has quit IRC09:38
*** errr has joined #openstack-ansible09:39
*** ysastri has joined #openstack-ansible09:39
*** errr has quit IRC09:39
*** errr has joined #openstack-ansible09:41
*** errr has quit IRC09:42
*** errr has joined #openstack-ansible09:43
*** errr has quit IRC09:44
*** errr has joined #openstack-ansible09:46
*** errr has quit IRC09:46
*** errr has joined #openstack-ansible09:47
*** errr has quit IRC09:47
*** errr has joined #openstack-ansible09:48
openstackgerritDuncan Martin Walker proposed openstack/openstack-ansible-ops master: Optional subdivision of elastic-logstash group  https://review.opendev.org/70784909:48
openstackgerritDuncan Martin Walker proposed openstack/openstack-ansible-ops master: Expose Elasticsearch discovery nodes  https://review.opendev.org/70882609:48
*** errr has quit IRC09:48
*** errr has joined #openstack-ansible09:50
*** errr has quit IRC09:51
*** errr has joined #openstack-ansible09:52
*** errr has quit IRC09:53
*** errr has joined #openstack-ansible09:55
*** errr has quit IRC09:55
*** errr has joined #openstack-ansible09:57
*** errr has quit IRC09:58
*** errr has joined #openstack-ansible10:00
*** errr has joined #openstack-ansible10:01
*** errr has quit IRC10:01
*** errr has joined #openstack-ansible10:03
*** errr has quit IRC10:04
*** errr has joined #openstack-ansible10:05
*** errr has quit IRC10:06
openstackgerritDuncan Martin Walker proposed openstack/openstack-ansible-ops master: Optional subdivision of elastic-logstash group  https://review.opendev.org/70784910:06
openstackgerritDuncan Martin Walker proposed openstack/openstack-ansible-ops master: Expose Elasticsearch discovery nodes  https://review.opendev.org/70882610:06
*** errr has joined #openstack-ansible10:06
*** errr has quit IRC10:07
*** errr has joined #openstack-ansible10:07
*** errr has quit IRC10:08
*** errr has joined #openstack-ansible10:08
*** errr has quit IRC10:09
*** errr has joined #openstack-ansible10:11
*** errr has quit IRC10:11
*** errr has joined #openstack-ansible10:13
*** errr has quit IRC10:14
*** errr has joined #openstack-ansible10:15
*** errr has quit IRC10:15
*** errr has joined #openstack-ansible10:16
*** errr has quit IRC10:17
*** errr has joined #openstack-ansible10:18
*** errr has quit IRC10:19
*** errr has joined #openstack-ansible10:21
*** errr has quit IRC10:21
*** errr has joined #openstack-ansible10:23
*** errr has joined #openstack-ansible10:24
openstackgerritArx Cruz proposed openstack/openstack-ansible-os_tempest master: DNM: Add support to tempest --worker-file option  https://review.opendev.org/70884410:25
*** errr has quit IRC10:25
*** errr has joined #openstack-ansible10:26
*** errr has quit IRC10:27
*** errr has joined #openstack-ansible10:29
*** errr has quit IRC10:30
*** errr has joined #openstack-ansible10:31
*** errr has quit IRC10:32
*** errr has joined #openstack-ansible10:33
*** errr has quit IRC10:34
*** errr has joined #openstack-ansible10:36
*** errr has quit IRC10:36
*** errr has joined #openstack-ansible10:38
*** errr has quit IRC10:38
*** errr has joined #openstack-ansible10:42
*** errr has quit IRC10:43
*** errr has joined #openstack-ansible10:45
*** errr has quit IRC10:46
*** errr has joined #openstack-ansible10:47
*** errr has quit IRC10:48
*** errr has joined #openstack-ansible10:49
*** errr has quit IRC10:50
*** errr has joined #openstack-ansible10:52
*** errr has quit IRC10:52
*** errr has joined #openstack-ansible10:54
*** errr has quit IRC10:55
*** errr has joined #openstack-ansible10:56
*** errr has joined #openstack-ansible10:59
*** errr has quit IRC11:00
*** errr has joined #openstack-ansible11:01
*** errr has quit IRC11:02
*** udesale_ has quit IRC11:03
*** errr has joined #openstack-ansible11:04
*** errr has quit IRC11:05
*** errr has joined #openstack-ansible11:07
*** errr has quit IRC11:07
*** errr has joined #openstack-ansible11:09
*** errr has quit IRC11:09
*** errr has joined #openstack-ansible11:10
*** errr has quit IRC11:11
*** cshen has quit IRC11:12
*** errr has joined #openstack-ansible11:13
*** errr has quit IRC11:14
*** errr has joined #openstack-ansible11:15
*** errr has quit IRC11:16
*** errr has joined #openstack-ansible11:18
*** errr has quit IRC11:19
*** errr has joined #openstack-ansible11:20
*** rpittau is now known as rpittau|bbl11:20
*** errr has quit IRC11:20
*** errr has joined #openstack-ansible11:21
*** errr has quit IRC11:22
*** sshnaidm|afk is now known as sshnaidm11:23
*** nicolasbock has joined #openstack-ansible11:23
*** errr has joined #openstack-ansible11:23
*** errr has quit IRC11:25
*** errr has joined #openstack-ansible11:26
*** errr has quit IRC11:26
*** errr has joined #openstack-ansible11:28
*** errr has quit IRC11:29
*** cshen has joined #openstack-ansible11:30
*** errr has joined #openstack-ansible11:31
jrossernoonedeadpunk: should we talk about the upgrade jobs?11:31
*** errr has quit IRC11:32
noonedeadpunkyeah11:32
*** errr has joined #openstack-ansible11:33
*** ansmith has joined #openstack-ansible11:33
jrosseris this the problem - that we mess with the state of the zuul provided repos so that the actual change we are testing isnt present in the 'upgraded' part of the job?11:34
noonedeadpunkI think yes. This is what happens now...11:34
*** errr has quit IRC11:34
noonedeadpunkbut I think that it's smth we can workaround11:34
jrosseryes, perhaps in upgrade jobs the installation of release N-1 needs to happen differently11:35
noonedeadpunkor maybe this jobs should be used only for integrated repo, but then they will be broken too often...11:35
*** errr has joined #openstack-ansible11:36
noonedeadpunkactually N-1 is ok, but after that N bootstrap is wrong11:36
jrosserwe could copy the repos from where zuul puts them into /etc/ansible/roles and checkout stable/<blah>11:36
*** errr has quit IRC11:36
jrosserthen when we do the upgrade we delete all that and use the existing process to symlink to the zuul checked out repos11:36
noonedeadpunkas we kinda do changes to N but not N-111:37
jrosserso that the patches applied by zuul remain untouched11:37
jrosserremeber we symlink though in N-1? that then messes with the state of the zuul repos?11:37
*** errr has joined #openstack-ansible11:38
noonedeadpunkoh, maybe... I can recall some copying from zuul but don't remember if it's linking or copying...11:38
noonedeadpunkanyway it's smth we can probably fix.11:38
*** errr has quit IRC11:39
noonedeadpunkbut I'm pretty bothered by 2 things except it - why in train CI we again have problems with placement migrations and why they broke for master as well11:39
jrosserjust on upgrades i think we should do something different here https://github.com/openstack/openstack-ansible/blob/master/scripts/get-ansible-role-requirements.yml#L50-L5711:39
noonedeadpunkyeah, agree11:40
jrossersome kind of copy/checkout for N-1 and then that symlink for N11:40
jrosserdo you have an example of the train placement migration error11:41
noonedeadpunkBut I'd say we should fix their current state at the first place? And after that fix overall logic for other than integrated repo11:41
*** errr has joined #openstack-ansible11:41
noonedeadpunksure11:41
*** errr has quit IRC11:41
noonedeadpunkhttps://zuul.opendev.org/t/openstack/build/e35a42f4e739453ea293fc0bc717c231/log/job-output.txt#1718511:42
*** errr has joined #openstack-ansible11:42
noonedeadpunkI've seen that before. But wasn't able to reproduce in spawned aio...11:42
*** errr has quit IRC11:42
noonedeadpunkiirc it happens with wrong mysql-migrate-db.sh11:43
*** errr has joined #openstack-ansible11:43
noonedeadpunk(or missing env vars with db creds)11:43
noonedeadpunkbut yes, I think youve pointed to the right reason and place we should change logic11:44
*** errr has quit IRC11:44
*** ansmith has quit IRC11:44
*** errr has joined #openstack-ansible11:46
*** errr has joined #openstack-ansible11:47
*** errr has quit IRC11:47
*** errr has joined #openstack-ansible11:49
*** errr has joined #openstack-ansible11:50
*** errr has quit IRC11:51
jrossernoonedeadpunk: just talking about the placement migration error with gshippey - we saw the exact same thing too11:51
noonedeadpunkbut I guess you've fixed it?11:51
jrossersequence of problems - lack of mysqldump, galera client etc we've patched all that11:51
jrosserbut it was reproducible in a 'clean' db11:52
jrosser*wasnt11:52
*** errr has joined #openstack-ansible11:52
*** errr has quit IRC11:52
noonedeadpunkactully I wasn't able to reproduce with gate-check-commit.sh either...11:52
noonedeadpunkso maybe smth ci-specific again....11:53
*** errr has joined #openstack-ansible11:53
jrosserwe were also never able to reporoduce, becasue in the lab we "fixed" it by ensuring the root user was used for the db connection11:54
jrossermessing with the my.cnf by hand11:54
*** errr has quit IRC11:54
jrosserand from memory there was something to do with db permissions when that "Table 'placement.resource_providers' doesn't exist\" error came up11:55
*** errr has joined #openstack-ansible11:55
noonedeadpunkwondring if I would be able to catch that without asking for autohold11:56
*** errr has quit IRC11:56
*** errr has joined #openstack-ansible11:58
*** errr has quit IRC11:59
*** errr has joined #openstack-ansible12:00
*** errr has quit IRC12:01
*** errr has joined #openstack-ansible12:02
*** errr has quit IRC12:03
*** errr has joined #openstack-ansible12:05
*** errr has quit IRC12:06
*** errr has joined #openstack-ansible12:07
*** errr has quit IRC12:08
*** errr has joined #openstack-ansible12:09
*** errr has quit IRC12:10
*** errr has joined #openstack-ansible12:12
*** errr has quit IRC12:12
*** errr has joined #openstack-ansible12:13
*** errr has joined #openstack-ansible12:14
*** errr has quit IRC12:14
*** errr has joined #openstack-ansible12:16
*** errr has quit IRC12:18
*** errr has joined #openstack-ansible12:19
*** errr has quit IRC12:19
*** errr has joined #openstack-ansible12:20
*** theintern_ has joined #openstack-ansible12:20
*** errr has quit IRC12:21
*** errr has joined #openstack-ansible12:22
*** errr has quit IRC12:23
*** errr has joined #openstack-ansible12:25
*** errr has quit IRC12:26
*** errr has joined #openstack-ansible12:27
*** mgariepy has joined #openstack-ansible12:27
*** errr has quit IRC12:28
*** errr has joined #openstack-ansible12:29
*** errr has quit IRC12:30
*** xakaitetoia has joined #openstack-ansible12:32
*** errr has joined #openstack-ansible12:32
*** errr has quit IRC12:33
*** udesale has joined #openstack-ansible12:34
*** errr has joined #openstack-ansible12:34
*** errr has quit IRC12:35
*** errr has joined #openstack-ansible12:36
*** errr has joined #openstack-ansible12:37
*** errr has quit IRC12:39
*** errr has joined #openstack-ansible12:40
*** errr has quit IRC12:40
*** errr has joined #openstack-ansible12:41
*** errr has quit IRC12:41
*** errr has joined #openstack-ansible12:42
*** errr has quit IRC12:42
*** errr has joined #openstack-ansible12:44
*** errr has quit IRC12:45
*** errr has joined #openstack-ansible12:46
*** errr has quit IRC12:46
*** errr has joined #openstack-ansible12:47
*** errr has quit IRC12:48
*** ysastri has quit IRC12:49
*** errr has joined #openstack-ansible12:49
*** errr has quit IRC12:50
*** ysastri has joined #openstack-ansible12:50
*** errr has joined #openstack-ansible12:52
*** errr has quit IRC12:53
*** errr has joined #openstack-ansible12:54
*** errr has quit IRC12:54
*** sshnaidm has quit IRC13:00
*** sshnaidm has joined #openstack-ansible13:01
*** rh-jelabarre has joined #openstack-ansible13:03
*** errr has joined #openstack-ansible13:03
*** errr has quit IRC13:04
*** errr has joined #openstack-ansible13:05
*** errr has quit IRC13:06
*** rpittau|bbl is now known as rpittau13:07
*** errr has joined #openstack-ansible13:08
*** errr has quit IRC13:09
*** errr has joined #openstack-ansible13:10
*** errr has quit IRC13:12
*** errr has joined #openstack-ansible13:16
*** errr has quit IRC13:17
*** errr has joined #openstack-ansible13:18
*** errr has quit IRC13:19
*** errr has joined #openstack-ansible13:20
*** errr has quit IRC13:21
*** errr has joined #openstack-ansible13:23
*** errr has quit IRC13:23
*** errr has joined #openstack-ansible13:25
*** errr has quit IRC13:26
*** errr has joined #openstack-ansible13:27
*** errr has quit IRC13:28
*** errr has joined #openstack-ansible13:30
*** errr has quit IRC13:30
*** errr has joined #openstack-ansible13:32
*** errr has quit IRC13:33
*** ansmith has joined #openstack-ansible13:34
*** errr has joined #openstack-ansible13:35
*** errr has quit IRC13:36
*** DanyC has joined #openstack-ansible13:37
*** errr has joined #openstack-ansible13:37
*** errr has quit IRC13:38
*** errr has joined #openstack-ansible13:40
*** errr has quit IRC13:41
*** DanyC_ has joined #openstack-ansible13:42
*** errr has joined #openstack-ansible13:42
*** errr has quit IRC13:43
*** DanyC has quit IRC13:44
*** errr has joined #openstack-ansible13:45
*** errr has quit IRC13:46
*** errr has joined #openstack-ansible13:47
*** errr has quit IRC13:48
*** errr has joined #openstack-ansible13:49
*** errr has quit IRC13:50
*** DanyC has joined #openstack-ansible13:50
*** DanyC_ has quit IRC13:52
*** errr has joined #openstack-ansible14:00
*** errr has quit IRC14:00
*** errr has joined #openstack-ansible14:01
openstackgerritMerged openstack/openstack-ansible-os_magnum stable/train: Download Magnum images once  https://review.opendev.org/70626114:02
*** errr has quit IRC14:02
*** errr has joined #openstack-ansible14:03
*** errr has quit IRC14:04
*** errr has joined #openstack-ansible14:06
*** errr has quit IRC14:07
*** errr has joined #openstack-ansible14:08
*** errr has quit IRC14:08
*** DanyC has quit IRC14:09
*** errr has joined #openstack-ansible14:11
*** errr has quit IRC14:12
*** errr has joined #openstack-ansible14:13
*** errr has quit IRC14:14
*** errr has joined #openstack-ansible14:14
*** errr has quit IRC14:15
*** errr has joined #openstack-ansible14:17
*** errr has quit IRC14:17
*** errr has joined #openstack-ansible14:19
*** errr has quit IRC14:20
*** ysastri has quit IRC14:21
*** errr has joined #openstack-ansible14:21
*** errr has quit IRC14:22
*** spatel has joined #openstack-ansible14:26
openstackgerritMohammed Naser proposed openstack/openstack-ansible-os_manila master: Add image download for manila  https://review.opendev.org/70497214:28
*** spatel has quit IRC14:31
openstackgerritMohammed Naser proposed openstack/openstack-ansible-os_manila master: Add image download for manila  https://review.opendev.org/70497214:32
openstackgerritMohammed Naser proposed openstack/openstack-ansible-os_manila master: Add Manila key generation and distribution  https://review.opendev.org/70501914:32
openstackgerritMerged openstack/openstack-ansible-openstack_hosts stable/train: Install gpg and sudo binary for Debian based systems  https://review.opendev.org/70865914:33
*** sshnaidm is now known as sshnaidm|afk14:34
*** errr has joined #openstack-ansible14:42
*** errr has quit IRC14:42
*** errr has joined #openstack-ansible14:43
*** errr has quit IRC14:43
*** errr has joined #openstack-ansible14:44
*** errr has quit IRC14:44
*** errr has joined #openstack-ansible14:46
*** errr has quit IRC14:47
*** errr has joined #openstack-ansible14:48
*** errr has quit IRC14:49
*** dave-mccowan has joined #openstack-ansible14:50
*** errr has joined #openstack-ansible14:51
*** errr has quit IRC14:52
*** errr has joined #openstack-ansible14:53
*** errr has quit IRC14:54
*** errr has joined #openstack-ansible14:55
*** errr has quit IRC14:56
*** errr has joined #openstack-ansible14:58
*** errr has quit IRC14:59
*** errr has joined #openstack-ansible15:00
*** errr has quit IRC15:00
*** errr has joined #openstack-ansible15:04
*** errr has quit IRC15:05
*** theintern_ has quit IRC15:06
*** errr has joined #openstack-ansible15:08
*** errr has quit IRC15:09
*** errr has joined #openstack-ansible15:10
*** errr has quit IRC15:11
*** errr has joined #openstack-ansible15:13
*** errr has quit IRC15:14
*** errr has joined #openstack-ansible15:15
*** errr has quit IRC15:16
noonedeadpunkmnaser: we kinda need https://review.opendev.org/#/c/705222/3 as after https://review.opendev.org/#/c/704942/ fixes things - this part actually breaks...15:17
*** errr has joined #openstack-ansible15:17
noonedeadpunkSo I was placing patches on top of 705222 with reason15:17
*** errr has quit IRC15:18
*** errr has joined #openstack-ansible15:20
*** errr has quit IRC15:21
*** errr has joined #openstack-ansible15:22
*** errr has quit IRC15:23
openstackgerritDmitriy Rabotyagov (noonedeadpunk) proposed openstack/openstack-ansible-os_manila master: Fix _manila_backends fact  https://review.opendev.org/70522215:23
*** errr has joined #openstack-ansible15:25
*** errr has quit IRC15:25
*** errr has joined #openstack-ansible15:26
*** errr has quit IRC15:27
*** errr has joined #openstack-ansible15:29
*** errr has quit IRC15:30
*** errr has joined #openstack-ansible15:31
*** errr has quit IRC15:32
*** errr has joined #openstack-ansible15:33
*** errr has quit IRC15:34
*** errr has joined #openstack-ansible15:36
*** errr has quit IRC15:36
*** errr has joined #openstack-ansible15:37
*** errr has quit IRC15:38
*** errr has joined #openstack-ansible15:39
*** errr has quit IRC15:39
cjloaderhi all15:41
cjloaderany update on upgrade jobs15:41
*** sshnaidm|afk is now known as sshnaidm15:43
*** errr has joined #openstack-ansible15:46
*** errr has quit IRC15:46
*** errr has joined #openstack-ansible15:48
*** errr has quit IRC15:49
*** errr has joined #openstack-ansible15:50
*** errr has quit IRC15:51
*** errr has joined #openstack-ansible15:53
*** errr has quit IRC15:53
*** errr has joined #openstack-ansible15:54
*** errr has quit IRC15:54
*** errr has joined #openstack-ansible15:56
*** errr has quit IRC15:57
*** errr has joined #openstack-ansible15:58
*** errr has quit IRC15:59
*** errr has joined #openstack-ansible16:00
*** errr has quit IRC16:01
*** errr has joined #openstack-ansible16:01
*** errr has quit IRC16:02
*** errr has joined #openstack-ansible16:04
*** errr has quit IRC16:04
*** errr has joined #openstack-ansible16:05
*** errr has quit IRC16:06
noonedeadpunkcjloader: not really16:06
cjloaderok16:06
*** errr has joined #openstack-ansible16:07
*** errr has quit IRC16:08
*** errr has joined #openstack-ansible16:09
*** errr has quit IRC16:10
*** errr has joined #openstack-ansible16:10
*** errr has quit IRC16:12
*** gyee has joined #openstack-ansible16:14
*** rgogunskiy has quit IRC16:17
noonedeadpunkjrosser: I wasn't able to reproduce placement failure even with resources in nova...16:21
noonedeadpunkneither without them....16:21
noonedeadpunkprobably worth asking for autohold unless you have ideas about how to reproduce16:22
cjloadernoonedeadpunk: might try rechecking to see if maybe an intermittent error16:26
jrossernoonedeadpunk: i think that a hold would be good16:39
* jrosser meetings right now16:40
*** cshen has quit IRC16:40
*** ivve has quit IRC16:41
*** errr has joined #openstack-ansible16:48
*** errr has quit IRC16:49
*** errr has joined #openstack-ansible16:49
*** errr has quit IRC16:50
*** errr has joined #openstack-ansible16:52
*** errr has quit IRC16:53
*** rpittau is now known as rpittau|afk16:53
*** DanyC has joined #openstack-ansible16:53
*** errr has joined #openstack-ansible16:54
*** errr has quit IRC16:54
*** errr has joined #openstack-ansible16:55
*** errr has quit IRC16:55
*** errr has joined #openstack-ansible16:56
*** errr has joined #openstack-ansible16:58
*** errr has quit IRC16:59
*** errr has joined #openstack-ansible17:01
*** errr has quit IRC17:01
*** errr has joined #openstack-ansible17:03
*** errr has quit IRC17:04
openstackgerritDmitriy Rabotyagov (noonedeadpunk) proposed openstack/openstack-ansible-os_manila master: Start using uWSGI role  https://review.opendev.org/70493517:05
*** errr has joined #openstack-ansible17:05
*** errr has quit IRC17:06
*** errr has joined #openstack-ansible17:08
*** errr has quit IRC17:09
*** errr has joined #openstack-ansible17:10
*** elenalindq has quit IRC17:11
*** errr has quit IRC17:11
*** errr has joined #openstack-ansible17:13
*** errr has joined #openstack-ansible17:14
*** errr has quit IRC17:14
*** DanyC has quit IRC17:15
*** errr has joined #openstack-ansible17:16
*** ckonstanski has quit IRC17:16
*** errr has quit IRC17:17
*** errr has joined #openstack-ansible17:18
*** errr has joined #openstack-ansible17:19
*** errr has quit IRC17:20
*** errr has joined #openstack-ansible17:22
*** DanyC has joined #openstack-ansible17:22
*** errr has quit IRC17:22
*** errr has joined #openstack-ansible17:24
*** errr has quit IRC17:25
*** elenalindq has joined #openstack-ansible17:26
*** errr has joined #openstack-ansible17:26
*** errr has quit IRC17:27
*** errr has joined #openstack-ansible17:29
*** errr has quit IRC17:29
*** errr has joined #openstack-ansible17:31
*** udesale has quit IRC17:33
*** evrardjp has quit IRC17:34
*** evrardjp has joined #openstack-ansible17:35
*** DanyC has quit IRC17:42
*** DanyC has joined #openstack-ansible17:43
*** errr has joined #openstack-ansible17:43
*** DanyC has quit IRC17:44
*** ivve has joined #openstack-ansible17:44
*** spatel has joined #openstack-ansible17:47
*** errr has joined #openstack-ansible17:55
*** errr has quit IRC17:56
*** errr has joined #openstack-ansible17:57
*** errr has quit IRC17:58
*** errr has joined #openstack-ansible17:59
*** errr has quit IRC18:00
*** errr has joined #openstack-ansible18:02
*** errr has quit IRC18:02
noonedeadpunkjrosser: but, I'd reproduced "{"msg": "'ansible.vars.hostvars.HostVarsVars object' has no attribute 'ansible_distribution'"}"18:03
*** errr has joined #openstack-ansible18:04
*** errr has quit IRC18:05
*** errr has joined #openstack-ansible18:06
*** errr has quit IRC18:07
jrossernoonedeadpunk: is that what’s breaking upgrades?18:07
noonedeadpunkyeah, on master18:08
noonedeadpunkbut... kinda once I've launched openstack-hosts-setup.yml again after failure it passed normally18:08
*** errr has joined #openstack-ansible18:09
*** errr has quit IRC18:09
*** errr has joined #openstack-ansible18:10
*** errr has quit IRC18:10
*** errr has joined #openstack-ansible18:12
*** errr has quit IRC18:13
*** chandankumar is now known as raukadah18:14
*** ckonstanski has joined #openstack-ansible18:14
noonedeadpunkI think this can be related https://opendev.org/openstack/openstack-ansible/src/branch/master/scripts/gate-check-commit.sh#L17418:14
*** errr has joined #openstack-ansible18:14
noonedeadpunk(wondering why did we need it)18:14
*** errr has quit IRC18:15
*** errr has joined #openstack-ansible18:17
*** errr has quit IRC18:17
*** errr has joined #openstack-ansible18:19
*** errr has quit IRC18:20
*** leo8a has quit IRC18:21
*** errr has joined #openstack-ansible18:21
*** errr has quit IRC18:22
*** errr has joined #openstack-ansible18:24
*** errr has quit IRC18:24
*** errr has joined #openstack-ansible18:26
*** errr has quit IRC18:27
*** errr has joined #openstack-ansible18:28
*** errr has quit IRC18:28
*** errr has joined #openstack-ansible18:29
*** elenalindq has quit IRC18:30
*** errr has quit IRC18:30
openstackgerritMerged openstack/openstack-ansible stable/train: Fix idempotency for os-lxc-container-setup.yml  https://review.opendev.org/70849318:31
*** errr has joined #openstack-ansible18:32
*** errr has joined #openstack-ansible18:32
*** errr has quit IRC18:33
*** errr has joined #openstack-ansible18:33
*** errr has quit IRC18:34
*** errr has joined #openstack-ansible18:34
*** errr has quit IRC18:35
*** errr has joined #openstack-ansible18:37
jrossernoonedeadpunk: facts are cached, but if placement is introduced new.....?18:37
*** errr has quit IRC18:38
*** spatel has quit IRC18:38
*** errr has joined #openstack-ansible18:39
*** errr has quit IRC18:40
*** errr has joined #openstack-ansible18:41
*** errr has quit IRC18:43
*** errr has joined #openstack-ansible18:43
*** errr has quit IRC18:44
noonedeadpunkjrosser: it's not about placement now... Placement fails on S->T and I can't reproduce that18:45
noonedeadpunkBut on T->U it fails on openstack hosts after merge of https://review.opendev.org/#/c/705849/718:45
noonedeadpunkand it's reproducable18:46
noonedeadpunk( https://zuul.opendev.org/t/openstack/build/ccea376864f843f8bce7d517b0d1a78e/log/job-output.txt#16203)18:47
noonedeadpunkso I think we can kinda clean facts and don't collect them again until that point...18:47
*** errr has joined #openstack-ansible18:48
*** errr has quit IRC18:49
openstackgerritDmitriy Rabotyagov (noonedeadpunk) proposed openstack/openstack-ansible master: DNM Gather facts for the setup-hosts.yml  https://review.opendev.org/70894918:49
*** errr has joined #openstack-ansible18:50
*** errr has quit IRC18:51
*** ansmith_ has joined #openstack-ansible18:56
*** mgariepy has quit IRC18:57
noonedeadpunkwould be great to get reviews for https://review.opendev.org/#/c/705222/418:58
*** ansmith has quit IRC18:59
*** errr has joined #openstack-ansible19:04
*** errr has joined #openstack-ansible19:05
*** errr has quit IRC19:06
*** errr has joined #openstack-ansible19:07
*** errr has quit IRC19:08
*** errr has joined #openstack-ansible19:09
*** errr has quit IRC19:10
*** errr has joined #openstack-ansible19:12
*** errr has quit IRC19:13
*** errr has joined #openstack-ansible19:14
*** errr has quit IRC19:15
*** errr has joined #openstack-ansible19:17
*** errr has quit IRC19:18
*** errr has joined #openstack-ansible19:19
*** errr has quit IRC19:20
*** errr has joined #openstack-ansible19:21
*** errr has quit IRC19:22
*** errr has joined #openstack-ansible19:24
*** errr has quit IRC19:24
*** pcaruana has quit IRC19:32
*** itsjg has joined #openstack-ansible19:34
*** miloa has quit IRC19:42
*** errr has joined #openstack-ansible19:43
*** errr has quit IRC19:44
*** ckonstanski has quit IRC19:45
*** ansmith_ has quit IRC19:55
*** errr has joined #openstack-ansible20:12
*** errr has quit IRC20:13
*** errr has joined #openstack-ansible20:19
*** errr has quit IRC20:20
*** errr has joined #openstack-ansible20:21
*** errr has quit IRC20:22
*** errr has joined #openstack-ansible20:23
*** errr has quit IRC20:24
*** errr has joined #openstack-ansible20:26
*** errr has quit IRC20:27
*** errr has joined #openstack-ansible20:28
*** errr has quit IRC20:29
*** errr has joined #openstack-ansible20:30
*** errr has quit IRC20:31
*** errr has joined #openstack-ansible20:33
*** errr has quit IRC20:34
*** errr has joined #openstack-ansible20:36
*** errr has quit IRC20:37
*** errr has joined #openstack-ansible20:38
*** errr has quit IRC20:39
*** errr has joined #openstack-ansible20:40
*** errr has quit IRC20:41
*** leo8a has joined #openstack-ansible20:41
*** errr has joined #openstack-ansible20:43
*** errr has quit IRC20:43
*** errr has joined #openstack-ansible20:45
jrossernoonedeadpunk: interesting https://github.com/ansible/ansible/issues/5172720:46
*** errr has joined #openstack-ansible20:46
*** errr has quit IRC20:47
*** errr has joined #openstack-ansible20:48
*** errr has quit IRC20:49
noonedeadpunkindeed20:50
*** errr has joined #openstack-ansible20:51
*** errr has quit IRC20:52
noonedeadpunkbtw https://review.opendev.org/#/c/708949/1/scripts/gate-check-commit.sh also fails upgrade...20:52
*** errr has joined #openstack-ansible20:54
*** errr has quit IRC20:55
*** errr has joined #openstack-ansible20:57
*** errr has quit IRC20:58
*** errr has joined #openstack-ansible21:00
*** errr has quit IRC21:00
*** errr has joined #openstack-ansible21:02
*** errr has quit IRC21:03
*** errr has joined #openstack-ansible21:04
*** errr has quit IRC21:05
*** ansmith_ has joined #openstack-ansible21:06
*** errr has joined #openstack-ansible21:07
*** errr has joined #openstack-ansible21:14
*** errr has quit IRC21:15
*** errr has joined #openstack-ansible21:16
*** errr has quit IRC21:17
jrossernoonedeadpunk: i wonder if we gather facts for aio1 but never for localhost21:17
*** errr has joined #openstack-ansible21:18
*** errr has quit IRC21:19
jrosserand then the whole task is delegated to localhost (deploy host) so i'm wondering if this is all just wrong21:20
*** errr has joined #openstack-ansible21:20
*** errr has quit IRC21:21
*** errr has joined #openstack-ansible21:23
*** errr has quit IRC21:24
*** errr has joined #openstack-ansible21:25
*** errr has quit IRC21:26
*** errr has joined #openstack-ansible21:30
*** jbadiapa has quit IRC21:31
*** errr has quit IRC21:31
*** errr has joined #openstack-ansible21:33
*** leo8a has quit IRC21:33
*** errr has quit IRC21:33
*** errr has joined #openstack-ansible21:35
*** errr has quit IRC21:36
*** strattao has joined #openstack-ansible21:36
strattao@jamesdenton are you in by chance?21:36
jamesdentonyes?21:37
jamesdentontied up at the moment, but can try to help. what's up?21:37
*** errr has joined #openstack-ansible21:37
*** ansmith_ has quit IRC21:38
strattaoI am losing my mind and could use a tip or pointer to something that we may have overlooked.... we're losing connectivity between the dhcp agents and the snat gateway on and off, the connection will stop working and then come back.  Nothing in the logs sticks out.21:38
*** errr has quit IRC21:38
strattaoWhen the connection is not working `ip neighbor` in the qdhcp namespace shows the entry for the snat gateway as FAILED21:40
*** errr has joined #openstack-ansible21:40
*** errr has quit IRC21:40
strattaoWhen are in the snat namespace and we ping the qdhcp ip address, the connectivity will start working again, but only for a minute or two.  Eventually it will start working on its own without the ping, but only for a period of time before dropping again.21:41
*** errr has joined #openstack-ansible21:42
*** errr has quit IRC21:42
strattaoWe've tried deleting and recreating the dhcp and l3 agents, deleting and recreating the network and router, and restarting all neutron services on the network hosts and hypervisors... and yet this issue still persists!21:43
strattaoAny other details I can give? Thoughts?21:46
jrosserso thats arp failure?21:46
*** errr has joined #openstack-ansible21:46
strattaoIt has to be arp related, but it's intermittent...21:47
jrosseri was just today having to fix arp related trouble on a l3 agent node21:47
*** errr has quit IRC21:47
jrosserwhich was related to multicast as we are using vxlan project networks21:48
*** errr has joined #openstack-ansible21:49
jamesdentonstrattao this is OVS, right?21:49
strattaoyessir21:49
strattaowe are also using vxlan21:49
jamesdentonstein? rocky?21:49
*** errr has quit IRC21:49
strattaowhere's my embarrassment emoji?... queens21:50
jamesdentonnaw, thats ok21:50
jamesdentoni'm battling some OVS demons myself right now21:50
*** errr has joined #openstack-ansible21:51
*** errr has quit IRC21:53
jamesdentoni am seeing similar behavior... things work OK, arp goes STALE->FAILED. In our case, pings stop working. a restart of the agent gets things going, only to start dropping again within minutes21:53
strattaovery similar to what we're seeing...21:53
jamesdentona pcap on the VM tap shows ARP replies, but they get dropped by OVS on the way out. we're seeing weird ovs flow cookie behavior21:53
jamesdentonthis is Stein w/ OVS 2.11.021:53
jamesdentonare you able to test with a VM? Or only between snat and qdhcp?22:02
strattaowe can test with a VM22:02
strattaothe problem is finding a project where this is happening22:02
jamesdentonahh ok, so it's not all?22:02
strattao(while it is happening)22:02
jamesdentongotcha22:03
strattaoit seems to be rolling22:03
*** xakaitetoia has quit IRC22:04
strattaoone thing that is interesting is that we have similar regions that were installed with queens and do not exhibit this issue... however, we run into this issue on the regions that were upgraded from Pike22:06
*** Neurognostic has joined #openstack-ansible22:06
strattaobut on those regions it is really frustrating because it only affects some interfaces, in some projects, some of the time...22:07
jamesdentoni understand22:08
*** spatel has joined #openstack-ansible22:08
*** spatel has quit IRC22:12
jamesdentonwell, if i get anywhere with this i will share what the fix is, could apply to you.22:13
strattaothank you for any help! Is there a launchpad bug you're tracking or do you think this is primarily an OVS issue?22:14
strattaojrosser, what did you ultimately have to do to fix your multicast issue for the l3 agent?  Maybe it's another thing we can poke around and look into as well...22:17
*** errr has joined #openstack-ansible22:18
jrosserstrattao: ours is linuxbridge so i don't know if it's relevant for OVS but we had a misconfigured multicast RP on one switch22:19
jamesdentonhavent gotten that far, yet22:19
jrosserwhich meant some but not all arp went awol22:19
*** errr has quit IRC22:19
jrosserand you could 'make it work' by going in the router namespace and manually pinging the right thing, which was ok till the arp cache timed out the entry22:20
*** errr has joined #openstack-ansible22:21
*** errr has quit IRC22:22
strattaojrosser, that really does sound exactly like what we're seeing...22:22
jrosseri have no idea if OVS is using mutlicast for mac learning or unicast replication22:24
strattaoYeah, how OVS handles the ARP and multicast traffic is still a bit of a black box for me...22:26
*** errr has joined #openstack-ansible22:27
*** errr has quit IRC22:27
*** errr has joined #openstack-ansible22:28
*** errr has joined #openstack-ansible22:30
*** errr has quit IRC22:31
*** errr has joined #openstack-ansible22:32
*** errr has quit IRC22:33
strattaojrosser This info is a few years old, but according to this blog: https://networkop.co.uk/blog/2016/05/06/neutron-l2pop/  Default OpenvSwitch implementation does not support VXLAN multicast flooding and uses unicast source replication instead.22:34
*** errr has joined #openstack-ansible22:35
*** errr has quit IRC22:36
*** errr has joined #openstack-ansible22:37
*** errr has joined #openstack-ansible22:38
*** errr has quit IRC22:39
jrosserstrattao: yes it feels like things are different here between OVS<>linuxbridge22:40
*** errr has joined #openstack-ansible22:40
*** errr has quit IRC22:41
*** errr has joined #openstack-ansible22:43
*** errr has quit IRC22:44
*** errr has joined #openstack-ansible22:45
openstackgerritJonathan Rosser proposed openstack/openstack-ansible-openstack_hosts master: Use ansible-runtime python to drop hosts file  https://review.opendev.org/70898922:45
*** errr has quit IRC22:46
openstackgerritJonathan Rosser proposed openstack/openstack-ansible master: DNM Gather facts for the setup-hosts.yml  https://review.opendev.org/70894922:46
*** errr has joined #openstack-ansible22:47
*** errr has quit IRC22:48
*** errr has joined #openstack-ansible22:50
*** errr has quit IRC22:50
*** errr has joined #openstack-ansible22:52
*** errr has quit IRC22:53
*** errr has joined #openstack-ansible22:54
*** itsjg has quit IRC22:55
*** errr has quit IRC22:55
*** errr has joined #openstack-ansible22:57
*** errr has quit IRC22:58
*** errr has joined #openstack-ansible22:59
*** errr has quit IRC23:00
*** errr has joined #openstack-ansible23:01
*** sshnaidm is now known as sshnaidm|afk23:02
*** errr has quit IRC23:02
*** errr has joined #openstack-ansible23:04
*** errr has quit IRC23:05
*** errr has joined #openstack-ansible23:07
*** errr has quit IRC23:08
*** errr has joined #openstack-ansible23:09
*** errr has quit IRC23:10
*** ahosam has quit IRC23:10
*** ahosam has joined #openstack-ansible23:11
*** errr has joined #openstack-ansible23:11
*** errr has quit IRC23:12
*** errr has joined #openstack-ansible23:14
*** errr has quit IRC23:14
*** errr has joined #openstack-ansible23:16
*** errr has quit IRC23:17
openstackgerritMerged openstack/openstack-ansible-os_manila master: Fix _manila_backends fact  https://review.opendev.org/70522223:17
*** errr has joined #openstack-ansible23:18
*** nicolasbock has quit IRC23:18
*** errr has quit IRC23:19
*** errr has joined #openstack-ansible23:21
*** errr has quit IRC23:22
*** errr has joined #openstack-ansible23:23
*** errr has quit IRC23:24
*** errr has joined #openstack-ansible23:25
*** errr has quit IRC23:26
*** errr has joined #openstack-ansible23:28
*** errr has quit IRC23:28
*** errr has joined #openstack-ansible23:30
*** errr has quit IRC23:31
*** errr has joined #openstack-ansible23:32
*** spatel has joined #openstack-ansible23:32
*** errr has quit IRC23:32
*** errr has joined #openstack-ansible23:33
*** errr has quit IRC23:35
*** spatel has quit IRC23:37
*** ahosam has quit IRC23:40
*** dave-mccowan has quit IRC23:51
*** dave-mccowan has joined #openstack-ansible23:52

Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!