| *** markvoelker has joined #openstack-ansible | 00:21 | |
| mnaser | dmsimard: send em our way | 00:36 |
|---|---|---|
| *** markvoelker has quit IRC | 00:54 | |
| *** markvoelker has joined #openstack-ansible | 01:51 | |
| *** dave-mccowan has joined #openstack-ansible | 01:55 | |
| dmsimard | I'm looking at the ssl stuff right now... it doesn't look like --insecure is working | 01:56 |
| dmsimard | but curl does | 01:56 |
| *** dave-mccowan has quit IRC | 02:00 | |
| dmsimard | oh, yuck, my fault.. a stray manual haproxy config I did to troubleshoot that last patch -- it was double-listening with different configs | 02:12 |
| dmsimard | nova-status from https://github.com/openstack/openstack-ansible-os_nova/blob/466ab0aecd3efa44d8a78fe856ad0f2cad9d1425/tasks/nova_db_post_setup.yml#L33 doesn't have an --insecure comand | 02:24 |
| dmsimard | command argument* | 02:24 |
| *** markvoelker has quit IRC | 02:24 | |
| dmsimard | so it's failing on the self signed certificate | 02:25 |
| dmsimard | insecure is set to true in nova.conf's keystone_authtoken but I don't think it's being taken into acconut | 02:27 |
| *** sdake has quit IRC | 03:08 | |
| *** sdake has joined #openstack-ansible | 03:11 | |
| *** markvoelker has joined #openstack-ansible | 03:22 | |
| *** markvoelker has quit IRC | 03:54 | |
| *** udesale has joined #openstack-ansible | 04:02 | |
| openstackgerrit | Merged openstack/openstack-ansible-os_cinder master: Add missing CLI_OPTIONS when setting up qos volume types https://review.openstack.org/636071 | 04:06 |
| *** aludwar has quit IRC | 04:12 | |
| openstackgerrit | Chandan Kumar proposed openstack/openstack-ansible-os_tempest master: Use the correct heat tests https://review.openstack.org/630695 | 04:29 |
| *** chandankumar is now known as chkumar|ruck | 04:32 | |
| *** ArchiFleKs has quit IRC | 04:42 | |
| *** markvoelker has joined #openstack-ansible | 04:51 | |
| *** ArchiFleKs has joined #openstack-ansible | 04:57 | |
| *** markvoelker has quit IRC | 05:24 | |
| *** sawblade6 has quit IRC | 05:29 | |
| openstackgerrit | Kevin Carter (cloudnull) proposed openstack/openstack-ansible-ops master: Add the ability to enable or disable rollups / indexes https://review.openstack.org/636090 | 05:58 |
| openstackgerrit | Kevin Carter (cloudnull) proposed openstack/openstack-ansible-ops master: Add logstash ingestion for collectd https://review.openstack.org/635418 | 06:14 |
| *** shyamb has joined #openstack-ansible | 06:19 | |
| *** markvoelker has joined #openstack-ansible | 06:22 | |
| openstackgerrit | Kevin Carter (cloudnull) proposed openstack/openstack-ansible-ops master: Change the overlay service start vars https://review.openstack.org/636095 | 06:51 |
| *** markvoelker has quit IRC | 06:55 | |
| *** shyamb has quit IRC | 06:57 | |
| *** DanyC has joined #openstack-ansible | 07:02 | |
| *** jbadiapa has joined #openstack-ansible | 07:04 | |
| openstackgerrit | Merged openstack/openstack-ansible-ops master: Add logstash ingestion for collectd https://review.openstack.org/635418 | 07:05 |
| *** DanyC has quit IRC | 07:07 | |
| *** kopecmartin|off is now known as kopecmartin | 07:07 | |
| openstackgerrit | Merged openstack/openstack-ansible-ops master: Change the overlay service start vars https://review.openstack.org/636095 | 07:14 |
| *** shyamb has joined #openstack-ansible | 07:24 | |
| openstackgerrit | Chandan Kumar proposed openstack/openstack-ansible-os_heat master: Fixed the egg name of heat to openstack_heat https://review.openstack.org/635518 | 07:29 |
| *** fnpanic has joined #openstack-ansible | 07:34 | |
| fnpanic | good morning | 07:34 |
| *** rgogunskiy has joined #openstack-ansible | 07:44 | |
| *** markvoelker has joined #openstack-ansible | 07:52 | |
| *** shyamb has quit IRC | 07:52 | |
| *** shyamb has joined #openstack-ansible | 08:00 | |
| *** gkadam has joined #openstack-ansible | 08:01 | |
| *** electrofelix has joined #openstack-ansible | 08:18 | |
| *** Emine has joined #openstack-ansible | 08:18 | |
| *** markvoelker has quit IRC | 08:24 | |
| *** electrofelix has quit IRC | 08:35 | |
| *** electrofelix has joined #openstack-ansible | 08:37 | |
| *** Emine has quit IRC | 08:37 | |
| *** Emine has joined #openstack-ansible | 08:42 | |
| *** shyamb has quit IRC | 08:50 | |
| *** tosky has joined #openstack-ansible | 08:54 | |
| *** shardy has joined #openstack-ansible | 09:04 | |
| chkumar|ruck | odyssey4me: need some help here http://logs.openstack.org/18/635518/5/check/openstack-ansible-functional-centos-7/aafb1d0/logs/ara-report/result/3b431320-b515-4e81-a763-da4167e217ed/ | 09:10 |
| *** DanyC has joined #openstack-ansible | 09:12 | |
| *** DanyC has quit IRC | 09:14 | |
| *** DanyC has joined #openstack-ansible | 09:14 | |
| *** shyamb has joined #openstack-ansible | 09:17 | |
| *** markvoelker has joined #openstack-ansible | 09:21 | |
| *** Emine has quit IRC | 09:29 | |
| *** Emine has joined #openstack-ansible | 09:36 | |
| fnpanic | just submitted this but https://bugs.launchpad.net/openstack-ansible/+bug/1815430 | 09:46 |
| openstack | Launchpad bug 1815430 in openstack-ansible "ceph_client role fails verifing keys when deploying behind a proxy" [Undecided,New] | 09:46 |
| fnpanic | i also added a diff for ubuntu in the bug report which works for ubuntu. i adopted it from the galera proxy patch | 09:47 |
| fnpanic | https://review.openstack.org/#/c/625291/ | 09:47 |
| fnpanic | looks like i am the only guy in the world deploying OSA behind a proxy? | 09:47 |
| jrosser | not the only one. i mirror as many of the upstream repos locally as possible so havent tripped over the ansible apt-key module not obeying the proxy settings | 09:50 |
| fnpanic | what do you use for mirroring? | 09:53 |
| fnpanic | aptly? | 09:53 |
| *** markvoelker has quit IRC | 09:54 | |
| jrosser | debmirror | 09:58 |
| jrosser | hrrm bionic/ceph integrated repo test is broken | 10:01 |
| jrosser | cinder distro packages pull in librados and librbd before the ceph_client role has run and set up the ceph repo and apt pinning | 10:02 |
| CeeMac | morning all | 10:15 |
| fnpanic | hi | 10:24 |
| *** mkuf_ has joined #openstack-ansible | 10:48 | |
| *** markvoelker has joined #openstack-ansible | 10:52 | |
| *** mkuf has quit IRC | 10:52 | |
| *** shyamb has quit IRC | 11:01 | |
| *** mkuf_ has quit IRC | 11:24 | |
| openstackgerrit | Jonathan Rosser proposed openstack/openstack-ansible-os_cinder master: Ensure cinder-volumes tool packages install correct ceph dependancies https://review.openstack.org/636115 | 11:24 |
| *** markvoelker has quit IRC | 11:25 | |
| openstackgerrit | Jonathan Rosser proposed openstack/openstack-ansible master: Fix typo btrfs -> zfs https://review.openstack.org/636035 | 11:40 |
| *** shyamb has joined #openstack-ansible | 11:42 | |
| *** sum12 has quit IRC | 11:44 | |
| *** sum12 has joined #openstack-ansible | 11:44 | |
| *** Emine has quit IRC | 11:46 | |
| CeeMac | is it possible/probable that using a custom SSL cert for HAproxy will have broken heat? | 11:52 |
| CeeMac | I'm getting 504 gateway time-outs through both dashboard and cli | 11:53 |
| mathlin | hi, what could stop a dhcp reply from the network node from reaching the nova compute node. Only request on the nova side, request and reply on the network node. Other traffic like a few 'who-has' can pass freely. | 11:57 |
| *** chhagarw has joined #openstack-ansible | 12:01 | |
| fnpanic | CeeMac: looks like we are in the same boat | 12:02 |
| CeeMac | fnpanic, at least I'm not alone | 12:02 |
| fnpanic | if it is a custom from a none public ca i guess so | 12:02 |
| CeeMac | ah, its from a public ca | 12:03 |
| fnpanic | ok | 12:03 |
| CeeMac | but different to the self-gen one created at deploymeny | 12:03 |
| fnpanic | good to know | 12:03 |
| fnpanic | different means? | 12:03 |
| CeeMac | i saw a bug that might have been related, but it doesn't seem to have gone anywhere | 12:03 |
| CeeMac | as in not the same :) | 12:03 |
| CeeMac | as in i used the override variable to provide a differnet cert | 12:04 |
| CeeMac | haproxy_user_ssl_cert etc | 12:04 |
| CeeMac | mathlin, is it a new deployment? I had a similar issue before | 12:04 |
| *** udesale has quit IRC | 12:05 | |
| mathlin | New deployment, yes. | 12:06 |
| *** udesale has joined #openstack-ansible | 12:06 | |
| openstackgerrit | Jonathan Rosser proposed openstack/openstack-ansible-os_cinder master: Ensure cinder-volumes tool packages install correct ceph dependancies https://review.openstack.org/636115 | 12:06 |
| CeeMac | mathlin, are you using vxlan for your tenant network? | 12:08 |
| *** shyamb has quit IRC | 12:10 | |
| *** shyamb has joined #openstack-ansible | 12:10 | |
| mathlin | yes | 12:12 |
| *** dave-mccowan has joined #openstack-ansible | 12:12 | |
| CeeMac | do you have multicast groups configured on your physical switches, or igmpsnooping enabled? | 12:13 |
| *** mkuf has joined #openstack-ansible | 12:13 | |
| CeeMac | fnpanic, do you get any specific error messages in any logs? I can't seem to find anything that definitely pinpoints where the problem is | 12:14 |
| mathlin | unsure, need to check that | 12:15 |
| CeeMac | mathlin, i had to enable l2population as I don't have multicast set up on the switches | 12:15 |
| CeeMac | fnpanic, i see a few "ERROR heat-api IOError: write error" but they don't seem to coincide with any attempt to connect to heat | 12:17 |
| *** markvoelker has joined #openstack-ansible | 12:21 | |
| fnpanic | CeeMac: i in the logs that heat gets an ssl verify error when it talks to keystone public endpoint | 12:27 |
| CeeMac | do you have debug enabled? or is that in the normal logs | 12:27 |
| CeeMac | if your cert is not public ca, do you have the appropriate ca certs installed so the cert chain can be validated? | 12:28 |
| fnpanic | you can try. open the heat endpoint in a browser | 12:29 |
| fnpanic | with debug it gives more details but error ist the same SSL verify error | 12:29 |
| fnpanic | yes | 12:29 |
| fnpanic | but had a talk with jrosser this morning | 12:29 |
| fnpanic | python does not use the ca-certificates by default | 12:30 |
| CeeMac | hmm | 12:30 |
| CeeMac | my endpoints are listed as http internally, but i cant load the page | 12:33 |
| fnpanic | it uses the extrnal api endpoint as far as i know | 12:34 |
| CeeMac | tried that to, no dice | 12:34 |
| CeeMac | no error in logs | 12:34 |
| *** Emine has joined #openstack-ansible | 12:35 | |
| jrosser | have you done simple debugging with wget/curl from the same container/host to the same endpoint to see if that works? | 12:39 |
| CeeMac | just noticed a firewall block from my laptop, just opened up port 8000 | 12:39 |
| CeeMac | if i connect to the :8000/v1 address i get an error about signature not matching aws standard | 12:40 |
| CeeMac | not sure if that url is supposed to work, or what it should show | 12:41 |
| jrosser | you can debug ssl issues against any endpoint, like keystone on port 5000, it's the same haproxy for all of them | 12:42 |
| fnpanic | :-) | 12:42 |
| CeeMac | the cert is working for the dashboard | 12:43 |
| CeeMac | just when I navigate to Orchestration I get 504 gateway time-out | 12:44 |
| CeeMac | same if i exectute openstack software cli commands | 12:44 |
| CeeMac | it could be complete coincident its not working after switch cert | 12:49 |
| CeeMac | or not, i can't work it out | 12:49 |
| CeeMac | if i curl from the host using the container IP it returns "versions" info, for both 8000 and 8004 | 12:50 |
| CeeMac | same if i use the internal vip | 12:50 |
| jrosser | can you try using python as well? | 12:51 |
| CeeMac | with guidance, sure | 12:51 |
| CeeMac | i don't python in general | 12:51 |
| jrosser | something like this, just from the command line in your container http://paste.openstack.org/show/744838/ | 12:52 |
| jrosser | obv. put in the url/port you want to test | 12:52 |
| CeeMac | sure :), 1 sec | 12:53 |
| jrosser | it'll either seem to do nothing, or blow up in a pile of SSL errors | 12:53 |
| *** shyamb has quit IRC | 12:53 | |
| CeeMac | ImportError: No module named requests | 12:54 |
| *** markvoelker has quit IRC | 12:54 | |
| jrosser | oh...... centos? | 12:56 |
| CeeMac | ubuntu | 12:57 |
| CeeMac | tried in the container and on the host | 12:57 |
| jrosser | well look - you've got errors in horizon and with the openstack cli | 12:57 |
| jrosser | it should be reasonable to debug both of those and find errors in logs somewhere | 12:58 |
| jrosser | without that this is all guesswork | 12:58 |
| CeeMac | yeah | 12:58 |
| CeeMac | nothing in the logs i've seen so far, that correlates anyway | 12:58 |
| CeeMac | i'll maybe double-check horizon logs and enable debug | 12:59 |
| jrosser | and once you find roughly where the issue is, trying to hit the same endpoints by hand with either wget/curl or a python hack is the normal approach | 12:59 |
| CeeMac | the only thing i can see in heat is an odd IOError: write error, nut not concurrent with timings of tests | 13:00 |
| CeeMac | *but not | 13:00 |
| CeeMac | i'll keep poking it with a stick, thanks | 13:00 |
| openstackgerrit | Michael Vollman proposed openstack/openstack-ansible-os_neutron master: Avoid distro installing unused services https://review.openstack.org/633277 | 13:20 |
| guilhermesp | chkumar|ruck: did you see this happening with other centos jobs? http://logs.openstack.org/79/635579/3/check/openstack-ansible-functional-centos-7/fb8c54f/logs/ara-report/result/9f277fa7-ee2b-4380-b542-c2b66e0704da/ | 13:42 |
| fnpanic | is it save to set pipelining = True for osa? | 13:42 |
| odyssey4me | fnpanic we already do it by default | 13:43 |
| odyssey4me | fnpanic https://github.com/openstack/openstack-ansible/blob/master/scripts/openstack-ansible.rc#L48-L50 | 13:43 |
| chkumar|ruck | guilhermesp: it is a known issue, we need to fix vlan stuff on os_nova side | 13:44 |
| chkumar|ruck | guilhermesp: on tempest side, we have fixed with this https://review.openstack.org/#/c/633732/ fixed by jrosser | 13:44 |
| jrosser | we need to talk with cloudnull about how to fix that centos bridge forwarding properly | 13:46 |
| jamesdenton | vollman submitted a patch here for networkd: https://review.openstack.org/#/c/635929/ | 13:47 |
| jrosser | i had a look but they way the netoworks are set up in the tests repo is a bit awkward to slip that extra var in | 13:47 |
| jamesdenton | indeed | 13:47 |
| jrosser | yes i stared for too long at how that pile of jinja creates the networks data structure, and it just made my head hurt | 13:48 |
| jamesdenton | lol i was headed down that path right now. slowly backing away | 13:48 |
| fnpanic | odyssey4me: thanks | 13:49 |
| vollman | jrosser: I was thinking this perhaps. https://github.com/mvollman/openstack-ansible-tests/commit/9d74b3dfb847d91ccb3b5828507dcaff7bda2f63#diff-5abaef8c358fed2100bf9007e4556c7a | 13:49 |
| *** mgariepy has joined #openstack-ansible | 13:51 | |
| mnaser | could i get eyes on https://review.openstack.org/#/c/634991/ again :< | 13:51 |
| vollman | statically setting ipforward: true for all test bridges | 13:51 |
| fnpanic | then it is strange that osa takes ages | 13:51 |
| *** markvoelker has joined #openstack-ansible | 13:52 | |
| odyssey4me | fnpanic ansible is slow, and we have *a lot* of tasks | 13:54 |
| fnpanic | i know but more then 4 hours? | 13:54 |
| odyssey4me | fnpanic if you can find ways to optimise what we're doing, then please do submit patches / bug reposrts with recommendations | 13:54 |
| fnpanic | for 3 controllers, 4 computes and 3 storage nodes? | 13:55 |
| fnpanic | the controllers have /var on ssd | 13:55 |
| odyssey4me | yeah, most of the work is on the control plane - that's where time is taken, the work from then on is much quicker | 13:55 |
| openstackgerrit | Merged openstack/openstack-ansible-nspawn_container_create master: Trivial: Fix the pep8 warning https://review.openstack.org/616527 | 13:55 |
| fnpanic | odyssey4me: i will take a deeper look were the time is lost and hope we can improve it | 13:57 |
| *** bgmccollum has quit IRC | 13:57 | |
| *** bgmccollum has joined #openstack-ansible | 13:58 | |
| odyssey4me | fnpanic FYI, the repo build process is one of those areas - I'm doing work in Stein to resolve that. | 14:04 |
| *** aludwar has joined #openstack-ansible | 14:06 | |
| jrosser | jamesdenton: i think the decision is either to set that forward flag everywhere using the tests repo, brute force | 14:06 |
| jamesdenton | path of least resistance | 14:07 |
| jrosser | or to set it selectively in the roles just on br-mgmt <- but idk how to make that work in the test repo code | 14:07 |
| jamesdenton | It's br-vlan, correct? | 14:07 |
| jrosser | oh sorry yes | 14:07 |
| jamesdenton | k | 14:07 |
| vollman | jrosser jamesdenton Ubuntu is defaulting forwarding=1 for all bridges so the brute force approach should be a noop | 14:09 |
| *** bgmccollum has quit IRC | 14:10 | |
| jamesdenton | vollman are the neutron tests passing for you with a distro install? | 14:23 |
| vollman | neutron or nova? | 14:23 |
| *** ArchiFleKs has quit IRC | 14:23 | |
| *** partlycloudy has quit IRC | 14:23 | |
| *** asettle has joined #openstack-ansible | 14:24 | |
| *** markvoelker has quit IRC | 14:24 | |
| jamesdenton | neutron | 14:25 |
| vollman | jamesdenton: Yea. all of the voting gate jobs passed on an os_neutron changeset last week | 14:26 |
| vollman | I just pushed another patch and the gates are running again now. should have fresh results in an hour or less | 14:27 |
| jamesdenton | yeah, i saw that. locally, i see that after it creates the router it tries to ping the qg interface, but all networks are 'vxlan' and there's no way to get there from the host. Just curious what it might have looked like for you | 14:28 |
| jamesdenton | this is related more to the distro patch than the ipforward thing. | 14:28 |
| *** ArchiFleKs has joined #openstack-ansible | 14:29 | |
| jamesdenton | may be a difference in the way i ran it compared to gate. running again | 14:30 |
| vollman | jamesdenton: I haven't manually run the neutron gate for this patch. I tested it with an integrated test and when that passed I let the gate run the neutron specific testing | 14:32 |
| openstackgerrit | Merged openstack/openstack-ansible-nspawn_container_create master: Fix the misspelling of "container" https://review.openstack.org/632644 | 14:33 |
| jrosser | cores: bionic/ceph is broken due to cinder package install trouble, slightly ugly fix in https://review.openstack.org/#/c/636115/ validated with a depends-on here https://review.openstack.org/#/c/636035/ | 14:41 |
| jrosser | ^ everything is blocked on master right now due to that | 14:42 |
| *** bgmccollum has joined #openstack-ansible | 14:44 | |
| odyssey4me | jrosser does that block rocky too? | 14:44 |
| jrosser | odyssey4me: no, it looks like only master | 14:46 |
| odyssey4me | ok, thankfully | 14:46 |
| * jrosser thinks about why that is, though....... | 14:46 | |
| chkumar|ruck | odyssey4me: Hello | 14:47 |
| odyssey4me | I guess the re-ordering we've done in the installs is affecting things | 14:47 |
| jrosser | also rocky may still be installing luminous, but i'd only expect that to do the same/worse | 14:47 |
| chkumar|ruck | odyssey4me: need some help on http://logs.openstack.org/18/635518/5/check/openstack-ansible-functional-centos-7/aafb1d0/logs/ara-report/result/3b431320-b515-4e81-a763-da4167e217ed/ mpi4py issue | 14:48 |
| odyssey4me | chkumar|ruck right, the missing deps - did you push a patch to change deps? | 14:48 |
| odyssey4me | or are these deps only for tempest plugins, perhaps? | 14:48 |
| chkumar|ruck | odyssey4me: sorry did not get change-deps part? | 14:49 |
| odyssey4me | chkumar|ruck well, there are missing distro packages to make those installs work - last we chatted I thought you were going to figure out which they were and do a patch to add them | 14:49 |
| chkumar|ruck | odyssey4me: for missing distro packages | 14:49 |
| dmsimard | I'm trying to get networking to work the way I want to in an AIO I set up last weekend and I'm not 100% sure I understand what is meant to be used where | 14:49 |
| chkumar|ruck | odyssey4me: https://review.openstack.org/#/c/635518/ i have added it here | 14:50 |
| chkumar|ruck | odyssey4me: or i need to put patches at some other place? | 14:50 |
| dmsimard | At the switch/firewall level, I have vlans 10, 20 and 30 set up and my network interfaces are similar to https://docs.openstack.org/openstack-ansible/latest/user/ceph/full-deploy.html#host-network-configuration | 14:51 |
| dmsimard | It would be pretty convenient if I could put everything on the same bond so I tried to do that but could get an instance to ping -- although I did see pings when doing a tcpdump on the bridge interface | 14:51 |
| odyssey4me | chkumar|ruck hmm, ok - I'll need to spend some time looking through what's going on there then if that's failing... I'm tied up with other work now, but will try to look at it in 2-3 hours. | 14:52 |
| jamesdenton | dmsimard Single bond is usually not an issue. The br-vlan bridge would need to have bond0 in it, in that case. | 14:54 |
| dmsimard | jamesdenton: yeah but I'm a bit confused as to what is meant to be routable or not | 14:54 |
| dmsimard | for example, should I be able to have a external network for 172.29.240.0/22 ? do I need to create a tenant network and set up a router between the two ? | 14:55 |
| dmsimard | I guess I'm confusing the vxlan and vlan networks | 14:56 |
| jamesdenton | The example assumes that VLAN 10 (mgmt) will be the primary interface of the node itself, with the default gateway off that interface. VLAN 20 and VLAN 30 are not usually routable. The br-vlan bridge connects to bond0 (untagged), and you would build out tagged provider networks with Neutron API. That provider network would need to have some kind of physical gateway (ie. router/firewall) | 14:56 |
| dmsimard | the "public" IPs for the VMs are meant to be on the vxlan subnet ? | 14:56 |
| dmsimard | yeah vlan 10 and mgmt is no issue, that works perfectly | 14:57 |
| jamesdenton | You can put VMs on that provider network, in which case they'd be right on the VLAN. Or, you build out tenant networks (may be vxlan by default), in which case you'd need a neutron router in the middle (one end connected to provider net, the other to tenant net) | 14:57 |
| jamesdenton | The IP configured on a VM is referred to as a 'fixed IP'. That would NOT be directly accessible if vxlan network type is used | 14:57 |
| dmsimard | jamesdenton: fwiw I'm pretty familiar with openstack, just my first time really trying OSA :p | 14:58 |
| mgariepy | can i have a final nudge on : https://review.openstack.org/#/c/632907/ and https://review.openstack.org/#/c/632908/ please :D | 14:58 |
| jamesdenton | So you create the router, plugin to both nets, create a floating IP and associate with the fixed IP, and bob's your uncle | 14:58 |
| jamesdenton | ok cool - sorry for the remediation then | 14:59 |
| jamesdenton | So in the example, 172.29.240.0/22 is the network used for VTEPs only. Not routable. | 15:01 |
| dmsimard | I think I may have seen that tempest uses 172.29.248.0/22 for the "public" subnet | 15:02 |
| jamesdenton | br-vxlan in OSA is just a consistent placeholder for vtep addrs. And when neutron agents are deployed in containers on the infra nodes, the container connects to br-vxlan via a veth pair. There's an interface inside the container, eth10 i think, that would have the vtep addr on it. | 15:02 |
| jamesdenton | I could be wrong, but i think tempest sets up 10.1.3.0/24 | 15:07 |
| jrosser | well role tests vs AIO vs whatever-you-want-in-your-lab may be a factor here | 15:08 |
| dmsimard | jamesdenton: found it here: http://git.openstack.org/cgit/openstack/openstack-ansible/tree/tests/roles/bootstrap-host/templates/user_variables.aio.yml.j2#n28 | 15:08 |
| jrosser | doesnt AIO set up all sorts of NAT for the "public" network, that might not be needed | 15:09 |
| jamesdenton | oh, right on | 15:09 |
| dmsimard | codesearch.openstack is my best friend :) | 15:09 |
| openstackgerrit | David Moreau Simard proposed openstack/openstack-ansible master: Remove an extra comment about Tempest settings https://review.openstack.org/636148 | 15:11 |
| openstackgerrit | Michael Vollman proposed openstack/ansible-role-systemd_networkd master: Add ipforward option to configure IPForward https://review.openstack.org/635929 | 15:13 |
| dmsimard | jrosser: something that is confusing -- and I haven't had the time to look it up yet is that it sets up a bunch of stuff in /etc/systemd/network which might conflict with stuff from /etc/network/interfaces | 15:13 |
| *** TxGirlGeek has joined #openstack-ansible | 15:13 | |
| dmsimard | I had to take the br-vxlan and br-vlan stuff out of there | 15:14 |
| jamesdenton | You were having issues with an AIO as-is? | 15:19 |
| *** markvoelker has joined #openstack-ansible | 15:22 | |
| cloudnull | mornings | 15:22 |
| jrosser | o/ cloudnull | 15:24 |
| jrosser | cloudnull: question about this https://github.com/openstack/openstack-ansible-ops/blob/master/overlay-inventories/osa-integration-inventory.yml#L123 | 15:24 |
| jrosser | that quite likley ends up with log1 already in log_hosts in a lot of deploys | 15:25 |
| jrosser | then the log1 host gets a ton of elk installed all over it | 15:25 |
| dmsimard | jamesdenton: it was a bumpy ride but I'm almost there | 15:25 |
| jamesdenton | Which OS? | 15:26 |
| dmsimard | ubuntu 18.04 with the latest rocky checkout | 15:26 |
| dmsimard | my problems started even before OSA though, the ubuntu server image ships with netplan which is kind of bad | 15:27 |
| jrosser | dmsimard: on 18.04 /etc/network/interfaces doesnt do anything unless you install ifupdown | 15:27 |
| dmsimard | yup | 15:27 |
| jamesdenton | netplan, that's all you had to say | 15:27 |
| dmsimard | I purged netplan and went ifupdown | 15:27 |
| jrosser | remember that an AIO does a bunch of stuff you might not want, like preparing the host in a particular way | 15:28 |
| dmsimard | I think my main complaint was around ssl (or lack thereof) | 15:29 |
| dmsimard | the defaults with aio aren't sufficient to provide something that works out of the box | 15:29 |
| dmsimard | this is what I have right now but I sort of lost track if they are really relevant with the trials/errors that I did http://paste.openstack.org/raw/744849/ | 15:31 |
| *** TxGirlGeek has quit IRC | 15:33 | |
| dmsimard | I'm probably going to end up re-deploying in a single go once I figure everything out | 15:34 |
| jamesdenton | jrosser i think this could be breaking some builds: https://review.openstack.org/#/c/633883/ | 15:34 |
| jrosser | jamesdenton: that is quite possible | 15:36 |
| jrosser | do you have an example? | 15:36 |
| jamesdenton | https://review.openstack.org/#/c/633277/ | 15:37 |
| jamesdenton | And locally having the same failures | 15:37 |
| jamesdenton | i just rechecked that, so those logs may disappear | 15:37 |
| chkumar|ruck | odyssey4me: sure | 15:38 |
| jrosser | jamesdenton: so, further on in that test would we expect it to create a VM and try to ssh to it, like the nova tests? | 15:39 |
| jamesdenton | Not from what i've seen. At that point it is just trying to ping the qg interface of the router | 15:39 |
| jamesdenton | locally, both networks (public and private) appear to be 'vxlan' type, which is likely the issue | 15:39 |
| openstackgerrit | Chandan Kumar proposed openstack/openstack-ansible-os_heat master: Fixed the egg name of heat to openstack_heat https://review.openstack.org/635518 | 15:45 |
| jamesdenton | actually - i retract that. let me dig a little. the qg interface is built in the namespace, but there's no bridge setup for it | 15:47 |
| jamesdenton | and the external net IS flat. I just saw two vxlan interfaces, for private and HA. | 15:47 |
| chkumar|ruck | odyssey4me: actually packages were not getting installed | 15:47 |
| chkumar|ruck | might be because of comments I have given, updated it again | 15:48 |
| *** gkadam has quit IRC | 15:49 | |
| jamesdenton | jrosser This may be related to the distro patch itself. Ignore me for now. | 15:49 |
| openstackgerrit | Merged openstack/ansible-role-systemd_networkd master: Add ipforward option to configure IPForward https://review.openstack.org/635929 | 15:49 |
| *** spatel has joined #openstack-ansible | 15:51 | |
| dmsimard | jamesdenton, jrosser: IT'S ALIVE! \o/ | 15:54 |
| jamesdenton | nice job :D | 15:54 |
| openstackgerrit | Michael Vollman proposed openstack/openstack-ansible-tests master: Ensure ipforward is set on all test bridges https://review.openstack.org/636162 | 15:54 |
| dmsimard | I ended up creating a vlan tagged network for the public network, easier to manage from my network gear | 15:55 |
| dmsimard | no floating ips or routers | 15:55 |
| jamesdenton | easy peasy | 15:55 |
| *** markvoelker has quit IRC | 15:56 | |
| dmsimard | now I even get to see the real mac addresses and hosts in my ubiquiti dashboard :p | 15:58 |
| vollman | jamesdenton: i'm seeing the same thing in the neutron gate on master. no route from localhost to the namespace to ping the gateway | 16:05 |
| *** electrofelix has quit IRC | 16:05 | |
| jamesdenton | vollman The bridge for the external network doesn't appear to be setup. | 16:06 |
| vollman | Yea only br-mgmt is configured | 16:07 |
| jamesdenton | yep. may wanna poke at it. If you run it locally, you can run 'toxenvs=docs,linters,functional; tox -e distro_install'. May be easier/faster than trying to gate it | 16:08 |
| *** macza has joined #openstack-ansible | 16:08 | |
| jamesdenton | Just be aware that there are some bugs that may keep you from easily repeating the test when it fails | 16:09 |
| *** macza has joined #openstack-ansible | 16:09 | |
| *** udesale has quit IRC | 16:15 | |
| *** TxGirlGeek has joined #openstack-ansible | 16:16 | |
| *** Emine has quit IRC | 16:21 | |
| *** chkumar|ruck is now known as chandankumar | 16:24 | |
| openstackgerrit | Merged openstack/openstack-ansible-os_cinder master: Ensure cinder-volumes tool packages install correct ceph dependancies https://review.openstack.org/636115 | 16:27 |
| odyssey4me | cloudnull https://review.openstack.org/#/c/636036/1/inventory/group_vars/all/all.yml :) nice - although perhaps that could just go into the venv build role itself? | 16:30 |
| *** spatel has quit IRC | 16:34 | |
| openstackgerrit | Jesse Pretorius (odyssey4me) proposed openstack/openstack-ansible-plugins master: Remove the config_template module https://review.openstack.org/635838 | 16:44 |
| openstackgerrit | Jesse Pretorius (odyssey4me) proposed openstack/openstack-ansible-os_glance master: [TEST] Role test with config_template repo https://review.openstack.org/636171 | 16:46 |
| *** dmsimard6 has joined #openstack-ansible | 16:48 | |
| *** dmsimard has quit IRC | 16:51 | |
| *** dmsimard6 is now known as dmsimard | 16:51 | |
| *** markvoelker has joined #openstack-ansible | 16:52 | |
| cloudnull | odyssey4me sure I can go add that to the venv build role itself | 17:03 |
| cloudnull | its testing well enough on the home lab which is ubuntu, opensuse, centos | 17:04 |
| odyssey4me | cloudnull excellent, I was hoping that someone would figure out a way to make that work - as I understand it, it'll use a same-distro repo server if one is available, but use the target host if not... that way we no longer need to mix up the control plane any more? | 17:05 |
| odyssey4me | cloudnull I think the only missing part now would be to make the repo servers sync their data between each other dynamically, rather than it being a one way sync from the first one. | 17:06 |
| cloudnull | yes that's the hope. I was having issues with the lsync things with master so i ended up disabling it. now builds just points at the appropriate repo server for the os family . | 17:07 |
| cloudnull | having just thought about it, I guess it would be good to add cpu arch to that tag as well | 17:08 |
| cloudnull | to ensure it would work with arm / power | 17:08 |
| odyssey4me | cloudnull yes, that'd be important | 17:08 |
| cloudnull | I'll abandon that patch and go make one for the python_venv role | 17:08 |
| odyssey4me | cloudnull I know what I need to do to further the work - I'll be working on it on Friday again. I'm fairly certain we'll be able to ditch repo-build quite soon. | 17:08 |
| cloudnull | sweet! | 17:09 |
| odyssey4me | However, if you can come up with a way to handle that sync better for the repo servers, then I look forward to seeing a patch. | 17:10 |
| cloudnull | I had to disable it | 17:10 |
| cloudnull | I worked on it a bit, cent and suse configs were so wildly different I found that it was better to simply turn it off | 17:11 |
| odyssey4me | Ugh. Maybe there's some other mechanism we can use instead. | 17:11 |
| cloudnull | NFS? | 17:11 |
| * cloudnull ^ not trolling | 17:12 | |
| odyssey4me | Yeah - a common file system source would work just fine - with some sort of backup and failover if a repo server goes down. | 17:12 |
| cloudnull | like enable if there's a shared storage backend available (ceph/nfs) | 17:12 |
| odyssey4me | I'd rather avoid DRBD if we can help it. | 17:12 |
| *** sdake has quit IRC | 17:12 | |
| cloudnull | ++ | 17:13 |
| cloudnull | gluster FTW! | 17:13 |
| * cloudnull ^ is trolling | 17:13 | |
| odyssey4me | :troll: | 17:13 |
| *** sdake has joined #openstack-ansible | 17:16 | |
| openstackgerrit | Michael Vollman proposed openstack/openstack-ansible-os_neutron master: Add br-vlan and br-vxlan bridges to gates https://review.openstack.org/636179 | 17:24 |
| *** gyee has joined #openstack-ansible | 17:24 | |
| chandankumar | odyssey4me: my review does not worked on os_heat, I am doing something wrong there | 17:24 |
| *** markvoelker has quit IRC | 17:24 | |
| openstackgerrit | Jesse Pretorius (odyssey4me) proposed openstack/openstack-ansible master: Use the config_template module from the dedicated repo https://review.openstack.org/636182 | 17:32 |
| openstackgerrit | Jesse Pretorius (odyssey4me) proposed openstack/openstack-ansible master: Use the config_template module from the dedicated repo https://review.openstack.org/636182 | 17:33 |
| odyssey4me | chandankumar ok, unfortunately I'm not able to look at it now - but will do so in the morning first thing | 17:34 |
| guilhermesp | does someone have seen this happening across metal jobs? http://logs.openstack.org/32/634032/2/check/openstack-ansible-deploy-aio_metal_heat-centos-7/ada9736/logs/ara-report/result/cad94618-c912-470b-88ef-6b1709380f32/ | 17:35 |
| guilhermesp | https://review.openstack.org/#/c/634032/ | 17:35 |
| chandankumar | guilhermesp: yes we are fixing here https://review.openstack.org/635518 | 17:37 |
| chandankumar | guilhermesp: it is also blocked on mpi4py stuff | 17:37 |
| chandankumar | which is not working for me | 17:38 |
| chandankumar | guilhermesp: feel free to take a look | 17:38 |
| guilhermesp | nice thanks chandankumar | 17:38 |
| * guilhermesp looking | 17:38 | |
| jrosser | i've seen this a couple of times for upgrades, git fails to checkout gnocchi at the specified SHA http://paste.openstack.org/show/744876/ | 17:39 |
| jrosser | anyone else seen that? adjusting the SHA forward/back by a commit usually makes it work | 17:39 |
| jrosser | ["fatal: reference is not a tree: 1e506094949d68b452d044941c0d6f85ef997b9e"] | 17:40 |
| odyssey4me | jrosser yeah, I've seen that once in a while too - not consistently - the solution always seems to be to initiative a git fetch or replace the repo clone | 17:40 |
| jrosser | ah, so on the repo server delete it and rerun | 17:41 |
| odyssey4me | jrosser yeah | 17:41 |
| dmsimard | Hmm, I think the last remaining thing to get to work are cinder volumes. Horizon is failing to retrieve the volume list due to self-signed ssl cert but OPENSTACK_SSL_NO_VERIFY is properly set to true in /etc/horizon/local_settings.py | 17:41 |
| *** macza has quit IRC | 17:42 | |
| *** macza_ has joined #openstack-ansible | 17:42 | |
| jrosser | and similarly the repo build is blowing up on molterniron which is probably legacy | 17:42 |
| jrosser | i need to dig into why it still thinks it needs to fetch that for rocky | 17:42 |
| *** DanyC has quit IRC | 17:42 | |
| *** shardy has quit IRC | 17:43 | |
| guilhermesp | chandankumar: https://review.openstack.org/#/c/635518/ I think you need to address mnaser 's commnet | 17:44 |
| dmsimard | odyssey4me, cloudnull: I've been meaning to try syncthing for synchronizing data between hosts :) | 17:45 |
| dmsimard | probably overkill for your repo server though | 17:46 |
| chandankumar | guilhermesp: updated it not working check patchset 2 | 17:48 |
| openstackgerrit | Kevin Carter (cloudnull) proposed openstack/ansible-role-python_venv_build master: Add venv build target hosts data structure https://review.openstack.org/636192 | 17:51 |
| cloudnull | odyssey4me ^ | 17:51 |
| odyssey4me | thanks cloudnull | 17:52 |
| odyssey4me | I'll peek at it in the morning. I take it that you've validated that it's working in your test lab? | 17:52 |
| openstackgerrit | Michael Vollman proposed openstack/openstack-ansible-os_cinder master: Avoid distro installing unused services https://review.openstack.org/633276 | 17:53 |
| cloudnull | I had the other one working in my test env, I've not specifically tested that PR in my local lab (is at the office where I dont have access ) | 17:54 |
| cloudnull | odyssey4me with that PR we should be able to simply point the pip config ( https://review.openstack.org/#/c/620339 ) at the build target ? | 17:54 |
| odyssey4me | cloudnull I had https://review.openstack.org/#/c/620340/ working with https://review.openstack.org/#/c/620339/ and https://review.openstack.org/#/c/620331/ - but on friday I'm going to tweak it up to be a little better... with that, no pip.conf is needed any more at all | 17:56 |
| cloudnull | cool! | 17:57 |
| odyssey4me | the basic plan will be to push up the AIO roles in a working state for role tests, and when we're there I'll just remove all pip.conf config from the integrated repo entirely | 17:57 |
| odyssey4me | We'll also be able to remove pypiserver, because it's not needed - we use find-links instead, because anything from a found link is preferred above an index | 17:57 |
| openstackgerrit | Kevin Carter (cloudnull) proposed openstack/openstack-ansible-os_keystone master: Fix multi-OS support https://review.openstack.org/636039 | 17:58 |
| cloudnull | ^ anyone want to give that a nudge :) | 17:58 |
| odyssey4me | Once that's all done, we can radically simplify the venv build role again to remove all the funky paths and things which are on the repo server right now... and the repo server can just have a flat folder for all wheels | 17:58 |
| cloudnull | also - https://review.openstack.org/#/c/635997 - I'd greatly appreciate a review there too | 18:00 |
| cloudnull | -cc mnaser | 18:00 |
| odyssey4me | I'm out for the night - chat again tomorrow. | 18:00 |
| cloudnull | take care odyssey4me | 18:00 |
| cloudnull | have a good one | 18:00 |
| *** sdake has quit IRC | 18:01 | |
| *** sdake has joined #openstack-ansible | 18:03 | |
| *** DanyC has joined #openstack-ansible | 18:11 | |
| *** kopecmartin is now known as kopecmartin|off | 18:12 | |
| *** DanyC has quit IRC | 18:16 | |
| *** markvoelker has joined #openstack-ansible | 18:21 | |
| openstackgerrit | Kevin Carter (cloudnull) proposed openstack/openstack-ansible-ops master: Update overlay inventory to resovle circular dep https://review.openstack.org/636204 | 18:25 |
| *** sdake has quit IRC | 18:27 | |
| *** sdake has joined #openstack-ansible | 18:30 | |
| *** sdake has quit IRC | 18:32 | |
| openstackgerrit | Merged openstack/openstack-ansible master: Fix typo btrfs -> zfs https://review.openstack.org/636035 | 18:32 |
| openstackgerrit | Merged openstack/openstack-ansible stable/pike: Bump SHAs for stable/pike https://review.openstack.org/636005 | 18:32 |
| openstackgerrit | Merged openstack/openstack-ansible stable/queens: Bump SHAs for stable/queens https://review.openstack.org/636004 | 18:32 |
| *** chhagarw has quit IRC | 18:41 | |
| openstackgerrit | Merged openstack/openstack-ansible stable/rocky: Bump SHAs for stable/rocky https://review.openstack.org/636003 | 18:43 |
| openstackgerrit | James Denton proposed openstack/openstack-ansible master: [docs] Apply provider network config on per-group basis https://review.openstack.org/635013 | 18:51 |
| *** openstackgerrit has quit IRC | 18:51 | |
| *** markvoelker has quit IRC | 18:55 | |
| *** sdake has joined #openstack-ansible | 18:56 | |
| cjloader | in master i'm seeing this error Could not find a version that satisfies the requirement heat (from versions: )\nNo matching distribution found for heat", "stderr_lines": [" Could not find a version that satisfies the requirement heat (from versions: )", "No matching distribution found for heat" | 18:57 |
| cjloader | TASK [python_venv_build : Upgrade the wheel build virtualenv pip/setuptools/wheel to the versions we want] | 18:58 |
| cjloader | on heat api container | 18:59 |
| *** sdake has quit IRC | 19:09 | |
| *** openstackgerrit has joined #openstack-ansible | 19:20 | |
| openstackgerrit | Guilherme Steinmuller Pimentel proposed openstack/openstack-ansible-os_heat master: Fixed the egg name of heat to openstack_heat https://review.openstack.org/635518 | 19:20 |
| cjloader | https://review.openstack.org/#/c/635518/ should fix my above issue | 19:34 |
| *** aedc has quit IRC | 19:39 | |
| *** gyee has quit IRC | 19:52 | |
| *** markvoelker has joined #openstack-ansible | 19:52 | |
| openstackgerrit | James Denton proposed openstack/openstack-ansible master: [docs] Apply provider network config on per-group basis https://review.openstack.org/635013 | 19:53 |
| jamesdenton | sorry cloudnull - one more time | 19:53 |
| cloudnull | done :) | 19:54 |
| guilhermesp | chandankumar: we have some progress https://review.openstack.org/#/c/635518/ | 19:57 |
| *** gyee has joined #openstack-ansible | 20:00 | |
| openstackgerrit | Michael Vollman proposed openstack/openstack-ansible-os_tempest master: Add option to disable router ping https://review.openstack.org/636211 | 20:12 |
| *** markvoelker has quit IRC | 20:24 | |
| openstackgerrit | Merged openstack/openstack-ansible master: [docs] Apply provider network config on per-group basis https://review.openstack.org/635013 | 20:39 |
| *** marst has joined #openstack-ansible | 20:41 | |
| *** asettle has quit IRC | 20:50 | |
| cloudnull | jamesdenton nice! | 20:59 |
| jamesdenton | wee | 21:00 |
| *** markvoelker has joined #openstack-ansible | 21:22 | |
| *** hwoarang has quit IRC | 21:32 | |
| *** hwoarang has joined #openstack-ansible | 21:34 | |
| *** DanyC has joined #openstack-ansible | 21:52 | |
| *** markvoelker has quit IRC | 21:55 | |
| *** ArchiFleKs has quit IRC | 21:59 | |
| *** ArchiFleKs has joined #openstack-ansible | 22:05 | |
| *** sdake has joined #openstack-ansible | 22:19 | |
| *** markvoelker has joined #openstack-ansible | 22:20 | |
| *** sdake has quit IRC | 22:51 | |
| *** sdake has joined #openstack-ansible | 22:54 | |
| *** sdake has quit IRC | 22:59 | |
| *** sdake has joined #openstack-ansible | 23:07 | |
| *** macza_ has quit IRC | 23:22 | |
| *** macza has joined #openstack-ansible | 23:26 | |
| *** marst has quit IRC | 23:29 | |
| *** sdake has quit IRC | 23:31 | |
| *** ironfoot_ has joined #openstack-ansible | 23:40 | |
| *** hwoarang_ has joined #openstack-ansible | 23:40 | |
| *** hwoarang has quit IRC | 23:46 | |
| *** mhayden has quit IRC | 23:46 | |
| *** mrhillsman has quit IRC | 23:46 | |
| *** ironfoot has quit IRC | 23:46 | |
| *** preece has quit IRC | 23:46 | |
| *** yetiszaf has quit IRC | 23:46 | |
| *** FrankZhang has quit IRC | 23:46 | |
| *** fyx has quit IRC | 23:46 | |
| *** Nick_A has quit IRC | 23:46 | |
| *** macza has quit IRC | 23:53 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!