| *** fatdragon has quit IRC | 00:14 | |
| *** speedmann has quit IRC | 00:31 | |
| *** gyee has quit IRC | 00:39 | |
| openstackgerrit | Merged openstack/openstack-ansible-ops master: Update curator to better metric storage https://review.openstack.org/591798 | 00:40 |
|---|---|---|
| *** speedmann has joined #openstack-ansible | 00:45 | |
| openstackgerrit | Kevin Carter (cloudnull) proposed openstack/openstack-ansible master: Replace tempest with healthchecks https://review.openstack.org/587544 | 01:35 |
| cloudnull | evrardjp jrosser: ^ i'll check back in on that a little later but seems to be working locally now | 01:37 |
| cloudnull | fixed the privatetmp issue, thanks again jamesdenton | 01:37 |
| cloudnull | and corrected a small typo in the healthchecks tasks which was resulting in no network getting dhcp | 01:37 |
| cloudnull | after that, everything goes. | 01:38 |
| openstackgerrit | Kevin Carter (cloudnull) proposed openstack/openstack-ansible-lxc_hosts master: Remove the checksum fills https://review.openstack.org/591880 | 01:44 |
| openstackgerrit | Kevin Carter (cloudnull) proposed openstack/openstack-ansible master: Remove checksum-fill, its not needed any longer https://review.openstack.org/591882 | 01:49 |
| openstackgerrit | Kevin Carter (cloudnull) proposed openstack/openstack-ansible master: Remove checksum-fill, its not needed any longer https://review.openstack.org/591882 | 01:50 |
| *** tacoboy has quit IRC | 02:01 | |
| *** gregoryo has joined #openstack-ansible | 02:07 | |
| *** dave-mccowan has quit IRC | 02:19 | |
| *** openstack has joined #openstack-ansible | 02:36 | |
| *** ChanServ sets mode: +o openstack | 02:36 | |
| openstackgerrit | Kevin Carter (cloudnull) proposed openstack/openstack-ansible master: Replace tempest with healthchecks https://review.openstack.org/587544 | 02:39 |
| openstackgerrit | Kevin Carter (cloudnull) proposed openstack/openstack-ansible master: Remove checksum-fill, its not needed any longer https://review.openstack.org/591882 | 02:40 |
| *** hakonw has quit IRC | 02:40 | |
| cloudnull | I need to go crash, but if folks have a moment to review https://review.openstack.org/#/q/starredby:cloudnull+status:open,n,z it'd be greatly appreciated. | 03:30 |
| prometheanfire | I suppose... | 03:32 |
| *** vnogin has joined #openstack-ansible | 03:39 | |
| *** vnogin has quit IRC | 03:44 | |
| openstackgerrit | jacky06 proposed openstack/openstack-ansible-os_tacker master: Update the project home to https https://review.openstack.org/584584 | 04:01 |
| openstackgerrit | jacky06 proposed openstack/openstack-ansible-os_congress master: Update the project home to https https://review.openstack.org/584585 | 04:01 |
| openstackgerrit | jacky06 proposed openstack/openstack-ansible-os_zaqar master: Add description section https://review.openstack.org/580866 | 04:01 |
| openstackgerrit | jacky06 proposed openstack/openstack-ansible-os_watcher master: Add the project source code repository in README https://review.openstack.org/584833 | 04:02 |
| openstackgerrit | jacky06 proposed openstack/openstack-ansible-os_blazar master: Add the project source code repository in README https://review.openstack.org/584834 | 04:02 |
| openstackgerrit | jacky06 proposed openstack/openstack-ansible-os_keystone master: Add the project source code repository in README https://review.openstack.org/584839 | 04:03 |
| openstackgerrit | jacky06 proposed openstack/openstack-ansible-os_panko master: Add the project source code repository in README https://review.openstack.org/584840 | 04:03 |
| openstackgerrit | jacky06 proposed openstack/openstack-ansible-os_monasca master: Add the project source code repository in README https://review.openstack.org/584841 | 04:03 |
| openstackgerrit | jacky06 proposed openstack/openstack-ansible-os_magnum master: Add the project source code repository in README https://review.openstack.org/584842 | 04:03 |
| openstackgerrit | jacky06 proposed openstack/openstack-ansible-os_rally master: Add the project source code repository in README https://review.openstack.org/584845 | 04:04 |
| openstackgerrit | jacky06 proposed openstack/openstack-ansible-os_tempest master: Add the project source code repository in README https://review.openstack.org/584846 | 04:04 |
| openstackgerrit | jacky06 proposed openstack/openstack-ansible-os_almanach master: Add the project source code repository in README https://review.openstack.org/584877 | 04:04 |
| openstackgerrit | jacky06 proposed openstack/openstack-ansible-os_tacker master: Add repository into README.rst https://review.openstack.org/584880 | 04:04 |
| *** poopcat has quit IRC | 04:09 | |
| openstackgerrit | jacky06 proposed openstack/openstack-ansible-os_glance master: Update the description for glance api https://review.openstack.org/584764 | 04:17 |
| *** mmalchuk_ has quit IRC | 04:23 | |
| *** oanson has quit IRC | 04:33 | |
| openstackgerrit | jacky06 proposed openstack/openstack-ansible-os_searchlight master: Add the project source code repository in README https://review.openstack.org/584843 | 04:34 |
| openstackgerrit | caoyuan proposed openstack/openstack-ansible-os_heat master: use include_tasks instead of include https://review.openstack.org/585265 | 04:47 |
| openstackgerrit | caoyuan proposed openstack/openstack-ansible-os_panko master: use include_tasks instead of include https://review.openstack.org/585269 | 04:53 |
| *** oanson has joined #openstack-ansible | 05:00 | |
| *** cshen has joined #openstack-ansible | 05:07 | |
| *** cshen has quit IRC | 05:12 | |
| *** pcaruana has joined #openstack-ansible | 05:12 | |
| *** Bhujay has joined #openstack-ansible | 05:18 | |
| *** mmalchuk has joined #openstack-ansible | 05:20 | |
| openstackgerrit | caoyuan proposed openstack/openstack-ansible-os_heat master: use include_tasks instead of include https://review.openstack.org/585265 | 05:31 |
| openstackgerrit | caoyuan proposed openstack/openstack-ansible-galera_server master: use include_tasks instead of include https://review.openstack.org/585267 | 05:40 |
| openstackgerrit | caoyuan proposed openstack/openstack-ansible-tests master: Use include_tasks instead of include https://review.openstack.org/585308 | 05:53 |
| openstackgerrit | caoyuan proposed openstack/openstack-ansible-tests master: Use include_tasks instead of include https://review.openstack.org/585308 | 05:59 |
| openstackgerrit | caoyuan proposed openstack/openstack-ansible-tests master: Use include_tasks instead of include https://review.openstack.org/585308 | 06:05 |
| openstackgerrit | weizj proposed openstack/openstack-ansible-tests master: Use include_tasks instead of include https://review.openstack.org/585308 | 06:08 |
| openstackgerrit | weizj proposed openstack/openstack-ansible-tests master: Use include_tasks instead of include https://review.openstack.org/585308 | 06:10 |
| openstackgerrit | caoyuan proposed openstack/openstack-ansible-tests master: Use include_tasks instead of include https://review.openstack.org/585308 | 06:14 |
| openstackgerrit | caoyuan proposed openstack/openstack-ansible-tests master: Use include_tasks instead of include https://review.openstack.org/585308 | 06:15 |
| openstackgerrit | caoyuan proposed openstack/openstack-ansible-tests master: Use include_tasks instead of include https://review.openstack.org/585308 | 06:17 |
| openstackgerrit | caoyuan proposed openstack/openstack-ansible-os_cloudkitty master: Use include_tasks instead of include https://review.openstack.org/585203 | 06:19 |
| openstackgerrit | Markos Chandras (hwoarang) proposed openstack/openstack-ansible-os_nova master: [DNM] - Testing openSUSE nova role https://review.openstack.org/495767 | 06:19 |
| openstackgerrit | caoyuan proposed openstack/openstack-ansible-os_cloudkitty master: Use include_tasks instead of include https://review.openstack.org/585203 | 06:22 |
| openstackgerrit | caoyuan proposed openstack/openstack-ansible-os_watcher master: Use include_tasks instead of include https://review.openstack.org/585210 | 06:25 |
| *** fghaas has joined #openstack-ansible | 06:27 | |
| openstackgerrit | caoyuan proposed openstack/openstack-ansible-os_watcher master: Use include_tasks instead of include https://review.openstack.org/585210 | 06:28 |
| *** dxiri has joined #openstack-ansible | 06:30 | |
| openstackgerrit | caoyuan proposed openstack/openstack-ansible-os_heat master: use include_tasks instead of include https://review.openstack.org/585265 | 06:30 |
| openstackgerrit | caoyuan proposed openstack/openstack-ansible-os_almanach master: use include_tasks instead of include https://review.openstack.org/585279 | 06:33 |
| openstackgerrit | caoyuan proposed openstack/openstack-ansible-os_almanach master: use include_tasks instead of include https://review.openstack.org/585279 | 06:36 |
| openstackgerrit | caoyuan proposed openstack/openstack-ansible-os_almanach master: use include_tasks instead of include https://review.openstack.org/585279 | 06:39 |
| *** cshen has joined #openstack-ansible | 06:41 | |
| *** hamzaachi has joined #openstack-ansible | 06:43 | |
| openstackgerrit | caoyuan proposed openstack/openstack-ansible-os_barbican master: use include_tasks instead of include https://review.openstack.org/585304 | 06:43 |
| openstackgerrit | caoyuan proposed openstack/openstack-ansible-os_barbican master: use include_tasks instead of include https://review.openstack.org/585304 | 06:45 |
| openstackgerrit | caoyuan proposed openstack/openstack-ansible-os_octavia master: use include_tasks instead of include https://review.openstack.org/585281 | 06:47 |
| openstackgerrit | caoyuan proposed openstack/openstack-ansible-os_octavia master: use include_tasks instead of include https://review.openstack.org/585281 | 06:50 |
| openstackgerrit | caoyuan proposed openstack/openstack-ansible-os_octavia master: use include_tasks instead of include https://review.openstack.org/585281 | 06:52 |
| *** vnogin has joined #openstack-ansible | 06:52 | |
| *** vnogin has quit IRC | 06:54 | |
| *** vnogin has joined #openstack-ansible | 06:55 | |
| *** mwe1 has quit IRC | 07:01 | |
| *** arbrandes has joined #openstack-ansible | 07:01 | |
| *** mwe1 has joined #openstack-ansible | 07:02 | |
| *** arbrandes1 has quit IRC | 07:04 | |
| jrosser | morning | 07:07 |
| hwoarang | good morning | 07:09 |
| *** electrofelix has joined #openstack-ansible | 07:40 | |
| odyssey4me | o/ | 07:46 |
| jrosser | hi | 07:49 |
| openstackgerrit | Markos Chandras (hwoarang) proposed openstack/openstack-ansible master: zuul: Add ceph jobs for distribution installations https://review.openstack.org/591949 | 08:01 |
| *** DanyC has joined #openstack-ansible | 08:04 | |
| odyssey4me | soo... what's the current state of things? anyone got any idea what the current blockers are? | 08:07 |
| odyssey4me | for master/rocky? | 08:07 |
| *** DanyC has quit IRC | 08:08 | |
| odyssey4me | it seems the ceph job for the integrated build is failing often | 08:08 |
| odyssey4me | ah hwoarang you were mentioning this yesterday http://logs.openstack.org/04/591304/1/check/openstack-ansible-deploy-ceph-ubuntu-xenial/8d3affa/job-output.txt.gz#_2018-08-13_14_24_51_619136 | 08:09 |
| hwoarang | yep | 08:09 |
| hwoarang | odyssey4me: https://github.com/ceph/ceph-ansible/pull/3015 | 08:09 |
| odyssey4me | any solution around for that? | 08:09 |
| hwoarang | but evrardjp afaik pinned ceph-ansible to an older version | 08:10 |
| hwoarang | here https://review.openstack.org/#/c/590778/ | 08:10 |
| odyssey4me | ok, that makes sense | 08:10 |
| odyssey4me | now we just need to get it merged | 08:10 |
| hwoarang | ;p | 08:11 |
| hwoarang | seems it needs some babysitting | 08:12 |
| openstackgerrit | Markos Chandras (hwoarang) proposed openstack/openstack-ansible master: zuul: Add ceph jobs for distribution installations https://review.openstack.org/591949 | 08:15 |
| odyssey4me | Yeah, I believe there's still some sort of issue with cinder which is causing a race condition of some kind. | 08:15 |
| odyssey4me | There were some patches up to the cinder role - have those ever gone through? | 08:16 |
| openstackgerrit | Markos Chandras (hwoarang) proposed openstack/openstack-ansible master: zuul: Add ceph jobs for distribution installations https://review.openstack.org/591949 | 08:16 |
| hwoarang | not sure to be honest | 08:19 |
| odyssey4me | any cores available to validate this backport? https://review.openstack.org/591039 | 08:19 |
| *** dxiri has quit IRC | 08:20 | |
| jrosser | odyssey4me: this was a relatively important find yesterday https://review.openstack.org/#/c/591821/ | 08:21 |
| odyssey4me | oh wow | 08:22 |
| jrosser | well that one is abandoned, but the problem itself is the thing to note there | 08:22 |
| openstackgerrit | Jesse Pretorius (odyssey4me) proposed openstack/ansible-role-systemd_service stable/rocky: Set private sandbox options to false by default https://review.openstack.org/591957 | 08:23 |
| jrosser | yes thats the one | 08:23 |
| hwoarang | aha we were seeing that in opnfv | 08:23 |
| hwoarang | thank you jrosser for pointing that out | 08:23 |
| hwoarang | this releases is being a true 'rocky' one | 08:24 |
| hwoarang | *release | 08:24 |
| odyssey4me | not sure if we're ever going to use branches in that role, but for now we should probably do backports of important fixes | 08:24 |
| jrosser | it was horribly breaking the healthcheck stuff | 08:24 |
| jrosser | this is needed in cinder https://review.openstack.org/#/c/589934/ | 08:27 |
| hwoarang | good point | 08:29 |
| odyssey4me | jrosser: yeah, was just looking at the tempest failures on those two ubuntu jobs | 08:29 |
| openstackgerrit | caoyuan proposed openstack/openstack-ansible-os_aodh master: use include_tasks instead of include https://review.openstack.org/585271 | 08:30 |
| hwoarang | the tempest failure in cinder only happens with distro packages | 08:30 |
| hwoarang | maybe we should just disable the job given we are still using queens for xenial | 08:30 |
| jrosser | tbh i am surprised we have distro installs voting, but there you go | 08:30 |
| hwoarang | that's my 'default' answer when i see xenial+distro failures | 08:31 |
| openstackgerrit | caoyuan proposed openstack/openstack-ansible-os_aodh master: use include_tasks instead of include https://review.openstack.org/585271 | 08:31 |
| odyssey4me | I think disabling the xenial distro install vote is a fair tactic. | 08:31 |
| hwoarang | jrosser: distros voting is not a bad thing provided they actually test the right thing. only centos does the right thing right now and we are lucky that opensuse works | 08:32 |
| odyssey4me | We should actually be removing xenial from master now. | 08:32 |
| hwoarang | because that uses queens too but we are working to release rocky right now | 08:32 |
| openstackgerrit | caoyuan proposed openstack/openstack-ansible-os_aodh master: use include_tasks instead of include https://review.openstack.org/585271 | 08:32 |
| openstackgerrit | Markos Chandras (hwoarang) proposed openstack/openstack-ansible-os_cinder master: zuul: Drop distro installation jobs for Xenial https://review.openstack.org/591961 | 08:34 |
| hwoarang | something like this?^ | 08:34 |
| hwoarang | i actually think we should drop it entirely from the template in the tests repo | 08:35 |
| hwoarang | but ok | 08:35 |
| odyssey4me | I was just thinking that. | 08:35 |
| odyssey4me | As long as there's a bionic version of it. | 08:35 |
| hwoarang | yeah we are not there yet... | 08:35 |
| hwoarang | but seeing all the work jrosser did with bionic we may try to turn it on :) | 08:36 |
| *** gregoryo has quit IRC | 08:36 | |
| odyssey4me | Happy if the bionic one is non-voting for now, just to expose the current state. | 08:36 |
| jrosser | imho this is ready to be nv https://review.openstack.org/#/c/591287/ | 08:36 |
| hwoarang | one sec | 08:36 |
| *** DanyC has joined #openstack-ansible | 08:38 | |
| hwoarang | hmm ok distro installs are not in the template | 08:38 |
| openstackgerrit | Markos Chandras (hwoarang) proposed openstack/openstack-ansible-tests master: zuul.d: Add bionic job for distro package installations https://review.openstack.org/591966 | 08:39 |
| jrosser | that'll need the bionic job creating wont it? | 08:40 |
| jrosser | oh its already there - interesting | 08:41 |
| openstackgerrit | Markos Chandras (hwoarang) proposed openstack/openstack-ansible-os_keystone master: zuul: Replace Xenial distro install job with Ubuntu bionic https://review.openstack.org/591968 | 08:42 |
| openstackgerrit | caoyuan proposed openstack/openstack-ansible-os_trove master: use include_tasks instead of include https://review.openstack.org/585300 | 08:43 |
| openstackgerrit | caoyuan proposed openstack/openstack-ansible-os_trove master: use include_tasks instead of include https://review.openstack.org/585300 | 08:44 |
| openstackgerrit | Markos Chandras (hwoarang) proposed openstack/openstack-ansible-os_cinder master: zuul: Drop distro installation jobs for Xenial https://review.openstack.org/591961 | 08:45 |
| hwoarang | jrosser: re https://review.openstack.org/#/c/591957/ ok to W+1 now? | 08:46 |
| hwoarang | i guess you removed it until you get the results back? | 08:47 |
| jrosser | done | 08:47 |
| hwoarang | ok great | 08:47 |
| jrosser | yeah i just hit the wrong button | 08:47 |
| openstackgerrit | caoyuan proposed openstack/openstack-ansible-os_cinder master: use include_tasks instead of include https://review.openstack.org/585307 | 08:47 |
| openstackgerrit | Markos Chandras (hwoarang) proposed openstack/openstack-ansible-ops master: Optimise vm_disk_mode conditionals https://review.openstack.org/591473 | 08:49 |
| odyssey4me | heh, thanks hwoarang :) | 08:50 |
| odyssey4me | jrosser: got a few mins to validate https://review.openstack.org/591039 ? | 08:51 |
| odyssey4me | hwoarang: I tried doing https://review.openstack.org/591339 but somehow the verbose output is still there in the log collection. Any idea why? | 08:52 |
| openstackgerrit | caoyuan proposed openstack/openstack-ansible-os_cinder master: use include_tasks instead of include https://review.openstack.org/585307 | 08:52 |
| hwoarang | let me check | 08:53 |
| jrosser | odyssey4me: just about to dive in a meeting for 30mins, will look after that | 08:55 |
| odyssey4me | okie dokey | 08:55 |
| openstackgerrit | caoyuan proposed openstack/openstack-ansible-os_masakari master: use include_tasks instead of include https://review.openstack.org/585298 | 08:55 |
| openstackgerrit | caoyuan proposed openstack/openstack-ansible-os_masakari master: use include_tasks instead of include https://review.openstack.org/585298 | 08:57 |
| hwoarang | odyssey4me: log_instance_info called later on in that function sets 'set -x' at the end | 08:57 |
| hwoarang | so it's turning it back on | 08:57 |
| hwoarang | :/ | 08:57 |
| odyssey4me | aha | 08:57 |
| hwoarang | boo | 08:57 |
| hwoarang | sneaky bash! | 08:58 |
| odyssey4me | I wonder if there's a way to capture the current settings, then restore them - instead of assuming like that. | 08:58 |
| odyssey4me | looks like it's possible: https://unix.stackexchange.com/questions/210158/how-can-i-list-bashes-options-for-the-current-shell | 08:59 |
| openstackgerrit | Merged openstack/ansible-role-systemd_service stable/rocky: Set private sandbox options to false by default https://review.openstack.org/591957 | 08:59 |
| openstackgerrit | caoyuan proposed openstack/openstack-ansible-os_searchlight master: use include_tasks instead of include https://review.openstack.org/585313 | 08:59 |
| hwoarang | maybe log_instance_info shouldn't mess with +x -x | 09:00 |
| hwoarang | actually it's the only place where we do set -x | 09:00 |
| hwoarang | rest of code uses set +x at the beginning of functions so seems like we need to turn +x globally and forget about it | 09:00 |
| openstackgerrit | caoyuan proposed openstack/openstack-ansible-os_ceilometer master: use include_tasks instead of include https://review.openstack.org/585305 | 09:01 |
| odyssey4me | hmm, we set -x in all the scripts which call it | 09:02 |
| openstackgerrit | caoyuan proposed openstack/openstack-ansible-os_ceilometer master: use include_tasks instead of include https://review.openstack.org/585305 | 09:02 |
| odyssey4me | I think set +x was used in the various functions to cut down on noise | 09:02 |
| odyssey4me | we should perhaps convert a ton of this into ansible instead, although then we lose what they do if ansible fails to install or whatever | 09:03 |
| odyssey4me | we could also just use zuul's ansible | 09:03 |
| openstackgerrit | caoyuan proposed openstack/openstack-ansible-os_designate master: use include_tasks instead of include https://review.openstack.org/585278 | 09:04 |
| hwoarang | not sure if there is any point in having such verbose out in that place unless we are looking to debug stuff which shouldn't be the default case | 09:06 |
| odyssey4me | true | 09:06 |
| openstackgerrit | caoyuan proposed openstack/openstack-ansible-os_designate master: use include_tasks instead of include https://review.openstack.org/585278 | 09:06 |
| odyssey4me | we could also just put status/debug information is as echo commands or something anyway | 09:06 |
| odyssey4me | right now our scripts are kinda messy to execute - it'd be nice if we could make them do cleaner output | 09:07 |
| openstackgerrit | caoyuan proposed openstack/openstack-ansible-os_blazar master: use include_tasks instead of include https://review.openstack.org/585268 | 09:08 |
| openstackgerrit | Jesse Pretorius (odyssey4me) proposed openstack/openstack-ansible master: Disable verbose output of log collection https://review.openstack.org/591339 | 09:10 |
| odyssey4me | well, perhaps at the PTG we can figure out how to cut down on the bash and move as much as possible to ansible | 09:10 |
| openstackgerrit | caoyuan proposed openstack/openstack-ansible-os_magnum master: use include_tasks instead of include https://review.openstack.org/585284 | 09:11 |
| openstackgerrit | caoyuan proposed openstack/openstack-ansible-os_magnum master: use include_tasks instead of include https://review.openstack.org/585284 | 09:12 |
| openstackgerrit | caoyuan proposed openstack/openstack-ansible-os_tacker master: use include_tasks instead of include https://review.openstack.org/585310 | 09:15 |
| openstackgerrit | caoyuan proposed openstack/openstack-ansible-os_tacker master: use include_tasks instead of include https://review.openstack.org/585310 | 09:16 |
| *** berendt_ has quit IRC | 09:18 | |
| *** xgerman_ has quit IRC | 09:18 | |
| openstackgerrit | Merged openstack/openstack-ansible-ops master: Optimise vm_disk_mode conditionals https://review.openstack.org/591473 | 09:18 |
| openstackgerrit | caoyuan proposed openstack/openstack-ansible-os_tempest master: use include_tasks instead of include https://review.openstack.org/585276 | 09:19 |
| openstackgerrit | caoyuan proposed openstack/openstack-ansible-os_tempest master: use include_tasks instead of include https://review.openstack.org/585276 | 09:19 |
| openstackgerrit | caoyuan proposed openstack/openstack-ansible-os_tempest master: use include_tasks instead of include https://review.openstack.org/585276 | 09:22 |
| openstackgerrit | caoyuan proposed openstack/openstack-ansible-os_tempest master: use include_tasks instead of include https://review.openstack.org/585276 | 09:23 |
| openstackgerrit | Merged openstack/openstack-ansible-repo_build master: Generate a filtered list of git repos to clone https://review.openstack.org/591368 | 09:24 |
| openstackgerrit | caoyuan proposed openstack/openstack-ansible-os_congress master: use include_tasks instead of include https://review.openstack.org/585266 | 09:24 |
| openstackgerrit | caoyuan proposed openstack/openstack-ansible-os_congress master: use include_tasks instead of include https://review.openstack.org/585266 | 09:25 |
| openstackgerrit | caoyuan proposed openstack/openstack-ansible-os_horizon master: use include_tasks instead of include https://review.openstack.org/585299 | 09:27 |
| openstackgerrit | caoyuan proposed openstack/openstack-ansible-os_horizon master: use include_tasks instead of include https://review.openstack.org/585299 | 09:28 |
| *** hamzaachi has quit IRC | 09:30 | |
| *** gkadam has joined #openstack-ansible | 09:30 | |
| openstackgerrit | caoyuan proposed openstack/openstack-ansible-os_swift master: use include_tasks instead of include https://review.openstack.org/585280 | 09:32 |
| *** hamzaachi has joined #openstack-ansible | 09:33 | |
| openstackgerrit | caoyuan proposed openstack/openstack-ansible-os_swift master: use include_tasks instead of include https://review.openstack.org/585280 | 09:35 |
| *** mjwales has joined #openstack-ansible | 09:38 | |
| *** sep has quit IRC | 09:41 | |
| *** sep has joined #openstack-ansible | 09:41 | |
| mjwales | Hey guys, so I have managed to get something sort of working. I can now spin up an instance and | 09:42 |
| *** sep has quit IRC | 09:43 | |
| *** sep has joined #openstack-ansible | 09:43 | |
| *** fghaas has quit IRC | 09:44 | |
| mjwales | which was not working before. The external gateway on the router is also showing as active which is was not before but my instances do not have any external network access. Nothing is sticking out in any of the neutron log files. Anyone got any ideas where else to look for error messages that might give some idea what is going on? | 09:44 |
| noonedeadpunk | odyssey4me: Hi, could you please check these "cherry-picks"? https://review.openstack.org/591335 https://review.openstack.org/591590 | 09:44 |
| odyssey4me | noonedeadpunk: https://review.openstack.org/591590 does not appear to be a cherry pick at all | 09:45 |
| noonedeadpunk | odyssey4me: yep, it's not, as queens has different format of tmeplate... | 09:46 |
| noonedeadpunk | but it's kinda similar and implements the same thing | 09:47 |
| odyssey4me | noonedeadpunk: then it should share the change ID at least, and the commit message must explain why it's not a clean cherry pick | 09:47 |
| noonedeadpunk | odyssey4me: my fault, fixing it | 09:48 |
| noonedeadpunk | should I use depends-on for referencing to the master? | 09:50 |
| *** mjwales has quit IRC | 09:50 | |
| jrosser | mjwales: what you probably should do is get on the appropriate infra/network node and use 'ip netns list' to see your network namespaces | 09:51 |
| *** sep has quit IRC | 09:51 | |
| jrosser | then ip netns exec <ns> <command> to examine the interfaces in the router namespace, and try to ping the router ip and your upstream network gateway | 09:52 |
| jrosser | that will tell you if you have everything wired up right to your external network | 09:52 |
| *** mjwales has joined #openstack-ansible | 09:54 | |
| odyssey4me | noonedeadpunk: no, it does not depend on the aster patch - the change id shows the link to the master patch | 09:54 |
| odyssey4me | depends-on is used to test patches together | 09:55 |
| noonedeadpunk | odyssey4me: so I need to pass change id same as for master, right? | 09:57 |
| odyssey4me | noonedeadpunk: yes, the change id in the commit msg must match that of the master branch | 09:57 |
| openstackgerrit | Dmitriy Rabotjagov (noonedeadpunk) proposed openstack/openstack-ansible-lxc_container_create stable/queens: Implements possibility to use physical networks in containers. https://review.openstack.org/591590 | 09:59 |
| noonedeadpunk | odyssey4me: ^ but for some reason by this change-id patch for queens is not present https://review.openstack.org/#/q/I299aa3fb26f5278d2c86bb18c5df5abddcf0ff65 | 10:02 |
| odyssey4me | noonedeadpunk: did you edit the patch in gerrit, or did you push it from your workstation? | 10:03 |
| odyssey4me | if you edited it in gerrit, it will do that - if you edit on your workstation and push it up then it'll do a new patch and you'll have to abandon the old one | 10:03 |
| odyssey4me | although now I'm not sure what it'll do | 10:04 |
| openstackgerrit | Dmitriy Rabotjagov (noonedeadpunk) proposed openstack/openstack-ansible-lxc_container_create stable/queens: Implements possibility to use physical networks in containers. https://review.openstack.org/591996 | 10:04 |
| odyssey4me | better, thanks | 10:05 |
| noonedeadpunk | odyssey4me: yep, I've edited with gerrit, as I hoped that it will be linked without creating new review) | 10:06 |
| noonedeadpunk | every day I'm learning smth new | 10:06 |
| odyssey4me | :) as do we all | 10:07 |
| odyssey4me | looks fine to me | 10:07 |
| noonedeadpunk | great. now I need to annoy some another core to get it merged) I've heard that evrardjp won't be here today? | 10:10 |
| *** dave-mccowan has joined #openstack-ansible | 10:12 | |
| odyssey4me | noonedeadpunk: yes, he's out today | 10:13 |
| jrosser | looks reasonble - has this been tested on queens? | 10:14 |
| noonedeadpunk | jrosser: yep, we're running queens now, and this was used for configuring containers with storage networks to use infiniband for ceph | 10:16 |
| mjwales | jrosser: any chance you could paste your openstack_user_config and network interface config of your infra nodes? Still trying to chase down the external networking issue from last night | 10:21 |
| jrosser | i can but like i said yesterday i've had to completely abandon openstack_user_config/provider_networks to do what i wanted | 10:24 |
| jrosser | which feels wrong really, but i was just unable to set things up as i wanted otherwise | 10:25 |
| mjwales | Ahh okay. I've got a semi working solution. Using tcpdump I can see traffic on my veth but it doesn't get passed to the external bridge which hints that I've missed something on the host interface configuration | 10:27 |
| jrosser | fwiw heres the config http://paste.openstack.org/show/728101/ | 10:28 |
| *** olivierbourdon38 has joined #openstack-ansible | 10:31 | |
| jrosser | mjwales: this is then how i have configured my external networking http://paste.openstack.org/show/728102/ | 10:33 |
| jrosser | it's been a long time since i did this so it really needs revisiting and making sure the comments are all correct | 10:34 |
| mjwales | My setup simply looks like network traffic is not passing from the veth to the physical bridge on the host. Once I get that solved it might work | 10:35 |
| jamesdenton | mjwales jrosser i'll be around in 45 min if you'd like to work on it some more | 10:36 |
| jrosser | i have meetings for the next 2hrs but will be back after that | 10:37 |
| mjwales | Just popping out to lunch but should be back in 30-45 mins. Hopefully can get this nailed today :) | 10:38 |
| jamesdenton | mjwales did you double check those veths were in an 'UP' state? You may need to 'ip link set ... up' them | 10:40 |
| jamesdenton | unless OSA created them, in which case that should be done for you | 10:41 |
| *** sep has joined #openstack-ansible | 10:44 | |
| *** priteau has joined #openstack-ansible | 10:45 | |
| noonedeadpunk | How galera_server is mapped to the release tag? By commit SHA? As bug on this line makes me crazy https://github.com/openstack/openstack-ansible-galera_server/blob/17.0.4/tasks/galera_cluster_state.yml#L51 | 11:02 |
| noonedeadpunk | And it fixed long time ago by cloudnull, but on 17.0.5 I still catch it. | 11:03 |
| jrosser | noonedeadpunk: that is interesting, galera_server appears to be missing some of the later tags | 11:05 |
| openstackgerrit | Merged openstack/openstack-ansible-os_tempest stable/rocky: Execute service setup against a delegated host https://review.openstack.org/591039 | 11:05 |
| jrosser | odyssey4me: should we expect to see a tag in each role for each tagged release in openstack-ansible? | 11:08 |
| openstackgerrit | Merged openstack/openstack-ansible-lxc_container_create stable/queens: Implements possibility to use physical networks in containers. https://review.openstack.org/591996 | 11:08 |
| noonedeadpunk | probably tags are placed only if some changes present, but there should definitely be changes for the last release | 11:11 |
| *** dante-as has joined #openstack-ansible | 11:30 | |
| mjwales | jamesdenton: on one of the infra nodes both br-ex-veth@eth20 and eth20@br-ex-veth are showing as up. If I run a tcpdump on one of the interface (i.e. br-ex-veth) and ping the external gateway from a instance output is: 11:36:03.412459 fa:16:3e:9e:35:d1 > ff:ff:ff:ff:ff:ff, ethertype ARP (0x0806), length 42: Request who-has XXX.XXX.XXX.XXX.1 tell XXX.XXX.XXX.21.So as far as I can tell the traffic is not being passed to the phys | 11:30 |
| mjwales | ical interface | 11:30 |
| jamesdenton | and the br-ex-veth bridge has what other interfaces in it? | 11:31 |
| dante-as | Hi guys, any ways to trick the os_subnet variable to omit if it is not existing in the variable files? Something like: dns_nameservers : - "{{ item.dns | default(omit) }}", but this one is not working since it is not in an IP format | 11:32 |
| mjwales | configuration for ifcfg-br-ex http://paste.openstack.org/show/728111/ | 11:35 |
| *** fghaas has joined #openstack-ansible | 11:36 | |
| jamesdenton | mjwales does a tcpdump on enp59s0f0 show anything? | 11:39 |
| openstackgerrit | Merged openstack/openstack-ansible-tests master: zuul.d: Add bionic job for distro package installations https://review.openstack.org/591966 | 11:39 |
| mjwales | jamesdenton: nothing similar to the traffic on br-ex-veth i.e. no requests for .1 from .21 | 11:42 |
| mjwales | no iptable rules are anything like that need to be added to route the traffic? | 11:42 |
| jamesdenton | ok. is this queens? and I assume the Neutron agent is running on baremetal and not inside a neutron_agents_container, right? | 11:42 |
| mjwales | Yes, this is queens and bare metal as far as I can tell | 11:43 |
| jamesdenton | k | 11:43 |
| odyssey4me | jrosser: nope, the release process has changed - we're no longer tagging the roles | 11:45 |
| jamesdenton | mjwales is it safe to assume that the IP configured on br-ex is in the same subnet as the instance? If so, can you ping the .1 addr from the host itself? | 11:45 |
| odyssey4me | noonedeadpunk: the sha for the role is in the a-r-r file, eg: https://github.com/openstack/openstack-ansible/blob/17.0.5/ansible-role-requirements.yml#L20 | 11:45 |
| jamesdenton | sourcing from that interface addr | 11:45 |
| odyssey4me | noonedeadpunk: you should update to the later tag if it has the fix | 11:45 |
| noonedeadpunk | odyssey4me: yep, I feel that I should upgrade, but wasn't sure that will help | 11:46 |
| odyssey4me | looks like 17.0.6 has that fix | 11:47 |
| mjwales | Instance is on a 192.168.0.0/24 network at the moment with the external network set on the router external net is 131.XXX.XXX.XXX/24 | 11:47 |
| mjwales | the instance currently does not have a floating IP assigned | 11:48 |
| *** hamzaachi has quit IRC | 11:49 | |
| jamesdenton | mjwales ok - so in that case we should expect the router to SNAT that traffic. Are you using VXLAN for the instance network? | 11:50 |
| jamesdenton | is x.x.x.21 the router's QG interface or the instance? | 11:51 |
| mjwales | Yes, admin_net (192.168.0.0/24) is using vxlan segmentation id 12. | 11:51 |
| mjwales | x.x.x.x.12 is the routers external gateway | 11:52 |
| jamesdenton | So what is x.x.x.21? | 11:53 |
| mjwales | Sorry the routers interface | 11:54 |
| jamesdenton | kk | 11:55 |
| *** olivierbourdon38 has quit IRC | 11:55 | |
| jamesdenton | from that router namespace, can you ping x.x.x.3? You can invoke the namespace using "ip netns exec qrouter-<uuid> ping x.x.x.3". I also can't tell from your pastebin if you're putting br-ex-veth inside the br-ex bridge | 11:56 |
| jamesdenton | you may need a 'post-up brctl addif br-ex br-ex-veth'. | 11:58 |
| ansmith | odyssey4me: hi, saw that https://review.openstack.org/#/c/590091/ has dependency on https://review.openstack.org/#/c/589234/ | 11:58 |
| ansmith | odyssey4me: does it make sense to abandon 589234 and remove dependency? | 11:59 |
| odyssey4me | ansmith: yeah, perhaps it does - lemme change that | 11:59 |
| openstackgerrit | Jesse Pretorius (odyssey4me) proposed openstack/openstack-ansible-tests master: Add qdrouterd user to MQ setup and update file https://review.openstack.org/590091 | 12:00 |
| mjwales | jamesdenton: so a 'ip netns exec qrouter-4654f5d6-48b8-4c59-a0ec-4470eb7066d2 ping x.x.x.x.3 ' resulted in 100% packet loss. Ran brctl addif br-ex br-ex-veth on the infra node set as active for the HA routers and tada ping works. | 12:01 |
| mjwales | Thank you so much:! :D | 12:01 |
| jamesdenton | sweet. | 12:02 |
| mjwales | Judging by what I have now it should be a case of adding another bridge (br-ex-private) to give the private floating IP pool. | 12:03 |
| mjwales | Once I have all this working on this deployment I will wipe everything clean and redeploy as this cloud has had too much tinkering with | 12:04 |
| jamesdenton | Well, that depends. Would you be using the bond for that network? | 12:04 |
| mjwales | It will go on to the same physical interface which did somehow work with fuel. I think our network team here have done some magic and i have no idea what | 12:05 |
| mjwales | I will give it a shot anyhoo and see where I get | 12:06 |
| *** fghaas has quit IRC | 12:06 | |
| odyssey4me | weird - I have systemd mounting a new partition (which matches a previous mount) when I didn't ask it to. Anyone seen something like that before on bionic? | 12:08 |
| jamesdenton | It may be as easy as creating a new veth pair and plugging it into the same br-ex bridge. That new pair 'br-ex-private<->eth21' could be used for the second flat network. But if both subnets are stacked on the same VLAN, then there are security implications there. Just FYI. | 12:08 |
| odyssey4me | hmm, looks like it's some sort of device auto mount | 12:12 |
| mjwales | jamesdenton: just checked with our network guru here and the address spaces are "overlaid on the same collision domain". so you we do have security implications but for our use case this isn't a problem | 12:13 |
| jamesdenton | ok cool. So then I would just create that second veth pair using the same method you used - the pre-up statements. And map 'physnet2:eth21', or something like that. And but br-ex-private into br-ex and call it a day | 12:14 |
| jamesdenton | *put | 12:14 |
| mjwales | Excellent! Again thanks for all the help deffo would not have got here without it | 12:15 |
| jamesdenton | Sure thing! We look forward to your documentation contributions :D | 12:15 |
| mjwales | 100% hopefully can make it easier for any newcomer like me to get something up and running | 12:16 |
| jamesdenton | right on | 12:18 |
| mnaser | morning everyone | 12:21 |
| mnaser | `openstack-ansible-functional-distro_install-ubuntu-xenial` is broken on cinder. it looks like it's fighting with httpd for listening on the port | 12:22 |
| mnaser | it looks like it installs httpd by default and then it listens on the cinder port, then uwsgi fails to go up | 12:22 |
| mnaser | it looks like rabbitmq was blocking cinder out | 12:27 |
| mnaser | hmm | 12:27 |
| mnaser | http://logs.openstack.org/34/589934/3/check/openstack-ansible-functional-ubuntu-xenial/052a993/logs/openstack/infra1/rabbitmq/rabbit@infra1.log.txt.gz#_2018-08-15_00_47_05_252 | 12:28 |
| mnaser | "missed heartbeats from client, timeout: 60s" | 12:28 |
| *** dxiri has joined #openstack-ansible | 12:28 | |
| mnaser | this is on packethost. did we ever end up resolving our mtu issues? | 12:28 |
| mnaser | (cc logan- cloudnull) | 12:28 |
| *** hamzaachi has joined #openstack-ansible | 12:31 | |
| *** zul has joined #openstack-ansible | 12:42 | |
| jrosser | jamesdenton: you got a moment? | 12:52 |
| jamesdenton | jrosser sure do | 12:52 |
| jrosser | i was in a similar position to mjwales wanting to have very specific interfaces for my external networks, and it nearly drove me nuts :)] | 12:52 |
| jamesdenton | hehe ok | 12:53 |
| jrosser | so this http://paste.openstack.org/show/728101/ doesnt even attempt to set them up | 12:53 |
| jrosser | becasue it was just too hard | 12:53 |
| jrosser | and i did this instead http://paste.openstack.org/show/728102/ | 12:53 |
| jamesdenton | you've got a leaf/spine setup? | 12:53 |
| jrosser | yeah | 12:53 |
| jamesdenton | yeah, that's tricky | 12:54 |
| jrosser | well not really | 12:54 |
| jrosser | the controlplane has its own leafs | 12:54 |
| jrosser | with vpc bonds to all the infra nodes | 12:54 |
| jrosser | it just happens that the compute nodes are routed from that | 12:55 |
| jrosser | i've gone for the same L2 resilience approach in the control plane that the example configs have | 12:56 |
| jrosser | and then addressed scalability of compute with L3/leaf+spine | 12:56 |
| jamesdenton | ok | 12:56 |
| jamesdenton | it's working, right? | 12:57 |
| jrosser | it is | 12:57 |
| * jamesdenton slowly backs away | 12:57 | |
| jamesdenton | hehe | 12:57 |
| jrosser | but i couldnt do it through provider_networks | 12:57 |
| jamesdenton | TBH this is the first time I'm seeing the pod config | 12:58 |
| *** hamzaachi has quit IRC | 12:59 | |
| openstackgerrit | Dmitriy Rabotjagov (noonedeadpunk) proposed openstack/openstack-ansible-ceph_client master: Added possibility to specify ssh username and port for ceph mons. https://review.openstack.org/583943 | 13:01 |
| jamesdenton | jrosser do you have any notes from when you tried provider networks (i.e. configs that didnt work)? | 13:01 |
| jrosser | perhaps - i'll have to look back | 13:01 |
| jrosser | i think it's all rooted in wanting a very specific interface to be used on the network nodes that doesnt exist elsewhere | 13:01 |
| jrosser | i have lots of nodes, and they quite strictly only carry the interfaces/bridges that are required | 13:02 |
| jrosser | so my group_vars/network_all.yml wires that very specifically for the network nodes | 13:02 |
| jamesdenton | Only the network nodes have bond1 and would have physnet_rd mapping? | 13:03 |
| jrosser | yes | 13:03 |
| odyssey4me | for future reference to all - if you have a fstab entry, systemd uses the systemd-fstab-generator to generate unit files to handle the mounting - so if you remove the fstab entry, you need to do 'systemctl daemon-reload' to remove the unit files automagically... otherwise systemd will very helpfully remount your device even though you didn't ask it to | 13:03 |
| jamesdenton | Sorry odyssey4me - memory full | 13:04 |
| jrosser | jamesdenton: perhaps my config is all a bit wierd anyway - but there have been a few folks turn up where their external network certainly isnt the flat network on br-vlan and then it all gets a bit difficult | 13:06 |
| odyssey4me | mnaser: I think we've come to the conclusion that the distro install for xenial is a lost cause, given that xenial's distro packages only get to queens | 13:07 |
| jamesdenton | jrosser You might be able to setup a provider network using 'net_name' of 'physnet_rd', 'container_interface' (maybe host_bind_override) of 'bond1' and a group_binds of network_all. | 13:07 |
| odyssey4me | mnaser: so the general idea should be for us to make the bionic distro install work for rocky onwards, and remove the xenial distro install (or disable voting for it at least) | 13:08 |
| mnaser | odyssey4me: should we either take it to non voting or remove the jobs for now? | 13:08 |
| odyssey4me | mnaser: I think hwoarang pushed up some patches earlier today to do that | 13:08 |
| * mnaser searches reviews | 13:08 | |
| odyssey4me | https://review.openstack.org/591961 | 13:09 |
| odyssey4me | https://review.openstack.org/591968 | 13:09 |
| jamesdenton | jrosser yes - the default config is a lowest-common-denominator sort-of thing. We tend to have a single nic/bond in br-vlan on all hosts, and let neutron do the tagging for external and non-external vlan provider/tenant networks. tend to not bother with flat. So the default 'vlan' provider network works for us on all hosts. But its a very homogenous setup. | 13:09 |
| mnaser | odyssey4me: wonderful. ok, so now the only things left in that would be the issue with ubuntu xenial source installs | 13:11 |
| *** hamzaachi has joined #openstack-ansible | 13:11 | |
| odyssey4me | yep, tempest failing due to a timeout | 13:11 |
| openstackgerrit | Merged openstack/openstack-ansible-os_ceilometer stable/rocky: Added drivers for coordination. https://review.openstack.org/591335 | 13:11 |
| jrosser | jamesdenton: sure, thats nice and simple...... i cant have the external net even on the same switch as internal bv-vlan and the wormhole kind of starts there! | 13:11 |
| guilhermesp | mnaser odyssey4me probably the missed heartbeats of rabbitmq... | 13:11 |
| jamesdenton | completely understand. i can play around with the provider config and try to mimic what you're looking to do and get back to you? | 13:12 |
| odyssey4me | guilhermesp mnaser pretty sure I've seen those in healthy deployments? | 13:12 |
| jrosser | that would be really useful yes | 13:12 |
| mnaser | odyssey4me, guilhermesp: "[b956eec0-15bb-4070-8a23-fd0f3c5e5a8e] AMQP server on 10.1.1.101:5672 is unreachable: <AMQPError: unknown error>. Trying again in 1 seconds." | 13:12 |
| mnaser | during a CI run, therefore cinder-volume drops out and the create/delete/etc dies | 13:12 |
| odyssey4me | aha, that's not good | 13:13 |
| jrosser | jamesdenton: i remember digging into a massive load of inline jinja that processed provider_networks | 13:13 |
| guilhermesp | I'm running it right now here to see things happening | 13:13 |
| mnaser | http://logs.openstack.org/61/591961/2/check/openstack-ansible-functional-ubuntu-xenial/f1781ce/logs/openstack/infra1/rabbitmq/rabbit@infra1.log.txt.gz#_2018-08-15_10_41_28_789 | 13:13 |
| mnaser | it is always happening | 13:13 |
| jamesdenton | the horror :D | 13:13 |
| mnaser | unfortunately the json logging in cinder right now is a bit hard on the yes | 13:13 |
| mnaser | s/yes/eyes/ | 13:13 |
| mnaser | https://bugs.launchpad.net/cinder/+bug/1703493 is the reason we have it | 13:14 |
| openstack | Launchpad bug 1703493 in Cinder "create volume with Chinese character raise error" [Undecided,In progress] - Assigned to hongwei.xu (dengxiafubi) | 13:14 |
| guilhermesp | this log says that vip is loosing connection with the infra container. So should we isolate it to a keepalived/haproxy and rabbit issue? | 13:15 |
| odyssey4me | hmm, I don't think the cinder role test uses haproxy/keepalived | 13:16 |
| mnaser | indeed, i dont think it does, even then, it connects directly to rabbitmq without haproxy | 13:16 |
| guilhermesp | yes, my bad | 13:17 |
| mnaser | all good | 13:17 |
| guilhermesp | just happening right now the missed heartbeats in my test env | 13:17 |
| *** mjwales has quit IRC | 13:21 | |
| openstackgerrit | Jesse Pretorius (odyssey4me) proposed openstack/openstack-ansible-ops master: MNAIO: Enable switching between lvm and file vm disk https://review.openstack.org/592027 | 13:21 |
| mnaser | i noticed it happened across 2 providers | 13:23 |
| mnaser | (in ci at least) | 13:24 |
| mnaser | did we bump versions | 13:24 |
| mnaser | for rabbitmq | 13:24 |
| mnaser | only for bionic | 13:25 |
| mnaser | wait no | 13:25 |
| mnaser | uhoh | 13:25 |
| * mnaser is at fault | 13:26 | |
| mnaser | jrosser: proposed https://review.openstack.org/#/c/586558/ -- odyssey4me proposed https://review.openstack.org/#/c/587765/ | 13:26 |
| mnaser | so we're installing xenial rabbitmq in bionic | 13:26 |
| mnaser | (unrelated, but could be) | 13:26 |
| mnaser | oh that was reverted already. damnit, i need more coffee | 13:26 |
| jrosser | oh yes that went a bit wonky | 13:28 |
| openstackgerrit | Jesse Pretorius (odyssey4me) proposed openstack/openstack-ansible-ops master: MNAIO: Enable using a data disk for file-backed VM's https://review.openstack.org/592027 | 13:29 |
| odyssey4me | wonky is a good technical term for it :p | 13:30 |
| mnaser | do we tweak or mess with default heartbeat numbers in cinder | 13:32 |
| * mnaser checks | 13:32 | |
| mnaser | oh man json logging leaks so much credentials. | 13:34 |
| guilhermesp | heheheh wonky from json | 13:35 |
| openstackgerrit | Merged openstack/openstack-ansible-tests master: Add qdrouterd user to MQ setup and update file https://review.openstack.org/590091 | 13:37 |
| *** throwsb1 has joined #openstack-ansible | 13:38 | |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/openstack-ansible-os_aodh master: Updated from OpenStack Ansible Tests https://review.openstack.org/592030 | 13:40 |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/openstack-ansible-os_barbican master: Updated from OpenStack Ansible Tests https://review.openstack.org/592031 | 13:40 |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/openstack-ansible-os_ceilometer master: Updated from OpenStack Ansible Tests https://review.openstack.org/592032 | 13:40 |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/openstack-ansible-os_cinder master: Updated from OpenStack Ansible Tests https://review.openstack.org/592033 | 13:40 |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/openstack-ansible-os_cloudkitty master: Updated from OpenStack Ansible Tests https://review.openstack.org/592034 | 13:40 |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/openstack-ansible-os_congress master: Updated from OpenStack Ansible Tests https://review.openstack.org/592035 | 13:40 |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/openstack-ansible-os_designate master: Updated from OpenStack Ansible Tests https://review.openstack.org/592036 | 13:40 |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/openstack-ansible-os_glance master: Updated from OpenStack Ansible Tests https://review.openstack.org/592037 | 13:40 |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/openstack-ansible-os_heat master: Updated from OpenStack Ansible Tests https://review.openstack.org/592038 | 13:40 |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/openstack-ansible-os_ironic master: Updated from OpenStack Ansible Tests https://review.openstack.org/592039 | 13:41 |
| openstackgerrit | Jesse Pretorius (odyssey4me) proposed openstack/openstack-ansible-tests stable/rocky: Add qdrouterd user to MQ setup and update file https://review.openstack.org/592040 | 13:41 |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/openstack-ansible-os_keystone master: Updated from OpenStack Ansible Tests https://review.openstack.org/592041 | 13:41 |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/openstack-ansible-os_magnum master: Updated from OpenStack Ansible Tests https://review.openstack.org/592042 | 13:41 |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/openstack-ansible-os_masakari master: Updated from OpenStack Ansible Tests https://review.openstack.org/592043 | 13:41 |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/openstack-ansible-os_neutron master: Updated from OpenStack Ansible Tests https://review.openstack.org/592044 | 13:41 |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/openstack-ansible-os_nova master: Updated from OpenStack Ansible Tests https://review.openstack.org/592045 | 13:41 |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/openstack-ansible-os_octavia master: Updated from OpenStack Ansible Tests https://review.openstack.org/592046 | 13:41 |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/openstack-ansible-os_sahara master: Updated from OpenStack Ansible Tests https://review.openstack.org/592047 | 13:42 |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/openstack-ansible-os_swift master: Updated from OpenStack Ansible Tests https://review.openstack.org/592048 | 13:42 |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/openstack-ansible-os_tacker master: Updated from OpenStack Ansible Tests https://review.openstack.org/592049 | 13:42 |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/openstack-ansible-os_trove master: Updated from OpenStack Ansible Tests https://review.openstack.org/592051 | 13:42 |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/openstack-ansible-os_watcher master: Updated from OpenStack Ansible Tests https://review.openstack.org/592052 | 13:42 |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/openstack-ansible-os_zun master: Updated from OpenStack Ansible Tests https://review.openstack.org/592053 | 13:42 |
| *** hamzaachi_ has joined #openstack-ansible | 13:42 | |
| *** hamzaachi has quit IRC | 13:45 | |
| openstackgerrit | Merged openstack/openstack-ansible-os_keystone master: zuul: Replace Xenial distro install job with Ubuntu bionic https://review.openstack.org/591968 | 13:49 |
| openstackgerrit | Jesse Pretorius (odyssey4me) proposed openstack/openstack-ansible-ops master: Use the Ansible runtime venv for pw-token-gen.py https://review.openstack.org/592055 | 13:53 |
| *** hvvcben has joined #openstack-ansible | 13:53 | |
| jrosser | \o/ ^ bionic all green there :) | 13:53 |
| *** vollman has joined #openstack-ansible | 13:54 | |
| guilhermesp | wonderful job jrosser ! | 13:54 |
| mnaser | next up is | 13:56 |
| * mnaser throws hands in air from cinder | 13:56 | |
| jrosser | if anyone has a mooment to look at this i'll do that same in a couple other places https://review.openstack.org/#/c/591412/ | 13:56 |
| mnaser | jrosser: forgive me if this is a bit of a pain but is there any reason why we cant do the merge inside defaults/main.yml and avoid duplicating code across all the other files | 13:59 |
| *** hvvcben has quit IRC | 14:00 | |
| mnaser | so lxc_cache_distro_packages: "{{ _lxc_cache_distro_packages + lxc_cache_extra_distro_packages }}" | 14:00 |
| mnaser | and change the var in vars/os.yml to be `_lxc_cache_distro_packages` instead of `_lxc_cache_distro_packages` ? | 14:00 |
| openstackgerrit | Jesse Pretorius (odyssey4me) proposed openstack/openstack-ansible-ops master: MNAIO: Enable using a data disk for file-backed VM's https://review.openstack.org/592027 | 14:01 |
| jrosser | mnaser: i'm just looking, its making my head hurt :) | 14:04 |
| jrosser | mnaser: just to be clear theres two things going on - packages in the cache and packages on the host | 14:15 |
| *** dxiri has quit IRC | 14:15 | |
| *** dxiri has joined #openstack-ansible | 14:16 | |
| *** cshen has quit IRC | 14:18 | |
| mnaser | jrosser: left comments | 14:18 |
| mnaser | which i think should give ya all you need | 14:19 |
| jrosser | fab thanks | 14:19 |
| openstackgerrit | Jesse Pretorius (odyssey4me) proposed openstack/openstack-ansible-ops master: MNAIO: Enable using a data disk for file-backed VM's https://review.openstack.org/592027 | 14:26 |
| openstackgerrit | Jesse Pretorius (odyssey4me) proposed openstack/openstack-ansible-ops master: MNAIO: Enable using a data disk for file-backed VM's https://review.openstack.org/592027 | 14:32 |
| *** Bhujay has quit IRC | 14:33 | |
| openstackgerrit | Andy Smith proposed openstack/openstack-ansible master: [WIP] Add qdrouterd role for messaging backend deployment https://review.openstack.org/570261 | 14:37 |
| *** zul has quit IRC | 14:39 | |
| openstackgerrit | Markos Chandras (hwoarang) proposed openstack/openstack-ansible-plugins master: strategy: linear: Inspect the delegated host on delegated tasks. https://review.openstack.org/591672 | 14:43 |
| openstackgerrit | Jonathan Rosser proposed openstack/openstack-ansible-lxc_hosts master: Allow user-defined extra distro packages https://review.openstack.org/591412 | 14:45 |
| guilhermesp | mnaser: I don't think the missing heartbeats could be the issue of cinder.... This is happening when cinder is connected successfully with rabbitmq http://paste.openstack.org/show/728113/ | 14:46 |
| mnaser | guilhermesp: i think cinder seems to lose connection when it is trying to create the volume | 14:47 |
| *** dante-as has quit IRC | 14:52 | |
| openstackgerrit | Markos Chandras (hwoarang) proposed openstack/openstack-ansible-plugins master: strategy: linear: Inspect the delegated host on delegated tasks. https://review.openstack.org/591672 | 14:53 |
| openstackgerrit | Merged openstack/openstack-ansible-os_masakari master: Updated from OpenStack Ansible Tests https://review.openstack.org/592043 | 14:59 |
| mnaser | AAAH | 15:00 |
| mnaser | guilhermesp, odyssey4me, jrosser: i think i found the cinder issue | 15:00 |
| mnaser | http://paste.openstack.org/raw/728114/ | 15:00 |
| mnaser | look at the times inbetween | 15:00 |
| mnaser | NotifyVolumeActionTask .. sounds rpc related | 15:01 |
| mnaser | and maybe that's timing out so the volume stays stuck forever | 15:01 |
| mnaser | blocks the main thread | 15:01 |
| mnaser | that doesnt do heartbeat anymore | 15:01 |
| mnaser | and we lose it | 15:01 |
| jrosser | so something fallen through the cracks in the mq/rpc work? | 15:02 |
| mnaser | i think it might be trying to notify rpc and failing to connet or something | 15:02 |
| mnaser | and blocking the main thread | 15:02 |
| openstackgerrit | Merged openstack/openstack-ansible-os_watcher master: Updated from OpenStack Ansible Tests https://review.openstack.org/592052 | 15:04 |
| jrosser | mnaser: is it me or does the commit message here describe more than there actually is in the patch? https://github.com/openstack/openstack-ansible-os_cinder/commit/da509488d73502fd8295e5801cf9e45ffe394d74 | 15:04 |
| *** mmalchuk has quit IRC | 15:05 | |
| openstackgerrit | Merged openstack/openstack-ansible-os_congress master: Updated from OpenStack Ansible Tests https://review.openstack.org/592035 | 15:05 |
| mnaser | yup you're right jrosser .. ssl missing there | 15:05 |
| *** cshen has joined #openstack-ansible | 15:05 | |
| openstackgerrit | Merged openstack/openstack-ansible-openstack_hosts stable/rocky: Remove pip_install role execution https://review.openstack.org/591327 | 15:06 |
| openstackgerrit | Merged openstack/openstack-ansible-openstack_hosts stable/rocky: Remove pip from packages installed https://review.openstack.org/591329 | 15:06 |
| mnaser | cinder_oslomsg_notify_use_ssl doesnt seem to be modified | 15:06 |
| jrosser | mnaser: idk is thats related at all, i was just surfing back through the history for anything looking suspect | 15:06 |
| *** schwicht has joined #openstack-ansible | 15:07 | |
| mnaser | yeah it doesn't seem to be.. idk if we run ci with ssl? | 15:07 |
| mnaser | i guess i can check that | 15:07 |
| openstackgerrit | Merged openstack/openstack-ansible-os_cloudkitty master: Updated from OpenStack Ansible Tests https://review.openstack.org/592034 | 15:07 |
| jrosser | oh crap! | 15:07 |
| jrosser | https://github.com/openstack/openstack-ansible-os_cinder/commit/cd4acef4228ff849a428bb6d0d117653b53e800f#diff-2444ad0870f91f17ca6c2a5e96b26823R83 | 15:08 |
| mnaser | looks like we don't | 15:08 |
| jrosser | line 83 | 15:08 |
| jrosser | read it and weep | 15:08 |
| mnaser | jrosser: i dont follow | 15:10 |
| openstackgerrit | Merged openstack/openstack-ansible-os_heat master: Updated from OpenStack Ansible Tests https://review.openstack.org/592038 | 15:10 |
| jrosser | no neither do i :) its right | 15:10 |
| mnaser | happens :P | 15:10 |
| jrosser | it just looks wrong! | 15:10 |
| *** cshen has quit IRC | 15:10 | |
| mnaser | it only configures notify vhost if ceilometer support is enabled | 15:10 |
| mnaser | which is reasonable because we don't want to notify things that are not consumed | 15:10 |
| mnaser | ok sorry these logs seem ok | 15:12 |
| *** cshen has joined #openstack-ansible | 15:12 | |
| mnaser | it went from pending to running to success | 15:12 |
| mnaser | idk why they would put new state first and old state after | 15:13 |
| mnaser | hmm i wonder if we're running out of disk? | 15:14 |
| openstackgerrit | Merged openstack/openstack-ansible-os_barbican master: Updated from OpenStack Ansible Tests https://review.openstack.org/592031 | 15:15 |
| * mnaser cloudnull: do you know how i can download journal files and read locally? | 15:16 | |
| jrosser | mnaser: that came up yesterday, logan- asked the same thing | 15:17 |
| jrosser | it seems surprisingly non functional generally doing that | 15:17 |
| mnaser | man the kernel logs are floooded with skb_warn_bad_offload | 15:17 |
| jrosser | oh theres a patch for that | 15:18 |
| mnaser | https://review.openstack.org/#/c/591882/ and https://review.openstack.org/#/c/591880/ | 15:18 |
| jrosser | https://review.openstack.org/#/c/591880/ | 15:18 |
| jrosser | yes | 15:18 |
| *** dxiri has quit IRC | 15:19 | |
| openstackgerrit | Mohammed Naser proposed openstack/openstack-ansible master: Remove checksum-fill, its not needed any longer https://review.openstack.org/591882 | 15:19 |
| mnaser | jrosser: great, i'm +2 on both, rebased teh second to get it to pass | 15:20 |
| jrosser | i certainly see that problem locally but couldnt see it in job dmesg.gz which is why i hesitated earlier | 15:20 |
| mnaser | it was even affecting packethost runs at some point too | 15:20 |
| mnaser | jrosser: it goes in kern.log | 15:21 |
| mnaser | http://logs.openstack.org/27/591327/2/gate/openstack-ansible-functional-opensuse-423/10cffe7/job-output.txt.gz (warning, 10MB) | 15:21 |
| jrosser | that'll be why then | 15:21 |
| mnaser | err wrong link | 15:21 |
| mnaser | http://logs.openstack.org/61/591961/2/check/openstack-ansible-functional-ubuntu-xenial/f1781ce/logs/host/kern.log.txt.gz (still 10MB file) | 15:21 |
| jrosser | ok cool | 15:21 |
| jrosser | thats not going to help anything | 15:22 |
| mnaser | well at least it'll make for a tiny bit more readable logs | 15:22 |
| openstackgerrit | Jesse Pretorius (odyssey4me) proposed openstack/openstack-ansible stable/rocky: Eliminate installing pip on host/containers https://review.openstack.org/591331 | 15:23 |
| openstackgerrit | Mohammed Naser proposed openstack/openstack-ansible-os_cinder master: Drop JSON logging for Cinder https://review.openstack.org/592085 | 15:25 |
| openstackgerrit | Mohammed Naser proposed openstack/openstack-ansible-os_cinder master: Drop JSON logging for Cinder https://review.openstack.org/592085 | 15:26 |
| openstackgerrit | Merged openstack/openstack-ansible-os_glance master: Updated from OpenStack Ansible Tests https://review.openstack.org/592037 | 15:28 |
| openstackgerrit | Mohammed Naser proposed openstack/openstack-ansible-os_cinder master: Fix cinder_environment_overrides https://review.openstack.org/589934 | 15:28 |
| openstackgerrit | Mohammed Naser proposed openstack/openstack-ansible-os_cinder master: Add target_helper to backend_defaults https://review.openstack.org/584604 | 15:28 |
| openstackgerrit | Merged openstack/openstack-ansible-os_keystone master: Updated from OpenStack Ansible Tests https://review.openstack.org/592041 | 15:31 |
| openstackgerrit | Mohammed Naser proposed openstack/openstack-ansible-os_cinder master: Drop JSON logging for Cinder https://review.openstack.org/592085 | 15:31 |
| openstackgerrit | Mohammed Naser proposed openstack/openstack-ansible-os_cinder master: Fix cinder_environment_overrides https://review.openstack.org/589934 | 15:32 |
| openstackgerrit | Mohammed Naser proposed openstack/openstack-ansible-os_cinder master: Add target_helper to backend_defaults https://review.openstack.org/584604 | 15:32 |
| guilhermesp | mnaser: ++ | 15:32 |
| mnaser | ok | 15:32 |
| mnaser | sorry for gerrit spam | 15:32 |
| mnaser | but lets watch the above | 15:32 |
| *** gyee has joined #openstack-ansible | 15:32 | |
| openstackgerrit | Merged openstack/openstack-ansible-os_aodh master: Updated from OpenStack Ansible Tests https://review.openstack.org/592030 | 15:42 |
| openstackgerrit | Merged openstack/openstack-ansible-os_zun master: Updated from OpenStack Ansible Tests https://review.openstack.org/592053 | 15:44 |
| *** DanyC has quit IRC | 15:45 | |
| openstackgerrit | Merged openstack/openstack-ansible-repo_build stable/rocky: Simplify pip/setuptools/wheel prep for venvs https://review.openstack.org/591328 | 15:45 |
| *** hamzaachi_ has quit IRC | 15:46 | |
| *** FracKen has joined #openstack-ansible | 15:53 | |
| noonedeadpunk | am i right, that dynamic_inventory should generate IP for tunneled interfaces? Because as a result I recieve this for cloud nodes http://paste.openstack.org/show/728123/ And as neutron_local_ip here https://github.com/openstack/openstack-ansible-os_neutron/blob/stable/queens/templates/plugins/ml2/linuxbridge_agent.ini.j2#L17 I receive just management IP | 15:58 |
| noonedeadpunk | But probably this is already fixed and I need just to upgrade... | 15:58 |
| openstackgerrit | Jesse Pretorius (odyssey4me) proposed openstack/openstack-ansible-ops master: MNAIO: Enable using a data disk for file-backed VM's https://review.openstack.org/592027 | 15:58 |
| jrosser | noonedeadpunk: https://docs.openstack.org/project-deploy-guide/openstack-ansible/latest/targethosts-networkconfig.html | 15:59 |
| jrosser | thats what OSA assumes you have done before running the playbooks | 15:59 |
| jrosser | so if i understand your question properly, you need to have assigned an ip to br-vxlan yourself as part of the host prep | 16:01 |
| noonedeadpunk | jrosser: I have a configured IP on the node (as vxlan "maps" to IP). Moreover it worked at some point, but after upgrade from 17.0.4 to 17.0.5 it puts in config management IP instead of tunnel one. And tunnel IP is not generated for computing/neutron nodes at all | 16:01 |
| jrosser | noonedeadpunk: have you put an IP on br-vxlan? | 16:04 |
| noonedeadpunk | jrosser: actually I don't have br-vxlan, instead I've put it on ib1.87d2 interface, and set openstack_user_config like this http://paste.openstack.org/show/728124/ | 16:08 |
| jrosser | noonedeadpunk: i think that this is the code responsible for finding the ip https://github.com/openstack/openstack-ansible/blob/stable/queens/playbooks/common-tasks/dynamic-address-fact.yml | 16:09 |
| jrosser | you should be able to find where that is used | 16:09 |
| noonedeadpunk | jrosser: oh, great, thanks! That's what I was looking for) | 16:10 |
| *** vnogin has quit IRC | 16:10 | |
| openstackgerrit | Jesse Pretorius (odyssey4me) proposed openstack/openstack-ansible-ops master: MNAIO: Use virt module to shut VM's down when saving images https://review.openstack.org/592131 | 16:11 |
| jrosser | noonedeadpunk: heres a shortcut to finding where that is used http://codesearch.openstack.org/?q=dynamic-address-fact.yml&i=nope&files=&repos= | 16:12 |
| noonedeadpunk | yeh, thanks, odyssey4me has already opened codesearch for me) | 16:13 |
| jrosser | ah great | 16:14 |
| noonedeadpunk | and this tool is really great and very helpful. Can't imagine life without it now) | 16:14 |
| mnaser | cores: https://review.openstack.org/#/q/%22use+include_tasks+instead+of+include%22+is:open is good to look at, i'll go over them now | 16:14 |
| openstackgerrit | Merged openstack/openstack-ansible-os_ceilometer master: Updated from OpenStack Ansible Tests https://review.openstack.org/592032 | 16:16 |
| cloudnull | mornings / afternoons | 16:20 |
| noonedeadpunk | or even evenings) Hi) | 16:24 |
| openstackgerrit | Kevin Carter (cloudnull) proposed openstack/openstack-ansible master: Replace tempest with healthchecks https://review.openstack.org/587544 | 16:26 |
| cloudnull | o/ :) | 16:26 |
| *** Bhujay has joined #openstack-ansible | 16:33 | |
| *** DanyC has joined #openstack-ansible | 16:38 | |
| openstackgerrit | Merged openstack/openstack-ansible-os_ironic master: Updated from OpenStack Ansible Tests https://review.openstack.org/592039 | 16:39 |
| cloudnull | jrosser logan- mnaser looks like the gz things in our role logs breaks the journal. | 16:40 |
| cloudnull | I downloaded our journals from the role logs and thats all broken | 16:40 |
| jrosser | interesting | 16:40 |
| cloudnull | it looks like we run it through gzip and that breaks everything | 16:41 |
| jrosser | i have tried it too and never made it work | 16:41 |
| cloudnull | I then downloaded the osa journals which had not been gzip'd and that all worked. | 16:41 |
| cloudnull | working - http://logs.openstack.org/15/585315/2/check/openstack-ansible-deploy-aio_metal-ubuntu-xenial/dafbd30/logs/host/journal/1daf114f10c44a9e9188b7f7812db184/ | 16:41 |
| cloudnull | broken - http://logs.openstack.org/51/588951/5/check/openstack-ansible-functional-ubuntu-bionic/2e562b8/logs/host/journal/5e8702d8bcc14d9799159951ee617753/ | 16:41 |
| jrosser | do you think there is any dependance on the version of systemd? | 16:41 |
| jrosser | idk if there is any guarantee of binary compatibility | 16:42 |
| cloudnull | so we should just not gzip the logs on the role tests | 16:42 |
| odyssey4me | the journals you mean | 16:42 |
| cloudnull | IDK if there's a dependency on systemd and journalctl | 16:42 |
| cloudnull | but i am able to read the journals I downloaded on my local system which is running opensuse tumbleweed (systemd 237) | 16:43 |
| cloudnull | odyssey4me ++ sorry yes. | 16:44 |
| cloudnull | we should not gzip the journals | 16:44 |
| odyssey4me | cloudnull: perhaps we should log the old way if debug is true, and log via journal when that's disabled? | 16:45 |
| odyssey4me | that way all our tests have the old log files, but production has journal files | 16:45 |
| cloudnull | we could, though we'd then have to put the log directory paths back | 16:45 |
| odyssey4me | or perhaps there should be a toggle specifically for this rather than lean on that global | 16:45 |
| cloudnull | I added journal dumps | 16:46 |
| odyssey4me | ah yes, those log paths | 16:46 |
| cloudnull | for specific units | 16:46 |
| odyssey4me | nvm | 16:46 |
| cloudnull | like http://logs.openstack.org/51/588951/5/check/openstack-ansible-functional-ubuntu-bionic/2e562b8/logs/host/networkd-dispatcher.service.journal.log.txt.gz | 16:46 |
| cloudnull | maybe we need to add more of those ? | 16:46 |
| cloudnull | another example more relevant to openstack http://logs.openstack.org/51/588951/5/check/openstack-ansible-functional-ubuntu-bionic/2e562b8/logs/host/neutron-linuxbridge-agent.service.journal.log.txt.gz | 16:47 |
| *** hamzaachi has joined #openstack-ansible | 16:48 | |
| openstackgerrit | Kevin Carter (cloudnull) proposed openstack/openstack-ansible-tests master: When archiving logs do not compress the journals https://review.openstack.org/592153 | 16:52 |
| * jrosser goes to sit in a dark corner after reviewing include|import|tasks|playbook | 16:53 | |
| jrosser | serves me right for starting at the end of the list and working backwards | 16:54 |
| openstackgerrit | Michael Vollman proposed openstack/ansible-config_template master: Compare dict vars to determine changed https://review.openstack.org/582532 | 16:56 |
| *** openstackstatus has joined #openstack-ansible | 16:56 | |
| *** ChanServ sets mode: +v openstackstatus | 16:56 | |
| *** mmalchuk has joined #openstack-ansible | 17:04 | |
| *** gkadam has quit IRC | 17:05 | |
| *** priteau has quit IRC | 17:06 | |
| *** mmalchuk_ has joined #openstack-ansible | 17:06 | |
| *** flaviosr has quit IRC | 17:07 | |
| *** zul has joined #openstack-ansible | 17:08 | |
| *** mmalchuk has quit IRC | 17:08 | |
| mnaser | i've made some progress | 17:09 |
| mnaser | cloudnull: the reason use_json was needed was because of an encoding failure, but the root cause is | 17:09 |
| mnaser | https://github.com/openstack/taskflow/blob/master/taskflow/types/notifier.py#L215-L217 it's trying to raise an exception | 17:09 |
| mnaser | it's probably not being caught in cinder (upstream) because whatever they're doing doesnt fail | 17:09 |
| *** mbuil has quit IRC | 17:10 | |
| openstackgerrit | Merged openstack/openstack-ansible-os_magnum master: Updated from OpenStack Ansible Tests https://review.openstack.org/592042 | 17:15 |
| *** gkadam has joined #openstack-ansible | 17:20 | |
| *** dariko has joined #openstack-ansible | 17:21 | |
| mnaser | ok so it looks like | 17:23 |
| mnaser | upstream cinder doesn't even run cinder_tempest_plugin ... | 17:23 |
| *** vollman has quit IRC | 17:25 | |
| guilhermesp | I'm back to the keyboard. mnaser so to summarize, use_json patch gave us some progress? I'm seeing that all tests failed | 17:26 |
| openstackgerrit | Mohammed Naser proposed openstack/openstack-ansible-os_cinder master: Drop JSON logging for Cinder https://review.openstack.org/592085 | 17:27 |
| openstackgerrit | Mohammed Naser proposed openstack/openstack-ansible-os_cinder master: Fix cinder_environment_overrides https://review.openstack.org/589934 | 17:27 |
| openstackgerrit | Mohammed Naser proposed openstack/openstack-ansible-os_cinder master: Add target_helper to backend_defaults https://review.openstack.org/584604 | 17:27 |
| openstackgerrit | Mohammed Naser proposed openstack/openstack-ansible-os_cinder master: Run smoke tests only in testing https://review.openstack.org/592156 | 17:27 |
| mnaser | guilhermesp: ^ | 17:27 |
| *** hamzaachi has quit IRC | 17:27 | |
| mnaser | so i propose to stop using cinder_tempest_plugin because it's not even being used to test upstream, and one of it's test is the very reason why cinder fails... we're not in the business of figuring out if cinder is working or not | 17:27 |
| mnaser | smoke tests will do basic in-repo tests, matching those that are ran usptream | 17:27 |
| mnaser | which should fix the json logging issue, but i dunno if that'll affect logging or not.. we'll wait and see | 17:28 |
| odyssey4me | mnaser: yep, or perhaps we should go ahead and ditch using tempest at all for general tests and use playbooks which exxercise things we care about | 17:29 |
| mnaser | odyssey4me: i think i'd prefer calling *specific* tempest checks rather than reimplementing tempest in ansible (but this is a good point to discuss in what alex proposed re: cross deployment tool roles) | 17:30 |
| mnaser | we all need to validate our deployment tools | 17:30 |
| odyssey4me | yep | 17:30 |
| mnaser | i mean the 'basic' set of tempest checks seem to be fairly health (from following other projects like nova, etc, they rarely fail) | 17:30 |
| chandankumar | mnaser: odyssey4me where we are running tempest currently in openstack-asible jobs? | 17:31 |
| odyssey4me | chandankumar: every single role, and the integrated build | 17:31 |
| mnaser | ^ | 17:31 |
| mnaser | https://review.openstack.org/#/c/587617/ this failed on an unrelated issue | 17:32 |
| mnaser | so id like to see how that works | 17:32 |
| odyssey4me | chandankumar: as an example, this installs and runs tempest https://github.com/openstack/openstack-ansible-os_magnum/blob/master/tests/test.yml#L57 and this config informs the role what to test and what plugins to use: https://github.com/openstack/openstack-ansible-os_magnum/blob/master/tests/os_magnum-overrides.yml#L13-L21 | 17:32 |
| guilhermesp | mnaser odyssey4me it is reasonable for me to not use *_tempest_plugins at all. We just need to care if what smoke tests covers. I had similar unicode issues for example using monasca_tempest_plugin, and I waste a lot of time trying to figure out what was the root cause of the problem | 17:34 |
| mnaser | guilhermesp: i think if upstream is using it, we should use it, but if upstream isn't even running it in their ci, i don't think its worth it, we'll see the smoke test results shortly, i believe it does things like create and delete a volume etc so that should give us a fairly reasonable pov | 17:35 |
| chandankumar | odyssey4me: thanks | 17:35 |
| chandankumar | in Tripleo, in each sceanrio we are running a set of tests http://codesearch.openstack.org/?q=tempest_whitelist&i=nope&files=&repos= | 17:35 |
| chandankumar | which tests most of the stuff | 17:35 |
| cloudnull | mnaser orly... | 17:36 |
| chandankumar | I like the magnum one | 17:36 |
| mnaser | chandankumar: super interesting, we should maybe share some of this stuff in terms of "this is some of the tests that we never had issues with" | 17:37 |
| chandankumar | https://github.com/openstack/tripleo-quickstart-extras/tree/master/roles/validate-tempest/vars it is the master skip list | 17:38 |
| mnaser | cloudnull, guilhermesp, odyssey4me ^ this is such an awesome list that is very valuable | 17:38 |
| mnaser | afaik tripleo CI is relatively stable | 17:39 |
| guilhermesp | yep, I was just looking now | 17:39 |
| guilhermesp | sounds a good approach | 17:39 |
| cloudnull | +1 | 17:41 |
| cloudnull | thats nice! | 17:41 |
| odyssey4me | yeah, if more eyes are on the same problems then it'll make life easier for everyone | 17:43 |
| cloudnull | anyone want to give https://review.openstack.org/#/q/starredby:cloudnull+status:open,n,z a review ? | 17:46 |
| spotz | cloudnull: checking | 17:49 |
| cloudnull | tyvm | 17:50 |
| *** gyee has quit IRC | 17:54 | |
| *** poopcat has joined #openstack-ansible | 18:00 | |
| *** vnogin has joined #openstack-ansible | 18:05 | |
| *** hamzaachi has joined #openstack-ansible | 18:07 | |
| *** vnogin has quit IRC | 18:09 | |
| jrosser | a few things seem to be failing on opensuse423 repo timeouts | 18:12 |
| openstackgerrit | Kevin Carter (cloudnull) proposed openstack/openstack-ansible-ops master: Remove anti-bind mount config https://review.openstack.org/592169 | 18:12 |
| cloudnull | ive also seen repeated failures w/ the lxc image on suse. | 18:17 |
| cloudnull | its the only os that is still using the lxc images | 18:18 |
| mnaser | cloudnull: how are the others deployed then? docker images? | 18:18 |
| mnaser | or $container_du_jour ? | 18:18 |
| *** zul has quit IRC | 18:18 | |
| cloudnull | cent is using the docker base image, ubuntu pulls the rootfs from ubuntu directly | 18:18 |
| *** gkadam has quit IRC | 18:19 | |
| cloudnull | suse has a rootfs that we could download from their openstack repos, but its a considerably large image | 18:19 |
| jrosser | did the ubuntu-minimal images ever get tried? | 18:21 |
| cloudnull | cent https://github.com/openstack/openstack-ansible-lxc_hosts/blob/master/vars/redhat-7.yml#L16, ubuntu https://github.com/openstack/openstack-ansible-lxc_hosts/blob/master/vars/ubuntu-16.04.yml#L16 | 18:21 |
| jrosser | ah, i remeber, i couldnt find a tgz for those iirc | 18:21 |
| cloudnull | jrosser on ubuntu the container images are SUPER minimal | 18:22 |
| openstackgerrit | Antony Messerli proposed openstack/openstack-ansible-ops master: Allow for Swift Leap Upgrade to be skipped https://review.openstack.org/591053 | 18:22 |
| odyssey4me | I keep meaning to change all that to prep images using DIB. I've got a hack of that working, but haven't really had time to put together a patch. | 18:22 |
| odyssey4me | Relying on an external source for images has, time and again, caused us problems. | 18:23 |
| *** electrofelix has quit IRC | 18:25 | |
| jrosser | odyssey4me: that would be great | 18:26 |
| openstackgerrit | Jesse Pretorius (odyssey4me) proposed openstack/openstack-ansible-ops master: MNAIO: Extend image saving to include manifest https://review.openstack.org/592171 | 18:27 |
| jrosser | it would build from the mirror i already have | 18:27 |
| odyssey4me | jrosser: yep, that's exactly it - build from the local mirror, works for online and offline installs | 18:27 |
| cloudnull | ++ | 18:29 |
| jrosser | odyssey4me: i could put some time on that if you want a hand | 18:29 |
| odyssey4me | jrosser: well, I could do one distro if you could help fill out the rest | 18:30 |
| odyssey4me | or perhaps we just do one per patch anyway | 18:30 |
| jrosser | sure | 18:31 |
| cloudnull | nvmd looks like the option lxc_hosts_container_image_download_legacy was removed for suse in lxc_hosts vars/suse-42.yml | 18:31 |
| cloudnull | so it's pulling from github | 18:31 |
| odyssey4me | the thing I kinda got stuck on was where to do it - I guess the simplest for now is to do it inside the lxc_hosts/nspawn_hosts roles | 18:31 |
| odyssey4me | perhaps use a common tasks file which is synchronised from the tests repo like the mq_setup tasks | 18:32 |
| odyssey4me | or perhaps use a role for it | 18:32 |
| cloudnull | ah nvmd again | 18:32 |
| cloudnull | looks like the github repo was changed to https://download.opensuse.org/repositories/Virtualization:/containers:/images:/ | 18:32 |
| cloudnull | so jrosser that would mirror the repo issue you had mentioned before | 18:32 |
| jrosser | right | 18:33 |
| odyssey4me | ok, I'll get moving on that on friday - I should be able to dedicate some time to it then | 18:33 |
| *** dariko has quit IRC | 18:34 | |
| openstackgerrit | Mohammed Naser proposed openstack/openstack-ansible-openstack_hosts master: Use variable with full URL for RDO repositories https://review.openstack.org/592172 | 18:34 |
| *** fghaas has joined #openstack-ansible | 18:35 | |
| noonedeadpunk | guys, what do you think about https://review.openstack.org/#/c/583943/ ? | 18:36 |
| *** priteau has joined #openstack-ansible | 18:36 | |
| openstackgerrit | Markos Chandras (hwoarang) proposed openstack/openstack-ansible-plugins master: DNM - Testing connection plugins https://review.openstack.org/577378 | 18:36 |
| openstackgerrit | Jesse Pretorius (odyssey4me) proposed openstack/openstack-ansible-ops master: MNAIO: Install jmespath https://review.openstack.org/592173 | 18:41 |
| openstackgerrit | Mohammed Naser proposed openstack/openstack-ansible-tests master: Use latest hashed RDO repository https://review.openstack.org/592174 | 18:42 |
| jrosser | is there any implication going from include: to import_playbook: ? | 18:43 |
| * jrosser nervous about that slew of reviews | 18:43 | |
| odyssey4me | jrosser: to include the playbooks, I don't think so | 18:45 |
| odyssey4me | however for the include -> {include,import}_tasks has broader implications | 18:46 |
| openstackgerrit | Kevin Carter (cloudnull) proposed openstack/openstack-ansible-nspawn_hosts master: Add SUSE 15 to the nspawn hosts role https://review.openstack.org/592177 | 18:46 |
| cloudnull | jrosser idk of any issues with moving to that | 18:46 |
| odyssey4me | I'm out for the night - cheers folks. | 18:48 |
| jrosser | if anyone fancies another look over the main repo review for that i'd appreciate it | 18:49 |
| jrosser | it was just arbitrary search/replace sadly | 18:50 |
| jrosser | theres some oddities too like the included thing being an include followed by a play | 18:51 |
| *** fghaas has quit IRC | 18:53 | |
| openstackgerrit | Cam J. Loader proposed openstack/openstack-ansible-ops master: Download get-pip.py for MNAIO https://review.openstack.org/592179 | 18:56 |
| *** aicilamd has joined #openstack-ansible | 18:57 | |
| openstackgerrit | Kevin Carter (cloudnull) proposed openstack/openstack-ansible-nspawn_hosts master: Add SUSE 15 to the nspawn hosts role https://review.openstack.org/592177 | 19:01 |
| *** aicilamd has quit IRC | 19:02 | |
| cloudnull | jrosser is there a review to look at ? | 19:04 |
| openstackgerrit | Mohammed Naser proposed openstack/openstack-ansible-tests master: Use latest hashed RDO repository https://review.openstack.org/592174 | 19:05 |
| *** hamzaachi has quit IRC | 19:05 | |
| *** DanyC has quit IRC | 19:06 | |
| *** hamzaachi has joined #openstack-ansible | 19:06 | |
| openstackgerrit | Merged openstack/openstack-ansible-ops master: Allow for Swift Leap Upgrade to be skipped https://review.openstack.org/591053 | 19:12 |
| *** aicilamd has joined #openstack-ansible | 19:15 | |
| *** openstackgerrit has quit IRC | 19:19 | |
| jrosser | cloudnull: https://review.openstack.org/#/c/585315/ | 19:19 |
| *** aicilamd has quit IRC | 19:20 | |
| jrosser | cloudnull: having thought about it more it may be preferable to split that review up. It touches everything at once. | 19:20 |
| *** openstackgerrit has joined #openstack-ansible | 19:27 | |
| openstackgerrit | Mohammed Naser proposed openstack/openstack-ansible-tests master: Use latest hashed RDO repository https://review.openstack.org/592174 | 19:27 |
| *** DanyC has joined #openstack-ansible | 19:30 | |
| *** hamzy has quit IRC | 19:43 | |
| *** hamzy has joined #openstack-ansible | 19:46 | |
| *** dariko has joined #openstack-ansible | 19:54 | |
| openstackgerrit | Kevin Carter (cloudnull) proposed openstack/openstack-ansible master: Removed conditional in test setup https://review.openstack.org/575748 | 19:57 |
| openstackgerrit | Merged openstack/openstack-ansible-ops master: Remove anti-bind mount config https://review.openstack.org/592169 | 20:04 |
| *** priteau has quit IRC | 20:05 | |
| *** aicilamd has joined #openstack-ansible | 20:16 | |
| *** aicilamd has quit IRC | 20:21 | |
| openstackgerrit | Markos Chandras (hwoarang) proposed openstack/openstack-ansible-plugins master: strategy: linear: Inspect the delegated host on delegated tasks. https://review.openstack.org/591672 | 20:27 |
| openstackgerrit | Kevin Carter (cloudnull) proposed openstack/openstack-ansible-nspawn_hosts master: Add SUSE 15 to the nspawn hosts role https://review.openstack.org/592177 | 20:28 |
| *** dariko has quit IRC | 20:28 | |
| *** hamzaachi has quit IRC | 20:28 | |
| *** hamzaachi has joined #openstack-ansible | 20:42 | |
| *** fghaas has joined #openstack-ansible | 20:45 | |
| *** hamzaachi has quit IRC | 20:47 | |
| *** hamzaachi has joined #openstack-ansible | 20:48 | |
| *** DanyC has quit IRC | 20:50 | |
| *** aicilamd has joined #openstack-ansible | 20:50 | |
| *** fghaas has quit IRC | 20:51 | |
| openstackgerrit | Andy Smith proposed openstack/openstack-ansible-os_glance master: Add extra distro packages for optional oslo.messaging amqp1 driver https://review.openstack.org/592195 | 20:52 |
| *** hamzaachi has quit IRC | 20:52 | |
| *** schwicht has quit IRC | 20:54 | |
| mnaser | cores: in order to resolve all of the recent problems about 'missing' packages.. can we land https://review.openstack.org/#/c/592172/ and https://review.openstack.org/#/c/592172/ for centos? | 20:55 |
| mnaser | err https://review.openstack.org/#/c/592174/3 and https://review.openstack.org/#/c/592172/ | 20:55 |
| mnaser | ignore the centos failure in openstack_hosts, it's the very failure we're trying to fix in the ansible-tests repo change, chicken and egg type of thing | 20:56 |
| mnaser | you can see centos-7 passed fine afterwards in the second change | 20:56 |
| mnaser | spotz: jrosser cloudnull if y'all are still around ^ | 20:57 |
| * jrosser looks | 20:57 | |
| spotz | mnaser: looking | 21:01 |
| prometheanfire | requirements branched, so we can merge the reqs tox update stuff | 21:03 |
| * cloudnull looks | 21:07 | |
| *** aicilamd has quit IRC | 21:07 | |
| *** aicilamd has joined #openstack-ansible | 21:08 | |
| *** aicilamd has quit IRC | 21:09 | |
| *** aicilamd has joined #openstack-ansible | 21:09 | |
| * jrosser learns more bash parameter expansion stuff | 21:09 | |
| *** vnogin has joined #openstack-ansible | 21:11 | |
| *** DanyC has joined #openstack-ansible | 21:12 | |
| *** aicilamd has quit IRC | 21:14 | |
| *** vnogin has quit IRC | 21:15 | |
| *** hamzaachi has joined #openstack-ansible | 21:17 | |
| jrosser | mnaser: do you want to push both through now or wait for the tests repo one to merge? | 21:20 |
| mnaser | jrosser: they're dependent so the test one wont merge till the openstack_hosts one does :p | 21:21 |
| jrosser | when you put it like that :) | 21:21 |
| *** throwsb1 has quit IRC | 21:22 | |
| openstackgerrit | Andy Smith proposed openstack/openstack-ansible-os_aodh master: Add install packages for optional oslo.messaging amqp1 driver https://review.openstack.org/592201 | 21:24 |
| openstackgerrit | Andy Smith proposed openstack/openstack-ansible-os_aodh master: Add install packages for optional oslo.messaging amqp1 driver https://review.openstack.org/592201 | 21:25 |
| *** hamzaachi has quit IRC | 21:28 | |
| openstackgerrit | Merged openstack/openstack-ansible-openstack_hosts master: Use variable with full URL for RDO repositories https://review.openstack.org/592172 | 21:30 |
| openstackgerrit | Mohammed Naser proposed openstack/openstack-ansible-tests master: Revert "Make CentOS tests non-voting" https://review.openstack.org/592207 | 21:42 |
| openstackgerrit | Mohammed Naser proposed openstack/openstack-ansible-tests master: Revert "Make CentOS tests non-voting" https://review.openstack.org/592207 | 21:42 |
| *** KeithMnemonic has quit IRC | 21:44 | |
| openstackgerrit | Mohammed Naser proposed openstack/openstack-ansible-os_cinder master: Run smoke tests only in testing https://review.openstack.org/592156 | 21:44 |
| *** FracKen has left #openstack-ansible | 22:17 | |
| *** cshen has quit IRC | 22:20 | |
| openstackgerrit | Merged openstack/openstack-ansible-tests master: Use latest hashed RDO repository https://review.openstack.org/592174 | 22:23 |
| *** DanyC has quit IRC | 22:25 | |
| *** schwicht has joined #openstack-ansible | 22:26 | |
| *** schwicht has quit IRC | 22:33 | |
| *** vnogin has joined #openstack-ansible | 22:56 | |
| *** schwicht has joined #openstack-ansible | 23:08 | |
| *** schwicht has quit IRC | 23:40 | |
| *** vnogin has quit IRC | 23:44 | |
| *** vnogin has joined #openstack-ansible | 23:45 | |
| *** schwicht has joined #openstack-ansible | 23:49 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!