| *** macza has quit IRC | 00:09 | |
| *** ThiagoCMC has quit IRC | 00:37 | |
| *** tosky has quit IRC | 00:40 | |
| *** rodolof has joined #openstack-ansible | 00:46 | |
| *** cshen has joined #openstack-ansible | 00:55 | |
| *** cshen has quit IRC | 01:01 | |
| *** gyee has quit IRC | 01:09 | |
| *** strattao has joined #openstack-ansible | 01:20 | |
| *** strattao has quit IRC | 01:24 | |
| *** dave-mccowan has quit IRC | 02:00 | |
| *** weezS has quit IRC | 02:17 | |
| *** cshen has joined #openstack-ansible | 02:20 | |
| *** cshen has quit IRC | 02:25 | |
| *** ThiagoCMC has joined #openstack-ansible | 02:34 | |
| *** dave-mccowan has joined #openstack-ansible | 03:11 | |
| *** CeeMac has quit IRC | 03:16 | |
| *** dave-mccowan has quit IRC | 03:19 | |
| *** cshen has joined #openstack-ansible | 04:21 | |
| *** lbragstad has quit IRC | 04:23 | |
| *** cshen has quit IRC | 04:26 | |
| *** weezS has joined #openstack-ansible | 04:53 | |
| *** rodolof has quit IRC | 05:10 | |
| *** weezS has quit IRC | 05:22 | |
| *** rodolof has joined #openstack-ansible | 05:25 | |
| *** rodolof has quit IRC | 05:31 | |
| *** vakuznet has quit IRC | 05:50 | |
| *** cshen has joined #openstack-ansible | 05:53 | |
| *** cshen has quit IRC | 05:59 | |
| *** cshen has joined #openstack-ansible | 06:02 | |
| *** hamzaachi has joined #openstack-ansible | 06:04 | |
| *** cshen has quit IRC | 06:07 | |
| *** cshen has joined #openstack-ansible | 06:36 | |
| *** cshen has quit IRC | 06:41 | |
| *** mma has joined #openstack-ansible | 06:53 | |
| *** hwoarang has quit IRC | 06:54 | |
| *** hwoarang has joined #openstack-ansible | 06:56 | |
| *** arbrandes has joined #openstack-ansible | 07:01 | |
| *** arbrandes1 has quit IRC | 07:03 | |
| openstackgerrit | Chandan Kumar proposed openstack/openstack-ansible-os_tempest master: Added support for running tempestconf from rpm https://review.openstack.org/622999 | 07:09 |
|---|---|---|
| *** cshen has joined #openstack-ansible | 07:14 | |
| *** cshen_ has joined #openstack-ansible | 07:21 | |
| *** cshen has quit IRC | 07:22 | |
| *** trident has quit IRC | 07:23 | |
| *** orderchaos has quit IRC | 07:23 | |
| *** orderchaos has joined #openstack-ansible | 07:25 | |
| *** trident has joined #openstack-ansible | 07:25 | |
| *** mma has quit IRC | 07:32 | |
| *** mma has joined #openstack-ansible | 07:35 | |
| *** mma has quit IRC | 07:36 | |
| *** mma has joined #openstack-ansible | 07:40 | |
| openstackgerrit | Chandan Kumar proposed openstack/openstack-ansible-os_tempest master: Use --profile feature for generating tempest.conf https://review.openstack.org/623413 | 07:45 |
| *** mma has quit IRC | 07:45 | |
| *** aedc has joined #openstack-ansible | 07:53 | |
| *** mma has joined #openstack-ansible | 08:02 | |
| *** mma has quit IRC | 08:03 | |
| *** mma has joined #openstack-ansible | 08:03 | |
| *** ppetit has joined #openstack-ansible | 08:04 | |
| *** ppetit has quit IRC | 08:06 | |
| *** ppetit has joined #openstack-ansible | 08:06 | |
| *** hamzaachi has quit IRC | 08:09 | |
| *** mma has quit IRC | 08:12 | |
| *** aedc has quit IRC | 08:28 | |
| *** aedc has joined #openstack-ansible | 08:31 | |
| chandan_kumar | odyssey4me: jrosser Hello | 08:36 |
| chandan_kumar | odyssey4me: jrosser please have a look at this one https://review.openstack.org/#/c/622999/ thanks :-) | 08:37 |
| *** aedc has quit IRC | 08:42 | |
| *** ppetit has quit IRC | 08:44 | |
| *** aedc has joined #openstack-ansible | 08:45 | |
| *** hamzaachi has joined #openstack-ansible | 08:51 | |
| *** shardy has joined #openstack-ansible | 08:52 | |
| *** KeithMnemonic has quit IRC | 09:00 | |
| *** hamzaachi has quit IRC | 09:01 | |
| *** KeithMnemonic has joined #openstack-ansible | 09:01 | |
| openstackgerrit | Frank Kloeker proposed openstack/openstack-ansible master: Change docs build to openstackdocstheme logic https://review.openstack.org/623429 | 09:17 |
| *** mma has joined #openstack-ansible | 09:18 | |
| *** DanyC has joined #openstack-ansible | 09:20 | |
| *** DanyC has quit IRC | 09:24 | |
| *** aedc has quit IRC | 09:32 | |
| *** DanyC has joined #openstack-ansible | 10:01 | |
| *** electrofelix has joined #openstack-ansible | 10:02 | |
| odyssey4me | jrosser so I was thinking this morning that doing the graphing on the check queue probably isn't best, because that will include bad patches - better to do it all o the gate queue because it will have passed gate at least once | 10:05 |
| *** aedc has joined #openstack-ansible | 10:10 | |
| odyssey4me | ThiagoCMC If you're sharing LDAP, then you don't need a single Horizon. But if you want a single Horizon, then have one region with horizon and keystone, register the keystone endpoints for the other regions in that keystone service catalog, and enable horizon multi-domain support: https://github.com/openstack/openstack-ansible-os_horizon/blob/master/templates/horizon_local_settings.py.j2#L77-L79 | 10:12 |
| odyssey4me | I think that'll do it. | 10:12 |
| jrosser | odyssey4me: yes there is certainly more to do on grafana- I put up another patch yesterday to separate out fail/success/timeout | 10:14 |
| jrosser | Adding gate graphs too would be great, think both are valid as you end up doing check/gate/check/gate anyway when things aren’t going through easily | 10:16 |
| *** trident has quit IRC | 10:18 | |
| chandan_kumar | odyssey4me: jrosser does tempest logs and stestr html files are indexed in logstash.openstack.org and visible in openstack-health? | 10:19 |
| odyssey4me | chandan_kumar no, only the console log is indexed as far as I know - although all the subunits are added into openstack-health, yes | 10:19 |
| odyssey4me | I think it's only for the integrated build though | 10:20 |
| *** trident has joined #openstack-ansible | 10:21 | |
| jrosser | odyssey4me: I also wanted to pull out the overall Zuul +/- vote status and chart that, which is actually the true pass fail rate from an end user POV | 10:24 |
| *** vnogin has joined #openstack-ansible | 10:25 | |
| chandan_kumar | odyssey4me: oh, we will fix that to get it indexed in logstash which in future will be helpful to generate elastic-recheck query from logstash. | 10:32 |
| chandan_kumar | odyssey4me: http://logs.openstack.org/13/623413/1/check/openstack-ansible-functional-tempestconf-centos-7/7712e47/job-output.txt.gz#_2018-12-07_09_22_09_731838 these timesout are related to centos 7.6 updrade? | 10:34 |
| odyssey4me | chandan_kumar we think so - but are not sure, they seem to have increased since the update | 10:36 |
| *** vnogin has quit IRC | 10:43 | |
| *** trident has quit IRC | 11:04 | |
| *** trident has joined #openstack-ansible | 11:06 | |
| odyssey4me | jrosser chandan_kumar mnaser so it looks like centos, after the new updates, is timing out - even for a non-container build | 11:25 |
| jrosser | Yes I saw 20mins for one step of the log collection on a timed out job that would otherwise have passed | 11:26 |
| odyssey4me | http://logs.openstack.org/73/622473/1/check/openstack-ansible-deploy-aio_metal-centos-7/b90d6ab/logs/ara-report/ shows 1:31 for setup-infrastructure, which is very long indeed - bionic takes ~20 mins | 11:27 |
| odyssey4me | check the duration here: http://logs.openstack.org/73/622473/1/check/openstack-ansible-deploy-aio_metal-centos-7/b90d6ab/logs/ara-report/result/68a3b4ce-a767-44ee-9798-d69042fc6617/ | 11:27 |
| odyssey4me | oh you can't see it - 45 mins to do the git clones | 11:28 |
| *** hamzaachi has joined #openstack-ansible | 11:28 | |
| *** aedc has quit IRC | 11:29 | |
| jrosser | Sounds like I/O sadness - would stack up with long time collecting logs too | 11:42 |
| jrosser | Maybe a quick patch to run bonnie++ right at the start of the job? | 11:44 |
| odyssey4me | hmm, it makes me wonder whether https://github.com/openstack/openstack-ansible/blob/master/tests/roles/bootstrap-host/templates/user_variables.aio.yml.j2#L199-L203 is taking effect or whether something is interfering with it | 11:44 |
| odyssey4me | but why would the IO be so different now, and only on one platform | 11:45 |
| jrosser | There’s been chat about timeouts for the last couple of days in #infra too | 11:48 |
| *** ansmith has joined #openstack-ansible | 11:48 | |
| *** tosky has joined #openstack-ansible | 11:52 | |
| noonedeadpunk | Morning everyone. | 11:52 |
| *** hamzaachi has quit IRC | 11:56 | |
| jamesdenton | ThiagoCMC Re: Single Horizon, Multiple Clouds - You might take a look at adding AVAILABLE_REGIONS config to Horizon: https://docs.openstack.org/horizon/rocky/admin/customize-configure.html | 12:14 |
| odyssey4me | jamesdenton ah yes, that's the thing I was thinking of | 12:15 |
| jamesdenton | It looks like it would do the trick | 12:15 |
| *** aedc has joined #openstack-ansible | 12:30 | |
| *** trident has quit IRC | 12:39 | |
| *** trident has joined #openstack-ansible | 12:41 | |
| *** hamzaachi has joined #openstack-ansible | 12:47 | |
| *** ahosam has joined #openstack-ansible | 12:49 | |
| *** cshen_ has quit IRC | 12:51 | |
| *** cshen has joined #openstack-ansible | 12:51 | |
| *** chandan_kumar is now known as chkumar|off | 13:12 | |
| mnaser | So it looks like only bare metal issues are still there on CentOS | 13:16 |
| mnaser | One of the OVH regions ya been really slow | 13:17 |
| mnaser | So is these jobs running on OVH possibly? | 13:17 |
| odyssey4me | mnaser well, no - vare metal or not - we're getting timeouts | 13:17 |
| odyssey4me | eg: https://review.openstack.org/#/c/622473/ | 13:17 |
| mnaser | Oh interesting those are rax too | 13:17 |
| mnaser | I saw something psss a few minutes ago in my email | 13:18 |
| odyssey4me | and the slow downs are not on a consistent task | 13:18 |
| mnaser | Wow it looks like it just slowed down so much | 13:19 |
| mnaser | On trivial things like systemd_service taking 2 minutes per | 13:19 |
| guilhermesp | morning! Should I recheck my pr? THere was post failure and timeout https://review.openstack.org/#/c/618820/48 | 13:20 |
| odyssey4me | earlier I saw one which took 45 mins to do the git clones in the repo build | 13:20 |
| mnaser | I wonder if it’s something related to DNS resolutions | 13:20 |
| mnaser | That’s the usual taking a long time behaviour | 13:20 |
| mnaser | guilhermesp: it’s broken right now, we’ll we don’t know if we have a 100% fix | 13:21 |
| guilhermesp | but is it broken in our side or infra side? | 13:21 |
| mnaser | I thought it was us, then I thought it was infra and now I think it might be us | 13:22 |
| mnaser | Regardless we need to figure it out | 13:22 |
| odyssey4me | mnaser for all metal builds, we don't touch any dns config | 13:22 |
| odyssey4me | so that takes all the containers out of the equation | 13:23 |
| odyssey4me | perhaps focus troubleshooting there, given that it removes a ton of extra things to think about | 13:23 |
| mnaser | odyssey4me: infra sets up unbound on boot so I thought maybe it wasn’t actually being started properly | 13:23 |
| mnaser | Tho after further digging it seems that the process is up and running | 13:23 |
| mnaser | Maybe I’ll ask infra to put a hold on a failing nose | 13:24 |
| mnaser | Nods | 13:24 |
| odyssey4me | sure, maybe - I'm just suggesting that given that the metal builds are failing and they have the least moving parts - look there | 13:24 |
| openstackgerrit | Justin Alford proposed openstack/openstack-ansible-os_cinder master: cinder-volume: rbd driver set backend_host value https://review.openstack.org/623490 | 13:36 |
| *** eumel8 has joined #openstack-ansible | 13:42 | |
| *** cshen has quit IRC | 14:04 | |
| *** lbragstad has joined #openstack-ansible | 14:06 | |
| mnaser | ok | 14:11 |
| mnaser | in the office now | 14:11 |
| mnaser | i can start digging | 14:11 |
| *** aedc has quit IRC | 14:19 | |
| *** lbragstad has quit IRC | 14:25 | |
| *** lbragstad has joined #openstack-ansible | 14:29 | |
| *** aedc has joined #openstack-ansible | 14:30 | |
| *** weezS has joined #openstack-ansible | 14:36 | |
| mnaser | finding this on a held job | 14:39 |
| mnaser | Dec 07 14:21:25 aio1 sshd[16959]: pam_systemd(sshd:session): Failed to create session: Failed to activate service 'org.freedesktop.login1': timed out | 14:39 |
| *** weezS has quit IRC | 14:39 | |
| odyssey4me | IIRC we do some stuff in the lxc_hosts cache prep which involves setting that up | 14:40 |
| *** weezS has joined #openstack-ansible | 14:40 | |
| odyssey4me | oh, but that won't be there in a metal build I don't think | 14:40 |
| mnaser | odyssey4me: im looking at this server logs | 14:40 |
| mnaser | and these errors start at 12:15 | 14:41 |
| *** strattao has joined #openstack-ansible | 14:42 | |
| ThiagoCMC | odyssey4me, thanks! I'll try that... :-D | 14:43 |
| * mnaser is tempted to pull mhayden in here if he's around and has a clue | 14:45 | |
| mnaser | cloudnull our systemd expert is not around :( | 14:45 |
| * mhayden clues in | 14:45 | |
| jamesdenton | look what the cat dragged in! | 14:46 |
| mnaser | ok | 14:48 |
| mnaser | https://github.com/openstack/openstack-ansible-lxc_hosts/blob/6eee41f123dd49d73ad2851b878c11efd6cfffa2/tasks/lxc_cache_preparation_systemd_old.yml | 14:48 |
| mnaser | i think that's what's messing it up | 14:48 |
| dmsimard | mnaser: if you look at the whole journalctl logs around "Dec 07 12:26:04" | 14:49 |
| dmsimard | that's where the issues start | 14:49 |
| *** cshen has joined #openstack-ansible | 14:49 | |
| mnaser | yeah im looking this file we drop in here https://github.com/openstack/openstack-ansible-lxc_hosts/blob/6eee41f123dd49d73ad2851b878c11efd6cfffa2/tasks/lxc_cache_preparation_systemd_old.yml | 14:50 |
| dmsimard | http://paste.openstack.org/raw/736824/ | 14:50 |
| dmsimard | oh in /etc/d-bus ? | 14:50 |
| dmsimard | yeah that looks very likely | 14:51 |
| odyssey4me | mnaser that's can't be the only thing though, because none of that runs on a machine with no containers | 14:52 |
| mnaser | odyssey4me: might be more than one issue? | 14:52 |
| odyssey4me | must be, because while that error may be happening on containered builds, the non-containered builds are also failing | 14:53 |
| mnaser | odyssey4me: yeah right now i just have a hold on a containerized deploy | 14:53 |
| mnaser | that im working on | 14:53 |
| openstackgerrit | Martin Kopec proposed openstack/openstack-ansible-os_tempest master: Create basic doc structure https://review.openstack.org/623512 | 14:55 |
| dmsimard | mnaser: I'm looking at a few running centos-based jobs outside of OSA | 14:55 |
| dmsimard | and grepping logs, not seeing the "Failed to activate service 'org.freedesktop.login1': timed out" | 14:56 |
| *** strattao has quit IRC | 14:57 | |
| mnaser | so this is the patch we're "carrying" and it seems like dbus dun broke after we restart adding this file https://cgit.freedesktop.org/systemd/systemd/commit/src/machine/org.freedesktop.machine1.conf?id=72c3897f77a7352618ea76b880a6764f52d6327b | 14:57 |
| mnaser | if i understand correctly, this really only allows non-privileged users to do machinectl operations | 14:58 |
| mnaser | but in our deploys and whatnot, we kinda rely on all of our work happening as privileged users anyways | 14:58 |
| mnaser | specificall this task | 14:59 |
| mnaser | https://github.com/openstack/openstack-ansible-lxc_hosts/blob/6eee41f123dd49d73ad2851b878c11efd6cfffa2/tasks/lxc_cache_preparation_systemd_old.yml#L28-L38 | 14:59 |
| odyssey4me | perhaps the new version has this fixed or something | 15:00 |
| mnaser | https://lists.freedesktop.org/archives/systemd-devel/2015-March/028922.html | 15:01 |
| mnaser | interesting | 15:01 |
| dmsimard | mnaser: ssh connections are quick again after restarting systemd-logind | 15:03 |
| ThiagoCMC | jamesdenton, oh, I see! Thank you! | 15:03 |
| mnaser | yeah it probably resolved that issue dmsimard | 15:03 |
| mnaser | let me remove all those workarounds and see if it builds fine. | 15:03 |
| dmsimard | mnaser: ok, I'll continue investigating | 15:03 |
| dmsimard | https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=770135 correlates the restart of dbus to the failure of systemd-logind and confirms that restarting systemd-logind fixes the issue | 15:07 |
| openstack | Debian bug 770135 in systemd "systemd: ssh logins considerably delayed (until PAM timeout) when systemd is upgraded but the system not rebooted" [Normal,Open] | 15:07 |
| openstackgerrit | Martin Kopec proposed openstack/openstack-ansible-os_tempest master: Create basic doc structure https://review.openstack.org/623512 | 15:11 |
| mnaser | dmsimard: yeah i rather not end up working around it by restarting more things, that code seems largely unused and unclean | 15:12 |
| openstackgerrit | Mohammed Naser proposed openstack/openstack-ansible-lxc_hosts master: prep: remove old machinectl workarounds https://review.openstack.org/623515 | 15:14 |
| dmsimard | mnaser: not suggesting to implement a restart, just collecting info about the bug | 15:14 |
| mnaser | dmsimard: ah yes :) i think the above should hopefully bring us in the right step.. i hope. | 15:14 |
| dmsimard | there's some bugzilla on the issue but nothing really came out of them | 15:15 |
| dmsimard | i.e, https://bugzilla.redhat.com/show_bug.cgi?id=1623651 and https://bugzilla.redhat.com/show_bug.cgi?id=1258557 | 15:15 |
| openstack | bugzilla.redhat.com bug 1623651 in systemd "Restarting dbus leaves systemd-logind in broken state" [Unspecified,New] - Assigned to systemd-maint | 15:15 |
| openstack | dmsimard: Error: Error getting bugzilla.redhat.com bug #1258557: NotPermitted | 15:15 |
| dmsimard | consensus appears to be "you shouldn't be restarting dbus" | 15:16 |
| dmsimard | ¯\_(ツ)_/¯ | 15:16 |
| mnaser | yeah maybe just reload it if this doesn't fix it | 15:18 |
| mnaser | so i guess we were doing something wrong and it never bit us until now | 15:18 |
| odyssey4me | mnaser well, bear in mind for containers we're downloading a docker image - adding systemd and all that, then booting a new container... so we don't restart anything, we boot fresh | 15:20 |
| mnaser | odyssey4me: right but with the code that was removed it seems to restart it on the host in order to add some machinectl features which dont seem to be used | 15:20 |
| mnaser | afaik? | 15:20 |
| odyssey4me | oh that's nice | 15:20 |
| mnaser | odyssey4me: oh boy | 15:25 |
| mnaser | we're running lxc_hosts in metal jobs. | 15:25 |
| *** trident has quit IRC | 15:25 | |
| mnaser | and so that's why it also broke, because the same code ran | 15:25 |
| mnaser | odyssey4me: on an aio_metal job, i'm seeing a 'aio1_neutron_ovn_northd_container-610a7fdf' container?! | 15:26 |
| *** trident has joined #openstack-ansible | 15:27 | |
| mnaser | jamesdenton: is neutron_ovn_northd technically a neutron agent? | 15:28 |
| jamesdenton | well, not quite | 15:28 |
| odyssey4me | mnaser hmm, that's a bug - thanks jamesdenton : | 15:29 |
| mnaser | because right now we have `neutron_agents_container`, `neutron_ovn_northd_container` and `neutron_server_container` | 15:29 |
| jamesdenton | and yes, it's there because the inventory was borked (by me), but nothing is deployed to it | 15:29 |
| odyssey4me | :p | 15:29 |
| mnaser | yeah no worries | 15:29 |
| mnaser | we approved it ;) | 15:29 |
| mnaser | im just wondering if we should add it to the agents (but if we do, itll run on metal from now on) | 15:29 |
| odyssey4me | mnaser ok, and because of that bug lxc_hosts is actually doing something - normally the host group is empty so it moves on without doing anything | 15:29 |
| mnaser | does it touch the data plane? | 15:29 |
| jamesdenton | it does not touch the data plane | 15:29 |
| jamesdenton | it runs ovsdb-server and some other ovn-related services | 15:30 |
| odyssey4me | ok, so we need to add whatever fix to https://review.openstack.org/#/c/622473/ as well | 15:30 |
| mnaser | yep that might be good, either shuffle it around in the inventory .. or add neutron_ovn_northd_container is_metal in etc/openstack_deploy/env.d/aio_metal.yml.example | 15:31 |
| odyssey4me | jamesdenton perhaps, as you suggested, we change to have those groups added by user config for now - until you find a nicer option? | 15:31 |
| jamesdenton | odyssey4me yes - i was just about to suggest that | 15:31 |
| odyssey4me | oh of course, it's missing in the etc/openstack_deploy/env.d/aio_metal.yml.example file! | 15:31 |
| cshen | d/quit | 15:31 |
| *** cshen has quit IRC | 15:31 | |
| mnaser | yep we have neutron_agents to is_metal: true | 15:32 |
| mnaser | so thats why i asked if we added it to neutron_agents we wouldnt have the second change but honestly, i'll trust jamesdenton decision fo rthat | 15:32 |
| mnaser | http://logs.openstack.org/15/623515/1/check/openstack-ansible-functional-centos-7/cbc82b1/ | 15:32 |
| mnaser | the lxc_hosts seem to have passed | 15:32 |
| mnaser | http://logs.openstack.org/15/623515/1/check/openstack-ansible-functional-centos-7/cbc82b1/logs/host/messages.txt.gz | 15:33 |
| mnaser | no timeouts | 15:33 |
| mnaser | and obviously no dbus restart | 15:33 |
| admin0 | jamesdenton, https://gist.github.com/a1git/4c7052f0b4ee0566e20bc03c40c2a1c5 -- i am constantly getting: fatal: [b1_neutron_server_container-e54d000e]: FAILED! => {"msg": "'dict object' has no attribute 'network_sriov_mappings'"} | 15:33 |
| odyssey4me | mnaser ok, could you add a release note to the patch describing the issue being fixed so that it goes into our releasenotes - we'll also have to backport this because of the new centos applying to all platforms | 15:34 |
| admin0 | i gave a network_sriov_mappings in the config of the compute node as you mentioned | 15:34 |
| jamesdenton | admin0 hang tight on that, if you don't mind | 15:34 |
| mnaser | odyssey4me: yep, one moment | 15:34 |
| * admin0 hangs on by the cliff | 15:34 | |
| odyssey4me | jamesdenton I think for now the best option is to pull the env.d file out of the default inventory and add it to etc/openstack_deploy/env.d/ as a .example file | 15:35 |
| jamesdenton | understood. i will update that patch, one sec | 15:35 |
| openstackgerrit | Mohammed Naser proposed openstack/openstack-ansible-lxc_hosts master: prep: remove old machinectl workarounds https://review.openstack.org/623515 | 15:37 |
| mnaser | odyssey4me: let met know if that's adequate | 15:37 |
| odyssey4me | mnaser looks fine to me | 15:38 |
| odyssey4me | perhaps jrosser could take a peek too :) | 15:38 |
| openstackgerrit | James Denton proposed openstack/openstack-ansible master: Only enable OVN VIPs when ml2.ovn plugin is used https://review.openstack.org/622473 | 15:40 |
| jamesdenton | ^^ odyssey4me take a look at that and let me know | 15:41 |
| * jrosser un-schoolruns...... something i need to look at? | 15:43 | |
| odyssey4me | jamesdenton you'll need to revert https://review.openstack.org/#/c/584069/14/tests/test_inventory.py too | 15:44 |
| jamesdenton | ok | 15:44 |
| odyssey4me | yep, with that it should be complete | 15:45 |
| odyssey4me | may as well make it depends-on https://review.openstack.org/623515 so that it gets passing sooner | 15:46 |
| odyssey4me | jrosser https://review.openstack.org/623515 looks like it resolves the centos build timeouts | 15:46 |
| jamesdenton | but i can leave inventory/inventory.ini alone? | 15:46 |
| odyssey4me | jamesdenton yeah, with those at least the groups exist - but that's not used for any container building/specification | 15:47 |
| jamesdenton | got it | 15:47 |
| odyssey4me | so it's a default group, at the very least | 15:47 |
| jrosser | shame theres been no jobs run in lxc_hosts repo since centos 7.6 as a reference | 15:48 |
| jamesdenton | should the depends-on reference the entire URL or just the change number? | 15:51 |
| odyssey4me | jamesdenton the URL: https://review.openstack.org/623515 | 15:51 |
| openstackgerrit | James Denton proposed openstack/openstack-ansible master: Only enable OVN VIPs when ml2.ovn plugin is used https://review.openstack.org/622473 | 15:53 |
| openstackgerrit | kourosh vivan proposed openstack/openstack-ansible-os_trove master: Add segmentation option https://review.openstack.org/623533 | 15:54 |
| odyssey4me | thanks jamesdenton lgtm - let's see how the gates go | 15:55 |
| jamesdenton | admin0 I need to take a look at the provider_networks module at some point, but I am guessing you did not create a neutron_provider_networks override and specify network_sriov_mappings? | 15:59 |
| jamesdenton | ideally, that module should bind net_name to sriov_host_interfaces to create that network_sriov_mappings relationship. | 16:00 |
| admin0 | jamesdenton, where do i do that ? | 16:00 |
| jamesdenton | well, if your nodes are configured the same, then you can create it in user_variables.yml | 16:01 |
| admin0 | for test, i have only 1 compute and 1 controller | 16:01 |
| *** weezS has quit IRC | 16:02 | |
| jamesdenton | it would look something like this: http://paste.openstack.org/show/736829/ | 16:02 |
| jamesdenton | can you post your openstack_user_config real quick? | 16:03 |
| admin0 | jamesdenton, here; https://gist.github.com/a1git/4c7052f0b4ee0566e20bc03c40c2a1c5 | 16:05 |
| admin0 | both config and variables | 16:05 |
| *** hamzaachi has quit IRC | 16:07 | |
| *** hamzaachi has joined #openstack-ansible | 16:07 | |
| jamesdenton | I would remove Line 43 and Line 54. And on line 142, remove vxlan:br-vxlan. You defining a network mapping that would never be used. After line 143, you can add 'network_sriov_mappings: "vlan: ens9f1"' | 16:08 |
| *** weezS has joined #openstack-ansible | 16:10 | |
| jamesdenton | the neutron_provider_networks override *shouldn't* be necessary when provider networks are defined in openstack_user_config.yml, but i know it was highlighted in early OVS wikis. I need to figure out what exactly is necessary and update some docs. | 16:10 |
| *** hamzaachi has quit IRC | 16:14 | |
| *** tosky has quit IRC | 16:14 | |
| admin0 | jamesdenton, can't sriov be tested east-west ? | 16:15 |
| jamesdenton | It only supports vlans | 16:15 |
| admin0 | aah | 16:15 |
| jamesdenton | tags are applied to the VF | 16:16 |
| mnaser | jrosser: around to look at https://review.openstack.org/#/c/623515/ ? | 16:16 |
| jrosser | yep | 16:16 |
| *** weezS has quit IRC | 16:17 | |
| mnaser | that should fix our centos-7 gates | 16:17 |
| jrosser | lgtm | 16:18 |
| mnaser | thank you sir | 16:21 |
| admin0 | jamesdenton, error, config and variable changes: https://gist.github.com/a1git/78c7a87ddefba40935a8de3e92f43bbd | 16:24 |
| *** gyee has joined #openstack-ansible | 16:25 | |
| admin0 | following your docs, i added the one for compute node as well | 16:26 |
| admin0 | ditto error | 16:26 |
| jamesdenton | Like 49 needs to move under line 63 with the other network overrides | 16:26 |
| admin0 | oh | 16:28 |
| admin0 | :D | 16:28 |
| admin0 | thanks .. retrying | 16:28 |
| jamesdenton | i;m a bit surprised it needs network_sriov_mappings for neutron-server | 16:29 |
| jamesdenton | so there may be something else up | 16:29 |
| *** rodolof has joined #openstack-ansible | 16:30 | |
| *** DanyC has quit IRC | 16:32 | |
| *** DanyC has joined #openstack-ansible | 16:33 | |
| *** aedc has quit IRC | 16:36 | |
| *** DanyC has quit IRC | 16:38 | |
| admin0 | something never seen before: [os_neutron : Setup External Network Provider Bridge] "ovs-vsctl: fail-mode must be \"standalone\" or \"secure\"" | 16:38 |
| mnaser | i have seen that | 16:39 |
| jamesdenton | hmm, yeah. That's new with 2.10.0 i think. | 16:39 |
| mnaser | and i hae fixed that | 16:39 |
| jamesdenton | there is a patch for that. may need backporting | 16:39 |
| admin0 | i am trying the 18.1.0 branch | 16:39 |
| *** macza has joined #openstack-ansible | 16:39 | |
| mnaser | https://review.openstack.org/#/c/613286/ | 16:40 |
| jamesdenton | yeah it needs to be backported to rocky | 16:40 |
| mnaser | other cores can vote on that to help admin0 unblock (it will require another release but yeah) | 16:40 |
| jamesdenton | but merged first. lol | 16:40 |
| mnaser | it merged in master, backport pending :) | 16:40 |
| jamesdenton | oh my bad | 16:41 |
| admin0 | i will cherry pick and move on :) | 16:41 |
| jamesdenton | stepping out for lunch. bbiaf | 16:42 |
| openstackgerrit | Merged openstack/openstack-ansible-lxc_hosts master: prep: remove old machinectl workarounds https://review.openstack.org/623515 | 16:42 |
| noonedeadpunk | reviews are appreciated https://review.openstack.org/#/c/622956/3 :) | 16:46 |
| *** weezS has joined #openstack-ansible | 16:52 | |
| mnaser | ill wait for the jamesdenton patch with depends-on to merge before touching too much | 16:53 |
| mnaser | and rechecking things | 16:53 |
| *** mma has quit IRC | 16:54 | |
| *** mma has joined #openstack-ansible | 16:55 | |
| *** mma has quit IRC | 16:57 | |
| *** aedc has joined #openstack-ansible | 17:03 | |
| *** DanyC has joined #openstack-ansible | 17:06 | |
| *** DanyC has quit IRC | 17:11 | |
| admin0 | thanks guys neutron is up and running and i can see the agent | 17:15 |
| admin0 | i dunno how to test, so i delivered it to those who wanted to test it . will wait for their feedback :D | 17:15 |
| jamesdenton | you will need to create a port whose type is 'direct'. then spin up an instance specifying that port | 17:18 |
| *** mwe1 has quit IRC | 17:18 | |
| ThiagoCMC | admin0, sounds like fun! | 17:20 |
| *** weezS has quit IRC | 17:36 | |
| *** weezS has joined #openstack-ansible | 17:37 | |
| *** weezS has quit IRC | 17:37 | |
| *** weezS has joined #openstack-ansible | 17:37 | |
| *** weezS has joined #openstack-ansible | 17:38 | |
| *** weezS has joined #openstack-ansible | 17:39 | |
| *** weezS has quit IRC | 17:39 | |
| *** weezS has joined #openstack-ansible | 17:40 | |
| *** weezS has quit IRC | 17:40 | |
| *** weezS has joined #openstack-ansible | 17:40 | |
| *** weezS has joined #openstack-ansible | 17:41 | |
| *** weezS has quit IRC | 17:42 | |
| *** weezS has joined #openstack-ansible | 17:42 | |
| *** weezS has quit IRC | 17:42 | |
| *** weezS has joined #openstack-ansible | 17:43 | |
| *** weezS has quit IRC | 17:43 | |
| *** weezS has joined #openstack-ansible | 17:44 | |
| *** weezS has quit IRC | 17:44 | |
| *** weezS has joined #openstack-ansible | 17:44 | |
| *** weezS has joined #openstack-ansible | 17:45 | |
| *** weezS has quit IRC | 17:45 | |
| *** weezS has joined #openstack-ansible | 17:46 | |
| *** weezS has quit IRC | 17:46 | |
| *** weezS has joined #openstack-ansible | 17:47 | |
| *** weezS has joined #openstack-ansible | 17:48 | |
| *** ahosam has quit IRC | 17:50 | |
| *** shardy has quit IRC | 17:57 | |
| guilhermesp | so as it is merged https://review.openstack.org/#/c/623515/ I think I just need to recheck mine https://review.openstack.org/#/c/618820/ as it is timing out because I rechecked before the fix being merged | 18:03 |
| guilhermesp | locally, our placement role is executing successful locally, I just can't see any placement process running. Maybe there are some uwsgi configuration bits missing https://review.openstack.org/#/c/618820/ | 18:04 |
| guilhermesp | I appreciate any review :) | 18:04 |
| *** udesale has joined #openstack-ansible | 18:26 | |
| *** udesale has quit IRC | 18:27 | |
| *** strattao has joined #openstack-ansible | 18:30 | |
| openstackgerrit | Merged openstack/openstack-ansible-ceph_client stable/rocky: Remove the dependency on SSH for monitors https://review.openstack.org/623234 | 18:35 |
| *** hamzaachi has joined #openstack-ansible | 18:44 | |
| *** hamzaachi has quit IRC | 18:49 | |
| *** fmulti has joined #openstack-ansible | 18:49 | |
| fmulti | Hi all, I have given an Ubuntu 16.04 laptop to users who are complaining that applications/windows/terminal/etc are disappearing within a set amount of time (approx 5-10 mins, they did not time it) of inactivity... I am at a loss and have not witnessed this myself, but what could be causing this? They said it does not KILL an application, but when they try to open a new one, it wont let them as it is already running (verifi | 18:50 |
| fmulti | (this is after I have ran ansible-hardening on it) | 18:50 |
| *** hamzaachi has joined #openstack-ansible | 18:50 | |
| openstackgerrit | Merged openstack/openstack-ansible-os_heat master: Enable overriding the service setup host python interpreter https://review.openstack.org/621201 | 18:52 |
| logan- | fmulti: there is a 10 minute timeout set in /etc/profile by ansible hardening, you'll see an env var being set called TMOUT. that is what is causing it | 19:00 |
| jamesdenton | Has anyone done an Ubuntul install recently? This hash has changed, which causes rabbitmq_server install to fail. https://github.com/openstack/openstack-ansible-rabbitmq_server/blob/e577e997a99b3152d6f090210facaf710629f539/vars/ubuntu.yml#L26 | 19:01 |
| jamesdenton | *Ubuntu | 19:01 |
| logan- | fmulti: https://github.com/openstack/ansible-hardening/blob/4648d7576dd574941791549a661ad1a8f595302b/tasks/rhel7stig/misc.yml#L228-L242 | 19:01 |
| logan- | fmulti: try setting the var security_rhel7_session_timeout to 0 | 19:02 |
| fmulti | logan-: even gui applications? they claim their IDE coding program dies | 19:02 |
| logan- | no idea i have not used ansible-hardening in a desktop env, but that is my guess | 19:03 |
| fmulti | logan-: okay thank you, ill give it a shot | 19:03 |
| *** rodolof has quit IRC | 19:06 | |
| *** rodolof has joined #openstack-ansible | 19:07 | |
| mnaser | yay | 19:07 |
| mnaser | centos jobs are passing | 19:07 |
| jamesdenton | nice work | 19:09 |
| *** electrofelix has quit IRC | 19:11 | |
| prometheanfire | so many rechecks | 19:12 |
| openstackgerrit | Merged openstack/openstack-ansible-os_ironic master: Enable overriding the service setup host python interpreter https://review.openstack.org/621208 | 19:15 |
| *** aedc has quit IRC | 19:17 | |
| *** hamzaachi has quit IRC | 19:19 | |
| *** hamzaachi has joined #openstack-ansible | 19:23 | |
| guilhermesp | Awesome mnaser ! | 19:24 |
| kaiokmo | guilhermesp: https://review.openstack.org/#/c/618820/ passing. w00t | 19:29 |
| guilhermesp | Yeah kaiokmo but I still think there is something else to fix/add as in my local tests I can't see placement running | 19:30 |
| openstackgerrit | Merged openstack/openstack-ansible-os_octavia master: Fix ansible deprecation warnings https://review.openstack.org/621658 | 19:31 |
| kaiokmo | jrosser: thank you for review :) | 19:48 |
| jrosser | :) | 19:50 |
| openstackgerrit | Merged openstack/openstack-ansible-os_ironic master: Update mailinglist from dev to discuss https://review.openstack.org/621759 | 19:57 |
| admin0 | jamesdenton, they want on br-vxlan :D so i will change br-vxlan to be from type: vxlan to vlan and rerun the playbooks :D | 20:11 |
| jamesdenton | because br-vxlan has that other NIC in it? | 20:11 |
| admin0 | br-vxlan has a diff nic .. for this test, the requirement is east-west and not north-soudh | 20:12 |
| jamesdenton | well, but ultimately they want to use SR-IOV, right? That will need a physical interface mapping | 20:12 |
| admin0 | south* | 20:12 |
| admin0 | having fun here :) | 20:12 |
| *** fmulti has quit IRC | 20:20 | |
| admin0 | if I change type: vxlan to type: vlan on the br-vxlan , is that enough to switch east-west to use vlans ? | 20:23 |
| admin0 | and give the vlan acceptable range ofcourse in the range: | 20:23 |
| *** strattao has quit IRC | 20:27 | |
| openstackgerrit | Kaio Kassiano Moura Oliveira proposed openstack/openstack-ansible master: [DOCS] Fix user guide to source admin openrc https://review.openstack.org/623576 | 20:28 |
| *** ahosam has joined #openstack-ansible | 20:41 | |
| *** DanyC has joined #openstack-ansible | 20:49 | |
| *** DanyC has quit IRC | 20:50 | |
| *** DanyC has joined #openstack-ansible | 20:50 | |
| *** ThiagoCMC has quit IRC | 20:51 | |
| lbragstad | 20:52 | |
| lbragstad | does osa populate the /etc/$service/policy.d/ directories during installation? | 20:52 |
| lbragstad | or at least lay them down? | 20:53 |
| openstackgerrit | Mohammed Naser proposed openstack/openstack-ansible-lxc_hosts stable/rocky: prep: remove old machinectl workarounds https://review.openstack.org/623578 | 20:53 |
| jamesdenton | admin0 I believe you'll need to change the default tenant network type, or at least the ordering. That 'network_types' override should just be vlan in this case | 20:56 |
| *** macza has quit IRC | 20:58 | |
| *** DanyC_ has joined #openstack-ansible | 20:59 | |
| admin0 | trying to find the exact variable name | 20:59 |
| *** aedc has joined #openstack-ansible | 21:00 | |
| *** DanyC_ has quit IRC | 21:00 | |
| jamesdenton | admin0 it was here: https://gist.github.com/a1git/78c7a87ddefba40935a8de3e92f43bbd#file-gistfile1-txt-L65 | 21:01 |
| *** DanyC has quit IRC | 21:03 | |
| admin0 | :D i was doing grep -ri network_type /etc/ansible/* | 21:03 |
| admin0 | :) ..looking for anything that said default | 21:03 |
| *** rodolof has quit IRC | 21:09 | |
| openstackgerrit | Merged openstack/openstack-ansible-os_neutron stable/rocky: Include dependency packages in distro install list https://review.openstack.org/616984 | 21:10 |
| openstackgerrit | Merged openstack/openstack-ansible-os_neutron stable/rocky: ovs: force to secure fail mode by default https://review.openstack.org/613286 | 21:10 |
| *** macza has joined #openstack-ansible | 21:24 | |
| *** DanyC has joined #openstack-ansible | 21:27 | |
| openstackgerrit | Merged openstack/openstack-ansible-os_neutron master: Install haproxy on ovn-controller nodes https://review.openstack.org/620994 | 21:59 |
| *** ThiagoCMC has joined #openstack-ansible | 22:10 | |
| *** KeithMnemonic has quit IRC | 22:16 | |
| *** rodolof has joined #openstack-ansible | 22:16 | |
| *** trident has quit IRC | 22:22 | |
| *** trident has joined #openstack-ansible | 22:22 | |
| jrosser | so zuul nodes in inap currently are centos 7.5 | 22:37 |
| jrosser | thats going to fail a bunch of stuff so don't be surprised if centos jobs still break a bit until that is fixed | 22:38 |
| *** rodolof has quit IRC | 22:52 | |
| *** hamzaachi has quit IRC | 23:10 | |
| *** hamzaachi has joined #openstack-ansible | 23:10 | |
| *** DanyC has quit IRC | 23:11 | |
| *** aedc has quit IRC | 23:22 | |
| openstackgerrit | Merged openstack/openstack-ansible master: Remove xenial from Vagrantfile https://review.openstack.org/622176 | 23:23 |
| openstackgerrit | Merged openstack/openstack-ansible stable/rocky: Docs: Fix VLAN bridge interface in example setup https://review.openstack.org/615795 | 23:29 |
| *** aedc has joined #openstack-ansible | 23:29 | |
| openstackgerrit | Merged openstack/openstack-ansible-os_neutron master: Fix ansible deprecation warnings https://review.openstack.org/621659 | 23:36 |
| redkrieg | hello all, I was wondering if anyone had experience with using flat networks with linux-bridge-agent that could shed some light on how their guests are bridged to the provider network. I'm able to see dhcp requests on the guest's bridge interface, but I don't see replies and the guest has no other signs of connectivity. | 23:46 |
| *** aedc has quit IRC | 23:52 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!