Thursday, 2018-12-13

*** Daemoen is now known as demone00:00
*** demone is now known as mmercer00:00
*** macza has quit IRC00:14
*** hwoarang has quit IRC00:15
*** hwoarang has joined #openstack-ansible00:17
*** spatel has joined #openstack-ansible00:17
*** spatel has quit IRC00:22
*** rodolof has quit IRC00:43
*** rodolof has joined #openstack-ansible00:44
*** tosky has quit IRC00:55
*** rodolof has quit IRC01:04
*** rodolof has joined #openstack-ansible01:04
*** gyee has quit IRC01:08
*** nurdie has joined #openstack-ansible01:11
*** vnogin has joined #openstack-ansible01:26
*** vnogin has quit IRC01:30
*** rodolof has quit IRC01:30
*** tinwood has quit IRC02:10
*** tinwood has joined #openstack-ansible02:11
*** cshen has joined #openstack-ansible02:36
*** cshen has quit IRC02:40
*** DanyC has joined #openstack-ansible02:40
*** DanyC has quit IRC02:45
*** maharg101 has quit IRC02:49
*** nurdie has quit IRC02:50
*** nurdie has joined #openstack-ansible02:52
*** nurdie_ has joined #openstack-ansible02:53
*** spatel has joined #openstack-ansible02:56
*** nurdie has quit IRC02:56
*** hw_wutianwei_ has joined #openstack-ansible03:17
*** fnpanic has quit IRC03:28
*** spatel has quit IRC03:39
*** nurdie_ has quit IRC04:47
*** nurdie has joined #openstack-ansible04:48
*** udesale has joined #openstack-ansible04:49
openstackgerritMerged openstack/openstack-ansible stable/rocky: Update all SHAs for 18.1.2  https://review.openstack.org/62440705:37
*** cshen has joined #openstack-ansible05:51
*** cshen has quit IRC05:56
*** nurdie_ has joined #openstack-ansible06:51
*** nurdie__ has joined #openstack-ansible06:53
*** nurdie has quit IRC06:54
*** nurdie_ has quit IRC06:55
*** pcaruana has joined #openstack-ansible07:12
*** nurdie has joined #openstack-ansible07:15
*** nurdie has quit IRC07:17
*** nurdie has joined #openstack-ansible07:18
*** nurdie__ has quit IRC07:18
*** nurdie has quit IRC07:22
*** cshen has joined #openstack-ansible07:22
*** kopecmartin|off is now known as kopecmartin07:35
*** shardy has joined #openstack-ansible07:55
*** maharg101 has joined #openstack-ansible08:33
*** nurdie has joined #openstack-ansible08:48
*** nurdie has quit IRC08:54
*** cshen has quit IRC09:06
*** cshen has joined #openstack-ansible09:06
*** trident has quit IRC09:16
*** DanyC has joined #openstack-ansible09:17
*** trident has joined #openstack-ansible09:19
*** DanyC has quit IRC09:21
*** dcdamien has joined #openstack-ansible09:29
*** DanyC has joined #openstack-ansible09:33
*** tosky has joined #openstack-ansible09:37
*** markvoelker has joined #openstack-ansible09:46
noonedeadpunkspotz: yeah, API has been removed, but it still collects data and sends it to gnocchi. I've filled in a bug https://bugs.launchpad.net/ceilometer/+bug/1801348 and placed a patch for it before Berlin summit, and try to ping in IRC at least once a week, but never saw anyone there09:56
openstackLaunchpad bug 1801348 in Ceilometer "notification agent fails with KeyError on volume create/delete" [Undecided,In progress] - Assigned to Dmitriy Rabotjagov (noonedeadpunk)09:56
*** rodolof has joined #openstack-ansible09:58
*** rgogunskiy has quit IRC10:03
noonedeadpunkActually, it's smth, that might be fixed by OSA like with https://review.openstack.org/#/c/623239/ but if the project is really dead, it'll look like project maintaining.10:04
*** electrofelix has joined #openstack-ansible10:04
*** trident has quit IRC10:15
*** trident has joined #openstack-ansible10:15
*** ppetit has joined #openstack-ansible10:20
openstackgerritMerged openstack/openstack-ansible-ops master: Chage cpu model to allow for nested virt to work  https://review.openstack.org/62478010:34
*** ppetit has quit IRC10:41
Miougegood morning folks10:46
*** nurdie has joined #openstack-ansible10:50
openstackgerritJesse Pretorius (odyssey4me) proposed openstack/openstack-ansible-lxc_hosts stable/pike: prep: remove old machinectl workarounds  https://review.openstack.org/62467410:51
odyssey4meo/ Miouge how're things going?10:52
*** DanyC has quit IRC10:53
*** DanyC has joined #openstack-ansible10:54
Miougeodyssey4me: All good, I’ve been testing an OSA Dockerfile :D10:54
*** DanyC has joined #openstack-ansible10:55
*** nurdie has quit IRC10:55
*** gisak has joined #openstack-ansible11:17
openstackgerritMaxime Guyot proposed openstack/openstack-ansible master: [WIP] Add doc for Alpine deployment host  https://review.openstack.org/62112711:27
openstackgerritMaxime Guyot proposed openstack/openstack-ansible master: [WIP] Add doc for Alpine deployment host  https://review.openstack.org/62112711:28
openstackgerritMaxime Guyot proposed openstack/openstack-ansible master: Add doc for Docker Alpine deployment host  https://review.openstack.org/62112711:33
*** ansmith has joined #openstack-ansible11:43
odyssey4meMiouge I'm curious why you didn't just use an Ubuntu/CentOS container base, and chose to use Alpine?11:47
Miougeodyssey4me: If you use Ubuntu as a core image, it’s exactly the same procedure as a standard deploy host, so nothing to document there.11:49
odyssey4meMiouge ah, fair enough11:50
*** DanyC has quit IRC11:59
odyssey4menoonedeadpunk I'd suggest trying to make contact with the PTL for telemetry to find out what's going on - mnaser can likely help with that.11:59
*** DanyC has joined #openstack-ansible12:00
*** DanyC has joined #openstack-ansible12:00
noonedeadpunkodyssey4me: I'll try to, thanks!12:02
noonedeadpunkbtw, should I place patch for removing git retrievment for ceilometer, and then add https://review.openstack.org/#/c/623239/ , or place such patch and rebase that on on top of it?12:03
openstackgerritJesse Pretorius (odyssey4me) proposed openstack/openstack-ansible master: Ensure that control plane hosts are also updated correctly  https://review.openstack.org/62496512:04
odyssey4menoonedeadpunk that one will also backport to queens right? if so - leave as-is and put the one to change it to use the venv on top of that12:04
*** django_ is now known as django12:05
odyssey4mejrosser this is a glaring bug in the upgrade process which will need porting back to rocky: https://review.openstack.org/62496512:05
* jrosser looks12:06
* odyssey4me is looking at http://zuul.openstack.org/builds?project=OpenStack%2Fopenstack-ansible&branch=stable%2Frocky&pipeline=periodic12:09
noonedeadpunkodyssey4me: I think it shouldn't be backported (even if I'd wish to), as it introduces override and configuration for 2 extra files, so it's kinda new functional.12:10
odyssey4menoonedeadpunk yep, I agree12:12
jrosserodyssey4me: i think somehow i dodged that during my Q->R upgrade12:12
jrosseri have dedicated neutron nodes12:12
odyssey4mejrosser ah, yes, that would have covered it12:16
odyssey4meI think, overall, we really need a better method12:17
jrosseri also did it manually12:17
jrossertoo much trust required to just let the script rip and sit back and watch12:17
odyssey4meit's ideal to be able to do a fan-out for all the deployment changes, but then only serialise the restarts - that'll be much, uch faster12:17
*** ansmith has quit IRC12:17
odyssey4meyeah, I agree - I prefer to execute manually, then inspect things after each step to confirm everything is as expected12:18
odyssey4mebut meh, some don't12:18
jrosserotoh the script (or equivalent) is needed for CI12:18
odyssey4meyep, that was the original reason it was created12:19
openstackgerritJesse Pretorius (odyssey4me) proposed openstack/openstack-ansible master: Ensure that control plane hosts are also updated correctly  https://review.openstack.org/62496512:21
*** markvoelker has quit IRC12:24
*** udesale has quit IRC12:25
*** udesale has joined #openstack-ansible12:26
openstackgerritJesse Pretorius (odyssey4me) proposed openstack/openstack-ansible-rabbitmq_server master: upgrade: start service before applying policies  https://review.openstack.org/62037812:43
*** mkuf_ has joined #openstack-ansible12:45
openstackgerritMerged openstack/openstack-ansible-os_keystone stable/pike: Ensure that LDAP config is deployed on all keystone hosts  https://review.openstack.org/62443412:45
gisakhi guys, one question: why would the public-network's bridge (in my case brq88038979-47) not bridge any interfaces at all ?12:49
*** mkuf has quit IRC12:49
*** nurdie has joined #openstack-ansible12:51
*** rodolof has quit IRC12:54
*** rodolof has joined #openstack-ansible12:54
*** nurdie has quit IRC12:56
jamesdentongisak you can check linuxbridge_agent.ini to see what the mapping should be. physical_interface_mappings is the option12:58
gisakthnx, in my case I have eno1 as internet access interface and it is bridged with br-vlan, how to manage the correct bridging topology?13:02
jamesdentonhow many interfaces are you working with?13:02
gisak4 on network node13:02
jamesdentonand what did that mapping look like13:02
gisakeno1 is br-vlan, eno2 br-mgmt eno3 br-vxlan and eno4 br-storage13:03
gisakshould i delete br-vlan and assign eno1 to physical_interface_mappings from linuxbridge_agent.ini?13:04
jamesdentonok. what is the current physical_interface_mappings value?13:04
spotzoff to dog related stuff back next week!13:07
openstackgerritChristian Zunker proposed openstack/openstack-ansible-os_neutron master: Set admin url endpoints insecure flag for nova client  https://review.openstack.org/62498713:09
gisakflat:eno1,vlan:br-vlan13:10
*** markvoelker has joined #openstack-ansible13:10
gisakI use flat, in this case i should remove vlan:br-vlan and then delete br-vlan brdige from node and try again ?13:11
jamesdentonwell, to get this working now you can set vlan:eno1 and remove eno1 from the br-vlan bridge13:11
jamesdentonbut to do it right, you'll need to update the provider networks in openstack_user_config.yml and add a 'host_bind_override: eno1' to the network block: https://docs.openstack.org/project-deploy-guide/openstack-ansible/draft/app-config-prod.html13:13
openstackgerritMerged openstack/openstack-ansible-lxc_container_create master: cleanup: remove rocky-only upgrade code  https://review.openstack.org/62034313:18
gisakin this case, if I set 'host_bind_override: eno1' in openstack_user_config.yml and rerun playbooks, will i still need the br-vlan ?13:18
gisakor br-vlan becomes useless in this case ?13:18
jamesdentonit would be used on infra/network nodes, but not computes13:19
jamesdentoni would recommend leaving itfor now13:19
jamesdentonas in, not deleting it13:19
odyssey4mejamesdenton have you managed to figure out how to get the neutron agents to install on the same network_host when doing the p->q upgrade?13:29
jamesdentonno13:29
odyssey4meI wonder if it wouldn't be as trivial as using the extra var 'on_metal=yes' when running the os-neutron-install playbook13:30
jamesdentoni can try13:31
odyssey4meotherwise it might have to be more subtle - like adding the hosts into the right group13:32
*** hamzaachi has joined #openstack-ansible13:33
jamesdentonyeah, i've noticed that the inventory post-upgrade looks like this: http://paste.openstack.org/show/737214/13:33
jamesdentonthe Network01 host is one i added after the upgrade, and it was added to the neutron_agent group as baremetal.13:34
odyssey4mejamesdenton and neutron_agent is the group that matters here, from a play/task/template targeting standpoint, right?13:36
gisakthank you jamesdenton13:37
*** gillesMo has joined #openstack-ansible13:37
gisakthe gateway finally is up, but floating IP's now dont work (13:38
gisakare unreachable13:38
gisakbut the instance has internet13:38
*** ansmith has joined #openstack-ansible13:38
jamesdentonodyssey4me i think so?13:40
gisakbtw, in openstack_user_config.yml the host_bind_override is set to eno1 )13:41
jamesdentonfor the vlan block as well as flat?13:42
jamesdentonodyssey4me the bare metal node would also need to be part of the other agent groups (neutron_linuxbridge_agent, ovs, etc)13:43
gisakthis is the openstack_user_config.yml I run the playbooks: http://paste.openstack.org/show/737217/13:43
gillesMogisak: sorry if I missed something, I'm just arriving ! But, are you sure that you just don't allow traffic via your security groups ?13:43
*** fnpanic has joined #openstack-ansible13:44
fnpanichi13:44
fnpanicquick question13:45
fnpanicis there any downside of disabling nat for the lxc containers?13:45
odyssey4mejamesdenton I have an idea - going to try it out now.13:45
jamesdentonodyssey4me so it looks like you can use inventory-manage.py -r to remove the container, then run the dynamic inventory script to update the inventory to include the baremetal infra node in the proper groups. But like you said, you would lose access to that container13:45
*** masterpe has joined #openstack-ansible13:45
odyssey4mefnpanic yes, they would lose internet access - unless you've made provision for that another way13:46
gillesMoHas someone tried to "host migrate" a compute node in Horizon (Queens) ? I've tried yesterday and my instances stay in migrating state since then (24h...). I see a ssh connection between two compute nodes, but there is WARNING in my nova-compute logs :13:46
jamesdentonwell you don't lose ssh access per say13:46
gisakok my bad here with sec groups :) neither ping nor ssh access was opened )13:46
odyssey4mejamesdenton yeah, but we lose the ability to target it via ansible directly as a host in the inventory13:46
fnpanicodyssey4me: thanks. we will use a quid in the mgmt network13:46
jamesdentonodyssey4me but at that point, does it matter?13:46
odyssey4mejamesdenton it does, because we still need to delete it later13:47
jamesdentonlxc-delete. hehe13:47
odyssey4meI mean, we could hack it away - but I think there's possibly a more elegant option - simply have a temporary ini file which adds network_hosts as a child to each of those other groups.13:48
gillesMoPb migrating instance while evacuate a host (migrate host in Horizon after disabling a compute node)... nova-compute logs shows "Instance not resizing, skipping migration"13:48
odyssey4methis would be something we drop into user space for during the upgrade, then remove after we've removed the containers13:49
jamesdentonso the ini will supplement, or be merged, against the existing inventory?13:49
odyssey4mejamesdenton ansible will merge the two - but the dynamic inventory will know nothing of it13:49
jamesdentonahh ok13:49
odyssey4megillesMo it's been a long time since I did any of that - but typically live/cold migrations will not happen if the base image the instances was created from was deleted - because it cannot pull it down to the target host... so you have to copy it over manually into the base cache13:52
jamesdentonodyssey4me alright, just so i understand. The approach could be to see which groups contain existing neutron agent containers, add the network_hosts group to those groups, install the bits, then remove the members of neutron_agents_container group from inventory?13:53
odyssey4mejamesdenton yep, that's basically my thinking13:53
odyssey4mebefore removing those members, we ensure that the agents are disabled, everything's migrated over, etc13:54
*** stuartgr has joined #openstack-ansible13:54
jamesdentonSo you want to automate that, too?13:54
odyssey4meyep, it seems automatable to me13:54
gillesMoodyssey4me: One additional info : I use Ceph everywhere. One of my instance is booted from a volume, the other is from image, which is still here (in Ceph)...13:54
jamesdentonIt's a bold strategy, let13:55
odyssey4mewhether anyone uses the automation is up to them - but the playbook would act as a guide for the procedure anyway13:55
jamesdentonlet's see if it pays off13:55
jamesdentonkk it's optional, then13:55
odyssey4megillesMo ah ok, then I dunno - live migrations always worked for me then... evacuation wasn't possible back when I operated a cloud ;)13:55
*** tosky has quit IRC13:56
odyssey4mejamesdenton it would be included in run-upgrade, but every part of run-upgrade can be run in the individual steps too - as preferred by the operator13:56
jamesdenton #yolo13:57
odyssey4me:)13:58
fnpanicso aio is broken currently?13:58
fnpanic?13:58
odyssey4mefnpanic what do you mean?13:58
jamesdentonso in the inventory... children and hosts... children simply refers to groups while hosts is individual machines?13:59
odyssey4mefnpanic the AIO is used daily to merge patches, so no - it's not broken13:59
odyssey4mejamesdenton yep14:00
*** tosky has joined #openstack-ansible14:00
*** irclogbot_0 has quit IRC14:00
*** markvoelker has quit IRC14:03
fnpanicodyssey4me: ok14:05
fnpanici was following the guide for quick aio14:06
fnpanicand it was not working14:06
fnpanicstrange14:06
odyssey4mefnpanic you'll have to be much more specific - what distro, where did it break?14:06
*** irclogbot_0 has joined #openstack-ansible14:07
odyssey4mealso, which specific guide are you following (master/latest, or rocky, or older?)14:08
jamesdentonodyssey4me New update to https://bugs.launchpad.net/openstack-ansible/+bug/1804770 confirms what we saw re: not migration to BM during upgrade. And another bug: https://bugs.launchpad.net/openstack-ansible/+bug/1785592. Just FYI14:09
openstackLaunchpad bug 1804770 in openstack-ansible "Pike -> Queens Upgrade: Documentation for OVS setup and neutron agent rebalancing missing" [High,Confirmed] - Assigned to James Denton (james-denton)14:09
openstackLaunchpad bug 1785592 in openstack-ansible "dynamic inventory doesn't handle is_metal: false->true change" [High,Confirmed] - Assigned to Kevin Carter (kevin-carter)14:09
*** mkuf has joined #openstack-ansible14:10
fnpanicodyssey4me: ubuntu 1804 with rocky14:10
odyssey4mejamesdenton ok, I've assigned that other one to me and will have a go at it to see if it works14:11
jamesdentonright on14:11
odyssey4meotherwise the other option is to remove the containers from the inventory as we discussed, but I suspect that'll be more hacky and prone to failure14:11
odyssey4mealso possibly more disruptive14:12
*** mkuf_ has quit IRC14:13
*** irclogbot_0 has quit IRC14:14
jamesdentonone might require a lot less work :D14:16
*** irclogbot_0 has joined #openstack-ansible14:23
*** thuydang has joined #openstack-ansible14:23
*** rodolof has quit IRC14:32
odyssey4mejamesdenton bother - the only imaged build of pike I have is RPC-O, which already has the env.d file from queens as an override, so I can't test this14:32
*** rodolof has joined #openstack-ansible14:32
jamesdentonare there some changes you want me to look at? I can redeploy Pike today14:33
odyssey4mejamesdenton oh no - if that's your plan, then I can just do an AIO now14:33
*** nurdie has joined #openstack-ansible14:33
jamesdentonWill an AIO tell the whole story, though? Won't the AIO already be baremetal since it runs compute?14:34
odyssey4mewell, I need to validate just ansible and the inventory behaviour before I go ahead and build a patch14:34
jamesdentonk14:34
odyssey4meI'll fire up a fresh MNAIO of pike too for testing the patch later.14:35
*** markvoelker has joined #openstack-ansible14:36
*** nurdie has quit IRC14:39
*** hamzaachi has quit IRC14:39
ansmithodyssey4me: if you have a minute, could you take glance at https://review.openstack.org/#/c/624184/14:40
ansmithodyssey4me: I started a review from the top for the qdrouterd integration, in the experimental job, no hosts matched for creating the qdrouterd container14:40
ansmithodyssey4me: http://logs.openstack.org/84/624184/4/experimental/openstack-ansible-deploy-aio_rpc-qdrouterd-ubuntu-bionic/d07c9c3/job-output.txt.gz14:41
ansmithodyssey4me: not quite sure what I am missing14:41
*** priteau has joined #openstack-ansible14:42
odyssey4meansmith ok, let me finish a thread here and I'll take a look14:43
ansmithodyssey4me: thanks14:44
*** cshen has quit IRC14:51
fnpanicso the aio guide is missing something?15:12
fnpanici am trying it on ubuntu 1804 with rocky15:12
jrosserfnpanic: is your question about the AIO also related to wanting to disable nat on the containers?15:14
odyssey4mefnpanic you still haven't said where it's failing for you or what broke when you tried to follow the intructions there15:15
odyssey4mefnpanic if you could share a transcript of any error output or the commands you ran or *anything* informative in a paste service or gist or something, that'd be helpful15:16
*** ThiagoCMC has joined #openstack-ansible15:18
ThiagoCMCcloudnull, hey man, are you around?15:19
odyssey4meThiagoCMC cloudnull is on holiday until next year afaik15:19
ThiagoCMCOh no!! LOL15:19
ThiagoCMCI'm seeing a problem with Ansible, when running it against Ubuntu LXD containers and, after googling about it, I can see that cloudnull faced this problem too but, I don't know how to fix it...15:20
ThiagoCMCThe thing is...15:20
ThiagoCMCWhen running Ansible against an Ubuntu LXD or nspawn container, the Ansible APT module fails to install packages, it shows state "ok:" but, it does nothing!15:21
ThiagoCMCAre you guys aware of this?15:21
ThiagoCMCI tried Ansible 2.5 from Ubuntu 18.04, and 2.7 from their PPA.15:21
ThiagoCMCBoth fails...15:21
jrosseri do ansible + lxd for non osa stuff and it works ok15:21
ThiagoCMCBut which OS inside of the container?15:22
ThiagoCMCUbuntu?15:22
jrosseryes15:22
ThiagoCMCOk, cool!15:22
ThiagoCMCI have no idea about what's happening here... :-(15:22
jrosseri set the lxd containers up to look just like hosts, no lxdbr nat or anything, they get a static IP when they're created15:22
jrosserand regular ansible + ssh15:22
odyssey4mejrosser mnaser happy with https://review.openstack.org/624965 ? I'd like to get that ported back asap.15:23
ThiagoCMCI have the very same RAM image, that if I boot on KVM, Ansible against it works, if I boot it with nspawn, it doesn't.15:23
ThiagoCMCjrosser, mtacvlan?15:23
jrosserno, bridge15:23
ThiagoCMCI mean, macvlan15:23
ThiagoCMCok15:23
ThiagoCMCsorry, auto corrector... =P15:23
ThiagoCMCI'll test it again today.15:24
ThiagoCMCThank you!15:24
mnaserodyssey4me: +215:25
jrosseri just kept it super simple, bridges and no lxd nat, static IP15:25
*** weezS has joined #openstack-ansible15:26
ThiagoCMCyes, I'm trying to keep it KISS.15:27
ThiagoCMCBut macvlan instead of bridge, container have IP from my LAN, even ipv6.15:28
ThiagoCMCI can ssh into container, run apt update and etc15:28
ThiagoCMCit's fine15:28
ThiagoCMCJust Ansible APT isn't working.15:28
ThiagoCMCOther tasks works.15:28
ThiagoCMCvery weird.15:28
odyssey4meThiagoCMC in that case, why bother with the lxd connection plugin - just treat it as a host15:28
ThiagoCMCI'm using it as if it was a regular server, via SSH, not lxd connection thing.15:29
odyssey4meok, perhaps you're hitting an issue relating to ansible_tmp - I seem to recall something along those lines being an issue15:30
ThiagoCMCThere is an issue, definitely! =P15:30
ThiagoCMCI remember about ansible_tmp as well!15:31
ThiagoCMCIt isn't the first time that I'm seeing this as well.15:31
ThiagoCMCLast time was in 201615:31
ThiagoCMCI thought that it wasn't an issue anymore...15:31
ThiagoCMCI'll try to change the ansible_tmp!15:31
*** cshen has joined #openstack-ansible15:32
openstackgerritweizj proposed openstack/openstack-ansible-os_heat stable/queens: Replace Chinese punctuation with English punctuation  https://review.openstack.org/62502715:34
openstackgerritweizj proposed openstack/openstack-ansible-ceph_client stable/queens: Replace Chinese punctuation with English punctuation  https://review.openstack.org/62502815:34
openstackgerritweizj proposed openstack/openstack-ansible-rsyslog_client stable/queens: Replace Chinese punctuation with English punctuation  https://review.openstack.org/62502915:34
jamesdentonmnaser you still out doing the Kube thing?15:34
openstackgerritweizj proposed openstack/openstack-ansible-repo_build stable/queens: Replace Chinese punctuation with English punctuation  https://review.openstack.org/62503015:34
*** cshen has quit IRC15:36
*** cshen has joined #openstack-ansible15:37
*** markvoelker has quit IRC15:38
mgariepyanyone here knows someone that works on ubuntu kernel team ?15:43
mgariepyi'm having quite a fun issue with the 4.15.0 kernel..15:44
odyssey4memgariepy not personally, but tinwood or jamespage might be able to point you in the right direction15:44
odyssey4meotherwise pop into #ubuntu/#ubuntu-devel and ask15:45
mgariepyi'm in #ubuntu-kernel and was in -virt before that, but seems to be more kernel related than virt..15:46
odyssey4meah ok :/15:46
*** hamzaachi has joined #openstack-ansible15:46
mnaserjamesdenton: yes, I’m heading back today at night (it’s 7 am here) but can I help with anything ?15:47
jamesdentonnaw, it's just been awful quiet in here :)15:47
ionimgariepy, what problem do you have with 4.15?15:53
ionii'm just curious because i'm on that as well on most of compute nodes15:54
mgariepyi have pci passthrough for some gamer pci video card15:55
mgariepyand i spawn 120G vms with 16 core and 4 titanx card .15:56
mgariepywhen i boot the vm the ram is pre-allocated and it goes fast for about 50% then is slows down15:56
mgariepyon the 4.13 kernel i don't have any issues15:57
ionidoes the qemu version changes?15:57
ionior only the kernel15:57
mgariepyonly changing the kernel15:58
ionimaybe it's due to l1tf15:58
ionidid you tried to disable the protection?15:58
mgariepyi did try an older version of qemu and libvirt but with the 4.15 kernel i have the issue.16:00
mgariepylooking into it.16:00
pabelangerYay, stable/rocky deployed from zuul: https://object-storage-ca-ymq-1.vexxhost.net/swift/v1/a0b4156a37f9453eb4ec7db5422272df/logs/4c/4c91f44a55cd37a5c80ddf7e147ba0252465195c/post/packet-ci-cloud-deploy/0aa4dbb/logs/ara-report/16:02
odyssey4mejamesdenton it looks like this inventory.ini puts all the network_hosts into the right groups - allowing us to use automation to get everything done. :) https://gist.github.com/odyssey4me/5e9df7ebce1eff5e9321bfbaf6959a3f16:02
odyssey4mepabelanger :) yay!16:02
ionimgariepy, https://www.kernel.org/doc/html/latest/admin-guide/l1tf.html16:03
jamesdentonodyssey4me genius16:05
ionihmm16:05
ionithis sounds something i need for P->Q16:06
odyssey4mejamesdenton ok, so now I'll wait for my pike deploy to complete - then image it - and get on with building out the actual automation, but I think this has a fighting chance of working16:07
odyssey4meioni right now it's something that has to be done by hand, but thanks to some inspiration from jamesdenton and two bug reports, I think we can automate it16:08
odyssey4meFYI jamesdenton - I updated the gist with more complete instructions16:09
jamesdentonodyssey4me let me know how i can help. i will reimage multinode w/ Pike and upgrade when you have the first set of patches16:09
mgariepyioni, seems to help.16:09
odyssey4mejamesdenton yep, once I have something that works for me - more testing and ironing out kinks for other deployment setups would be nice - I'll be working with linuxbridge only, but you're able to try more configs out and provide feedback16:10
jamesdentonok yeah, i've been doing OVS16:11
jamesdentonwhich lines up with the recent biug report16:11
odyssey4meexcellent, so we'll be able to confirm it for at least those two options16:11
odyssey4meit seems to me that the procedure will be the same regardless, but it's better when confirmed16:11
ioniodyssey4me, do you mind giving me the bug reports?16:12
gillesMoDoes someone knows how to get rid of messages like "Instance 3689d4c2-20b1-438e-ab0d-429c1f352485 has allocations against this compute host but is not found in the database". After failed migrations...16:17
mgariepyioni, well. not so much after the reboot..16:17
mgariepythat's weird..16:17
ionimgariepy, i don't have ideas. sorry16:19
ioniit's the last version of package for 4.13? because all versions have fixes hardware bugs in procs16:20
odyssey4meioni I've noted them in the gist16:20
ioniodyssey4me, thanks16:20
odyssey4meok, let me take a peek at ansmith's things before I dive into that16:25
*** nurdie has joined #openstack-ansible16:31
mgariepyioni, latest 4.13 works fine without the l1tf setting16:36
mgariepyho, there is no l1tf patch in 4.13 it seems.16:37
*** udesale has quit IRC16:41
*** tosky has quit IRC16:43
*** tosky has joined #openstack-ansible16:44
mnaserjrosser: https://review.openstack.org/#/c/620378/2 if you got a few sec, i potato'd and almost +2d my own thing16:48
mnaser:p16:48
* jrosser looks16:52
jrosserlgtm16:54
*** cshen has quit IRC17:01
*** cshen has joined #openstack-ansible17:02
*** hamzy_ has quit IRC17:10
*** hamzy_ has joined #openstack-ansible17:10
ionimgariepy, interesting17:11
odyssey4meansmith ok, reviewed and made some suggestions to change it up to ensure it's opt-in, but that qdrouterd is preferred if the right inventory entries are found17:11
ionimaybe you didn't disable all the workarounds?17:11
odyssey4meansmith it'll also make it simpler to configure17:11
ansmithodyssey4me: thanks for checking it out, i will take action on your suggestions17:12
odyssey4meansmith the test result was success, so I think we're now down to refining it and finalising it :)17:12
odyssey4meansmith I also reviewed the depends-on patches17:12
mgariepyioni, :  noibrs noibpb nospectre_v2 nopti17:13
*** gisak has quit IRC17:26
odyssey4memnaser I'm not sure what to say about http://logs.openstack.org/65/624965/2/gate/openstack-ansible-deploy-aio_lxc-centos-7/6b66364/logs/ara-report/result/3c5c77ea-9d8d-429c-9b84-047aaa343bbb/ :/17:28
odyssey4mewhich is from https://review.openstack.org/#/c/624965/17:28
odyssey4methat is a very odd error17:29
*** gillesMo has quit IRC17:30
mnaserodyssey4me: i wonder if that was another change that broke it17:33
mnaserthe one that used loops in the incluhde_role rather than looping the include_role17:34
*** arxcruz is now known as arxcruz|off|next17:34
mnaserodyssey4me: https://review.openstack.org/#/c/607814/ this?17:34
*** arxcruz|off|next is now known as arxcruz|next_yea17:34
openstackgerritMerged openstack/openstack-ansible-rabbitmq_server master: upgrade: start service before applying policies  https://review.openstack.org/62037817:35
*** arxcruz|next_yea is now known as arxcruz|next_yr17:35
odyssey4memnaser then why do all other platforms work, and centos does not :p17:37
mnaservalid question17:37
mnaserlog digging17:37
odyssey4meof course all other platforms might be silently ignoring things17:37
odyssey4mehowever, that very same patch passed check earlier today17:37
mnaserDec 13 16:51:07 localhost cinder-volume: 2018-12-13 16:51:07.821 4729 ERROR cinder.cmd.volume [-] Configuration for cinder-volume does not specify "enabled_backends". Using DEFAULT section to configure drivers is not supported since Ocata.#033[00m17:38
odyssey4mehow nice :)17:38
mnaser"cinder-rtstool is not installed correctly: OSError: [Errno 2] No such file or directory"17:39
mnaserthats what is breaking it17:39
odyssey4meI thought that was fixed some time ago.17:40
mnaserwhat the hell17:40
mnaserhttps://review.openstack.org/#/c/607814/17:41
mnaserhttp://logs.openstack.org/14/607814/5/check/openstack-ansible-functional-centos-7/bfdf67e/logs/openstack/keystone/stestr_results.html17:41
mnaserthat failed17:41
mnaserbut reported as passed17:41
mnaserwe fixed it but that patch i remember wasnt working or needed some work to make it work with the env variables17:41
mnaserand i thought it was fixed because job passed17:41
mnaserbut tempest failed17:41
*** ianychoi has quit IRC17:42
odyssey4meodd though, that merged two days ago, and centos has been passing master patches since17:42
mnaser"Fail if tempest tests did not succeed"17:42
mnaser"All assertions passed"17:42
mnaseri think we messed up tempest17:42
*** dcdamien has quit IRC17:42
mnaserand even failing tempest shows up as passing17:42
mnaserhttp://logs.openstack.org/14/607814/5/check/openstack-ansible-functional-centos-7/bfdf67e/job-output.txt.gz17:42
mnasersame job doesnt fail but failed tempest17:42
odyssey4meah, must be https://github.com/openstack/openstack-ansible-os_tempest/commit/ea8ae41f6146d762bfe4b146a71dbda6740dcf0817:43
odyssey4methe 'always:' should not have been removed there17:44
odyssey4melemme push that up17:44
mnaserodyssey4me: darn, a bit of a poor yaml spacing made it seem confusing17:45
mnaserwe should maybe run a yaml linter in the future so the spacing is more visible17:46
mnaserodyssey4me: are you sure that is the issue though? it looks like the "Fail if tempest tests did not succeed" task ran anyways17:47
mnaserand asserted true, so it did return 017:47
odyssey4meyeah, I'm actually inclined to revert, because it seems to me that tempest returns differently with the subunit output17:48
mnaserhttps://github.com/openstack/tempest/blob/f9650269a32800fdcb873ff63f366b7bc914b3d7/tempest/cmd/run.py#L174-L18317:49
mnaserdoesnt look like it would affect too much17:49
openstackgerritJesse Pretorius (odyssey4me) proposed openstack/openstack-ansible-os_tempest master: Revert "Use tempest run for generating subunit results"  https://review.openstack.org/62507017:49
mnaserlemme ping mtreinish for a sec17:50
odyssey4meyeah, that's pretty odd - that all seems right17:50
odyssey4mehowever, perhaps the subunit generation is overriding the test result return code17:51
mnaserodyssey4me: that is my guess17:52
openstackgerritJesse Pretorius (odyssey4me) proposed openstack/openstack-ansible-os_tempest master: Return the 'always' block  https://review.openstack.org/62507317:52
odyssey4meif that is the case, then we found a bug in tempest which could affect everyone - so we did good. :)17:52
openstackgerritMerged openstack/openstack-ansible-galera_server stable/queens: Fix SSL support  https://review.openstack.org/62463317:52
openstackgerritMerged openstack/openstack-ansible-galera_server stable/queens: Fix Galera self-signed SSL functionality  https://review.openstack.org/62448217:52
mnaseri pinged mtreinish who is behing stestr but i dunno how much before a response17:53
mnaserwe wont be able to reach arxcruz|next_yr until next year =)17:53
odyssey4mewell, he's already approved the revert :p17:53
mnaseroh :p17:54
odyssey4mewe should probably get that merged asap, and we may have to deal with fallout in other roles as a result of it17:56
pabelangermnaser:17:59
pabelangermnaser: sorry, did you see we are now CDing openstack-ansible from zuul, in ansible-network18:00
pabelangerhttps://object-storage-ca-ymq-1.vexxhost.net/v1/a0b4156a37f9453eb4ec7db5422272df/logs/4c/4c91f44a55cd37a5c80ddf7e147ba0252465195c/post/packet-ci-cloud-deploy/0aa4dbb/18:00
pabelangerlogs/ara-report for osa bits18:00
*** kopecmartin is now known as kopecmartin|off18:01
pabelangerthe next step for me, is to turn https://github.com/ansible-network/packet-ci-cloud/blob/master/openstack_deploy/openstack_inventory.json into a static inventory file18:01
pabelangerbut, haven't figured out that step yet18:01
*** gyee has joined #openstack-ansible18:09
*** sep has quit IRC18:23
*** sep has joined #openstack-ansible18:24
*** ansmith has quit IRC18:30
*** ansmith has joined #openstack-ansible18:30
*** electrofelix has quit IRC18:34
*** ansmith has quit IRC18:35
*** electrofelix has joined #openstack-ansible18:44
*** dcdamien has joined #openstack-ansible18:45
*** shardy has quit IRC18:46
*** electrofelix has quit IRC18:51
goldenfriDoes anyone happen to have an example network interfaces file using netplan for OSA? Not looking forward to translating all of this.19:14
noonedeadpunkI had, but reverted everything back, as netplan does not support infiniband, and it's a case for me19:15
noonedeadpunkI'd said, it's not rreally complicated19:15
goldenfrithanks, thats good to know19:15
goldenfriyea not complicated but annoying19:15
goldenfrialso not sure how the eth12 stuff will translate19:16
noonedeadpunkI have example only for computing node though http://paste.openstack.org/show/737250/19:18
noonedeadpunkbut it's kinda container related, while you should just configure right bridges with netplan19:18
goldenfriOk thanks, yea it looks like you aren't creating the veth pair for eth1219:20
jamesdentonyou don't need to. that happens automatically19:20
noonedeadpunklxc should do it for you, based on it's interfaces config files19:21
goldenfrioh hrm19:22
jamesdentongoldenfri here is a two interface xample: http://paste.openstack.org/show/737252/19:22
*** ansmith has joined #openstack-ansible19:22
*** DanyC_ has joined #openstack-ansible19:24
*** DanyC has quit IRC19:24
goldenfriI use eth12 for my provider network19:24
jamesdentoneth12 is usually only used on an infra node in a neutron_agent container19:25
jamesdentonbut i have seen some users recently create a veth with one end named eth12, and the other end in br-vlan. Kinda lines up with the docs, but not the best approach19:26
*** weezS has quit IRC19:27
noonedeadpunkI'm not using eth12 at all, as it's required only by linuxbridge-agent (in my case) wich is bare-metal19:28
noonedeadpunkso yeah, every setup differs:)19:30
*** vnogin has joined #openstack-ansible19:30
*** DanyC_ has quit IRC19:33
*** fnpanic_ has joined #openstack-ansible19:33
fnpanic_hi19:33
fnpanic_i am still trying quickstart aio on ubuntu 18.0419:34
fnpanic_without luck19:34
fnpanic_it looks like the /etc/openstack_deploy is not populated19:34
fnpanic_i also needed to install ansible from universe prior running the bootstrap.sh19:35
*** vnogin has quit IRC19:35
goldenfrijamesdenton: yes that is how I am doing it. Was the only way I could get it to work at the time based the docs etc19:35
jamesdentonfnpanic_ you ran scripts/bootstrap-ansible.sh first?19:35
fnpanic_yes19:36
fnpanic_but it fails because ansible is not there19:36
*** rodolof has quit IRC19:36
jamesdentongoldenfri when you define the provider networks in openstack_user_config.yml, specify 'host_bind_override: <interface>' to use a physical interface of some kind. Then you bypass that eth12 veth and br-vlan19:36
fnpanic_here is the logput from setup-hosts19:36
fnpanic_http://paste.openstack.org/show/737257/19:36
*** rodolof has joined #openstack-ansible19:37
fnpanic_and /etc/openstack_deploy only hase ansible_facts dir19:37
fnpanic_so why is the bootstrap.sh failing on 18.0419:37
fnpanic_it was freshly installed :-)19:38
fnpanic_does ist produce a log?19:38
jrosserhave you run bootstrap-aio.sh?19:38
fnpanic_YES19:39
goldenfrijamesdenton: right but if I remember why I did it this way, my interfaces are not consistent across hosts so I use eth1219:39
fnpanic_yes19:39
fnpanic_sorry :-) caps was a fault19:39
jrosserif you don't have a populated /etc/openstack_deploy then something went wrong at the bootstrap-aio point19:40
fnpanic_i guess so19:40
jamesdentongoldenfri You can override those interfaces on a per-host basis if you want to. Or, the method you're using is fine for now. I'm guessing you're asking about netplan because you want something akin to a 'post-up' to configure that veth?19:40
fnpanic_the playbook finished without error19:41
fnpanic_i am re-running it19:41
fnpanic_localhost                  : ok=131  changed=70   unreachable=0    failed=019:43
*** cshen has quit IRC19:43
goldenfrijamesdenton: I'm just asking about netplan because I am starting to test out rocky and 18.04 and need to translate my queens config19:43
fnpanic_now the files are there19:44
fnpanic_wtf?19:44
jamesdentonfair enough. i ask because running post-up equivalent commands with netplan isn't really intuitive.19:44
fnpanic_this is crazy19:44
fnpanic_the first run was also without errors19:44
jamesdentonanyway, that link i sent earlier has an example of setting up the bridges and whatnot19:44
fnpanic_and now the files are there19:44
fnpanic_i am not sure what it is but i had a similar "problem" when i first cloned the git and run bootstrap-aio.sh it was not able to find the sshd role19:46
fnpanic_then i wiped the /opt/openstack-ansible on the same machine and it worked....19:47
*** pcaruana has quit IRC19:47
fnpanic_any idea?19:47
jrosseri think that was a bug which got fixed19:47
fnpanic_???19:47
fnpanic_what bug?19:47
fnpanic_i just re-run the script19:47
mgariepyioni, transparent_hugepage=never seems to help.19:48
goldenfriThanks, jamesdenton that will get me started, but yea I will need to figure out that pre-up/post-down stuff sometime soon19:48
*** cshen has joined #openstack-ansible19:49
jamesdentongoldenfri i ended up creating a service file to do it. Ghetto, but works.: http://paste.openstack.org/show/737258/19:49
jrosserfnpanic_: i had to dig a bit but it shouldnt affect you with a recent branch, ansible-sshd messed with it's repo and broke some older osa stuff19:53
jrosseranyway - the aio build process should be really solid because it is what's used for all the CI job gating19:54
fnpanic_jrosser: so fixed in master?19:54
jrosserno, becasue it only affects old releases19:54
jrosserhence not something to worry about19:54
fnpanic_i am confused.... old?19:54
jrosserpike/ocata19:54
fnpanic_mhhh but this was rocky i was using19:54
jrosseranyway the point is the aio should work out of the box19:55
jrosserit's run many times a day as part of the osa CI19:55
fnpanic_for me it is strange that when i have no ansible installed and run bootstrap-aio.sh it fails saying ansible-playbook command not found19:55
fnpanic_then i install it and it does not find the sshd role19:55
jamesdentondid you run them in the right order?19:55
jrosserdid you first do bootstrap-ansible.sh ?19:56
odyssey4mefnpanic_ I'm confused - there is no bootstrap.sh... what documentation are you reading - can you provide a URL?19:56
fnpanic_then i wipe the /opt/openstack-ansible19:56
fnpanic_and clone the git again19:56
*** tosky has quit IRC19:56
fnpanic_it works but does not copy the config to /etc/openstack_deploy19:56
fnpanic_when i run it a second time it works19:56
fnpanic_this is reproducable19:56
*** tosky has joined #openstack-ansible19:57
odyssey4mefnpanic_ cloning the git repo does not create anything other than a copy of the git repo - you appear to be missing steps19:57
fnpanic_tried it on two different vms and hosts (kvm and vbox)19:57
odyssey4meeither that or your over summarising19:57
jrosserfnpanic_: just hold on - you shoudnt have to install ansible yourself19:57
fnpanic_bootstrap-aio.sh19:57
fnpanic_yes19:57
odyssey4mefnpanic_ did you do bootstrap-ansible before that?19:57
jamesdentonfnpanic_ https://docs.openstack.org/openstack-ansible/rocky/user/aio/quickstart.html19:58
fnpanic_no19:58
fnpanic_bootstrap-aio.sh19:58
odyssey4meclone repo - bootstrap-ansible- - bootstrap-aio... the docs are pretty clear on that19:58
odyssey4meright, so the issue is that you have not done the ansible bootstrap as specified by the docs19:58
fnpanic_ohhhhhhhhh19:58
fnpanic_i see19:58
fnpanic_damn19:58
fnpanic_i am stupid and cannot read19:58
fnpanic_:-(19:58
odyssey4methe ansible bootstrap puts ansible down, and puts the roles in place19:58
fnpanic_sorry19:58
fnpanic_my brain was only reading bootstrap and did autocomplete the rest19:59
fnpanic_sorry19:59
jamesdentonit's all good19:59
odyssey4methe deploy guide also makes it clear: https://docs.openstack.org/project-deploy-guide/openstack-ansible/rocky/deploymenthost.html#install-the-source-and-dependencies19:59
fnpanic_you are absolutly right....20:00
fnpanic_also the quickstart AIO20:00
odyssey4me:) IT gives us a curse - we skim read, instead of reading thoroughly20:00
fnpanic_i earned a channel ban for the next 5 days....20:00
odyssey4methere is always too much waffle20:00
odyssey4me*but* thankfully we figured out what was going wrong, and you're a step forward :)20:01
fnpanic_you are very kind....20:01
fnpanic_even helping the dumpest people20:02
odyssey4melol, I am probably the dumbest of all :p20:02
fnpanic_btw the unbound role, does it install dns in lxc and then points the containers to the ip?20:02
fnpanic_dns resolver20:03
fnpanic_i mean20:03
*** hamzaachi has quit IRC20:03
fnpanic_where does it pick the upstream dns servers from? the infra hosts?20:04
odyssey4mefnpanic_ the default will modify /etc/hosts on the hosts to allow dns resolution everywhere... however if you add unbound hosts then I think it sets up unbound on the hosts and makes all containers use it... although logan- would have to explain better given he set it up and uses it20:07
odyssey4meI suspect there may be some bugs for queens+ for that setup, because it's not gate tested.20:08
odyssey4meanyway, I'm out for the night - will be back online tomorrow to work out a way to finalise that P2Q upgrade stuff for neutron for jamesdenton and ioni :)20:09
jamesdentonthanks odyssey4me! have a great night20:09
fnpanic_ok20:11
fnpanic_thanks20:11
fnpanic_have a great night20:11
fnpanic_we will stick with host files :-)20:11
prometheanfireodyssey4me: I think https://review.openstack.org/#/c/623240/ already merged :P20:13
*** priteau has quit IRC20:15
*** fnpanic_ has quit IRC20:18
openstackgerritMatthew Thode proposed openstack/openstack-ansible-haproxy_server master: Force force-tlsv12 only  https://review.openstack.org/62241120:20
*** nurdie has quit IRC20:25
*** nurdie has joined #openstack-ansible20:25
*** nurdie has quit IRC20:30
*** vnogin has joined #openstack-ansible20:31
*** vnogin has quit IRC20:35
*** ianychoi has joined #openstack-ansible20:59
*** cshen has quit IRC21:02
redkriegHas anyone experienced memcache import errors when trying to set up the IdP for a federated setup in openstack-ansible?  http://paste.openstack.org/show/737263/21:03
*** macza has joined #openstack-ansible21:09
*** markvoelker has joined #openstack-ansible21:11
*** cshen has joined #openstack-ansible21:24
*** DanyC has joined #openstack-ansible21:26
*** DanyC has quit IRC21:31
*** vnogin has joined #openstack-ansible21:32
*** ansmith has quit IRC21:33
*** vnogin has quit IRC21:37
*** DanyC has joined #openstack-ansible21:47
*** DanyC has quit IRC21:51
openstackgerritJonathan Rosser proposed openstack/openstack-ansible-haproxy_server master: Allow backend and backup node rise/fall parameters to be set  https://review.openstack.org/62512421:55
*** DanyC has joined #openstack-ansible21:59
*** markvoelker has quit IRC22:06
*** vnogin has joined #openstack-ansible22:33
*** vnogin has quit IRC22:37
*** lbragstad has quit IRC22:37
redkriegI was able to complete the federation setup by installing the following packages in the keystone container: python-keystoneclient python-memcache python-crypto22:39
redkriegIs there an accepted way to list additional packages for install in a specific container?22:39
*** dcdamien has quit IRC22:45
*** cshen has quit IRC22:57
*** DanyC has quit IRC22:58
*** cshen has joined #openstack-ansible23:00
*** noonedeadpunk has quit IRC23:02
*** noonedeadpunk has joined #openstack-ansible23:03
*** cshen has quit IRC23:08
openstackgerritMerged openstack/openstack-ansible-os_ceilometer master: gnocchi_resources override fixed  https://review.openstack.org/62295623:16
openstackgerritMerged openstack/openstack-ansible-os_ceilometer master: Add ability to override meters.yaml and event_definitions.yaml  https://review.openstack.org/62323923:16
redkriegI've hit another issue that I'23:23
redkriegm really not sure how best to work around.  keystone-manage doesn't seem to have saml2 support accessible: http://paste.openstack.org/show/737273/23:24
*** nurdie has joined #openstack-ansible23:26
*** tosky has quit IRC23:27
redkriegI've also noticed that keystone isn't even running on port 5000 on the SP.  The IdP seems fine.  My user_federation.yml from my SP: http://paste.openstack.org/show/737274/23:28
*** nurdie has quit IRC23:31
*** vnogin has joined #openstack-ansible23:33
*** vnogin has quit IRC23:38
*** thuydang has quit IRC23:56
*** thuydang has joined #openstack-ansible23:56

Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!