| *** Kurlee has joined #openstack-ansible | 00:00 | |
| *** pcaruana has quit IRC | 00:02 | |
| openstackgerrit | Mohammed Naser proposed openstack/openstack-ansible-lxc_hosts master: debian: add support https://review.openstack.org/650552 | 00:02 |
|---|---|---|
| *** nurdie has joined #openstack-ansible | 00:03 | |
| *** nurdie has quit IRC | 00:08 | |
| openstackgerrit | Mohammed Naser proposed openstack/openstack-ansible-lxc_hosts master: debian: add support https://review.openstack.org/650552 | 00:23 |
| *** nurdie has joined #openstack-ansible | 00:24 | |
| *** Kurlee has quit IRC | 00:29 | |
| *** nurdie has quit IRC | 00:29 | |
| openstackgerrit | Mohammed Naser proposed openstack/openstack-ansible-lxc_hosts master: debian: add support https://review.openstack.org/650552 | 00:39 |
| kplant | mnaser: removing those lines seems to have helped, deployment failed for other reasons but i'll figure that out | 00:45 |
| *** nurdie has joined #openstack-ansible | 00:45 | |
| *** Kurlee has joined #openstack-ansible | 00:46 | |
| *** nurdie has quit IRC | 00:50 | |
| *** kplant has quit IRC | 01:06 | |
| *** nurdie has joined #openstack-ansible | 01:07 | |
| *** Kurlee has quit IRC | 01:09 | |
| *** nurdie has quit IRC | 01:11 | |
| *** kplant has joined #openstack-ansible | 01:13 | |
| mnaser | kplant: cool! | 01:17 |
| mnaser | cloudnull: is there a historic reason behind why we pull down the ubuntu images and instead not do something like use debootstrap | 01:24 |
| *** nurdie has joined #openstack-ansible | 01:28 | |
| *** nurdie has quit IRC | 01:32 | |
| *** marst has joined #openstack-ansible | 01:37 | |
| *** cshen has joined #openstack-ansible | 01:41 | |
| *** cshen has quit IRC | 01:45 | |
| *** markvoelker has joined #openstack-ansible | 01:46 | |
| *** nurdie has joined #openstack-ansible | 01:49 | |
| *** nurdie has quit IRC | 01:53 | |
| cloudnull | mnaser we did use deboostrap for a long while in the icehouse, juno, kilo timeframe. it was just slow so we changed to pulling the lxc ubuntu image and then later the ubuntu base image | 02:06 |
| cloudnull | that said, debootsrap worked fine. | 02:06 |
| cloudnull | and assuming we could make it go faster it may be a good option to return to ? | 02:06 |
| mnaser | cloudnull: the reason is that Debian doesn’t publish a docker or lxc image or tar | 02:08 |
| cloudnull | both centos and suse can create container images using a similar process so maybe its something we can do across the board? | 02:09 |
| mnaser | cloudnull: im working on Debian support which has been trivial and already passing metal gate.. so that’s why I’m stuck | 02:09 |
| cloudnull | awesome ! | 02:09 |
| *** nurdie has joined #openstack-ansible | 02:10 | |
| mnaser | But yeah I’m a bit blocked now on this. Not sure the best path to go now | 02:10 |
| cloudnull | looks like https://github.com/debuerreotype/docker-debian-artifacts/tree/dist-amd64/buster - thats the official image repo | 02:15 |
| *** nurdie has quit IRC | 02:15 | |
| cloudnull | you should be able to use the rootfs | 02:15 |
| cloudnull | in queens we added the ability to use docker images directly in lxc hosts | 02:16 |
| cloudnull | found the official image from here https://github.com/docker-library/official-images/blob/master/library/debian | 02:17 |
| *** markvoelker has quit IRC | 02:19 | |
| mnaser | cloudnull: ooooh that's awesome | 02:21 |
| mnaser | okay ill get going there | 02:21 |
| cloudnull | we may be missing something to make debian go, however, it should just work with the rootfs.xz file | 02:22 |
| mnaser | cloudnull: yeah that should be enough to make me get over that step | 02:22 |
| mnaser | yay thanks! | 02:22 |
| cloudnull | https://github.com/cloudnull/os-ansible-deployment/tree/debian-compat | 02:23 |
| cloudnull | ^ funny enough | 02:23 |
| cloudnull | CRAZY OLD | 02:23 |
| mnaser | with Debian support someone would be able to have a full free stack from OS to openstack | 02:23 |
| mnaser | and free as in freedom :) | 02:24 |
| cloudnull | you just said almost exactly what I was typing :) | 02:24 |
| mnaser | https://review.openstack.org/#/q/topic:osa/debian-support | 02:24 |
| mnaser | https://review.openstack.org/#/c/650485/ | 02:24 |
| mnaser | passing and even 3 minutes faster than Ubuntu :P | 02:25 |
| cloudnull | I wanted to do that what feels like a million years ago, but didn't have time and enough tractions. very cool that its happening now | 02:25 |
| mnaser | well, I gotta admit, 99% of the work is done now | 02:25 |
| mnaser | most of these patches are mv ubuntu.yml debian.yml | 02:25 |
| cloudnull | ++ | 02:25 |
| cloudnull | that's awesome! | 02:25 |
| mnaser | we probably will move towards it and away from centos | 02:26 |
| mnaser | there's a lot of things driving me away from centos right now :< | 02:26 |
| mnaser | but I've been burned by ubuntu a lot with the lack of control, aka if something is broken, I just push up a bug and hope that someone takes action | 02:26 |
| mnaser | with Debian, we can just go and fix it. | 02:27 |
| mnaser | ok ill test that change here locally and try it out the link you provided, thanks a lot | 02:29 |
| cloudnull | ++ | 02:29 |
| cloudnull | "but I've been burned by ubuntu a lot with the lack of control, aka if something is broken, I just push up a bug and hope that someone takes action" - like a canonical bug and its sitting forever? | 02:30 |
| mnaser | cloudnull: yeah. | 02:31 |
| *** nurdie has joined #openstack-ansible | 02:31 | |
| mnaser | like in Debian world, given it's a community project, kinda like openstack, I can go and submit a patch for the packaging, or propose an update for a version, or a back ported patch, etc | 02:31 |
| *** hwoarang has quit IRC | 02:31 | |
| cloudnull | yeah , canonical is terrible about that stuff, unless you give them money, then they are only awful | 02:31 |
| mnaser | haha | 02:32 |
| mnaser | yeah, so that's kinda why as we consider moving away from centos, I want us to jump straight into Debian, cut everything out and be able to control our stack entirely | 02:32 |
| mnaser | the same right now is with centos, we are at mercy of redhat | 02:32 |
| mnaser | (except for RDO packages of course, which is nice that it's the other way around) | 02:32 |
| *** hwoarang has joined #openstack-ansible | 02:33 | |
| mnaser | aka RDO is upstream of RH's product, vs RHEL is upstream or CentOS | 02:33 |
| cloudnull | source builds for the win :) | 02:34 |
| cloudnull | unless your package issues are system and not openstack related then your in a bad way no matter what. | 02:35 |
| kplant | if you wanted to stay EL and have the upstream product of RHEL you could go fedora | 02:35 |
| kplant | however, not the greatest production idea | 02:35 |
| mnaser | kplant: yeah but running fedora for servers... *shrug* :p | 02:35 |
| kplant | exactly | 02:35 |
| *** nurdie has quit IRC | 02:36 | |
| mnaser | cloudnull: yeah, I have been unlucky lately with kernel issues | 02:36 |
| mnaser | we recently had something that when the openmanage service started, the entire server crashed :( | 02:36 |
| cloudnull | yeah... thats tough, we pin kernels in ubuntu, otherwise its kinda kernel roulette | 02:36 |
| cloudnull | but the fact you need to sucks | 02:36 |
| mnaser | and then R640's crashing the second a VXLAN packet egresses | 02:36 |
| cloudnull | :( | 02:37 |
| mnaser | when I install kernel-ml | 02:37 |
| mnaser | it works fine | 02:37 |
| mnaser | sooooo slowly but surely I was losing my sanity and getting tired and wanting to be more upstream-y | 02:37 |
| *** marst has quit IRC | 02:37 | |
| cloudnull | ++ debian is a good choice. | 02:37 |
| cloudnull | I still have personal servers running debian wheezy and squeeze, they've been rock solid | 02:38 |
| cloudnull | no need or want to upgrade them. | 02:39 |
| kplant | so it sounds like i shouldn't consider moving from tripleo+centos7 to osa+centos7 if you're considering dropping support for it :x | 02:39 |
| cloudnull | dropping support i wouldnt think so . | 02:40 |
| cloudnull | mnaser ? | 02:40 |
| mnaser | kplant: I don't think we're going to drop support | 02:40 |
| openstackgerrit | Merged openstack/openstack-ansible-galera_client master: debian: add support https://review.openstack.org/650488 | 02:41 |
| cloudnull | kplant sounds like mnaser is considering moving away from it but the community actively uses it / maintains it | 02:41 |
| mnaser | it's pretty stable and I think we'd probably continue to maintain it as much as we can, I'm a lot more comfortable keeping it alive than sure for example :p I know the inner movings of it | 02:41 |
| kplant | ah, you meant personally and not the project | 02:41 |
| kplant | my bad | 02:41 |
| mnaser | ah yes, sorry, I personally would move towards debian but I wouldn't rip out centos support, there's a few big centos users | 02:41 |
| openstackgerrit | Merged openstack/openstack-ansible-openstack_hosts master: debian: add support https://review.openstack.org/650487 | 02:41 |
| kplant | good to hear, i'm kind of afraid to let anyone at my org touch a tripleo deployment | 02:42 |
| kplant | it's way to easy to just irreversibly destroy it | 02:42 |
| mnaser | kplant: the team behind tripleo is great, I just don't think the architecture is that great. I was the PTL for the puppet openstack modules that tripleo uses :) | 02:42 |
| mnaser | but yeah, it has this terrifying bus factor | 02:42 |
| mnaser | heat stack-delete cloud | 02:43 |
| mnaser | "oops" | 02:43 |
| kplant | i think the usage of heat is a great idea | 02:43 |
| kplant | however.. being cuffed by redhat (no magnum because openshift exists) is just a major turn off as well | 02:43 |
| openstackgerrit | Mohammed Naser proposed openstack/openstack-ansible-lxc_hosts master: debian: add support https://review.openstack.org/650552 | 02:43 |
| mnaser | oh cool, you are looking for magnum? | 02:43 |
| kplant | yeah for sure | 02:43 |
| mnaser | we have it in prod, as well as deployed it for a few customers | 02:43 |
| mnaser | cloudnull: ^ that passed locally :D thanks for the help! | 02:44 |
| kplant | we wrote our own heat stack to leverage kubespray but magnum is just cleaner | 02:44 |
| cloudnull | woot! | 02:44 |
| kplant | i know cern is using it extensively as well, magnum | 02:44 |
| mnaser | kplant: yep, there's a lot of cool stuff happening with magnum | 02:45 |
| mnaser | cloudnull: btw I finally was able to do my "re-use deploy node" dream :P | 02:46 |
| mnaser | and it really wasn't hard | 02:46 |
| mnaser | https://review.openstack.org/#/c/649709/ | 02:46 |
| mnaser | oh you +W'd it | 02:46 |
| mnaser | oops | 02:46 |
| cloudnull | yuo! | 02:46 |
| cloudnull | yup | 02:46 |
| kplant | are you using magnum deployed by the osa role? or at least a fork of it? | 02:47 |
| kplant | seems pretty straightforward | 02:47 |
| mnaser | kplant: upstream everything here :-) | 02:47 |
| mnaser | so you'll get our prod deployment by using magnum role :) | 02:47 |
| cloudnull | :D ^ | 02:47 |
| cloudnull | ours too | 02:47 |
| kplant | v.nice | 02:48 |
| mnaser | cloudnull: do you know about PTG or not yet? :< | 02:48 |
| openstackgerrit | Merged openstack/openstack-ansible-galera_server master: debian: add support https://review.openstack.org/650490 | 02:50 |
| guilhermesp | debian goooooing mnaser nice work! | 02:52 |
| guilhermesp | how's going everyone? | 02:52 |
| * mnaser just hacking on Debian stuff till 12am to 3am maintenance window | 02:53 | |
| guilhermesp | warming up and having fun I bet! | 02:54 |
| openstackgerrit | Merged openstack/openstack-ansible-rabbitmq_server master: debian: add support https://review.openstack.org/650491 | 02:59 |
| openstackgerrit | Mohammed Naser proposed openstack/openstack-ansible-lxc_container_create master: debian: add support https://review.openstack.org/650553 | 03:00 |
| openstackgerrit | Mohammed Naser proposed openstack/openstack-ansible-tests master: debian: add jobs https://review.openstack.org/650533 | 03:01 |
| mnaser | ahaha warm up is a good way to put it :P | 03:01 |
| cloudnull | guilhermesp having a blast! | 03:01 |
| cloudnull | mnaser I'm still unsure about the PTG | 03:02 |
| cloudnull | though I'm threatening to drive to denver at this point :) | 03:02 |
| mnaser | cloudnull: how long is the drive? | 03:04 |
| cloudnull | ~13 hours | 03:04 |
| *** masterpe has quit IRC | 03:05 | |
| mnaser | I mean it's not fun but it's not wild | 03:06 |
| mnaser | im sure the drive might be gorgeous tho because of the mountains | 03:06 |
| cloudnull | ~15 hours if I pick up mordred on my way :) https://twitter.com/cloudnull/status/1113488784241438720 | 03:07 |
| kplant | if it's logistically feasible, i highly recommend perry's steakhouse in park meadows | 03:07 |
| kplant | we have a few offices in englewood and i make it a point to go everytime i'm out there | 03:07 |
| * cloudnull will check it out if i can | 03:08 | |
| mnaser | kplant: will you be at the summit? | 03:09 |
| kplant | unfortunately not | 03:09 |
| mnaser | aw, maybe in Shanghai the next one? :P | 03:10 |
| kplant | haha - i don't think i could swing an international trip on the ol' expense report | 03:11 |
| *** nurdie has joined #openstack-ansible | 03:12 | |
| openstackgerrit | Mohammed Naser proposed openstack/openstack-ansible-lxc_container_create master: debian: add support https://review.openstack.org/650553 | 03:12 |
| mnaser | kplant: where ya based out of? | 03:13 |
| kplant | charlotte, nc | 03:13 |
| kplant | usa | 03:13 |
| mnaser | oh, right next door to RH | 03:14 |
| mnaser | :P | 03:14 |
| kplant | yeah they're here all of the time, we're too close to the triangle | 03:14 |
| mnaser | :P | 03:14 |
| kplant | how about you? | 03:15 |
| * mnaser is in montreal canada | 03:15 | |
| kplant | oh neat, i have friends in boucherville and trois-rivieres | 03:15 |
| kplant | i grew up not terrible far away in new york | 03:16 |
| *** markvoelker has joined #openstack-ansible | 03:16 | |
| kplant | terribly** | 03:16 |
| mnaser | ah yes, we have some people out in Sherbrooke in the OSA team :P | 03:16 |
| *** nurdie has quit IRC | 03:17 | |
| kplant | unfortunately for you guys you've been too friendly so i may hang out more often and launch random questions about osa :> | 03:18 |
| mnaser | please do :D | 03:19 |
| mnaser | the cool thing is we don't necessarily have "osa devs" | 03:19 |
| mnaser | we're all deployers just using it as tooling | 03:19 |
| mnaser | so we're all in the same boat :P | 03:19 |
| guilhermesp | cloudnull 13 hours driving seems ok, let's go to the PTG :P | 03:20 |
| kplant | yeah that is quite nice | 03:20 |
| kplant | i would love to contribute once i feel i'm not so ignorant to the arch | 03:21 |
| guilhermesp | would be nice to have an osa user reporting the use for kplant ! Come more often when the rest of the team is around :) | 03:21 |
| kplant | i will for sure | 03:22 |
| kplant | the 10,000m view is i want to move to an openstack private cloud. i'm still in the research stage of how i want to deploy | 03:23 |
| mnaser | kplant: we have code review process so there's no need to worry, our playbooks and roles are pretty darn straightforward imho | 03:23 |
| mnaser | if you find something not making sense, we should probs document it | 03:24 |
| kplant | okay - that's fair | 03:25 |
| *** nurdie has joined #openstack-ansible | 03:34 | |
| mnaser | cloudnull: another 'history' lesson maybe but is there any reason why our gate/aio doesn't use connection_type=local for the physical host? | 03:37 |
| *** nurdie has quit IRC | 03:38 | |
| *** cshen has joined #openstack-ansible | 03:41 | |
| openstackgerrit | Merged openstack/openstack-ansible-os_glance master: debian: add support https://review.openstack.org/650493 | 03:41 |
| openstackgerrit | Merged openstack/openstack-ansible-os_neutron master: debian: add support https://review.openstack.org/650497 | 03:44 |
| openstackgerrit | Merged openstack/openstack-ansible-os_tempest master: debian: add support https://review.openstack.org/650499 | 03:44 |
| *** cshen has quit IRC | 03:45 | |
| openstackgerrit | Mohammed Naser proposed openstack/openstack-ansible-lxc_container_create master: debian: add support https://review.openstack.org/650553 | 03:46 |
| kplant | for centos would you recommend using the cloud sig repositories for rdo or leave it as the osa default pulling from source? | 03:47 |
| openstackgerrit | Mohammed Naser proposed openstack/openstack-ansible-lxc_hosts master: debian: add support https://review.openstack.org/650552 | 03:47 |
| kplant | when using tripleo, i noticed the cloud sig repositories moved very slowly for openstack | 03:47 |
| mnaser | kplant: are you deploying from distro pkgs or source | 03:48 |
| mnaser | if you're doing source, we deploy cloud-sig/rdo pkgs for dependencies | 03:48 |
| openstackgerrit | Merged openstack/openstack-ansible-os_nova master: debian: add support https://review.openstack.org/650496 | 03:49 |
| *** markvoelker has quit IRC | 03:49 | |
| kplant | right now i'm just doing source, was wondering if using rdo for all of it might make more sense | 03:49 |
| mnaser | kplant: we put in the repos we need | 03:49 |
| mnaser | so you shouldn't need to worry about it | 03:49 |
| openstackgerrit | Merged openstack/openstack-ansible-os_cinder master: debian: add support https://review.openstack.org/650495 | 03:51 |
| kplant | nice, okay | 03:51 |
| kplant | this is an easy one, do you think nspawn will be declared stable in the stein release of osa? | 03:52 |
| kplant | or maybe not so easy | 03:52 |
| mnaser | kplant: cloudnull says that he's deployed with nspawn and it works for him, but we don't have any gating for it, so you might run into some corner cases here and there | 03:53 |
| mnaser | but its like | 03:53 |
| mnaser | 99.9999% there | 03:53 |
| mnaser | so maybe just a few edge cases | 03:53 |
| openstackgerrit | Merged openstack/openstack-ansible-os_keystone master: debian: add support https://review.openstack.org/650492 | 03:54 |
| openstackgerrit | Merged openstack/openstack-ansible-os_keystone master: Set default address to wait for https://review.openstack.org/650186 | 03:54 |
| kplant | that sounds in line with what i've heard from a few talks, i think one of them might've even been yours | 03:54 |
| *** nurdie has joined #openstack-ansible | 03:55 | |
| mnaser | ouf.. maybe, I talk a lot :) | 03:55 |
| kplant | i think it'd make more sense for me to eat the potential of bugs in testing then going with somerthing i know is being replaced and guaranteeing myself more work in migrating off of lxc | 03:55 |
| mnaser | yeah, that'd probably be a great way to get started contributing too :) | 03:56 |
| kplant | yeah for sure | 03:57 |
| kplant | although i won't be doing anything exotic for a while | 03:57 |
| kplant | do you guys use aio for ci currently? | 03:59 |
| mnaser | kplant: yep | 03:59 |
| mnaser | kplant: https://review.openstack.org/#/c/650485/6 you can see for example here is my patch adding Debian support | 03:59 |
| mnaser | you can see all the jobs that we run | 03:59 |
| *** nurdie has quit IRC | 04:00 | |
| mnaser | job keywords are 'lxc' = lxc, 'metal' = runs on bare metal instead of inside containers, 'distro' = it uses distro packaging instead of build from source, 'Ceph' = deploy ceph as part of it, 'proxy' = runs job behind squid proxy to verify we can do things behind http proxy | 04:00 |
| kplant | is there any merrit in a multi-node ci environment, something similar to oooq? | 04:06 |
| mnaser | kplant: I am hoping we can land something like that this cycle | 04:13 |
| mnaser | we did some cleaning up which should make it possible :) | 04:14 |
| *** nurdie has joined #openstack-ansible | 04:16 | |
| *** nurdie has quit IRC | 04:20 | |
| *** nurdie has joined #openstack-ansible | 04:37 | |
| kplant | nice | 04:40 |
| kplant | that would be very helpful, i quickly wrote some crappy scripts and playbooks to spinup and teardown libvirt networks and guests to get familiar with osa | 04:41 |
| kplant | way easier than pxe booting baremetal over and over again | 04:41 |
| *** nurdie has quit IRC | 04:42 | |
| *** nurdie has joined #openstack-ansible | 04:58 | |
| *** nurdie has quit IRC | 05:03 | |
| openstackgerrit | Mohammed Naser proposed openstack/openstack-ansible-lxc_container_create master: debian: add support https://review.openstack.org/650553 | 05:04 |
| openstackgerrit | Mohammed Naser proposed openstack/openstack-ansible-lxc_container_create master: suse: move jobs to non-voting https://review.openstack.org/650558 | 05:04 |
| *** hwoarang has quit IRC | 05:04 | |
| openstackgerrit | Mohammed Naser proposed openstack/openstack-ansible-lxc_container_create master: Updated from OpenStack Ansible Tests https://review.openstack.org/645904 | 05:05 |
| openstackgerrit | Mohammed Naser proposed openstack/openstack-ansible-lxc_container_create master: fix tox python3 overrides https://review.openstack.org/606778 | 05:05 |
| *** hwoarang has joined #openstack-ansible | 05:06 | |
| mnaser | kplant: a 'hack' is just running ./scripts/gate-check-<something-i-cant-remember>.sh | 05:07 |
| mnaser | it'll just do everything same as our testing gate | 05:07 |
| mnaser | kplant: also, there is work in our ops repo around this thing called MNAIO, multi node aio | 05:08 |
| mnaser | I haven't played much with it but | 05:08 |
| mnaser | https://github.com/openstack/openstack-ansible-ops/tree/master/multi-node-aio | 05:08 |
| mnaser | it takes a bare metal machine and uses libvirt to build a cluster inside it | 05:08 |
| kplant | oh awesome | 05:08 |
| kplant | is there a way to globally override container_tech or set a default container_tech instead of changing it individually for each host? | 05:09 |
| kplant | i've tried container_vars: { container_tech: nspawn } in global_overrides but that doesn't seem to be effective | 05:11 |
| *** nurdie has joined #openstack-ansible | 05:19 | |
| *** nurdie has quit IRC | 05:23 | |
| *** hwoarang has quit IRC | 05:38 | |
| *** nurdie has joined #openstack-ansible | 05:40 | |
| *** cshen has joined #openstack-ansible | 05:41 | |
| *** hwoarang has joined #openstack-ansible | 05:43 | |
| *** nurdie has quit IRC | 05:45 | |
| *** cshen has quit IRC | 05:45 | |
| *** markvoelker has joined #openstack-ansible | 05:46 | |
| *** cshen has joined #openstack-ansible | 05:53 | |
| openstackgerrit | Mohammed Naser proposed openstack/openstack-ansible-haproxy_server master: handlers: reload instead of restart https://review.openstack.org/650561 | 05:56 |
| openstackgerrit | Mohammed Naser proposed openstack/openstack-ansible-haproxy_server master: debian: add ci and update meta https://review.openstack.org/650562 | 05:57 |
| *** cshen has quit IRC | 05:58 | |
| mnaser | kplant: whoops sorry I don't have this window up | 05:59 |
| mnaser | you can drop it into user_variables.yml | 05:59 |
| mnaser | every ansible run, everything in user_variables.yml overrides everything | 05:59 |
| kplant | i think i tried it in there as well, maybe not under container_vars though | 06:00 |
| mnaser | kplant: in user_variables.yml everything is like 'root' level | 06:00 |
| mnaser | so you can just add | 06:00 |
| mnaser | container_tech: nspawn | 06:00 |
| mnaser | and that's all, you should be good to go | 06:00 |
| *** nurdie has joined #openstack-ansible | 06:01 | |
| kplant | grep -rnw /etc -e _nspawn_hosts_container_image_url | 06:05 |
| kplant | derp - wrong window | 06:05 |
| *** nurdie has quit IRC | 06:06 | |
| *** cshen has joined #openstack-ansible | 06:10 | |
| openstackgerrit | Mohammed Naser proposed openstack/openstack-ansible-haproxy_server master: debian: add ci and update meta https://review.openstack.org/650562 | 06:15 |
| *** markvoelker has quit IRC | 06:20 | |
| *** cshen has quit IRC | 06:21 | |
| kplant | nspawn may be a little sick on stable/rocky: 'The base container image "centos-7-amd64" does not exist. Check the name and try again.' | 06:22 |
| *** nurdie has joined #openstack-ansible | 06:22 | |
| kplant | i'm about to pass out though - good night! | 06:22 |
| *** kplant has quit IRC | 06:22 | |
| *** nurdie has quit IRC | 06:27 | |
| *** nurdie has joined #openstack-ansible | 06:43 | |
| *** D3v1lRy has joined #openstack-ansible | 06:47 | |
| *** nurdie has quit IRC | 06:48 | |
| *** hwoarang has quit IRC | 06:49 | |
| *** hwoarang has joined #openstack-ansible | 06:51 | |
| *** nurdie has joined #openstack-ansible | 07:04 | |
| *** nurdie has quit IRC | 07:09 | |
| *** markvoelker has joined #openstack-ansible | 07:17 | |
| *** nurdie has joined #openstack-ansible | 07:26 | |
| *** nurdie has quit IRC | 07:30 | |
| *** luksky has joined #openstack-ansible | 07:36 | |
| *** jbadiapa has joined #openstack-ansible | 07:42 | |
| *** cshen has joined #openstack-ansible | 07:43 | |
| *** nurdie has joined #openstack-ansible | 07:47 | |
| *** jbadiapa has quit IRC | 07:47 | |
| *** markvoelker has quit IRC | 07:50 | |
| *** nurdie has quit IRC | 07:52 | |
| *** cshen has quit IRC | 07:52 | |
| *** hwoarang has quit IRC | 07:52 | |
| *** hwoarang has joined #openstack-ansible | 07:59 | |
| *** jbadiapa has joined #openstack-ansible | 08:00 | |
| *** nurdie has joined #openstack-ansible | 08:03 | |
| *** jbadiapa has quit IRC | 08:06 | |
| *** jbadiapa has joined #openstack-ansible | 08:08 | |
| *** nurdie has quit IRC | 08:10 | |
| *** hamzaachi has joined #openstack-ansible | 08:14 | |
| *** jbadiapa has quit IRC | 08:23 | |
| *** Weifan has joined #openstack-ansible | 09:41 | |
| *** cshen has joined #openstack-ansible | 09:46 | |
| *** markvoelker has joined #openstack-ansible | 09:47 | |
| *** Weifan has quit IRC | 09:47 | |
| *** luksky has quit IRC | 09:49 | |
| *** cshen has quit IRC | 09:51 | |
| *** Adri2000 has quit IRC | 10:04 | |
| *** Adri2000 has joined #openstack-ansible | 10:06 | |
| *** markvoelker has quit IRC | 10:21 | |
| *** cshen has joined #openstack-ansible | 10:31 | |
| *** brtknr has joined #openstack-ansible | 10:38 | |
| *** hwoarang has quit IRC | 10:39 | |
| *** hwoarang has joined #openstack-ansible | 10:40 | |
| *** cshen has quit IRC | 10:54 | |
| *** pcaruana has joined #openstack-ansible | 10:56 | |
| *** markvoelker has joined #openstack-ansible | 11:18 | |
| *** nicolasbock has joined #openstack-ansible | 11:33 | |
| *** nicolasbock has quit IRC | 11:37 | |
| *** nicolasbock has joined #openstack-ansible | 11:39 | |
| cloudnull | mnaser I have no historical context for that one. | 11:51 |
| *** markvoelker has quit IRC | 11:51 | |
| cloudnull | changing to connection "local" for aio1 would probably be benificial | 11:51 |
| *** hwoarang has quit IRC | 12:04 | |
| *** hwoarang has joined #openstack-ansible | 12:08 | |
| *** luksky has joined #openstack-ansible | 12:30 | |
| *** mmercer_ has quit IRC | 12:41 | |
| *** mmercer_ has joined #openstack-ansible | 12:42 | |
| *** cshen has joined #openstack-ansible | 12:50 | |
| *** cshen has quit IRC | 12:54 | |
| *** kplant has joined #openstack-ansible | 13:02 | |
| *** cshen has joined #openstack-ansible | 13:16 | |
| *** cshen has quit IRC | 13:22 | |
| *** hamzaachi has quit IRC | 13:24 | |
| *** klamath has joined #openstack-ansible | 13:28 | |
| *** markvoelker has joined #openstack-ansible | 13:49 | |
| *** cshen has joined #openstack-ansible | 14:06 | |
| *** markvoelker has quit IRC | 14:22 | |
| *** cshen has quit IRC | 14:59 | |
| *** tosky has joined #openstack-ansible | 15:10 | |
| *** hwoarang has quit IRC | 15:16 | |
| *** hwoarang has joined #openstack-ansible | 15:18 | |
| *** cshen has joined #openstack-ansible | 15:22 | |
| *** kplant has quit IRC | 15:27 | |
| *** kplant has joined #openstack-ansible | 15:27 | |
| *** cshen has quit IRC | 15:39 | |
| *** D3v1lRy has quit IRC | 16:07 | |
| logan- | wouldn't that break the use of the container aware connection plugin | 16:13 |
| logan- | also with connection=local, ansible would use the ansible-runtime venv python for all tasks targeting aio1 unless configured otherwise. | 16:16 |
| mnaser | logan-: I thought about that. I was thinking we set that connection type for the physical machines only | 16:17 |
| logan- | i typically try to use connection=local only for tasks that are meant to target the host running ansible (never any target hosts) even in a single node gate scenario because it changes the environment that the tasks are executed in so much (python interpreter, env inherited from ansible-playbook run, etc) | 16:17 |
| logan- | i'm not sure if that will work mnaser, the container aware connection plugin relies on ssh to the physical machine to attach to the container | 16:18 |
| logan- | i'm not sure if it supports connection=local for the physical machine hosting the containers | 16:18 |
| logan- | but considering it is a wrapper of the ssh connection plugin I kind of doubt it | 16:19 |
| mnaser | logan-: is the connection plugin for containers stays ssh but the physical host used localhosy | 16:19 |
| mnaser | If the ** | 16:19 |
| logan- | gotcha | 16:20 |
| mnaser | That way, physical host tasks happen using local and it pushes the container tasks through the ssh connection which will use container aware | 16:20 |
| mnaser | But yeah, I do agree that we do end up with a slightly different not 100% environment. | 16:20 |
| mnaser | I think our plugin has a lot of ways it can be optimized when I went over it the other day.. but then again we can just use mitogen | 16:21 |
| kplant | mnaser: i tried 'container_tech: nspawn' in user_variables.yml but it still attempted to deploy with lxc. the only way i could get osa to use nspawn was to use container_vars: { container_tech: nspawn } with each node | 16:30 |
| kplant | which then still led me to the missing image error - is there further config i need to do to use nspawn instead of lxc? | 16:31 |
| mnaser | kplant: weird. I may be wrong there then. | 16:31 |
| mnaser | The missing image shouldn’t really happen. | 16:32 |
| mnaser | Can you share the logs until that failure? | 16:32 |
| logan- | were you working from an inventory that was generated before you switched container_tech? | 16:32 |
| kplant | nope, each change i test with a fresh environment | 16:33 |
| kplant | i'll teardown and do it again to get fresh logs | 16:33 |
| *** luksky has quit IRC | 16:51 | |
| openstackgerrit | Logan V proposed openstack/openstack-ansible master: [DNM] Debug host /etc collection failures https://review.openstack.org/646009 | 16:56 |
| kplant | which logs in particular should i be looking at? | 17:04 |
| mnaser | kplant: just the playbook log | 17:15 |
| mnaser | I can have a look nd try to help | 17:15 |
| kplant | failure is different now but seems to be a similar cause: https://pastebin.com/e4Lp7c77 | 17:22 |
| kplant | Apr 07 17:22:54 osa-infra-01 systemd-nspawn[16445]: Failed to add new macvlan interfaces: Device or resource busy | 17:23 |
| kplant | there we go | 17:23 |
| mnaser | what interface do you have defined kplant ? | 17:27 |
| mnaser | aka what's your openstack_user_config looking like | 17:27 |
| kplant | it's pretty much the ceph-prod example: https://pastebin.com/2DNuSP54 | 17:29 |
| openstackgerrit | Mohammed Naser proposed openstack/openstack-ansible-lxc_container_create master: debian: add ci and update meta https://review.openstack.org/650587 | 17:30 |
| kplant | and i have matched each bridge to a nic on an untagged virt network | 17:31 |
| kplant | the nics in the nodes are: eth0: external unbridged, eth1: br-mgmt, eth2: br-vxlan, eth3: br-vlan, eth4: br-storage | 17:32 |
| jrosser | mnaser: i've been wondering if we are trying to bite off too much in one go with lxc -> nspawn+macvlan | 17:32 |
| mnaser | jrosser: I was thinking that was actually the cause of the complications | 17:33 |
| mnaser | if we went from lxc+bridge -> nspawn+bridge would have probably been better | 17:33 |
| jrosser | i'd be totally in favour of that ^ | 17:33 |
| mnaser | jrosser: when I tried to fix it up and clean it up, it was mostly macvlan related issue | 17:34 |
| mnaser | kplant: https://opendev.org/openstack/openstack-ansible/src/branch/master/etc/openstack_deploy/openstack_user_config.yml.aio-nspawn.j2 | 17:34 |
| mnaser | kplant: container_bridge should be an actual physical interface with nspawn (and using macvlan) | 17:35 |
| *** cshen has joined #openstack-ansible | 17:35 | |
| jrosser | mnaser: a bit of a nit on these https://review.openstack.org/#/c/650548/ they don't actually add the jobs, just update the meta? | 17:35 |
| kplant | ah, so my config is wrong | 17:35 |
| mnaser | jrosser: the depends-on adds the job so I figured it would kinda clarify that a little bit | 17:36 |
| jrosser | yeah, i was a bit confused by the commit message and went digging through the depends-on | 17:36 |
| mnaser | jrosser: I can change the message for the future ones? I'd rather not go over all of the patches again :p | 17:37 |
| mnaser | just to have CI spin for a commit message | 17:37 |
| jrosser | mnaser: no worries :) | 17:37 |
| mnaser | jrosser: I have a whole topic if you nothing better to do on a Sunday :) https://review.openstack.org/#/q/topic:osa/debian-support+is:open | 17:39 |
| jrosser | i'm on it | 17:39 |
| *** cshen has quit IRC | 17:39 | |
| jrosser | mnaser: this fedora job fails like every time https://review.openstack.org/#/c/650534/ | 17:41 |
| mnaser | jrosser: grr, let me try to hack on it.. | 17:42 |
| jrosser | and thats in the tree of depends-on | 17:42 |
| mnaser | yeah | 17:42 |
| mnaser | that will be necessary to unblock it | 17:42 |
| mnaser | Linking /home/zuul/.ansible/plugins/callback/ara to /home/zuul/src/git.openstack.org/openstack/ansible-hardening/.tox/functional/lib/python2.7/site-packages/ara/plugins/callbacks/ | 17:42 |
| mnaser | it fails then.. | 17:42 |
| mnaser | jrosser: hmm I think it might be failing here https://opendev.org/openstack/openstack-ansible-tests/src/branch/master/test-ansible-env-prep.sh#L255-L262 | 17:46 |
| openstackgerrit | Merged openstack/openstack-ansible-lxc_hosts master: debian: add support https://review.openstack.org/650552 | 17:53 |
| openstackgerrit | Jonathan Rosser proposed openstack/openstack-ansible-lxc_container_create master: Remove Xenial support and CI jobs https://review.openstack.org/650588 | 17:56 |
| *** ivve has quit IRC | 18:12 | |
| *** cshen has joined #openstack-ansible | 18:14 | |
| *** cshen has quit IRC | 18:20 | |
| jsquare | anyone seen this? | 18:21 |
| jsquare | fatal: [infra-03_gnocchi_container-385c7335]: FAILED! => {"msg": "The task includes an option with an undefined variable. The error was: 'venv_base_download_url' is undefined\n\nThe error appears to have been in '/etc/ansible/roles/os_gnocchi/tasks/gnocchi_install.yml': line 36, column 3, but may\nbe elsewhere in the file depending on the exact syntax problem.\n\nThe offending line appears to | 18:21 |
| jsquare | be:\n\n\n- name: Retrieve checksum for venv download\n ^ here\n"} | 18:21 |
| mnaser | jsquare: nope.. what release? | 18:24 |
| jsquare | stable/rocky | 18:24 |
| *** nicolasbock has quit IRC | 18:25 | |
| mnaser | jsquare: never seen that. how did you run the playbook? | 18:25 |
| jsquare | mnaser: we see that every single time | 18:26 |
| jsquare | mnaser: via setup-openstack.yml playbook | 18:26 |
| jsquare | mnaser: distro install btw | 18:27 |
| mnaser | jsquare: I think not all of our roles support distro install right now, it may seem gnocchi is one of them | 18:28 |
| mnaser | adding distro support is only a matter of defining some variables though | 18:28 |
| jsquare | hmm | 18:30 |
| mnaser | jsquare: I can help you out if you wanna pick that up, it should be relatively triival | 18:32 |
| openstackgerrit | Mohammed Naser proposed openstack/openstack-ansible-tests master: fedora: fix selinux configuration https://review.openstack.org/650590 | 18:34 |
| openstackgerrit | Mohammed Naser proposed openstack/ansible-hardening master: debian: drop stable job https://review.openstack.org/650534 | 18:35 |
| mnaser | jrosser: ^ that fixed/worked locally :) | 18:36 |
| jsquare | mnaser: yes, that'd be great | 18:37 |
| openstackgerrit | Jonathan Rosser proposed openstack/openstack-ansible-lxc_container_create master: Remove Xenial support and CI jobs https://review.openstack.org/650588 | 18:38 |
| mnaser | jsquare: this could be inspiration https://review.openstack.org/#/c/569457/ | 18:39 |
| mnaser | jrosser: https://review.openstack.org/#/c/650558/ to get your lxc_container_create changes to pass | 18:39 |
| jrosser | ahha | 18:39 |
| jrosser | i was just looking at how all the jobs in there are nv and that is sad | 18:40 |
| mnaser | jrosser: yeah... | 18:40 |
| jsquare | mnaser: thanks, will take a look | 18:45 |
| openstackgerrit | Merged openstack/openstack-ansible-lxc_container_create master: suse: move jobs to non-voting https://review.openstack.org/650558 | 19:00 |
| openstackgerrit | Merged openstack/openstack-ansible-lxc_container_create master: debian: add support https://review.openstack.org/650553 | 19:00 |
| openstackgerrit | Merged openstack/openstack-ansible-lxc_container_create master: fix tox python3 overrides https://review.openstack.org/606778 | 19:00 |
| openstackgerrit | Jonathan Rosser proposed openstack/openstack-ansible-lxc_container_create master: Remove Xenial support and CI jobs https://review.openstack.org/650588 | 19:01 |
| mnaser | jrosser: https://review.openstack.org/#/c/650590/ is the dependency to fix Ansible-hardening too | 19:12 |
| jrosser | thats just about to pass tests i think | 19:13 |
| mnaser | \o/ | 19:15 |
| mnaser | cools | 19:15 |
| jrosser | bingo | 19:18 |
| *** markvoelker has joined #openstack-ansible | 19:21 | |
| *** markvoelker has quit IRC | 19:25 | |
| *** markvoelker has joined #openstack-ansible | 19:35 | |
| *** hamzaachi has joined #openstack-ansible | 19:43 | |
| *** hwoarang has quit IRC | 19:50 | |
| *** luksky has joined #openstack-ansible | 19:52 | |
| *** hwoarang has joined #openstack-ansible | 19:55 | |
| *** cshen has joined #openstack-ansible | 20:16 | |
| *** cshen has quit IRC | 20:21 | |
| kplant | i keep getting different deployment errors on each clean attempt with the same config: https://pastebin.com/0KJ4w5GB | 20:29 |
| kplant | it's pretty much a rip-off of the ceph example, is there anything blatantly wrong with my config? | 20:30 |
| *** pcaruana has quit IRC | 20:47 | |
| *** hwoarang has quit IRC | 20:50 | |
| mnaser | kplant: do you havve specific errors | 20:51 |
| *** hwoarang has joined #openstack-ansible | 20:52 | |
| kplant | they seem to be different each time, the last one i got was one of the magnum containers failed to be added to the db | 20:54 |
| kplant | i'm ignoring any of the errors that might be caused by a mirror, but i've been getting some other random errors that don't appear to be related to a mirror | 20:55 |
| mnaser | kplant: I'd need to look at the playbook logs to help :) | 20:55 |
| kplant | yeah for sure, i didn't want to waste your time chasing something that wasn't consistent | 20:55 |
| kplant | i was hoping there was a blatant mistake that i made in my configs that was causing the unexpected behaviro | 20:55 |
| *** hamzaachi has quit IRC | 21:04 | |
| mnaser | kplant: its easier for me to see logs and track it back to a config being wrong than parsing a config myself =P | 21:07 |
| kplant | yeah... easier to know what you're looking for | 21:10 |
| kplant | i have it running again, if it fails i'll be more responsible to report what happened :] | 21:10 |
| *** djhankb has joined #openstack-ansible | 22:10 | |
| *** marst has joined #openstack-ansible | 22:13 | |
| *** cshen has joined #openstack-ansible | 22:17 | |
| *** nurdie has joined #openstack-ansible | 22:17 | |
| *** nurdie has quit IRC | 22:20 | |
| *** cshen has quit IRC | 22:21 | |
| *** tosky has quit IRC | 22:40 | |
| *** nurdie has joined #openstack-ansible | 22:40 | |
| *** markvoelker has quit IRC | 22:41 | |
| *** nurdie has quit IRC | 22:45 | |
| *** marst has quit IRC | 22:59 | |
| kplant | ah i remember why i didn't mention this one: TASK [os_magnum : Create DB for service] :: fatal: [infra1_magnum_container-edc682ce]: FAILED! => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} | 23:04 |
| kplant | should i rerun with -vvv ? | 23:04 |
| kplant | i'm going to remove the no_log from that task in /etc/ansible/roles/os_magnum/tasks/magnum_db_setup.yml - maybe that'll shed some insight | 23:12 |
| *** nurdie has joined #openstack-ansible | 23:22 | |
| *** luksky has quit IRC | 23:24 | |
| *** nurdie has quit IRC | 23:26 | |
| mnaser | kplant: im gonna go ahead and guess maybe you're missing password there | 23:30 |
| kplant | wouldn't it fail earlier without galera_root_password? | 23:36 |
| *** hwoarang has quit IRC | 23:40 | |
| *** hwoarang has joined #openstack-ansible | 23:43 | |
| mnaser | kplant: magnum database password | 23:45 |
| kplant | https://github.com/openstack/openstack-ansible-os_magnum/blob/master/tasks/magnum_db_setup.yml | 23:51 |
| kplant | it's failing on the first guy | 23:51 |
| kplant | hopefully without no_log i'll have an idea which variable it's unhappy with | 23:52 |
| mnaser | kplant: to speed things up | 23:53 |
| mnaser | you can run individual playbooks | 23:53 |
| mnaser | you can run openstack-ansible playbooks/os-magnum-install.yml | 23:53 |
| kplant | yeah that will speed this up immensely, thanks for the suggestion | 23:54 |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!