| *** vnogin has quit IRC | 00:26 | |
| openstackgerrit | Mohammed Naser proposed openstack/openstack-ansible-os_cinder master: zuul: Drop distro jobs for Xenial and switch to smoke tests https://review.openstack.org/591961 | 00:36 |
|---|---|---|
| openstackgerrit | Mohammed Naser proposed openstack/openstack-ansible-os_cinder master: Drop JSON logging for Cinder https://review.openstack.org/592085 | 00:36 |
| openstackgerrit | Mohammed Naser proposed openstack/openstack-ansible-os_cinder master: Fix cinder_environment_overrides https://review.openstack.org/589934 | 00:36 |
| openstackgerrit | Mohammed Naser proposed openstack/openstack-ansible-os_cinder master: Add target_helper to backend_defaults https://review.openstack.org/584604 | 00:36 |
| *** gregoryo has joined #openstack-ansible | 00:52 | |
| *** dave-mccowan has quit IRC | 01:02 | |
| *** dave-mccowan has joined #openstack-ansible | 01:03 | |
| cloudnull | evenings | 01:38 |
| *** schwicht has quit IRC | 01:55 | |
| *** spatel has joined #openstack-ansible | 02:09 | |
| spatel | Hey folks! | 02:09 |
| *** tinwood has quit IRC | 02:10 | |
| spatel | cloudnull: I want to set MTU 9000 for br-storage | 02:10 |
| spatel | I am reading this https://docs.openstack.org/project-deploy-guide/openstack-ansible/newton/app-networking.html | 02:10 |
| spatel | I can set container_mtu: "9000" in user_confi file | 02:11 |
| *** tinwood has joined #openstack-ansible | 02:11 | |
| spatel | after that do i need to run setup-host playbook? | 02:11 |
| openstackgerrit | Kevin Carter (cloudnull) proposed openstack/openstack-ansible-ops master: Add basic index templates https://review.openstack.org/592249 | 02:37 |
| cloudnull | spatel the lxc-container create playbook should be all you need to run | 02:37 |
| cloudnull | just make sure that the mtu is set accordingly on the underlying interfaces | 02:38 |
| spatel | Okay! you mean Host machine | 02:38 |
| spatel | cloudnull: thanks for the hint :) | 02:39 |
| spatel | Do i need to set MTU 9000 on lxcbr0 interface also? | 02:40 |
| spatel | cloudnull: ^^ | 02:40 |
| cloudnull | only if you want to | 02:40 |
| cloudnull | the storage traffic shouldnt travel over that interface | 02:41 |
| spatel | i think on infra* node there is not real use for MTU9000 | 02:41 |
| spatel | you are right! | 02:41 |
| cloudnull | for simplicity, when using jumbo frames, i set mtu 9000 on all hosts for a given interface | 02:42 |
| spatel | oh! wait in that case i don't need to do anything.. all i need to do just set MTT 9000 on compute node by hand | 02:42 |
| cloudnull | yea if there's no storage on the cinder-volume hosts | 02:42 |
| cloudnull | ip link set mtu 9000 dev $interface | 02:42 |
| spatel | I am not using Cinder-volume | 02:42 |
| cloudnull | and you're good to go | 02:43 |
| cloudnull | then save that in the ifcfg files so it persists post reboot | 02:43 |
| spatel | my application need CPU & memory, we have no use of storage | 02:43 |
| spatel | yup!! | 02:43 |
| cloudnull | should be an easy change :) | 02:43 |
| cloudnull | no need to change anything in OSA | 02:44 |
| spatel | glad!! | 02:44 |
| spatel | One more question! what is the best way to migrate all my VMware guest VM to Openstack instance? | 02:45 |
| spatel | I did google and all but look like no easy way :( | 02:45 |
| spatel | I have 22 VMware host and 200 VMs trying to move over openstack | 02:45 |
| cloudnull | you can export the vmdk, import the image into glance, and build a new vm? | 02:46 |
| spatel | but in that case i will have 200 images in glance right? | 02:46 |
| cloudnull | yes | 02:47 |
| cloudnull | no good way to directly convert the vms, that im aware of | 02:47 |
| spatel | that is what i found on google | 02:47 |
| cloudnull | yea, that wouldnt be a great experience | 02:47 |
| cloudnull | i wrote this tool a long time ago | 02:48 |
| cloudnull | https://github.com/cloudnull/InstanceSync | 02:48 |
| spatel | i am trying to avoid creating 200 images in glance with 4TB of storage for just images.. yike!! | 02:48 |
| cloudnull | might need to be updated for newer OS's | 02:48 |
| cloudnull | but that was used a lot when i wrote it to migrate servers to various places. | 02:49 |
| cloudnull | its basically fancy rsync | 02:49 |
| spatel | oh!! i like that | 02:49 |
| spatel | so it does like DD ? | 02:49 |
| cloudnull | create a VM in the new place that is of the same OS then run the script | 02:50 |
| cloudnull | I had a version tahat did DD but it had very inconsistent results | 02:50 |
| spatel | it will like disk to disk copy right? | 02:50 |
| cloudnull | this is literally rsync with a bunch of extra things | 02:50 |
| cloudnull | the target VM will need to be the same OS with a similar size | 02:51 |
| spatel | let me understand your script and try that out.. | 02:51 |
| cloudnull | its not a block level copy | 02:51 |
| spatel | its file level copy | 02:51 |
| cloudnull | though you could do that too. https://cloudnull.io/2011/12/dd-and-the-mighty-disk-destroyer-or-duplicator/ | 02:51 |
| cloudnull | bottom of the post | 02:52 |
| cloudnull | dd to gzip over ssh | 02:52 |
| cloudnull | assuming nothing interupts the transfer that does work well | 02:52 |
| spatel | hmm! interesting | 02:53 |
| cloudnull | the file level copy of the instancesync script was used a lot and had much better results. | 02:53 |
| cloudnull | the post is old as is the script so YMMV | 02:53 |
| spatel | i think i got clue from your script...! | 02:54 |
| spatel | i can live with that if it works :) | 02:54 |
| cloudnull | even if you dont use the script it should get you most of the way there. | 02:54 |
| cloudnull | yea live copying worked | 02:54 |
| cloudnull | during huricane sandy that was used to move a lot of workloads out of Datacenters in NYC | 02:55 |
| spatel | interesting.. | 02:55 |
| cloudnull | anyways that might be a better way forward, instead of doing the vmdk export | 02:56 |
| spatel | cloudnull: thanks man!! it was nice hint | 02:56 |
| cloudnull | anytime | 02:56 |
| spatel | no i am not going to import 200 vmdk.. that is really ugly way | 02:56 |
| spatel | I am going to do rsync that is safe and easy | 02:57 |
| *** spatel has quit IRC | 03:06 | |
| *** spatel has joined #openstack-ansible | 03:07 | |
| spatel | cloudnull: Good nite! thanks for your help! | 03:07 |
| cloudnull | you too | 03:08 |
| cloudnull | take care | 03:08 |
| *** spatel has quit IRC | 03:08 | |
| *** udesale has joined #openstack-ansible | 03:36 | |
| *** shyamb has joined #openstack-ansible | 03:40 | |
| *** Bhujay has quit IRC | 03:43 | |
| *** shyamb has quit IRC | 03:52 | |
| *** dave-mccowan has quit IRC | 04:04 | |
| *** poopcat has quit IRC | 04:04 | |
| *** mmalchuk_ has quit IRC | 04:18 | |
| *** shyamb has joined #openstack-ansible | 04:20 | |
| openstackgerrit | Kevin Carter (cloudnull) proposed openstack/openstack-ansible-ops master: Extend the embedded bootstap process https://review.openstack.org/592261 | 04:27 |
| openstackgerrit | Kevin Carter (cloudnull) proposed openstack/openstack-ansible-ops master: Make journalbeat install detect if it should install https://review.openstack.org/592264 | 04:30 |
| openstackgerrit | Merged openstack/openstack-ansible-ops master: Add basic index templates https://review.openstack.org/592249 | 04:30 |
| *** Bhujay has joined #openstack-ansible | 04:38 | |
| openstackgerrit | Kevin Carter (cloudnull) proposed openstack/openstack-ansible-ops master: Make journalbeat install detect if it should install https://review.openstack.org/592264 | 04:45 |
| cloudnull | the ceph checks in the integrated repo seem to be failing here - http://logs.openstack.org/44/587544/22/check/openstack-ansible-deploy-ceph-ubuntu-xenial/cb2a689/job-output.txt.gz#_2018-08-15_18_09_01_054177 - is that something that we're aware of or know what to do to resolve ? | 04:50 |
| cloudnull | if we can pacify ceph, evrardjp healthcheck change will get the gate going in a seemingly more robust and healthy way | 04:51 |
| cloudnull | https://review.openstack.org/#/c/587544/ | 04:51 |
| * cloudnull headed to bed | 04:51 | |
| cloudnull | but it'd be great if folks could have a look | 04:51 |
| openstackgerrit | Merged openstack/openstack-ansible-lxc_hosts master: Remove the checksum fills https://review.openstack.org/591880 | 04:53 |
| cloudnull | also https://review.openstack.org/#/c/592153/ - that would be good to get in so we can download and read journals | 04:55 |
| cloudnull | in the role tests | 04:55 |
| openstackgerrit | Merged openstack/openstack-ansible-ops master: Use the Ansible runtime venv for pw-token-gen.py https://review.openstack.org/592055 | 05:04 |
| *** Bhujay has quit IRC | 05:07 | |
| *** Bhujay has joined #openstack-ansible | 05:08 | |
| openstackgerrit | Merged openstack/openstack-ansible-ops master: Extend the embedded bootstap process https://review.openstack.org/592261 | 05:14 |
| *** mmalchuk has joined #openstack-ansible | 05:15 | |
| openstackgerrit | Merged openstack/openstack-ansible-ops master: Make journalbeat install detect if it should install https://review.openstack.org/592264 | 05:22 |
| *** pcaruana has quit IRC | 05:26 | |
| *** shyamb has quit IRC | 05:43 | |
| *** shyamb has joined #openstack-ansible | 05:46 | |
| *** DanyC has joined #openstack-ansible | 05:47 | |
| *** shyamb has quit IRC | 06:01 | |
| *** Bhujay has quit IRC | 06:02 | |
| *** Bhujay has joined #openstack-ansible | 06:03 | |
| *** DanyC has quit IRC | 06:12 | |
| *** mbuil has joined #openstack-ansible | 06:25 | |
| openstackgerrit | Markos Chandras (hwoarang) proposed openstack/openstack-ansible-plugins master: [DNM] - strategy: linear: Inspect the delegated host on delegated tasks. https://review.openstack.org/591672 | 06:32 |
| hwoarang | good morning | 06:42 |
| *** cshen has joined #openstack-ansible | 06:43 | |
| *** gkadam has joined #openstack-ansible | 06:48 | |
| *** pcaruana has joined #openstack-ansible | 06:48 | |
| evrardjp | morning hwoarang | 06:57 |
| openstackgerrit | Merged openstack/openstack-ansible-galera_client master: Use include_tasks instead of include https://review.openstack.org/585288 | 06:57 |
| jrosser | morning | 07:02 |
| openstackgerrit | Markos Chandras (hwoarang) proposed openstack/openstack-ansible-plugins master: [DNM] - strategy: linear: Inspect the delegated host on delegated tasks. https://review.openstack.org/591672 | 07:07 |
| openstackgerrit | Merged openstack/openstack-ansible-galera_server master: use include_tasks instead of include https://review.openstack.org/585267 | 07:08 |
| *** fghaas has joined #openstack-ansible | 07:16 | |
| openstackgerrit | Merged openstack/openstack-ansible-os_heat master: use include_tasks instead of include https://review.openstack.org/585265 | 07:18 |
| evrardjp | jrosser: did you had a look at the networking, or filed a bug for cloudnull ? | 07:20 |
| jrosser | it is fixed | 07:21 |
| jrosser | look at the systemd service role | 07:21 |
| jrosser | privatetmp | 07:21 |
| *** shardy has joined #openstack-ansible | 07:21 | |
| jrosser | evrardjp: i will fire off a fresh aio right now and try that all from scratch again | 07:22 |
| jrosser | also patch for lxcbr0 checksums should get rid of dmeag spam | 07:23 |
| evrardjp | jrosser: should help on the IO | 07:25 |
| evrardjp | jrosser: do you have that patch handy? | 07:25 |
| jrosser | gimme 10mins just on the tram :) | 07:26 |
| evrardjp | jrosser: does that look like I didn't watch the repos during my day off? :p | 07:26 |
| jrosser | netns is this according to cloudnull https://github.com/openstack/ansible-role-systemd_service/commit/00394f42dbf8f79f5346752fdca3951020a105df | 07:27 |
| evrardjp | jrosser: we need to get the freeze in ASAP | 07:27 |
| evrardjp | else the freeze will look like an ugly beast real fast | 07:28 |
| evrardjp | thinking about https://review.openstack.org/#/c/591957/1 that should get in | 07:28 |
| jrosser | checksums here https://review.openstack.org/#/c/591880/ | 07:29 |
| evrardjp | oh well I thought this merged and we had to revert it | 07:31 |
| evrardjp | good thing is not what I think | 07:31 |
| evrardjp | it is not* | 07:31 |
| evrardjp | yeah I will rebuild an AIO too | 07:31 |
| openstackgerrit | Markos Chandras (hwoarang) proposed openstack/openstack-ansible-plugins master: [DNM] - strategy: linear: Inspect the delegated host on delegated tasks. https://review.openstack.org/591672 | 07:36 |
| openstackgerrit | Merged openstack/openstack-ansible-os_horizon master: use include_tasks instead of include https://review.openstack.org/585299 | 07:38 |
| openstackgerrit | Markos Chandras (hwoarang) proposed openstack/openstack-ansible-plugins master: [DNM] - strategy: linear: Inspect the delegated host on delegated tasks. https://review.openstack.org/591672 | 07:45 |
| *** olivierbourdon38 has joined #openstack-ansible | 07:48 | |
| openstackgerrit | Chandan Kumar proposed openstack/openstack-ansible-os_tempest master: [WIP] Unifing tempest role for Tripleo https://review.openstack.org/591424 | 08:01 |
| chandankumar | evrardjp: odyssey4me mnaser ^^ | 08:01 |
| openstackgerrit | Jesse Pretorius (odyssey4me) proposed openstack/openstack-ansible stable/rocky: Revert "Revert "Update UPPER_CONSTRAINTS_FILE for stable/rocky"" https://review.openstack.org/592312 | 08:04 |
| *** keithmnemonic[m] has quit IRC | 08:11 | |
| *** RossSchulman[m] has quit IRC | 08:11 | |
| *** evrardjp[m] has quit IRC | 08:11 | |
| *** linshuicheng[m] has quit IRC | 08:11 | |
| *** flaviosr has joined #openstack-ansible | 08:11 | |
| openstackgerrit | Jesse Pretorius (odyssey4me) proposed openstack/openstack-ansible master: Update ansible to 2.5.7 https://review.openstack.org/592315 | 08:17 |
| openstackgerrit | Jesse Pretorius (odyssey4me) proposed openstack/openstack-ansible-tests master: Update ansible to 2.5.7 https://review.openstack.org/592317 | 08:18 |
| *** electrofelix has joined #openstack-ansible | 08:20 | |
| *** DanyC has joined #openstack-ansible | 08:26 | |
| openstackgerrit | Merged openstack/openstack-ansible-tests master: When archiving logs do not compress the journals https://review.openstack.org/592153 | 08:29 |
| *** rpittau has joined #openstack-ansible | 08:30 | |
| *** gregoryo has quit IRC | 08:41 | |
| evrardjp | chandankumar: reviewed | 08:42 |
| *** ianychoi_ has quit IRC | 08:42 | |
| evrardjp | thanks for having a go at this | 08:42 |
| *** mjwales has joined #openstack-ansible | 08:42 | |
| mjwales | Hi guys, when using an LDAP backend do you need to manually copy the tls_cacertfile into the keystone container? | 08:47 |
| *** shyamb has joined #openstack-ansible | 08:55 | |
| evrardjp | mjwales: it's better to distribute valid certs everywhere | 09:12 |
| odyssey4me | evrardjp: interesting error in the update of ansible: http://logs.openstack.org/15/592315/1/check/openstack-ansible-varstest-aio_lxc-ubuntu-xenial/36a7999/job-output.txt.gz#_2018-08-16_08_37_20_686301 | 09:17 |
| odyssey4me | I think it's because the action plugin can't be found, but am not sure. | 09:17 |
| chandankumar | evrardjp: odyssey4me regarding python-tempestconf https://github.com/openstack/python-tempestconf we use this to manage tempest config generation automatically | 09:19 |
| chandankumar | recently we have integrated with refstack-client https://github.com/openstack/refstack-client/commit/e1744ea1e4276de04c7e37399a96c506e24e9d63 for wider adoption | 09:19 |
| odyssey4me | ok, what does it mean to 'manage tempest configuration automatically' ? | 09:19 |
| chandankumar | odyssey4me: it discovers the minimal values from the deployed cloud for tempest.conf automatically if admin credentails are used it create resources for the same which can be used by tempest while testing | 09:20 |
| odyssey4me | OK, so you're suggesting that this can replace our templating of tempest.conf and our implementation of resources? | 09:21 |
| chandankumar | odyssey4me: I am not suggesting to replace it , let's have the support of both | 09:22 |
| odyssey4me | ok, sounds good - I guess it'll need some sort of task routing for that then | 09:22 |
| chandankumar | if user wants to use devstack provided tempest.conf generation https://github.com/openstack-dev/devstack/blob/master/lib/tempest that can use it with the role | 09:22 |
| odyssey4me | I suggest that you do that in a seperate patch to the one you've pushed up. | 09:23 |
| chandankumar | odyssey4me: since we are unifing I am thinking of reusing our role here https://github.com/openstack/tempest/tree/master/roles also | 09:23 |
| chandankumar | but step by step | 09:24 |
| jrosser | evrardjp: you seen this? http://logs.openstack.org/57/591357/2/check/openstack-ansible-deploy-aio_nspawn-opensuse-423/2be5d60/job-output.txt.gz#_2018-08-16_08_19_40_941737 | 09:24 |
| odyssey4me | chandankumar: interesting, what's the scope of those roles and their purpose? are they intended for use outside of zuul? | 09:24 |
| chandankumar | odyssey4me: currenlty those are tied to openstack zuul jobs and re-used where tempest gates are triggered with devstack | 09:25 |
| odyssey4me | aha | 09:25 |
| odyssey4me | so would you be porting what they do over - or would we consume those after they've been generalised? | 09:26 |
| chandankumar | odyssey4me: what i am thinking, let's make our piece better and then propose there to reuse it | 09:26 |
| odyssey4me | sure, we just need to bear in mind that testing inside zuul is different from running things in production | 09:28 |
| odyssey4me | although this is tempest, which kinda sites between things | 09:28 |
| odyssey4me | anyway - sure, anything that helps to improve and broadens the audience is good in my books | 09:29 |
| chandankumar | but the role should be written in such a way it can be used anywhere | 09:29 |
| chandankumar | irrespective of the enviornment | 09:29 |
| odyssey4me | yep | 09:29 |
| odyssey4me | what I was trying to say is that some things done for zuul are very zuul specific, and we shouldn't allow the role we use to do that - it needs to be universal | 09:31 |
| odyssey4me | so I think we're agreeing. :) | 09:31 |
| chandankumar | yup | 09:32 |
| evrardjp | odyssey4me: yeah it seems we need to remove static: | 09:35 |
| odyssey4me | evrardjp: that's a warning, not the error | 09:36 |
| evrardjp | jrosser: not sure what should I see | 09:36 |
| odyssey4me | the error appears to me that the module is config_template in that task, and ansible doesn't find it | 09:37 |
| evrardjp | odyssey4me: yeah | 09:37 |
| evrardjp | I haven't changed anything in the path | 09:38 |
| evrardjp | did we change things? | 09:38 |
| evrardjp | is that master? | 09:38 |
| evrardjp | wait let me check your patch | 09:38 |
| odyssey4me | nope, this is all working with the current ansible - but the patch update fails with that | 09:38 |
| odyssey4me | so it seems that something changed in ansible I guess | 09:38 |
| jrosser | evrardjp: from that point on it all goes horribly wrong with opensuse mirrors | 09:38 |
| evrardjp | yeah with a "minor" update | 09:39 |
| jrosser | another thing that cropped up yesterday, some of the locations looked like they had changed | 09:39 |
| evrardjp | odyssey4me: maybe there is something that needs to be looked over, is that it's properly trying to load it but fails to load | 09:39 |
| evrardjp | due to invalid code for that version | 09:40 |
| odyssey4me | that's possible too | 09:40 |
| odyssey4me | evrardjp: interesting that https://review.openstack.org/592317 passed just fine | 09:41 |
| odyssey4me | so I suspect that the issue is that we do not set where the plugins should be sourced in the integrated repo | 09:41 |
| evrardjp | odyssey4me: maybe https://github.com/openstack/ansible-config_template/blob/master/action/config_template.py#L660-L668 could be cherry picked to plugins and adjusted to 2.5.x | 09:41 |
| odyssey4me | in the role tests we do | 09:41 |
| evrardjp | if that's the root cause | 09:41 |
| evrardjp | I haven't looked | 09:41 |
| evrardjp | odyssey4me: that could be the cause | 09:42 |
| evrardjp | or | 09:43 |
| evrardjp | /etc/ansible/roles/ceph-ansible/library is having config-template too and conflicting | 09:44 |
| evrardjp | I think it needs a more thorough look | 09:45 |
| evrardjp | I am stabbing in the dark right now | 09:45 |
| *** cshen has quit IRC | 09:46 | |
| odyssey4me | yeah, it's clearly not urgent - I'll find some time later to look into it | 09:47 |
| evrardjp | odyssey4me: I think it's worth bumping to a more recent version anyway | 09:47 |
| odyssey4me | well, that's why I proposed it ;) | 09:47 |
| evrardjp | I think the most interesting thing right now, is how the hell did tempest work while the (almost similar, if not the same) ansible equivalent doesn't work | 09:48 |
| evrardjp | odyssey4me: I meant 2.6 | 09:48 |
| odyssey4me | Ah yes, there was a patch for that which also failed spectacularly. | 09:49 |
| *** olivierbourdon38 has quit IRC | 09:52 | |
| *** olivierbourdon38 has joined #openstack-ansible | 09:53 | |
| evrardjp | that's why I think we need to cherry pick the 2.6 support into plugins, or (better!) move to use the ansible-config_template repo | 09:53 |
| *** shyambiradar has joined #openstack-ansible | 09:57 | |
| noonedeadpunk | hi everyone | 09:58 |
| noonedeadpunk | evrardjp: thanks for review! odyssey4me: can you also take a look at https://review.openstack.org/#/c/586318/ one more time? | 09:58 |
| evrardjp | noonedeadpunk: sorry for the trouble I gave you. I think it's the best way forward to do this | 09:59 |
| evrardjp | we can even clean the role up later :) | 09:59 |
| noonedeadpunk | I'd be glad to help with it) | 10:00 |
| *** shyamb has quit IRC | 10:00 | |
| evrardjp | noonedeadpunk: oh then I have a few ideas :) | 10:00 |
| evrardjp | I can give you a few of the code guidelines and you typey typey? :D | 10:01 |
| noonedeadpunk | evrardjp: yes, why not? | 10:01 |
| noonedeadpunk | I hove some time for this | 10:01 |
| evrardjp | haha that was a joke but why not :) | 10:01 |
| evrardjp | let's merge your patch first, to avoid conflicts | 10:01 |
| noonedeadpunk | but still your ideas are highly appriciated :) | 10:02 |
| evrardjp | so yeah, one thing I think we should do there, is to cleanup the vars/main.yml of the symlinking to what I briefly checked (may need further analysis) all the symlinked files can be discovered by proper find module, with a proper glob, searching for *.py files excluding the _init | 10:04 |
| noonedeadpunk | as it seems that my vision can be too complicated without any reason on it | 10:04 |
| evrardjp | noonedeadpunk: well, simplicity is ultime complication :) | 10:04 |
| evrardjp | ultimate* | 10:05 |
| *** stuartgr has quit IRC | 10:05 | |
| evrardjp | with that find in horizon, there is no need to have conditionals | 10:06 |
| noonedeadpunk | evrardjp: ok, than I'll try to manage with this) sounds very reasonable, as this list of files should be maintained and it'll become more flexible in case of some custom panels | 10:08 |
| evrardjp | noonedeadpunk: precisely | 10:09 |
| evrardjp | if we make the pattern extensible, there is 0 limit to this, but I don;t think it should be extensible by default | 10:10 |
| evrardjp | else we always carry a ton of variables for no reason :) | 10:10 |
| evrardjp | less code, less to maintain :) | 10:11 |
| noonedeadpunk | and things like this https://github.com/openstack/openstack-ansible-os_horizon/blob/master/vars/main.yml#L35-L37 should be put as a separate task? | 10:13 |
| evrardjp | jrosser: https://review.openstack.org/#/c/587544/ shows that even with latest version of the cinder role that includes your patch to the environment, we have an race condition | 10:13 |
| evrardjp | noonedeadpunk: well I was thinking about that one | 10:14 |
| mjwales | evrardjp: thanks, will manually add the certs for our LDAP server to the keystone containers post deployment | 10:14 |
| evrardjp | it depends on where we are in policy in code, and auto discovery in horizon. That's where we need to check if this is still relevant | 10:14 |
| *** cshen has joined #openstack-ansible | 10:15 | |
| evrardjp | mjwales: I am not sure it's a great idea for your operations later, but I understand that urgencies happen. | 10:15 |
| evrardjp | noonedeadpunk: if there are other repos that are using that, then we should do a similar thing | 10:16 |
| evrardjp | else yeah, just add this to the list of files to link | 10:16 |
| jrosser | evrardjp: does the cinder role include my environment fix yet? | 10:17 |
| evrardjp | jrosser: horizon role could definitely benefit from mitogen's approach :D | 10:17 |
| evrardjp | jrosser: it does | 10:17 |
| evrardjp | well | 10:17 |
| evrardjp | I think it has | 10:17 |
| evrardjp | that's master, and unfrozen | 10:17 |
| evrardjp | and your patch merged right? | 10:18 |
| jrosser | you mean this? https://review.openstack.org/#/c/589934/ | 10:18 |
| jrosser | theres i ton of stuff i have in flight that just wont merge | 10:18 |
| evrardjp | oh still not | 10:18 |
| evrardjp | I just commented it in my run! | 10:18 |
| evrardjp | that's too bad | 10:18 |
| *** jrosser has quit IRC | 10:19 | |
| evrardjp | I mean I commented your patch out. | 10:19 |
| evrardjp | darn. | 10:19 |
| evrardjp | oh but you got a timeout exception in that same patch: http://logs.openstack.org/34/589934/7/check/openstack-ansible-functional-ubuntu-xenial/5cf4f46/job-output.txt.gz :) | 10:20 |
| evrardjp | kindra proves the point :) | 10:21 |
| evrardjp | jroll: to what I check, only the scheduler needs restart. | 10:21 |
| noonedeadpunk | evrardjp: got it | 10:21 |
| evrardjp | woops that was not for jroll but for jrosser | 10:21 |
| *** jrosser has joined #openstack-ansible | 10:25 | |
| *** shyambiradar has quit IRC | 10:35 | |
| *** shyambiradar has joined #openstack-ansible | 10:35 | |
| *** jrosser_ has joined #openstack-ansible | 10:38 | |
| *** jrosser has quit IRC | 10:38 | |
| *** jrosser_ is now known as jrosser | 10:38 | |
| *** cshen has quit IRC | 10:39 | |
| *** olivierbourdon38 has quit IRC | 10:40 | |
| *** Bhujay has quit IRC | 10:45 | |
| *** olivierbourdon38 has joined #openstack-ansible | 10:47 | |
| *** shyambiradar has quit IRC | 10:51 | |
| *** jrosser has quit IRC | 10:59 | |
| *** Laszlo-74 has joined #openstack-ansible | 10:59 | |
| *** jrosser has joined #openstack-ansible | 10:59 | |
| Laszlo-74 | hello | 11:00 |
| Laszlo-74 | we have problems running OSA. We frequently run out of memory. | 11:01 |
| Laszlo-74 | is there some recommendation about the amount of RAM to be used based on the number of hosts where we're installing openstack? | 11:02 |
| *** shyambiradar has joined #openstack-ansible | 11:02 | |
| odyssey4me | Laszlo-74: specifically with regards to the ansible deployment host, right? | 11:03 |
| noonedeadpunk | Laszlo-74: 3GB is pretty enough for me | 11:03 |
| jrosser | evrardjp: my aio+healthcheck ran all the way through ok | 11:03 |
| noonedeadpunk | but it's a minimum i think | 11:04 |
| Laszlo-74 | yes, I'm talking about the ansible deployment host | 11:06 |
| Laszlo-74 | that's where we run out of memory, and the ansible-pplaybook processes get killed by the OOM killer | 11:07 |
| noonedeadpunk | Laszlo-74: I had same problems with 2Gb of ram, but with 3Gb the problem has gone | 11:08 |
| odyssey4me | Laszlo-74: perhaps it's good to share the size of your deployment, as that has an impact | 11:09 |
| noonedeadpunk | yep, I was talking about absolutely basic one - with 3 computing nodes and 3 mgmt nodes | 11:10 |
| *** schwicht has joined #openstack-ansible | 11:14 | |
| Laszlo-74 | I'm talking about 57 physical servers of which we have 3 controllers, 3 ceph monitors, 9 ceph osd, 2 haproxy servers, 2 zabbix, 2 network nodes, and the rest (42) is compute | 11:15 |
| *** dave-mccowan has joined #openstack-ansible | 11:16 | |
| jrosser | odyssey4me: i did openstack-ansible playbooks/repo-build.yml -e 'repo_build_wheel_rebuild=true' and this happened..... http://paste.openstack.org/show/728182/ | 11:16 |
| jrosser | surprised to see it putting rax mirrors in there | 11:16 |
| odyssey4me | jrosser: is that for an AIO? | 11:17 |
| jrosser | yes | 11:17 |
| odyssey4me | outside of zuul, right? | 11:17 |
| jrosser | yep | 11:17 |
| *** shyambiradar has quit IRC | 11:17 | |
| odyssey4me | https://github.com/openstack/openstack-ansible/blob/master/tests/roles/bootstrap-host/tasks/prepare_aio_config.yml#L123-L133 | 11:18 |
| jrosser | ok i'll see where that leads | 11:18 |
| odyssey4me | it's a little script which finds the fastest infra mirror, then adds it as an index to help AIO's build faster | 11:18 |
| jrosser | for outside zuul? | 11:19 |
| odyssey4me | yep | 11:19 |
| jrosser | ooooh | 11:19 |
| odyssey4me | inside zuul we are given the closest mirror by the mirror script | 11:19 |
| openstackgerrit | Merged openstack/openstack-ansible stable/rocky: Temporarily use tempest master for all builds https://review.openstack.org/591357 | 11:20 |
| odyssey4me | outside zuul we seek one out | 11:20 |
| odyssey4me | it's perhaps not all that appropriate to assume that - but the AIO is a dev/test tool | 11:20 |
| odyssey4me | omg that patch finally merged! | 11:20 |
| odyssey4me | probably just in time to revert it :p | 11:20 |
| *** cshen has joined #openstack-ansible | 11:21 | |
| openstackgerrit | Jesse Pretorius (odyssey4me) proposed openstack/openstack-ansible stable/rocky: AIO: Tune the nova scheduler workers down https://review.openstack.org/591063 | 11:21 |
| openstackgerrit | Jesse Pretorius (odyssey4me) proposed openstack/openstack-ansible stable/rocky: AIO: Tune down the glance uwsgi processes https://review.openstack.org/591062 | 11:22 |
| openstackgerrit | Jesse Pretorius (odyssey4me) proposed openstack/openstack-ansible stable/queens: AIO: Tune down the glance uwsgi processes https://review.openstack.org/592347 | 11:22 |
| *** shyambiradar has joined #openstack-ansible | 11:23 | |
| *** schwicht has quit IRC | 11:25 | |
| *** galaxyblr has joined #openstack-ansible | 11:27 | |
| *** jrosser has quit IRC | 11:29 | |
| *** udesale has quit IRC | 11:30 | |
| *** Bhujay has joined #openstack-ansible | 11:35 | |
| *** olivierbourdon38 has quit IRC | 11:38 | |
| mjwales | evrardjp: yes less than ideal. Looking at the keystone role for LDAP it does not copy any certificates into the container so probably needs some work done there | 11:38 |
| *** olivierbourdon38 has joined #openstack-ansible | 11:39 | |
| *** jrosser has joined #openstack-ansible | 11:53 | |
| mnaser | chandankumar: is there a tool that generates tempest configuration which you use u | 11:54 |
| mnaser | Inside tripleo? | 11:54 |
| * mnaser is getting kinda frustrated with cinder role | 11:56 | |
| chandankumar | mnaser: https://github.com/openstack/python-tempestconf | 11:58 |
| chandankumar | mnaser: https://docs.openstack.org/python-tempestconf/latest/ | 11:58 |
| chandankumar | let us know if anything missing we can work together to add the support there | 11:58 |
| chandankumar | consumed in tripleo here https://github.com/openstack/tripleo-quickstart-extras/blob/master/roles/validate-tempest/templates/configure-tempest.sh.j2#L119 | 11:59 |
| openstackgerrit | Merged openstack/ansible-hardening stable/rocky: Update UPPER_CONSTRAINTS_FILE for stable/rocky https://review.openstack.org/590832 | 12:00 |
| chandankumar | odyssey4me: regarding suse rpm for tempestconf, I am pushing the spec file in openstack/rpm-packaging project | 12:00 |
| mnaser | chandankumar: does the tripleo team absolutely need to use it or could we just template things with Ansible instead to do the same thing? | 12:01 |
| *** schwicht has joined #openstack-ansible | 12:02 | |
| chandankumar | mnaser: tripleo is using the tool from long time, we are doing a new release which is a complete rewrite for tempest conf generation. I would keep the both way let's the user choose what they want | 12:02 |
| openstackgerrit | Merged openstack/openstack-ansible-os_searchlight stable/rocky: Update UPPER_CONSTRAINTS_FILE for stable/rocky https://review.openstack.org/590968 | 12:05 |
| chandankumar | mnaser: sorry I didnot get the template thing you mean tempest config generation without using that tool as currently done in os_tempest role? | 12:07 |
| *** zul has joined #openstack-ansible | 12:07 | |
| noonedeadpunk | guys, can anyone review https://review.openstack.org/#/c/586318/ ? | 12:10 |
| openstackgerrit | Merged openstack/openstack-ansible-os_ceilometer stable/rocky: Update UPPER_CONSTRAINTS_FILE for stable/rocky https://review.openstack.org/590901 | 12:13 |
| odyssey4me | mnaser: I think a tool that does discovery and then configures openstack for tempest testing is awesome. It'd be even nicer if that was in tempest itself, actually. | 12:14 |
| odyssey4me | For now, though, enabling both methods seems sensible to me. | 12:14 |
| openstackgerrit | Merged openstack/openstack-ansible-os_cloudkitty stable/rocky: Update UPPER_CONSTRAINTS_FILE for stable/rocky https://review.openstack.org/590907 | 12:15 |
| openstackgerrit | Merged openstack/openstack-ansible-os_sahara stable/rocky: Update UPPER_CONSTRAINTS_FILE for stable/rocky https://review.openstack.org/590965 | 12:17 |
| openstackgerrit | Merged openstack/openstack-ansible-os_aodh stable/rocky: Update UPPER_CONSTRAINTS_FILE for stable/rocky https://review.openstack.org/590892 | 12:18 |
| odyssey4me | mnaser: with regards to the cinder role - what do we know, and where are we stuck? | 12:18 |
| odyssey4me | We could perhaps disable some tests temporarily to allow patches to the role to move again while we work on figuring out the issue? | 12:19 |
| mnaser | odyssey4me: I’m getting into the office just now so I’m going to look at CI results | 12:20 |
| odyssey4me | mnaser: ok, let me know how I can help - when I see patches stuck for more than a week I get concerned about it falling behind and it causing downstream issues in the integrated repo. | 12:20 |
| mnaser | odyssey4me, chandankumar: my question was that why don’t we just bring the tempestconf functionality into a role inside os_tempest.. that way we don’t have that external dependency and it can work on all distros | 12:21 |
| guilhermesp | odyssey4me: yesterday we just changed to use smoke tests and re-enable voting to bionic functional tests https://review.openstack.org/#/c/591961/3 | 12:21 |
| guilhermesp | I'm looking at the results now too | 12:21 |
| mnaser | I started squashing things but it looks like one passes and others fail. With nothing interesting in logs | 12:21 |
| mnaser | odyssey4me: it would be cool if we had some sort of promotion system for integrated repo to always have latest working master hashes. That way master always works regardless of other stuff breaking | 12:22 |
| odyssey4me | mnaser: yep, evrardjp and I have discussed and will talk at the PTG about an idea to change how we release so that we just release the last working thing, rather than have to merge SHA pins and such. | 12:23 |
| mnaser | ++ | 12:23 |
| odyssey4me | Being held hostage to having to merge a patch before releasing is not good. Sometimes CI fails or is slow and it puts a lot of pressure on the release manager. | 12:24 |
| mnaser | It would be good to bring in some infra/Zuul experts to see what’s the best way of doing something like that | 12:24 |
| odyssey4me | yep, evrardjp has already been discussing options with the releases and infra/zuul crew | 12:24 |
| mnaser | I agree. And with a busy gate, you probably have 3 shots of retrying. | 12:24 |
| mnaser | In a day | 12:24 |
| odyssey4me | Well, of concern is also that we now have 20+ tests at once running, and any one of them could fail due to external factors - so it's like running a gauntlet | 12:25 |
| odyssey4me | So we really need to figure out what to do there. Making tests non-voting will result in things merging that break other things, so that's not really a solution. | 12:25 |
| odyssey4me | Reducing external dependencies can help, but our whole model currently uses external deps all over the place. | 12:26 |
| openstackgerrit | Merged openstack/openstack-ansible-os_barbican stable/rocky: Update UPPER_CONSTRAINTS_FILE for stable/rocky https://review.openstack.org/590895 | 12:27 |
| openstackgerrit | Merged openstack/openstack-ansible-tests stable/rocky: Update UPPER_CONSTRAINTS_FILE for stable/rocky https://review.openstack.org/591014 | 12:28 |
| odyssey4me | d34dh0r53 cloudnull when you're in, please take a peek at https://review.openstack.org/#/q/topic:file-backed-vms+status:open | 12:30 |
| *** schwicht has quit IRC | 12:37 | |
| openstackgerrit | Merged openstack/openstack-ansible-os_horizon stable/rocky: Update UPPER_CONSTRAINTS_FILE for stable/rocky https://review.openstack.org/590926 | 12:37 |
| *** aicilamd has joined #openstack-ansible | 12:42 | |
| mnaser | odyssey4me: most of those tests get skipped anyways and our failure rate was so low in puppet world back then | 12:43 |
| odyssey4me | mnaser: ok, my reference to tests was that we have 20+ functional builds going at any one time... not that each build has 20+ tests in it | 12:44 |
| mnaser | Oh yes, sorry | 12:45 |
| mnaser | Yeah, in that way it kinda sucks that one bad job slash node puts us a few hours behind | 12:45 |
| evrardjp | I think we need to think about changing our functional testing | 12:45 |
| evrardjp | that's why I proposed some healthchecks | 12:46 |
| evrardjp | because eventually these, if they are accepted, can be included into each roles, as tests/healtchecks.yml | 12:46 |
| evrardjp | then called independently | 12:46 |
| evrardjp | integration would simply be limited to a limited set of scenarios that are always ensuring the integrated passes | 12:47 |
| odyssey4me | healthchecks will make absolutely no difference to the problem I'm describing | 12:47 |
| evrardjp | then the integrated gates can be summarized to periodic jobs | 12:48 |
| evrardjp | or to a simple set | 12:48 |
| odyssey4me | The problem I'm describing is that we have a lot of very long running tests, and any one can fail due to a bad mirror - wasting what ends up very often being days of our time doing rechecks. | 12:48 |
| evrardjp | that's kinda my point | 12:48 |
| evrardjp | by doing this you separate into small bits | 12:48 |
| odyssey4me | Yeah, make it smaller and spread it over the repositories. | 12:49 |
| evrardjp | that's the first step | 12:49 |
| odyssey4me | Then also every build pulls in extra things it doesn't need - like cloning all the roles and all that. | 12:49 |
| chandankumar | mnaser: odyssey4me I am not sure about replacing tempestconf with ansible role would be a good idea, at the end of the day it is a python project consuming it from pip or git solve the issue, what we can improve is additional resources which needs to be created while runnign different scenario tests | 12:49 |
| odyssey4me | If we make that into a more pluggable system, then we'd be cutting down more opportunities for failure. | 12:50 |
| openstackgerrit | Merged openstack/openstack-ansible-os_tempest stable/rocky: Update UPPER_CONSTRAINTS_FILE for stable/rocky https://review.openstack.org/590977 | 12:50 |
| evrardjp | yeah that's kinda my second part. | 12:50 |
| evrardjp | I think we all agree that I can't properly express my vision in english, but I am sure you understood me :) | 12:50 |
| evrardjp | haha | 12:50 |
| odyssey4me | Then of course there's the debate over the use of extra repositories - rabbit, mariadb, etc. If we use less of those, there are less opportunities for failures too. | 12:51 |
| odyssey4me | less moving parts = less chance of failure | 12:51 |
| evrardjp | agreed | 12:51 |
| evrardjp | odyssey4me: don't forget for mariadb we were waiting for openstack changes | 12:51 |
| evrardjp | we couldn't bump to 10.2 | 12:51 |
| evrardjp | well | 12:52 |
| evrardjp | we tried | 12:52 |
| odyssey4me | If we flip to using nspawn everywhere, then we could also change more things to build faster and more reliably. | 12:52 |
| odyssey4me | orly? I didn't know about that? | 12:52 |
| evrardjp | more reliably I don't know but faster I agree | 12:52 |
| evrardjp | yeah, that's why there are so many questions about 10.1 and packages. | 12:53 |
| evrardjp | 10.1 is supposed to be dead, but our last bump was not successful. | 12:53 |
| evrardjp | I will try again 10.2 | 12:53 |
| odyssey4me | well, we can make it more reliable then by pre-building images for CI that have all the software - then the build just uses the latest images available | 12:53 |
| evrardjp | docker docker docker? | 12:53 |
| evrardjp | :) | 12:53 |
| odyssey4me | nspawn nspawn nspawn | 12:53 |
| evrardjp | hahah | 12:53 |
| evrardjp | harder to pronounce you can't win | 12:54 |
| odyssey4me | I reject your reality and substitute it with my own. :p | 12:54 |
| evrardjp | hahaha | 12:54 |
| *** cshen has quit IRC | 12:55 | |
| odyssey4me | lol, just as I thought - tempest 19.0.0 has just been proposed for release https://review.openstack.org/592276 | 12:58 |
| *** schwicht has joined #openstack-ansible | 13:06 | |
| *** shyambiradar has quit IRC | 13:08 | |
| *** shardy has quit IRC | 13:11 | |
| odyssey4me | evrardjp: ok, so do we want to go ahead and implement https://review.openstack.org/579371 so that we can key all the branch settings off it? | 13:11 |
| *** shardy has joined #openstack-ansible | 13:11 | |
| odyssey4me | If so, I'll fire up patches to all the roles to ensure the tox changes are done. | 13:12 |
| *** udesale has joined #openstack-ansible | 13:12 | |
| mnaser | chandankumar: i see. well, to be honest, i'm not *super* opposed to OSA leveraging python-tempestconf if that means that tripleo will consume it and therefore we'll have more hands on the tempest role (because, i think it'd be useful) | 13:14 |
| mnaser | odyssey4me: yeah, flipping to nspawn + finding a way to do cow for the containers, i think that would speed things up so much | 13:15 |
| *** olivierbourdon38 has quit IRC | 13:15 | |
| mnaser | odyssey4me, evrardjp: the one thing about prebuilding images, etc.. i start feeling like we're pretty much reinventing the whole ecosystem around docker already :X | 13:16 |
| *** mjwales has left #openstack-ansible | 13:17 | |
| *** vollman has joined #openstack-ansible | 13:17 | |
| odyssey4me | mnaser: I know what you're saying, but it's not quite the same. And if we commit to nspawn we can also commit to the ecosystem it has for building images, pushing them out, etc. | 13:17 |
| mnaser | odyssey4me: oh yes, absolutely, i see what you mean with that | 13:18 |
| mnaser | my (kinda) concern is how we might end up with such a vast range of systemd versions we'd need to deal with | 13:19 |
| mnaser | because centos | 13:19 |
| odyssey4me | My only concern at this stage is that CentOS isn't keeping up with the other platforms, so we'll be stuck with the lowest common demoninator's capabilities. :/ | 13:19 |
| mnaser | yup, my thoughts exactly | 13:19 |
| jrosser | i already had to change stuff to make the networkd bits work on bionic | 13:19 |
| jrosser | this will be the new norm i think | 13:20 |
| *** shardy has quit IRC | 13:20 | |
| evrardjp | odyssey4me: yeah | 13:23 |
| evrardjp | I know | 13:24 |
| evrardjp | (about tempest) | 13:24 |
| evrardjp | I am following that so... | 13:24 |
| *** shardy has joined #openstack-ansible | 13:25 | |
| evrardjp | odyssey4me: I don't see how https://review.openstack.org/#/c/579371/6 is linked to tempest though :p | 13:25 |
| evrardjp | mnaser: that's my point. If ppl want to use pre-built images, they can already. | 13:25 |
| odyssey4me | evrardjp: https://review.openstack.org/#/c/579371/6 is not linked to tempest - it's linked to actually testing rocky, right now all our rocky branches are testing master, not rocky | 13:29 |
| evrardjp | yeah. | 13:30 |
| evrardjp | well | 13:30 |
| evrardjp | I have patches that are pending merging | 13:30 |
| evrardjp | we need to get them in | 13:30 |
| odyssey4me | the question I have is whether we want to go ahead with that change so that all tests key off the .gitreview file | 13:31 |
| odyssey4me | to go ahead though we need to change the tox config in every repo | 13:31 |
| *** nicolasbock has joined #openstack-ansible | 13:32 | |
| evrardjp | I am fine with that change nowadays | 13:32 |
| evrardjp | we are so close to have the freeze in stable/rocky, and everything is already keyed to stable branch | 13:32 |
| evrardjp | we will just backport it | 13:32 |
| evrardjp | as we'll do | 13:33 |
| evrardjp | sorry it took so long | 13:33 |
| evrardjp | because now we all have a duty to backport if necessary | 13:33 |
| odyssey4me | so is that a yes, I should go ahead and push up the tox changes? | 13:33 |
| evrardjp | yes. | 13:33 |
| odyssey4me | okie dokey - will do that now | 13:33 |
| *** KeithMnemonic has joined #openstack-ansible | 13:34 | |
| evrardjp | odyssey4me: haha you already pushed the revert of the revert of the upper constraints branch! | 13:37 |
| evrardjp | team team team | 13:37 |
| mnaser | ok so | 13:40 |
| mnaser | cinder-api gets the actual create request | 13:40 |
| mnaser | but it never makes it to cinder-volume | 13:40 |
| odyssey4me | mnaser: does the scheduler get it? | 13:42 |
| mnaser | oh good question, i forgot about the scheduler | 13:42 |
| mnaser | [e30ca7ae-598b-4403-92f2-ad0458f6535e] AMQP server on 10.1.1.101:5672 is unreachable: [Errno 32] Broken pipe. Trying again in 1 seconds.: error: [Errno 32] Broken pipe | 13:42 |
| mnaser | [req-10d2f84f-4610-4832-9e0f-da53a21ed1e2 - - - - -] [e30ca7ae-598b-4403-92f2-ad0458f6535e] Reconnected to AMQP server on 10.1.1.101:5672 via [amqp] client with port 33706. | 13:42 |
| mnaser | this is when happening on opensuse | 13:42 |
| mnaser | this is even happening on opensuse* | 13:42 |
| mnaser | it looks like it loses connectivity, which means the 'volume created' message gets lost i think | 13:43 |
| mnaser | http://logs.openstack.org/34/589934/7/check/openstack-ansible-functional-opensuse-423/85cd5a8/logs/host/cinder-volume.service.journal.log.txt.gz look at the end here | 13:43 |
| odyssey4me | hmm, I wonder whether the inventory there is using correct ip addresses | 13:44 |
| odyssey4me | and consistent with the correct ranges for the cidr | 13:44 |
| odyssey4me | or perhaps there's a conflicting address | 13:45 |
| mnaser | it has to.. this fails non deterministically | 13:45 |
| mnaser | it would just not work at all if it wasn't working i think | 13:45 |
| jrosser | why does it have a 10.x address at all? | 13:46 |
| mnaser | i think in ci, we bring up this small mini network with 10.x ips for the different containers | 13:46 |
| jrosser | unless it's not an AIO of course | 13:46 |
| mnaser | in the role ci | 13:46 |
| jrosser | oh right, sure | 13:46 |
| mnaser | http://logs.openstack.org/34/589934/7/check/openstack-ansible-functional-opensuse-423/85cd5a8/logs/ethtool-eth0-cfg.txt.gz | 13:47 |
| mnaser | we should install ethtool in our bootstrap | 13:47 |
| jrosser | i have this for the integrated build https://review.openstack.org/#/c/590424/ | 13:47 |
| *** throwsb1 has joined #openstack-ansible | 13:47 | |
| jrosser | perhaps the same thing needs to go on the tests to just check that the network isnt borked from the word go | 13:48 |
| mnaser | jrosser: the thing is, the rabbitmq logs show that it goes up and recieves connetions | 13:49 |
| mnaser | so something blips at some poit | 13:49 |
| mnaser | point | 13:49 |
| mnaser | at least we're almost never failing on deploys | 13:50 |
| mnaser | its just tempest | 13:50 |
| mnaser | http://logs.openstack.org/85/592085/5/check/openstack-ansible-functional-distro_install-ubuntu-bionic/8e29416/logs/host/cinder-volume.service.journal.log.txt.gz#_Aug_16_01_35_32 | 13:51 |
| odyssey4me | these are the vars used by lxc for the nat interface: https://github.com/openstack/openstack-ansible-tests/blob/master/test-vars.yml#L69-L72 | 13:51 |
| mnaser | it's always the same issue, the connection gets reset | 13:51 |
| mnaser | happens across all operating systems | 13:51 |
| odyssey4me | 10.1.1.1 is the br-mgmt bridge, and the containers use 10.1.1.10x | 13:55 |
| odyssey4me | so no interference there | 13:55 |
| mnaser | i wonder if its the fact we restart the service a few times | 13:56 |
| mnaser | cause it looks like we're dropping the systemd file | 13:56 |
| mnaser | enable it | 13:56 |
| mnaser | it spins trying to keep starting up | 13:56 |
| mnaser | and then somehow the disconnect/connect again from all the restarts is messing up rabbitmq.. | 13:57 |
| mnaser | cause it even goes up before the lvms are up and raises and exception and shutsd own again | 13:57 |
| odyssey4me | perhaps we need to change some ordering there, or make the unit depend on another service | 13:58 |
| mnaser | i'm thinking of moving the systemd include_role out to the end of main.yml | 13:58 |
| mnaser | but we flush handlers once | 13:58 |
| *** throwsb1 has quit IRC | 13:58 | |
| mnaser | im assuming we flush handlers to bring up the api service so we can create teh cinder backends | 13:58 |
| odyssey4me | yes | 13:59 |
| mnaser | cinder_lvm_config.yml is totally not needed to run while things are up | 13:59 |
| mnaser | so i can move that one back up, say after pre install | 13:59 |
| odyssey4me | it seems sensible to prepare the underlying storage prior to setting up the service that relies on it | 14:00 |
| mnaser | spam inc | 14:04 |
| openstackgerrit | Mohammed Naser proposed openstack/openstack-ansible-os_cinder master: zuul: Drop distro jobs for Xenial and switch to smoke tests https://review.openstack.org/591961 | 14:04 |
| openstackgerrit | Mohammed Naser proposed openstack/openstack-ansible-os_cinder master: Drop JSON logging for Cinder https://review.openstack.org/592085 | 14:04 |
| openstackgerrit | Mohammed Naser proposed openstack/openstack-ansible-os_cinder master: Fix cinder_environment_overrides https://review.openstack.org/589934 | 14:04 |
| openstackgerrit | Mohammed Naser proposed openstack/openstack-ansible-os_cinder master: Add target_helper to backend_defaults https://review.openstack.org/584604 | 14:04 |
| mnaser | it'll at least reduce the amount of weird restarts happening | 14:04 |
| openstackgerrit | Jesse Pretorius (odyssey4me) proposed openstack/ansible-config_template master: Use the TESTING_BRANCH env var for constraints https://review.openstack.org/592476 | 14:04 |
| mnaser | you know what would be nice | 14:05 |
| chandankumar | mnaser: sure | 14:05 |
| mnaser | `when_or` for ansible conditionals that are OR'd instead of AND'd | 14:05 |
| openstackgerrit | Jesse Pretorius (odyssey4me) proposed openstack/ansible-role-python_venv_build master: Use the TESTING_BRANCH env var for constraints https://review.openstack.org/592477 | 14:06 |
| *** mmalchuk has quit IRC | 14:06 | |
| evrardjp | jrosser: ok I have my machine up now, and I had a few minutes to check | 14:07 |
| evrardjp | it seems the namespaces are better now | 14:07 |
| mnaser | chandankumar: but the initial review looks good, evrardjp gave some good reviews on it, so if you can follow up with those | 14:07 |
| evrardjp | so I can actually do commands | 14:07 |
| mnaser | i'm excited | 14:07 |
| evrardjp | jrosser: what I noticed is that traffic flows perfectly fine from outside to the veth, then to bridges, and fails into the router namespace. The iptables rules are well set, but the instance doesn't seem to be pingable on its local (private) network | 14:08 |
| evrardjp | jrosser: will dig deeper | 14:08 |
| jrosser | evrardjp: i ran it this morning and the heathcheck passed | 14:09 |
| evrardjp | jrosser: oh interesting | 14:09 |
| evrardjp | my next step was to tap into the tap interface of the vm | 14:10 |
| jrosser | nothing but cloudnulls latest tweek on your healthcheck patch, on xenial/master | 14:10 |
| openstackgerrit | Jesse Pretorius (odyssey4me) proposed openstack/ansible-role-systemd_mount master: Use the TESTING_BRANCH env var for constraints https://review.openstack.org/592478 | 14:10 |
| chandankumar | mnaser: I will update the patch tomorrow | 14:10 |
| mnaser | chandankumar: thank you very much :) | 14:10 |
| evrardjp | jrosser: latest tweak? | 14:10 |
| evrardjp | HAHA | 14:11 |
| evrardjp | that's it | 14:11 |
| openstackgerrit | Jesse Pretorius (odyssey4me) proposed openstack/ansible-role-systemd_networkd master: Use the TESTING_BRANCH env var for constraints https://review.openstack.org/592479 | 14:11 |
| jrosser | see he pushed a few fixes https://review.openstack.org/#/c/587544/ | 14:11 |
| evrardjp | yeah | 14:11 |
| evrardjp | ofc | 14:11 |
| evrardjp | I was running my patch 18 | 14:11 |
| evrardjp | ofc this is the failure reason: | 14:11 |
| evrardjp | item[1]. was not there | 14:11 |
| evrardjp | so dhcp was false | 14:11 |
| evrardjp | so the private net doesn't ahve the ip | 14:12 |
| evrardjp | so that's why it can't be pinged on private network | 14:12 |
| evrardjp | silly me | 14:12 |
| jrosser | well good job really in a way | 14:12 |
| *** Bhujay has quit IRC | 14:12 | |
| jrosser | becasue it made us look and find the netns thing | 14:12 |
| openstackgerrit | Jesse Pretorius (odyssey4me) proposed openstack/ansible-role-systemd_service master: Use the TESTING_BRANCH env var for constraints https://review.openstack.org/592480 | 14:12 |
| odyssey4me | evrardjp: that role sha pin for the RC will likely require updates to all the role sha's now to pick up those changes | 14:13 |
| evrardjp | odyssey4me: yup | 14:14 |
| evrardjp | but now it's the regular process of bumping shas | 14:14 |
| jrosser | odyssey4me: got a momoent for repo build things? | 14:14 |
| evrardjp | jrosser: without netns command it was freaking harder :p | 14:15 |
| odyssey4me | well, there's no role pin on the rocky branch right now | 14:15 |
| odyssey4me | and the pins there are fro prior to the ns fix | 14:15 |
| jamesdenton | evrardjp rookie | 14:15 |
| evrardjp | jamesdenton: hahah | 14:15 |
| odyssey4me | jrosser: sure, shoot | 14:15 |
| evrardjp | odyssey4me: we'll update | 14:15 |
| evrardjp | it passes gates for now | 14:15 |
| jrosser | odyssey4me: i've been having a poke at why the wheel build takes really so much longer in the gate check than in my local vm | 14:16 |
| openstackgerrit | Jesse Pretorius (odyssey4me) proposed openstack/openstack-ansible master: Use the TESTING_BRANCH env var for constraints https://review.openstack.org/592481 | 14:16 |
| jrosser | and the last phase where pip wheel does the actual builds does a truly gigantic amount of i/o to /tmp | 14:16 |
| odyssey4me | yep, that's to be expected | 14:16 |
| jrosser | but /tmp is slooooow | 14:17 |
| openstackgerrit | Jean-Philippe Evrard proposed openstack/openstack-ansible stable/rocky: Freeze all SHAs for RC1 https://review.openstack.org/590503 | 14:17 |
| evrardjp | okay I didn't see the freeze in zuul, so I am reproposing | 14:17 |
| evrardjp | weird | 14:17 |
| jrosser | odyssey4me: so i was wondering if making /tmp actual tmpfs rather than just a normal directory would help | 14:18 |
| jrosser | as it wouldnt hit actual disks | 14:18 |
| evrardjp | jrosser: that's not a bad idea | 14:18 |
| evrardjp | or mount some place as tmpfs temporarily | 14:18 |
| jrosser | i had a go here and see a marginal improvement in my vm | 14:18 |
| jrosser | by symlinking /tmp/ to /dev/shm/tmp as a hack | 14:19 |
| jrosser | and also setting TMPDIR environment | 14:19 |
| odyssey4me | hmm, would this be something we apply to all containers? | 14:19 |
| evrardjp | jrosser: we absolutely need to cleanup that space later though, as we are on the edge of OOM everytime | 14:19 |
| jrosser | evrardjp: yes indeed | 14:19 |
| evrardjp | and only during repo build | 14:19 |
| jrosser | so i certainly see it ~10% faster here, but in my local tests the wheel build stage is always super quick | 14:20 |
| jrosser | comparing that with what happens in the gate the wheel build is always by far the slowest stage | 14:20 |
| jrosser | so i have a hunch we are being hurt by pointless i/o there | 14:20 |
| evrardjp | jrosser: if you want to improve speed we can make git sparse clone by using versions, it's faster to process. | 14:20 |
| *** galaxyblr has quit IRC | 14:20 | |
| evrardjp | this is also a series of seconds shaved | 14:21 |
| openstackgerrit | Jesse Pretorius (odyssey4me) proposed openstack/openstack-ansible-apt_package_pinning master: Use the TESTING_BRANCH env var for constraints https://review.openstack.org/592483 | 14:21 |
| openstackgerrit | Jesse Pretorius (odyssey4me) proposed openstack/openstack-ansible-ceph_client master: Use the TESTING_BRANCH env var for constraints https://review.openstack.org/592484 | 14:22 |
| evrardjp | odyssey4me: why did you want to push a specific patch for the netns on the freeze, vs doing what we said about backporting role per role, carefully? | 14:22 |
| openstackgerrit | Jesse Pretorius (odyssey4me) proposed openstack/openstack-ansible-galera_client master: Use the TESTING_BRANCH env var for constraints https://review.openstack.org/592485 | 14:22 |
| evrardjp | I mean it's technically a stable branch now, so we justify the things that need to get in : ) | 14:23 |
| odyssey4me | evrardjp: the backport's already in the role - but the sha pin you have does not include it | 14:23 |
| evrardjp | I don't mind doing a bump of all the roles just after the freeze merges | 14:23 |
| odyssey4me | ok, as long as it's not affecting the tests, sure | 14:24 |
| evrardjp | oh yeah if it breaks tests no choice | 14:24 |
| odyssey4me | I thought it might be affecting tests in some way. | 14:24 |
| openstackgerrit | Jesse Pretorius (odyssey4me) proposed openstack/openstack-ansible-galera_server master: Use the TESTING_BRANCH env var for constraints https://review.openstack.org/592487 | 14:24 |
| evrardjp | now I understand your point | 14:25 |
| openstackgerrit | Jesse Pretorius (odyssey4me) proposed openstack/openstack-ansible-haproxy_server master: Use the TESTING_BRANCH env var for constraints https://review.openstack.org/592490 | 14:25 |
| openstackgerrit | Jesse Pretorius (odyssey4me) proposed openstack/openstack-ansible-lxc_container_create master: Use the TESTING_BRANCH env var for constraints https://review.openstack.org/592491 | 14:25 |
| openstackgerrit | Jesse Pretorius (odyssey4me) proposed openstack/openstack-ansible-lxc_hosts master: Use the TESTING_BRANCH env var for constraints https://review.openstack.org/592492 | 14:26 |
| openstackgerrit | Jesse Pretorius (odyssey4me) proposed openstack/openstack-ansible-memcached_server master: Use the TESTING_BRANCH env var for constraints https://review.openstack.org/592493 | 14:26 |
| openstackgerrit | Jesse Pretorius (odyssey4me) proposed openstack/openstack-ansible-nspawn_container_create master: Use the TESTING_BRANCH env var for constraints https://review.openstack.org/592494 | 14:27 |
| openstackgerrit | Jesse Pretorius (odyssey4me) proposed openstack/openstack-ansible-nspawn_hosts master: Use the TESTING_BRANCH env var for constraints https://review.openstack.org/592495 | 14:27 |
| evrardjp | so odyssey4me jrosser : should we split https://review.openstack.org/#/c/587544/ in two patches: one for bringing healthchecks for openstack, and one for using them instead of tempest? | 14:27 |
| openstackgerrit | Jesse Pretorius (odyssey4me) proposed openstack/openstack-ansible-openstack_hosts master: Use the TESTING_BRANCH env var for constraints https://review.openstack.org/592496 | 14:27 |
| odyssey4me | evrardjp: yes, I think so - because I'm not so sure that replacing them is something that's going to get blessing from the PTL ;) | 14:28 |
| odyssey4me | but the health checks are valuable to operators | 14:28 |
| evrardjp | agreed | 14:28 |
| openstackgerrit | Jesse Pretorius (odyssey4me) proposed openstack/openstack-ansible-openstack_openrc master: Use the TESTING_BRANCH env var for constraints https://review.openstack.org/592498 | 14:29 |
| openstackgerrit | Jesse Pretorius (odyssey4me) proposed openstack/openstack-ansible-ops master: Use the TESTING_BRANCH env var for constraints https://review.openstack.org/592499 | 14:29 |
| openstackgerrit | Jesse Pretorius (odyssey4me) proposed openstack/openstack-ansible-os_almanach master: Use the TESTING_BRANCH env var for constraints https://review.openstack.org/592501 | 14:31 |
| openstackgerrit | Jesse Pretorius (odyssey4me) proposed openstack/openstack-ansible-os_aodh master: Use the TESTING_BRANCH env var for constraints https://review.openstack.org/592502 | 14:31 |
| openstackgerrit | Jesse Pretorius (odyssey4me) proposed openstack/openstack-ansible-os_barbican master: Use the TESTING_BRANCH env var for constraints https://review.openstack.org/592504 | 14:32 |
| openstackgerrit | Andy Smith proposed openstack/openstack-ansible-os_barbican master: Add install packages for optional oslo.messaging amqp1 driver https://review.openstack.org/592506 | 14:36 |
| openstackgerrit | Jesse Pretorius (odyssey4me) proposed openstack/openstack-ansible-os_ceilometer master: Use the TESTING_BRANCH env var for constraints https://review.openstack.org/592509 | 14:41 |
| openstackgerrit | Jesse Pretorius (odyssey4me) proposed openstack/openstack-ansible-os_cinder master: Use the TESTING_BRANCH env var for constraints https://review.openstack.org/592510 | 14:41 |
| openstackgerrit | Jesse Pretorius (odyssey4me) proposed openstack/openstack-ansible-os_cloudkitty master: Use the TESTING_BRANCH env var for constraints https://review.openstack.org/592511 | 14:41 |
| openstackgerrit | Jesse Pretorius (odyssey4me) proposed openstack/openstack-ansible-os_congress master: Use the TESTING_BRANCH env var for constraints https://review.openstack.org/592512 | 14:41 |
| openstackgerrit | Jesse Pretorius (odyssey4me) proposed openstack/openstack-ansible-os_designate master: Use the TESTING_BRANCH env var for constraints https://review.openstack.org/592513 | 14:42 |
| mnaser | yeah i think replacing health checks by tempest is a bit more of a discussion | 14:51 |
| mnaser | if tempest cant create a volume, the 'healthcheck' playbook wont either, so it'll just be an extra thing we're carrying to maintain | 14:51 |
| openstackgerrit | Markos Chandras (hwoarang) proposed openstack/openstack-ansible-tests master: tests-ansible-env-prep: Fix plugin and ops testing on Vagrant https://review.openstack.org/592527 | 14:58 |
| openstackgerrit | Markos Chandras (hwoarang) proposed openstack/openstack-ansible-plugins master: [DNM] - strategy: linear: Inspect the delegated host on delegated tasks. https://review.openstack.org/591672 | 14:59 |
| openstackgerrit | Markos Chandras (hwoarang) proposed openstack/openstack-ansible-plugins master: tests: Use 'ansible_connection=local' for connecting to localhost https://review.openstack.org/592529 | 14:59 |
| openstackgerrit | Victor Palma proposed openstack/openstack-ansible-galera_server master: fix bootstrap of galera 10.2+ clusters https://review.openstack.org/592531 | 15:02 |
| openstackgerrit | Markos Chandras (hwoarang) proposed openstack/openstack-ansible-tests master: tests-ansible-env-prep: Fix plugin, tests and ops testing on Vagrant https://review.openstack.org/592527 | 15:08 |
| openstackgerrit | Victor Palma proposed openstack/openstack-ansible-galera_server master: fix bootstrap of galera 10.2+ clusters https://review.openstack.org/592531 | 15:10 |
| *** spatel has joined #openstack-ansible | 15:12 | |
| spatel | Quick question if i am running Ceph-mon on Infra node and it has br-storage interface in that case do you think i need to set MTU9000 there? | 15:12 |
| spatel | my all OSD storage node running MTU 9000 | 15:13 |
| spatel | Does Ceph-mon need MTU 9000? | 15:13 |
| openstackgerrit | Merged openstack/openstack-ansible-ops master: MNAIO: Enable using a data disk for file-backed VM's https://review.openstack.org/592027 | 15:13 |
| spatel | anyone running ceph-mon on Infra* nodes? | 15:14 |
| openstackgerrit | Jonathan Rosser proposed openstack/openstack-ansible master: Mount heavily used temporary directories onto the host tmpfs https://review.openstack.org/592540 | 15:14 |
| noonedeadpunk | spatel: not me. we're running ceph on infiniband, so it's not case for me... | 15:15 |
| spatel | noonedeadpunk: that is cool!! | 15:15 |
| jamesdenton | jrosser re: limiting provider networks to certain hosts (network-only vs all). Best I can tell, the templates do not regard group membership. If neutron_provider_networks.network_mappings is defined the host gets all of them when the respective agent is configured. The provider_networks plugin may need to be updated to expose group_binds and then we can check that within the neutron plugin templates? | 15:16 |
| jrosser | jamesdenton: yes i think that was my conclusion, so i was unable to drive specific & different config to the compute and network hosts | 15:17 |
| jamesdenton | yeah. i'll try and come up with something | 15:18 |
| spatel | jamesdenton: quick question related F5, everything working great but my novac console not working on F5, i did create 6080 VIP on F5 but it somehow doesn't like.. what else NOVAC need ? any other dependency if you know otherwise i have to do some tcpdump stuff | 15:18 |
| jamesdenton | is the URL correct when you try to console from Horizon? Or what does the API return for the console URL? | 15:20 |
| evrardjp | mnaser: with the TC discussion I am convinced we should split my commit in two. Then I let you decide where to run what. | 15:23 |
| *** DanyC has quit IRC | 15:24 | |
| mnaser | evrardjp: thanks, ill link it here for others once eavesdrop publishes | 15:24 |
| evrardjp | sure. I think it's good to document mordred answer here about the risk of hidden successes. | 15:25 |
| *** dave-mccowan has quit IRC | 15:26 | |
| spatel | jamesdenton: I am getting this URL https://10.30.2.9:6080/vnc_lite.html?token=6abb3bed-9990-4c15-b01f-2b836726b3c8&title=centos7-tux-1(6f868285-4641-4600-b131-bc3b5872da08) | 15:28 |
| spatel | And console screen is blank with "something went wrong, connection is closed" | 15:28 |
| evrardjp | mnaser: also we'll need to fix the cinder issue | 15:28 |
| spatel | in tcpdump i can see data going back and forth | 15:29 |
| jamesdenton | is 10.30.2.9 the correct VIP? also - maybe it doesn't work over SSL? not sure | 15:29 |
| spatel | yes that is external VIP | 15:29 |
| spatel | hmmm! | 15:29 |
| spatel | i have tried to change 10.30.2.9 with name also manually but it doesn't like that too | 15:30 |
| spatel | jamesdenton: just wondering if any other ports i need to open for external VIP? | 15:31 |
| jamesdenton | not for VNC | 15:31 |
| *** luksky has joined #openstack-ansible | 15:32 | |
| spatel | oh! wait.. | 15:33 |
| spatel | i have two instance on openstack, just found VM-1 is not working but VM-2 is working... | 15:34 |
| spatel | strange | 15:34 |
| spatel | let me dig more | 15:34 |
| spatel | let me spin more VMs | 15:35 |
| spatel | new VM works!! | 15:36 |
| openstackgerrit | David Wilde proposed openstack/openstack-ansible-ops master: Deploy ELK in MNAIO https://review.openstack.org/586612 | 15:38 |
| jamesdenton | spatel nice work | 15:41 |
| openstackgerrit | Andy Smith proposed openstack/openstack-ansible-os_ceilometer master: Add install packages for optional oslo.messaging amqp1 driver https://review.openstack.org/592560 | 15:42 |
| mnaser | evrardjp: i pushed up another attempt | 15:46 |
| mnaser | but i will reach out to cinder team if it doesnt go through | 15:46 |
| mnaser | so far so good though | 15:47 |
| mnaser | nope, centos-7 distro just failed, so | 15:47 |
| mnaser | so close | 15:47 |
| evrardjp | what do you mean there mnaser ? | 15:48 |
| evrardjp | which patch? | 15:48 |
| *** dave-mccowan has joined #openstack-ansible | 15:49 | |
| mnaser | the whole stack | 15:51 |
| mnaser | it hasnt published yet, i am looking on zuul.o.o | 15:51 |
| mnaser | btw, any reason why we're not using the rabbitmq that's shipped in UCA/RDO ? | 15:51 |
| mnaser | that one is pre-integrated/gated with openstack before they release it | 15:51 |
| mnaser | it's also the same one that is used in openstack ci | 15:52 |
| *** aicilamd has quit IRC | 15:52 | |
| *** gyee has joined #openstack-ansible | 15:53 | |
| odyssey4me | mnaser: because we've got stuck on using the same release across distributions instead | 15:54 |
| *** mmalchuk has joined #openstack-ansible | 15:54 | |
| mnaser | is the difference in releases really that major to be problematic for us | 15:54 |
| *** mmalchuk has quit IRC | 15:55 | |
| mnaser | i.e. i doubt that our ansible rabbit_* modules wouldn't work | 15:55 |
| odyssey4me | personally I wouldn't mind switching our default to using UCA/RDO/OBS and letting those who want to use rabbitmq/erlang repositories directly do so as opt-in | 15:55 |
| *** mmalchuk has joined #openstack-ansible | 15:55 | |
| mnaser | odyssey4me: it's one less dependency to worry about | 15:55 |
| odyssey4me | I think it might be a hangover from when we didn't use UCA and the distro version was universally terrible | 15:55 |
| mnaser | i.e. i legit just realized that we're downloading rabbitmq from the internets | 15:56 |
| odyssey4me | I agree, but it's cloudnull you need to convince. ;) | 15:56 |
| mnaser | i mean UCA has rabbitmq that ubuntu tests/deploys, and im pretty sure upstream openstack uses that same one | 15:56 |
| *** fghaas has quit IRC | 15:57 | |
| odyssey4me | the trouble with differing versions between distributions is that testing gets hairy, because we have to cater to those differences in a single role. | 15:57 |
| mnaser | but are they really that major of differences | 15:57 |
| odyssey4me | But perhaps we can figure out a suitable strategy for that. | 15:57 |
| *** mmalchuk_ has joined #openstack-ansible | 15:57 | |
| odyssey4me | Certainly something worth discussing at the PTG. | 15:57 |
| mnaser | i cant imagine rabbitmq in centos and uca being so wildly far away from each other that things wont work interchangibly | 15:57 |
| cloudnull | don't we have an option to not use the rabbit packages from pivital ? | 15:58 |
| chandankumar | odyssey4me: what about creating something like openstack-ansible-rpm-packaging-hardening or reuse rdoinfo for version control | 15:58 |
| chandankumar | ? | 15:58 |
| chandankumar | * version pinning | 15:58 |
| mnaser | cloudnull: well the default seems to be use directly from pivotal | 15:59 |
| *** mmalchuk has quit IRC | 16:00 | |
| mnaser | rabbitmq-server seems to be at 3.6.10 in uca | 16:00 |
| cloudnull | I think we use 3.7.5 | 16:01 |
| cloudnull | and while functionally there's likley not a lot of difference between those versions, there's quite a bit of difference under the hood | 16:01 |
| cloudnull | especially with HA and clustering | 16:02 |
| mnaser | looks like rabbitmq-server in rdo is 3.6.16 | 16:02 |
| mnaser | so we're really doing something on our own | 16:02 |
| mnaser | and i know at least the rdo one is very heavily integrated in the ci | 16:03 |
| cloudnull | is there an issue with 3.7.x ? | 16:03 |
| mnaser | well, it's being pulled from the internet so it takes ~8-9 minutes to install | 16:04 |
| mnaser | so we don't have any mirrors | 16:04 |
| mnaser | and then we're seeing this | 16:04 |
| mnaser | http://logs.openstack.org/04/584604/10/check/openstack-ansible-functional-ubuntu-bionic/d303a81/logs/host/cinder-volume.service.journal.log.txt.gz | 16:04 |
| mnaser | with these other timeouts http://logs.openstack.org/04/584604/10/check/openstack-ansible-functional-ubuntu-bionic/d303a81/logs/openstack/infra1/rabbitmq/rabbit@infra1.log.txt.gz#_2018-08-16_15_13_07_889 | 16:05 |
| *** udesale has quit IRC | 16:05 | |
| mnaser | and at this point i'm at a bit of a loss other than i think it might be the rabbitmq version | 16:05 |
| evrardjp | mnaser: probably worth asking hwoarang I think he is the last one to have tried bumping rmq | 16:06 |
| cloudnull | looks like the root of that first stack was from DuplicateMessageError: Found duplicate message(01f71fa100df4a4d8468471a85bc03f6) ? | 16:06 |
| cloudnull | likely an issue with nested try/except in oslo ? | 16:07 |
| evrardjp | mnaser: but at some point we thought: why not avoiding bumping until distros catch up | 16:07 |
| mnaser | oh i didnt notice that cloudnull ! | 16:07 |
| evrardjp | so taht upgrades are easy | 16:07 |
| * mnaser might have been looking at this for too long | 16:07 | |
| mnaser | evrardjp: makes sense | 16:07 |
| *** Laszlo-74 has quit IRC | 16:07 | |
| evrardjp | mnaser: I generally try to ;p | 16:07 |
| mnaser | cloudnull: there seems to be a connection reset above that though | 16:08 |
| evrardjp | I am not sure how hard it is to downgrade rabbit. | 16:09 |
| evrardjp | But I remember one of the major upgrades brought a lot of perf improvements | 16:09 |
| evrardjp | not sure if it's 3.6, 3.7 or 3.8+? | 16:10 |
| mnaser | well maybe we can convince distros to go up | 16:10 |
| mnaser | rdo is more of a community based thing, we have suse | 16:10 |
| mnaser | but for ubuntu.. idk | 16:10 |
| evrardjp | we can ask the community at the PTG | 16:10 |
| evrardjp | we know ppl :) | 16:10 |
| cloudnull | ++ | 16:10 |
| *** dxiri has joined #openstack-ansible | 16:11 | |
| cloudnull | mnaser I wonder if that connection reset issue could be related to the checksums we were adding? (and are in the process of removing) | 16:12 |
| *** aicilamd has joined #openstack-ansible | 16:12 | |
| mnaser | cloudnull: it could be, but the openstack-ansible-lxc_hosts change landed so i think it should no longer be an issue? | 16:12 |
| *** shardy has quit IRC | 16:12 | |
| mnaser | cloudnull: nope, it's still a thing http://logs.openstack.org/04/584604/10/check/openstack-ansible-functional-ubuntu-bionic/d303a81/logs/host/kern.log.txt.gz (warning 2mb file) | 16:13 |
| mnaser | http://logs.openstack.org/04/584604/10/check/openstack-ansible-functional-ubuntu-bionic/d303a81/job-output.txt.gz#_2018-08-16_14_43_23_891599 | 16:13 |
| mnaser | i think we need a patch to openstack-ansible-tests | 16:13 |
| openstackgerrit | Kevin Carter (cloudnull) proposed openstack/openstack-ansible master: Remove checksum-fill, its not needed any longer https://review.openstack.org/591882 | 16:14 |
| cloudnull | ^ that one still needs to get in | 16:14 |
| cloudnull | but its got to be something to do with the general networking given its returning a connection reset | 16:15 |
| mnaser | cloudnull: https://github.com/openstack/openstack-ansible-tests/blob/master/test-prepare-host.yml#L239-L258 | 16:15 |
| mnaser | can we remove all this because its in lxc_hosts ? | 16:16 |
| mnaser | "Add iptables rule to ensure traffic checksum is correct" is gone .. "Add iptables rule to provide internet connectivity to instances" seems to be already part of lxc_hosts here http://git.openstack.org/cgit/openstack/openstack-ansible-lxc_hosts/tree/templates/lxc-system-manage.j2 | 16:16 |
| cloudnull | yup we should remove that checksum fill call too | 16:16 |
| mnaser | is the /usr/local/bin/lxc-system-manage call still necessary or no cloudnull ? | 16:17 |
| * cloudnull wonders if there's anymore of those | 16:17 | |
| cloudnull | nope not required | 16:17 |
| cloudnull | in fact we could probably remove all that code and replace it with bits like we have in the integrated repo | 16:18 |
| openstackgerrit | Mohammed Naser proposed openstack/openstack-ansible-tests master: Remove tasks that mangle iptables (checksum, NAT) https://review.openstack.org/592571 | 16:18 |
| mnaser | cloudnull: ^ | 16:18 |
| mnaser | yeah all of openstack-ansible-tests has a lot of duplication with integrated repo | 16:19 |
| noonedeadpunk | cloudnull: can you kindly check https://review.openstack.org/#/c/586318/ one more time? | 16:20 |
| *** luksky has quit IRC | 16:20 | |
| spatel | cloudnull: I don't think we need CHECKSUM --checksum-fill anywhere, i think its OLD school thing and recent DHCP doesn't need those too | 16:20 |
| cloudnull | ^ totally agree | 16:21 |
| cloudnull | I think we're on track to purging all of that | 16:21 |
| mnaser | spatel: you should vote on that patch then :) | 16:21 |
| spatel | I am glade to test them out, anyway i did test checksum stuff and found it cause lots of issue so that is no brain | 16:21 |
| mnaser | cloudnull: i really hope that we can eliminate most of openstack-ansible-tests by straight up calling the integrated repo | 16:22 |
| spatel | I am going to VOTE sure | 16:22 |
| mnaser | import_playbook: os-keystone-install.yml | 16:22 |
| openstackgerrit | Mohammed Naser proposed openstack/openstack-ansible-tests master: Clone integrated repo into the base jobs https://review.openstack.org/592573 | 16:24 |
| spatel | mnaser: I just voted on checksum-fill patch | 16:26 |
| mnaser | nice :) | 16:26 |
| spatel | I am build new lab to test rocky on CentOS7..! soon you will see flood of bug request from me :) | 16:28 |
| *** vnogin has joined #openstack-ansible | 16:31 | |
| *** cshen has joined #openstack-ansible | 16:33 | |
| *** pcaruana has quit IRC | 16:34 | |
| *** vnogin has quit IRC | 16:36 | |
| mnaser | spatel: it might not work at all | 16:37 |
| spatel | ? | 16:37 |
| spatel | mnaser: ^^ | 16:39 |
| bgmccollum | rocky might be "rocky" on CentOS7... | 16:41 |
| mnaser | centos ci has broken all cycle and ive been trying to fix it but i couldnt in time for stable branch cut | 16:41 |
| mnaser | and still is | 16:42 |
| mnaser | speaking of, evrardjp any reason why we cut stable/rocky so early? | 16:42 |
| mnaser | i assume osa is cycle trailing | 16:42 |
| odyssey4me | mnaser: why not? | 16:43 |
| mnaser | i figured if we did it after release, then we can cut stable/rocky after we stabilize things to upstream releases packaged | 16:44 |
| mnaser | i.e. rdo would have their packages out, uca too | 16:44 |
| odyssey4me | but our source builds would suffer because the tests for those would be building master, not rocky | 16:45 |
| odyssey4me | the packaged builds aren't affected by the branch name - but source builds are | 16:45 |
| mnaser | right | 16:46 |
| mnaser | but our dependencies that we install from distros is what i was thinking of | 16:46 |
| odyssey4me | sure, but that's not affected by the branch name at all | 16:46 |
| odyssey4me | it does mean we need to port fixes back and all, but that's fine | 16:47 |
| mnaser | yeah the backports are a bit of a pain lol | 16:47 |
| odyssey4me | over time we can try to move towards the roles not being series specific, but for now they're pretty tightly coupled | 16:47 |
| odyssey4me | it'd be nice to work towards that not being a problem - supporting 2-3 releases or something | 16:48 |
| odyssey4me | we could then perhaps even switch away from naming our branches like we do | 16:48 |
| odyssey4me | they could be ansible release related, for example | 16:48 |
| *** openstackgerrit has quit IRC | 16:49 | |
| mnaser | odyssey4me: yeah but that might become difficult as things get deprecated/removed/etc | 16:50 |
| * mnaser cant imagine the mess of compatiblity layer that might become | 16:50 | |
| mnaser | hwoarang: are you making any progress on opensuse-150 jobs? they have been nv for a very long time and take up a fairly significant amount of resources right now | 16:51 |
| odyssey4me | yeah, it might be - but might not ;) | 16:51 |
| odyssey4me | it's food for thought, not very actionable right now | 16:51 |
| mnaser | odyssey4me: oh agreed, it's a discussion | 16:51 |
| * mnaser really hopes the checksum stuff helps fix cinder | 16:51 | |
| mnaser | https://review.openstack.org/#/c/592571/ if you dont mind giving an advance +W odyssey4me | 16:51 |
| odyssey4me | we could, for example, make the rmq role be totally unrelated to openstack series and instead focused on rmq release major.minor - then each distro could use the one meant for it or a deployer could use the upstream packages instead of distribution packages and choose the right role version for that | 16:52 |
| mnaser | odyssey4me: that i totally agree on. our infra roles should be totally usable and id love for them to be decoupled from osa | 16:52 |
| mnaser | i.e: ansible-role-rabbitmq | 16:53 |
| mnaser | i think they are relatively un-opinionated right now | 16:53 |
| odyssey4me | we have kinda agreed to that in the past, but I don't think we've had time to action it, really | 16:54 |
| odyssey4me | we can't just rename the repo - we'd have to seed a new one and retire the old | 16:54 |
| mnaser | i'm looking at a 'sample' to see what the runs are like | 16:57 |
| mnaser | took 22 seconds to deploy memcached.. seems reasonable.. but with_items seems to the biggest thing that hurts, esp combined with template | 16:58 |
| mnaser | 1 second per file transferred as it looks like it does it operation by operation | 16:58 |
| *** fghaas has joined #openstack-ansible | 17:00 | |
| odyssey4me | yep, ansible prepares a script on the deploy host, then fires up a connection to the remote host and copies the script over, then executes it | 17:00 |
| mnaser | odyssey4me: i think mitogen is the type of thing that might help resolve these issues i guess | 17:01 |
| mnaser | especially with it's built-in delegation into containers! | 17:01 |
| odyssey4me | pipelining helps speed it up a little, but this is the price to be paid for doing this sort of thing without an agent on the target host | 17:01 |
| mnaser | i think that's a lot of what mitogen does | 17:01 |
| * dw scans up | 17:02 | |
| mnaser | and gets rid of the latency too | 17:02 |
| dw | mnaser: indeed mitogen helps loops if the looped task is naturally 'fast' and it's ansible overhead killing things, but let me read ^ first :) | 17:02 |
| odyssey4me | I think that if we really want to speed things up, we have to fundamentally change how things are done instead. | 17:02 |
| mnaser | dw: in this case a `with_items` + `template` | 17:02 |
| mnaser | so we can get rid of our connection and strategy plugin and rely on mitogen to do that | 17:03 |
| *** gkadam has quit IRC | 17:03 | |
| mnaser | i think by looking at profiling it just seems like every little iteration in ansible takes a second or two | 17:03 |
| mnaser | and with a large # of tasks they add up | 17:03 |
| dw | mnaser: with_items over a template is generally a big winner | 17:04 |
| odyssey4me | we do a lot of that | 17:04 |
| dw | but please give me at least over the weekend -- there are serious bugs in the stuff osa needs just now :S | 17:04 |
| mnaser | for example our "update the local file system crud" is a common job | 17:04 |
| mnaser | all it does is ensure directory/file in with_items | 17:04 |
| mnaser | dw: no worries, we won't be switching for a little while but i think there is a big benefit in us taking advantage of it | 17:05 |
| dw | mnaser: that much is clear from my local runs so far.. we just need to flush out all the issues identified so far, but my energy is a little low this week | 17:05 |
| dw | i have a whole new branch with osa's name on it, but it's realistically probably at least a month out | 17:05 |
| mnaser | dw: hey no worries :) | 17:05 |
| mnaser | i have all sorts of fun things im dealing with too | 17:05 |
| odyssey4me | dweaver: we appreciate the work you're doing? | 17:06 |
| odyssey4me | lol, that went wrong | 17:06 |
| odyssey4me | dw ^ | 17:06 |
| jrosser | i just noticed that .ansible is on real disks not tmpfs | 17:06 |
| jrosser | thats really not great | 17:06 |
| odyssey4me | jrosser: .ansible? you mean for the role tests? | 17:06 |
| jrosser | where it drops all the temporary files for each task | 17:07 |
| jrosser | in all the containers and everywhere | 17:07 |
| odyssey4me | ah yes, I think that can be changed | 17:07 |
| jrosser | that feels like super low hanging fruit | 17:07 |
| *** fghaas has quit IRC | 17:07 | |
| * jrosser travels | 17:07 | |
| odyssey4me | but on a overburdened host, things in /tmp disappear (perhaps due to some garbage collection?) - so ideally we need something a little more protected | 17:07 |
| spatel | I am trying to increase MTU 9000 so i did container_mtu: "9000" in openstack_user_config and now running openstack-ansible lxc-hosts-setup.yml but it didn't change MTU on br-storage interface | 17:09 |
| spatel | Do i need to run any other playbook ? | 17:09 |
| odyssey4me | spatel: lxc-containers-create I think | 17:09 |
| odyssey4me | that's the one that actually does all the container things | 17:10 |
| spatel | Let me run that! | 17:10 |
| *** openstackgerrit has joined #openstack-ansible | 17:13 | |
| openstackgerrit | Merged openstack/openstack-ansible-os_horizon master: Implements custom theme distribution https://review.openstack.org/586318 | 17:13 |
| *** DanyC has joined #openstack-ansible | 17:13 | |
| *** electrofelix has quit IRC | 17:15 | |
| *** goldenfri has joined #openstack-ansible | 17:17 | |
| openstackgerrit | Jesse Pretorius (odyssey4me) proposed openstack/openstack-ansible-os_glance master: Use the TESTING_BRANCH env var for constraints https://review.openstack.org/592587 | 17:18 |
| openstackgerrit | Jesse Pretorius (odyssey4me) proposed openstack/openstack-ansible-os_gnocchi master: Use the TESTING_BRANCH env var for constraints https://review.openstack.org/592588 | 17:18 |
| openstackgerrit | Jesse Pretorius (odyssey4me) proposed openstack/openstack-ansible-os_heat master: Use the TESTING_BRANCH env var for constraints https://review.openstack.org/592589 | 17:18 |
| openstackgerrit | Jesse Pretorius (odyssey4me) proposed openstack/openstack-ansible-os_horizon master: Use the TESTING_BRANCH env var for constraints https://review.openstack.org/592590 | 17:18 |
| openstackgerrit | Jesse Pretorius (odyssey4me) proposed openstack/openstack-ansible-os_ironic master: Use the TESTING_BRANCH env var for constraints https://review.openstack.org/592592 | 17:19 |
| openstackgerrit | Jesse Pretorius (odyssey4me) proposed openstack/openstack-ansible-os_keystone master: Use the TESTING_BRANCH env var for constraints https://review.openstack.org/592593 | 17:19 |
| openstackgerrit | Jesse Pretorius (odyssey4me) proposed openstack/openstack-ansible-os_magnum master: Use the TESTING_BRANCH env var for constraints https://review.openstack.org/592594 | 17:19 |
| openstackgerrit | Jesse Pretorius (odyssey4me) proposed openstack/openstack-ansible-os_masakari master: Use the TESTING_BRANCH env var for constraints https://review.openstack.org/592595 | 17:21 |
| openstackgerrit | Kevin Carter (cloudnull) proposed openstack/openstack-ansible-tests master: Convert test network setup to use the networkd role https://review.openstack.org/592596 | 17:22 |
| openstackgerrit | Jesse Pretorius (odyssey4me) proposed openstack/openstack-ansible-os_molteniron master: Use the TESTING_BRANCH env var for constraints https://review.openstack.org/592597 | 17:22 |
| openstackgerrit | Jesse Pretorius (odyssey4me) proposed openstack/openstack-ansible-os_monasca master: Use the TESTING_BRANCH env var for constraints https://review.openstack.org/592598 | 17:22 |
| openstackgerrit | Jesse Pretorius (odyssey4me) proposed openstack/openstack-ansible-os_monasca-agent master: Use the TESTING_BRANCH env var for constraints https://review.openstack.org/592599 | 17:22 |
| openstackgerrit | Jesse Pretorius (odyssey4me) proposed openstack/openstack-ansible-os_monasca-ui master: Use the TESTING_BRANCH env var for constraints https://review.openstack.org/592600 | 17:23 |
| openstackgerrit | Jesse Pretorius (odyssey4me) proposed openstack/openstack-ansible-os_neutron master: Use the TESTING_BRANCH env var for constraints https://review.openstack.org/592601 | 17:23 |
| openstackgerrit | Jesse Pretorius (odyssey4me) proposed openstack/openstack-ansible-os_nova master: Use the TESTING_BRANCH env var for constraints https://review.openstack.org/592602 | 17:23 |
| openstackgerrit | Jesse Pretorius (odyssey4me) proposed openstack/openstack-ansible-os_octavia master: Use the TESTING_BRANCH env var for constraints https://review.openstack.org/592603 | 17:24 |
| openstackgerrit | Jesse Pretorius (odyssey4me) proposed openstack/openstack-ansible-os_panko master: Use the TESTING_BRANCH env var for constraints https://review.openstack.org/592604 | 17:24 |
| openstackgerrit | Jesse Pretorius (odyssey4me) proposed openstack/openstack-ansible-os_rally master: Use the TESTING_BRANCH env var for constraints https://review.openstack.org/592605 | 17:24 |
| openstackgerrit | Jesse Pretorius (odyssey4me) proposed openstack/openstack-ansible-os_sahara master: Use the TESTING_BRANCH env var for constraints https://review.openstack.org/592606 | 17:25 |
| openstackgerrit | Jesse Pretorius (odyssey4me) proposed openstack/openstack-ansible-os_searchlight master: Use the TESTING_BRANCH env var for constraints https://review.openstack.org/592607 | 17:25 |
| openstackgerrit | Jesse Pretorius (odyssey4me) proposed openstack/openstack-ansible-os_swift master: Use the TESTING_BRANCH env var for constraints https://review.openstack.org/592608 | 17:25 |
| openstackgerrit | Jesse Pretorius (odyssey4me) proposed openstack/openstack-ansible-os_tacker master: Use the TESTING_BRANCH env var for constraints https://review.openstack.org/592609 | 17:25 |
| openstackgerrit | Jesse Pretorius (odyssey4me) proposed openstack/openstack-ansible-os_tempest master: Use the TESTING_BRANCH env var for constraints https://review.openstack.org/592610 | 17:26 |
| openstackgerrit | Jesse Pretorius (odyssey4me) proposed openstack/openstack-ansible-os_trove master: Use the TESTING_BRANCH env var for constraints https://review.openstack.org/592611 | 17:26 |
| openstackgerrit | Jesse Pretorius (odyssey4me) proposed openstack/openstack-ansible-os_trove master: Use the TESTING_BRANCH env var for constraints https://review.openstack.org/592611 | 17:26 |
| *** mmalchuk_ has quit IRC | 17:27 | |
| openstackgerrit | Jesse Pretorius (odyssey4me) proposed openstack/openstack-ansible-os_watcher master: Use the TESTING_BRANCH env var for constraints https://review.openstack.org/592612 | 17:28 |
| *** mmalchuk has joined #openstack-ansible | 17:28 | |
| openstackgerrit | Jesse Pretorius (odyssey4me) proposed openstack/openstack-ansible-os_zaqar master: Use the TESTING_BRANCH env var for constraints https://review.openstack.org/592613 | 17:28 |
| openstackgerrit | Jesse Pretorius (odyssey4me) proposed openstack/openstack-ansible-os_zun master: Use the TESTING_BRANCH env var for constraints https://review.openstack.org/592614 | 17:28 |
| *** gkadam has joined #openstack-ansible | 17:29 | |
| openstackgerrit | Jesse Pretorius (odyssey4me) proposed openstack/openstack-ansible-pip_install master: Use the TESTING_BRANCH env var for constraints https://review.openstack.org/592615 | 17:29 |
| mnaser | nice, checksum fix made it to gate | 17:29 |
| openstackgerrit | Jesse Pretorius (odyssey4me) proposed openstack/openstack-ansible-plugins master: Use the TESTING_BRANCH env var for constraints https://review.openstack.org/592616 | 17:29 |
| openstackgerrit | Jesse Pretorius (odyssey4me) proposed openstack/openstack-ansible-rabbitmq_server master: Use the TESTING_BRANCH env var for constraints https://review.openstack.org/592617 | 17:29 |
| openstackgerrit | Jesse Pretorius (odyssey4me) proposed openstack/openstack-ansible-repo_build master: Use the TESTING_BRANCH env var for constraints https://review.openstack.org/592618 | 17:30 |
| openstackgerrit | Jesse Pretorius (odyssey4me) proposed openstack/openstack-ansible-repo_server master: Use the TESTING_BRANCH env var for constraints https://review.openstack.org/592619 | 17:30 |
| mnaser | is anyone looking at the ceph issues, i remember seeing some discussions around it | 17:30 |
| noonedeadpunk | offtopic question - does anyone has a discount, which is known not to be used, for Berlin Summit? | 17:30 |
| openstackgerrit | Jesse Pretorius (odyssey4me) proposed openstack/openstack-ansible-rsyslog_client master: Use the TESTING_BRANCH env var for constraints https://review.openstack.org/592620 | 17:30 |
| openstackgerrit | Jesse Pretorius (odyssey4me) proposed openstack/openstack-ansible-rsyslog_server master: Use the TESTING_BRANCH env var for constraints https://review.openstack.org/592621 | 17:30 |
| mnaser | noonedeadpunk: didnt get an ATC code? :( | 17:34 |
| openstackgerrit | Merged openstack/openstack-ansible-ops master: MNAIO: Use virt module to shut VM's down when saving images https://review.openstack.org/592131 | 17:35 |
| openstackgerrit | Merged openstack/openstack-ansible-ops master: MNAIO: Extend image saving to include manifest https://review.openstack.org/592171 | 17:35 |
| openstackgerrit | Merged openstack/openstack-ansible-ops master: MNAIO: Install jmespath https://review.openstack.org/592173 | 17:35 |
| odyssey4me | mnaser: for ceph https://review.openstack.org/590778 contains a ceph-ansible pin to tide us over | 17:36 |
| odyssey4me | so we need to babysit that one | 17:36 |
| mnaser | gr | 17:37 |
| mnaser | github failed to resolve | 17:37 |
| mnaser | for ansible-keepalived | 17:37 |
| noonedeadpunk | mnaser: actually I get, thanks to you) But probably someone has one, and he knows, that he won't use it (because of recieving free entrance or as he's not going to come) for a co-worker. As my company is not very interested in sending staff to conferences | 17:37 |
| mnaser | noonedeadpunk: let me see what i can do try to do | 17:37 |
| mnaser | noonedeadpunk: can you email me to mnaser@vexxhost.com with some info and ill try to get something | 17:38 |
| *** fghaas has joined #openstack-ansible | 17:38 | |
| noonedeadpunk | mnaser: oh, great) it's not really required, but if it's possible, I'd be really grateful | 17:39 |
| mnaser | i'll try to see what i can do but i can't guarantee anything :) | 17:39 |
| *** hamzaachi has joined #openstack-ansible | 17:39 | |
| *** dave-mccowan has quit IRC | 17:39 | |
| odyssey4me | ok folks, I'm out for the night - back online tomorrow, see you soon! | 17:42 |
| mnaser | later odyssey4me | 17:44 |
| mnaser | https://review.openstack.org/#/q/%22+use+include_tasks+instead+of+include%22+is:open+label:Code-Review%252B2+label:Verified%252B1 | 17:44 |
| mnaser | easy +W's if possible | 17:44 |
| openstackgerrit | Dmitriy Rabotjagov (noonedeadpunk) proposed openstack/openstack-ansible-os_horizon stable/rocky: Implements custom theme distribution https://review.openstack.org/591336 | 17:47 |
| dw | jrosser_: assuming i get a gust of productivity over the weekend, how do i actually go about validating the finished OSA deploy is sane? | 17:49 |
| openstackgerrit | Andy Smith proposed openstack/openstack-ansible-os_swift master: Add install packages for optional oslo.messaging amqp1 driver https://review.openstack.org/592630 | 17:49 |
| jrosser | dw: it will run a test suite at the end - tempest | 17:50 |
| jrosser | Which is the source of much trouble for us currently btw | 17:50 |
| *** cshen has quit IRC | 17:51 | |
| mnaser | jrosser: yeah, if you want to push up the code, we can help review it in some way (if you made changes to osa) | 17:51 |
| mnaser | dw: ^ | 17:51 |
| dw | aah :) | 17:51 |
| jrosser | I pushed up a test patch to enable mitogen | 17:52 |
| dw | mnaser: mostly i want to play catch-up with the issues jrosser has already found | 17:52 |
| mnaser | jrosser: ooooh do you have a link | 17:52 |
| jrosser | Just for folks to see where/what got changed | 17:52 |
| jrosser | mnaser https://review.openstack.org/#/c/591236/ | 17:53 |
| mnaser | yep looks like it died on galera -- EOF on stream; last 300 bytes received: u'ssh: Could not resolve hostname {{ galera_server_bootstrap_node }}: Temporary failure in name resolution\r\n' | 17:54 |
| mnaser | https://github.com/openstack/openstack-ansible-galera_server/blob/master/handlers/main.yml#L65 | 17:55 |
| jrosser | The trouble all starts with delegation I think | 17:55 |
| mnaser | i guess the delegate_to being templated is not something that's supported | 17:55 |
| jrosser | We had a good poke at it in #mitogen the other day | 17:56 |
| *** fghaas has quit IRC | 17:57 | |
| *** cshen has joined #openstack-ansible | 17:58 | |
| jrosser | dw: check this out for driving openstack with ansible modules https://review.openstack.org/#/c/587544/ | 17:59 |
| *** poopcat has joined #openstack-ansible | 17:59 | |
| jrosser | And top tip, look at the "download" link top right of all these patch pages, it's really easy to drop them onto your working tree | 18:00 |
| *** hamzaachi has quit IRC | 18:05 | |
| openstackgerrit | Dmitriy Rabotjagov (noonedeadpunk) proposed openstack/openstack-ansible-os_horizon stable/queens: Implements custom theme distribution https://review.openstack.org/592635 | 18:05 |
| openstackgerrit | Kevin Carter (cloudnull) proposed openstack/openstack-ansible-tests master: Convert test network setup to use the networkd role https://review.openstack.org/592596 | 18:07 |
| openstackgerrit | Merged openstack/openstack-ansible-tests master: Remove tasks that mangle iptables (checksum, NAT) https://review.openstack.org/592571 | 18:09 |
| noonedeadpunk | cloudnull: if it's possible, check please cherry-pick to rocky as well https://review.openstack.org/#/c/591336/ | 18:09 |
| openstackgerrit | Kevin Carter (cloudnull) proposed openstack/openstack-ansible-nspawn_hosts master: Add SUSE 15 to the nspawn hosts role https://review.openstack.org/592177 | 18:10 |
| spatel | odyssey4me: Thanks for you help! MTU 9000 works after running lxc-create playbook | 18:15 |
| mnaser | do we have more attendees at the ptg (cloudnull anyone else from rax coming too?) | 18:23 |
| cloudnull | odyssey4me d34dh0r53 devx cjloader prometheanfire, i think | 18:23 |
| cjloader | yep | 18:24 |
| mnaser | cool, eyes on https://etherpad.openstack.org/p/osa-stein-ptg (and if you can add yourself as attendees too :)) | 18:24 |
| d34dh0r53 | yep | 18:24 |
| prometheanfire | I'm on the list | 18:24 |
| mnaser | cool, evrardjp will hwoarang be attending (as well as any other suse folks?) | 18:25 |
| mnaser | would have been nice to have jrosser :( | 18:25 |
| guilhermesp | I want my visaaa =( heheh | 18:25 |
| cjloader | I just put my name down mnaser | 18:26 |
| mnaser | yeah guilhermesp is pending travel stuff (hopefully) | 18:26 |
| mnaser | i dunno who else to ping who was there last time | 18:26 |
| mnaser | i'm not sure if andymccr will continue doing openstack stuff within redhat | 18:26 |
| mnaser | logan-: too perhaps? | 18:27 |
| openstackgerrit | Kevin Carter (cloudnull) proposed openstack/openstack-ansible-nspawn_hosts master: Add SUSE 15 to the nspawn hosts role https://review.openstack.org/592177 | 18:31 |
| *** gkadam_ has joined #openstack-ansible | 18:38 | |
| *** gkadam has quit IRC | 18:42 | |
| *** hamzaachi has joined #openstack-ansible | 18:43 | |
| spotz | mnaser: me too! | 18:58 |
| *** DanyC has quit IRC | 18:59 | |
| *** gkadam_ has quit IRC | 19:00 | |
| openstackgerrit | Jonathan Rosser proposed openstack/openstack-ansible master: [TEST] bionic https://review.openstack.org/586607 | 19:00 |
| spotz | Poor mnaser is gonna be sick of me:) | 19:03 |
| *** hamzaachi has quit IRC | 19:08 | |
| *** hamzaachi has joined #openstack-ansible | 19:09 | |
| *** Leo_m has joined #openstack-ansible | 19:16 | |
| openstackgerrit | James Denton proposed openstack/openstack-ansible master: Provides group_names for use with provider_network plugin https://review.openstack.org/592659 | 19:19 |
| *** Leo_m has quit IRC | 19:21 | |
| openstackgerrit | James Denton proposed openstack/openstack-ansible-plugins master: Modify provider_network plugin to compare group_binds to group_names https://review.openstack.org/592661 | 19:25 |
| jamesdenton | jrosser maybe this will do what you're thinking: https://review.openstack.org/#/q/status:open+topic:bug/1787462. Accompanying bug: https://bugs.launchpad.net/openstack-ansible/+bug/1787462 | 19:26 |
| openstack | Launchpad bug 1787462 in openstack-ansible "group_binds for provider networks has no effect on neutron provider mappings" [Undecided,In progress] - Assigned to James Denton (james-denton) | 19:26 |
| guilhermesp | https://review.openstack.org/#/c/591961/4 | 19:27 |
| guilhermesp | success :P | 19:27 |
| *** Leo_m has joined #openstack-ansible | 19:27 | |
| guilhermesp | I was waiting for this day | 19:27 |
| *** kevinshort has joined #openstack-ansible | 19:28 | |
| jrosser | jamesdenton: ahha cool - we are redeploying the lab very soon so i can try those out for real | 19:31 |
| jamesdenton | yes, please do! | 19:34 |
| bgmccollum | so it was ignoring the group_binds...no wonder | 19:34 |
| jamesdenton | aye | 19:34 |
| spatel | jamesdenton: or other i have question, my infra-02 is primary node in OSA, i have notice if i shutdown that node for testing, my horizon is just stuck and wheel is spinning, i have check F5 LB also and F5 removed that node from pool | 19:36 |
| spatel | Do you guys have seen issue like that? | 19:36 |
| jamesdenton | bgmccollum i don't see group_binds being used anywhere, honestly | 19:36 |
| jamesdenton | but i could be overlooking it | 19:36 |
| spatel | it seem this is galera issue.. | 19:37 |
| jamesdenton | spatel Interesting. i would recommend, at a minimum, ensuring you have connection mirroring enabled on the galera VS | 19:37 |
| jamesdenton | and have the mirror interfaces setup properly on the F5s | 19:38 |
| *** Leo_m has quit IRC | 19:38 | |
| jamesdenton | well - that would only apply to F5 failover, not pool member failure | 19:38 |
| mnaser | omg | 19:39 |
| mnaser | ITS PASSING?! | 19:39 |
| mnaser | THEY ALL PASSED | 19:39 |
| mnaser | WITH NO TIMEOUTS | 19:39 |
| mnaser | !!! | 19:39 |
| openstack | mnaser: Error: "!!" is not a valid command. | 19:39 |
| mnaser | i think a lot of our slowness had to do with the checksum stuff cloudnull | 19:39 |
| spatel | mirroring is disabled on F5 | 19:40 |
| spatel | jamesdenton: ^^ | 19:40 |
| mnaser | https://review.openstack.org/#/c/591961/4 https://review.openstack.org/#/c/592085/6 https://review.openstack.org/#/c/589934/8 https://review.openstack.org/#/c/584604/10 please | 19:40 |
| *** Leo_m has joined #openstack-ansible | 19:40 | |
| mnaser | appreciate reviews on those | 19:40 |
| spatel | why do we need mirroring? HAproxy doesn't use mirroring | 19:40 |
| jamesdenton | spatel ok - i would enable it for production, but it doesn't make a difference in this case. Can you hit the galera VIP thru the F5 when you shutdown infra02? | 19:40 |
| mnaser | cc jrosser ^ :> | 19:41 |
| spatel | let me try some basic test and i get back to you in min... | 19:41 |
| mnaser | and spotz if oyu're around ^ | 19:41 |
| jamesdenton | spatel The F5 will reset all of those connections and cause services to create new ones, and you may hit max_connections on the galera node | 19:41 |
| * jrosser looks | 19:41 | |
| jamesdenton | just FYI | 19:41 |
| spotz | mnaser: looking | 19:41 |
| spatel | jamesdenton: interesting if max_connection issue.. can i see that in log? let me check | 19:42 |
| guilhermesp | mnaser: reviews asap otherwise cinder will frustrate us hahaha | 19:42 |
| mnaser | i reviewed the ones i'm allowed to ;p | 19:42 |
| mnaser | i authored the ones i didnt | 19:42 |
| jamesdenton | spatel you would see something logged. But anyway, i don't suspect that here. it was the first thing that came to mind, but doesn't apply to this scenario | 19:42 |
| spotz | heheh, they looked straightforward and all cindery:) | 19:44 |
| mnaser | thanks for the reviews jrosser and spotz <3 | 19:45 |
| guilhermesp | mnaser: but I'm still wondering we have solid conclusions of what happaned with cinder role...was it the last change in defaults/main determinant? | 19:46 |
| mnaser | guilhermesp: it was a combination of things | 19:46 |
| mnaser | every review on that stack had an issue somewhere | 19:46 |
| spotz | I try to be just a distracted ping away:) | 19:46 |
| *** fghaas has joined #openstack-ansible | 19:47 | |
| *** fghaas has quit IRC | 19:53 | |
| guilhermesp | Yeah but, every review with specific changes but all of them with transient failings across different distros.. that's why I'm still wondering what makes all the patches pass | 19:55 |
| *** cshen has quit IRC | 20:06 | |
| *** pcaruana has joined #openstack-ansible | 20:13 | |
| *** cshen has joined #openstack-ansible | 20:21 | |
| *** hamzaachi has quit IRC | 20:22 | |
| openstackgerrit | Merged openstack/openstack-ansible-os_cinder master: zuul: Drop distro jobs for Xenial and switch to smoke tests https://review.openstack.org/591961 | 20:29 |
| openstackgerrit | Merged openstack/openstack-ansible-os_cinder master: Drop JSON logging for Cinder https://review.openstack.org/592085 | 20:29 |
| *** kevinshort has quit IRC | 20:33 | |
| openstackgerrit | Merged openstack/openstack-ansible-os_cinder master: Fix cinder_environment_overrides https://review.openstack.org/589934 | 20:35 |
| openstackgerrit | Merged openstack/openstack-ansible-os_cinder master: Add target_helper to backend_defaults https://review.openstack.org/584604 | 20:35 |
| cloudnull | mnaser are you seeing faster returns now ? | 20:41 |
| mnaser | cloudnull: i think the jobs might be running faster | 20:41 |
| * cloudnull just got back to a computer | 20:41 | |
| mnaser | like that was pretty fast to build.. | 20:41 |
| mnaser | took 40m for the cinder stuff, worked flawless | 20:41 |
| mnaser | look at the stack, centos took 7 minutes more only | 20:42 |
| mnaser | cloudnull: https://review.openstack.org/#/c/592207/2 would be nice | 20:42 |
| mnaser | we gotta make sure this is all backported to stable/rockty | 20:43 |
| mnaser | rocky* | 20:43 |
| cloudnull | ++ | 20:43 |
| openstackgerrit | Mohammed Naser proposed openstack/openstack-ansible-openstack_hosts stable/rocky: Use variable with full URL for RDO repositories https://review.openstack.org/592683 | 20:43 |
| openstackgerrit | Mohammed Naser proposed openstack/openstack-ansible-tests stable/rocky: Use latest hashed RDO repository https://review.openstack.org/592684 | 20:43 |
| * cloudnull shgould have made those prs use the same topic | 20:43 | |
| openstackgerrit | Mohammed Naser proposed openstack/openstack-ansible-tests stable/rocky: Remove tasks that mangle iptables (checksum, NAT) https://review.openstack.org/592685 | 20:44 |
| mnaser | yeah | 20:44 |
| mnaser | all good | 20:44 |
| openstackgerrit | Mohammed Naser proposed openstack/openstack-ansible-os_cinder stable/rocky: zuul: Drop distro jobs for Xenial and switch to smoke tests https://review.openstack.org/592686 | 20:44 |
| openstackgerrit | Mohammed Naser proposed openstack/openstack-ansible-os_cinder stable/rocky: Drop JSON logging for Cinder https://review.openstack.org/592687 | 20:45 |
| openstackgerrit | Mohammed Naser proposed openstack/openstack-ansible-os_cinder stable/rocky: Fix cinder_environment_overrides https://review.openstack.org/592688 | 20:45 |
| openstackgerrit | Mohammed Naser proposed openstack/openstack-ansible-os_cinder stable/rocky: Add target_helper to backend_defaults https://review.openstack.org/592689 | 20:45 |
| *** ssbarnea has quit IRC | 20:45 | |
| openstackgerrit | Mohammed Naser proposed openstack/openstack-ansible-os_cinder stable/rocky: Drop JSON logging for Cinder https://review.openstack.org/592687 | 20:45 |
| openstackgerrit | Mohammed Naser proposed openstack/openstack-ansible-os_cinder stable/rocky: Fix cinder_environment_overrides https://review.openstack.org/592688 | 20:45 |
| openstackgerrit | Mohammed Naser proposed openstack/openstack-ansible-os_cinder stable/rocky: Add target_helper to backend_defaults https://review.openstack.org/592689 | 20:46 |
| openstackgerrit | Mohammed Naser proposed openstack/openstack-ansible-lxc_hosts stable/rocky: Remove the checksum fills https://review.openstack.org/592690 | 20:47 |
| openstackgerrit | Mohammed Naser proposed openstack/openstack-ansible-lxc_hosts stable/queens: Remove the checksum fills https://review.openstack.org/592692 | 20:47 |
| openstackgerrit | Mohammed Naser proposed openstack/openstack-ansible-lxc_hosts stable/pike: Remove the checksum fills https://review.openstack.org/592693 | 20:47 |
| openstackgerrit | Mohammed Naser proposed openstack/openstack-ansible-lxc_hosts stable/ocata: Remove the checksum fills https://review.openstack.org/592694 | 20:47 |
| *** cshen has quit IRC | 20:47 | |
| *** Darcidride has joined #openstack-ansible | 20:51 | |
| openstackgerrit | Alexandre Bassel proposed openstack/openstack-ansible-ops master: fix intermittent failures when deploying base glance images https://review.openstack.org/592696 | 20:53 |
| *** pcaruana has quit IRC | 20:53 | |
| mnaser | cloudnull: im thinking we should squash https://review.openstack.org/#/c/590778/ with https://review.openstack.org/#/c/591882/ | 20:55 |
| mnaser | because your change will never merge cause it needs the ceph one, and i think its not being stable because of the iptables stuff | 20:55 |
| cloudnull | lets do it | 20:55 |
| *** spatel has quit IRC | 20:55 | |
| *** Darcidride has quit IRC | 20:55 | |
| cloudnull | you want me to do that or you? | 20:55 |
| mnaser | cloudnull: do you mind doing it please? | 20:56 |
| cloudnull | np at all | 20:56 |
| cloudnull | one sec | 20:56 |
| mnaser | hopefully its far more stable | 20:56 |
| openstackgerrit | Kevin Carter (cloudnull) proposed openstack/openstack-ansible master: Temporarily use tempest master for all builds https://review.openstack.org/590778 | 20:58 |
| cloudnull | lets see how that goes | 20:58 |
| openstackgerrit | Alexandre Bassel proposed openstack/openstack-ansible-ops master: fix intermittent failures when deploying base glance images https://review.openstack.org/592696 | 21:00 |
| mnaser | cloudnull: ill keep an eye out and +A if it goes through smoothly in ci | 21:02 |
| mnaser | im excited, i feel good about this | 21:03 |
| mnaser | the way all 4 patches in a stack passed flawlessly in check and gate made me happy | 21:03 |
| openstackgerrit | Alexandre Bassel proposed openstack/openstack-ansible-ops master: fix intermittent failures when deploying base glance images https://review.openstack.org/592696 | 21:03 |
| *** madorn has quit IRC | 21:18 | |
| *** madorn has joined #openstack-ansible | 21:19 | |
| *** aicilamd has quit IRC | 21:20 | |
| *** aicilamd has joined #openstack-ansible | 21:20 | |
| jrosser | mnaser: looks like there is an ordering issue with swift | 21:22 |
| mnaser | jrosser: context? | 21:23 |
| jrosser | as in the various services start / fail with exception / restart about 200-300 times before a valid config gets put down | 21:23 |
| mnaser | cloudnull: do you think there's someway to get nspawn images for centos somewhere other than github? | 21:23 |
| mnaser | oh ouch | 21:23 |
| *** DanyC has joined #openstack-ansible | 21:24 | |
| mnaser | im not all that familiar with it | 21:24 |
| cloudnull | mnaser if there's a better source, absolutly | 21:24 |
| mnaser | i'm wondering if there is somewhere else we can look for them | 21:24 |
| cloudnull | I just took the official cent docker image source | 21:25 |
| *** aicilamd has quit IRC | 21:25 | |
| jrosser | search this for "Started swift" http://logs.openstack.org/07/586607/21/check/openstack-ansible-deploy-aio_lxc-ubuntu-bionic/c4f9187/logs/host/syslog.txt.gz | 21:25 |
| mnaser | cloudnull: yeah im wondering if it can be stored somewhere outside github that we can easily mirror | 21:26 |
| cloudnull | id love it if there was a published rootfs somewhere | 21:26 |
| cloudnull | i couldnt find one | 21:26 |
| cloudnull | but if theres a place we should change to that | 21:27 |
| mnaser | cloudnull: yeah im googling and i cant seem to find anywhere | 21:27 |
| bgmccollum | jrosser looks like the rings need to be dropped before the swift services are started | 21:27 |
| mnaser | i wonder if we can somehow pull it in from docker.io | 21:27 |
| mnaser | jrosser: yeah i think bgmccollum might be on it | 21:27 |
| jrosser | mnaser: cool - it just stood out in syslog when i was looking for something else | 21:28 |
| cloudnull | thats where I got the current image | 21:28 |
| cloudnull | which pulls from githubn | 21:28 |
| *** hamzaachi has joined #openstack-ansible | 21:30 | |
| mnaser | cloudnull: what are your thoughts of bootstrapping a container using yum | 21:31 |
| cloudnull | that would work | 21:31 |
| cloudnull | i think odyssey4me had some work in that direction too | 21:31 |
| mnaser | i think i can find some old ansible code where i got that to work | 21:31 |
| mnaser | and itll pull from mirrors too | 21:31 |
| cloudnull | using bindep LIR | 21:31 |
| mnaser | oh thats an interesting approach | 21:32 |
| mnaser | cloudnull: i see that we've moved to using docker images rather than stuff out of https://us.images.linuxcontainers.org/ .. any history behind this? | 21:35 |
| mnaser | im thinking if we use https://us.images.linuxcontainers.org/ behind infra cache/mirrors, we might see speed ups | 21:36 |
| mnaser | and infra's cache is there to save us from any network issues | 21:36 |
| *** aicilamd has joined #openstack-ansible | 21:36 | |
| mnaser | http://mirror.ca-ymq-1.vexxhost.openstack.org:8080/images.linuxcontainers/ is a thing too | 21:39 |
| openstackgerrit | Merged openstack/openstack-ansible-tests master: Revert "Make CentOS tests non-voting" https://review.openstack.org/592207 | 21:41 |
| openstackgerrit | Byron McCollum proposed openstack/openstack-ansible-os_swift master: [Test] Move swift ring tasks before systemd tasks https://review.openstack.org/592748 | 21:48 |
| *** goldenfri has quit IRC | 21:49 | |
| *** aicilamd has quit IRC | 21:51 | |
| *** aicilamd has joined #openstack-ansible | 21:51 | |
| openstackgerrit | Byron McCollum proposed openstack/openstack-ansible-os_swift master: [Test] Move swift ring tasks before systemd tasks https://review.openstack.org/592748 | 21:52 |
| *** aicilamd has quit IRC | 21:56 | |
| bgmccollum | jrosser ^ we'll see if that helps | 21:56 |
| openstackgerrit | Kevin Carter (cloudnull) proposed openstack/openstack-ansible-tests master: Convert test network setup to use the networkd role https://review.openstack.org/592596 | 21:56 |
| cloudnull | mnaser I'm pretty sure that we have a reverse proxy in infra for the lxc container images | 21:58 |
| cloudnull | however for the most part we dont use those images any longer. | 21:58 |
| mnaser | cloudnull: any reason why? they seem to be fairly up to date | 21:58 |
| cloudnull | the current images are SUPER minimal vs those images. | 21:58 |
| mnaser | ah | 21:58 |
| mnaser | was hoping we can leverage http://mirror.ca-ymq-1.vexxhost.openstack.org:8080/images.linuxcontainers/ again :( | 21:59 |
| cloudnull | we can | 21:59 |
| cloudnull | its a variable | 21:59 |
| mnaser | i just almost finished most of the work in nspawn-hosts :p | 21:59 |
| mnaser | and it's still possible to go the old way | 21:59 |
| cloudnull | https://github.com/openstack/openstack-ansible-lxc_hosts/blob/master/defaults/main.yml#L188-L189 | 21:59 |
| mnaser | with the option of specifying the url still there | 22:00 |
| cloudnull | both lxc and nspawn use the same images | 22:00 |
| cloudnull | yes with the legacy image option, the url can be set | 22:02 |
| cloudnull | https://github.com/openstack/openstack-ansible-lxc_hosts/blob/master/defaults/main.yml#L186 | 22:02 |
| cloudnull | same for nspawn | 22:02 |
| cloudnull | https://github.com/openstack/openstack-ansible-nspawn_hosts/blob/master/defaults/main.yml#L114-L118 | 22:02 |
| openstackgerrit | Mohammed Naser proposed openstack/openstack-ansible-tests master: Clone integrated repo into the base jobs https://review.openstack.org/592573 | 22:04 |
| openstackgerrit | Mohammed Naser proposed openstack/openstack-ansible-tests master: Add NODEPOOL_LXC_IMAGE_PROXY for faster and more reliable images https://review.openstack.org/592752 | 22:04 |
| openstackgerrit | Mohammed Naser proposed openstack/openstack-ansible-nspawn_hosts master: Switch to downloading images from LXC mirrors https://review.openstack.org/592753 | 22:06 |
| mnaser | cloudnull: ^ your thoughts welcome, id want to rerun it twice to see performance difference | 22:06 |
| * mnaser anxiously watched the openstack-ansible patch | 22:11 | |
| mnaser | so far so good | 22:11 |
| mnaser | could it have been the checksum the whole time | 22:11 |
| *** schwicht has quit IRC | 22:12 | |
| openstackgerrit | Mohammed Naser proposed openstack/openstack-ansible-nspawn_hosts master: Switch to downloading images from LXC mirrors https://review.openstack.org/592753 | 22:19 |
| mnaser | https://us.images.linuxcontainers.org/images/centos/7/amd64/default/20180816_02:16/ | 22:21 |
| mnaser | looks like it's 67M for centos | 22:21 |
| mnaser | bionic is 80M | 22:22 |
| mnaser | and it downloaded in 2s in a job, i bet it'd be even faster and more reliable with the proxy and mirrors | 22:22 |
| mnaser | http://zuul.openstack.org/builds.html?job_name=openstack-ansible-deploy-aio_metal-centos-7 | 22:27 |
| *** chyka has joined #openstack-ansible | 22:27 | |
| mnaser | im hoping to see the first green in months | 22:27 |
| *** vnogin has joined #openstack-ansible | 22:39 | |
| openstackgerrit | Merged openstack/openstack-ansible-os_swift master: Move centos tests to voting https://review.openstack.org/587712 | 22:44 |
| *** hamzaachi has quit IRC | 22:52 | |
| *** DanyC has quit IRC | 23:04 | |
| *** DanyC has joined #openstack-ansible | 23:04 | |
| *** vnogin has quit IRC | 23:05 | |
| *** chyka has quit IRC | 23:13 | |
| cloudnull | mnaser the http://zuul.openstack.org/builds.html?job_name=openstack-ansible-deploy-aio_metal-centos-7 jobs shouldn't be using containers. | 23:20 |
| cloudnull | all of those would be gating on host only | 23:20 |
| mnaser | cloudnull: yeah it's the one that is least likely to fail | 23:20 |
| cloudnull | ah ok | 23:20 |
| mnaser | it was failing last time because of the cinder stuff | 23:20 |
| mnaser | but we just got the fixes in so fingers crossed | 23:20 |
| cloudnull | nice! | 23:20 |
| mnaser | for some reason the build got aborted and restarted? | 23:20 |
| *** Leo_m has quit IRC | 23:20 | |
| mnaser | :( | 23:20 |
| mnaser | but still all green for that squashed one | 23:21 |
| *** Leo_m has joined #openstack-ansible | 23:21 | |
| cloudnull | mnaser in the case of ubuntu the images from http://cdimage.ubuntu.com/ubuntu-base/releases/ are about half the size of the ones on the lxc index. | 23:31 |
| cloudnull | maybe it makes sense to change cent to lxc index | 23:31 |
| cloudnull | suse and ubuntu are sourcing from their respective repos | 23:32 |
| * cloudnull going to eat dinner, bbl | 23:33 | |
| *** DanyC has quit IRC | 23:34 | |
| *** aicilamd has joined #openstack-ansible | 23:52 | |
| *** gyee has quit IRC | 23:53 | |
| *** aicilamd has quit IRC | 23:57 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!