| *** ShannonM has quit IRC | 00:01 | |
| openstackgerrit | Kevin Carter proposed openstack/openstack-ansible-os_nova: Implement base configuration for independent repository https://review.openstack.org/286736 | 00:07 |
|---|---|---|
| -cloudnull- if you have review cycles the following reviews could use your left over cycles https://review.openstack.org/#/q/status:open+branch:master+topic:bp/independent-role-repositories and https://review.openstack.org/#/q/status:open+branch:liberty+topic:liberty_upgrade -- TYVM :) | 00:09 | |
| stevelle | and the gate plays Sad Trombone for us | 00:21 |
| *** jmckind has joined #openstack-ansible | 00:23 | |
| *** skamithi has joined #openstack-ansible | 00:28 | |
| jmccrory | hmmm so --no-download is only in 14.x virtulenv | 00:30 |
| *** thorst_ has joined #openstack-ansible | 00:30 | |
| jmccrory | an option in* | 00:31 |
| cloudnull | yes ... its never-download | 00:33 |
| stevelle | devilish details, yeah | 00:33 |
| openstackgerrit | Kevin Carter proposed openstack/openstack-ansible-repo_build: Updated venv build process for virtualenv compat https://review.openstack.org/286923 | 00:35 |
| cloudnull | that issue is caused by the updated constraints in the repo-server limiting us to virtualenv from global requirements. | 00:36 |
| cloudnull | jmccrory stevelle ^^ that should fix it for us | 00:36 |
| stevelle | cloudnull: watching it, but it's happy hour :) | 00:37 |
| *** evilrob has left #openstack-ansible | 00:37 | |
| cloudnull | its passing local checks. | 00:37 |
| * cloudnull babysitting the gate. | 00:37 | |
| *** thorst_ has quit IRC | 00:38 | |
| *** thorst_ has joined #openstack-ansible | 00:38 | |
| lane_kong | hi, guys, i am working on developing ansible role for zaqar service, I know a little ansible, but not very familiar with. I came across a problem, please see http://paste.openstack.org/show/488879/ | 00:39 |
| lane_kong | the project structure of the ansible role is very like other services. | 00:40 |
| cloudnull | lane_kong: it would seem that your venv path is not being detected as a venv | 00:40 |
| *** thorst_ has quit IRC | 00:40 | |
| *** thorst_ has joined #openstack-ansible | 00:41 | |
| *** help has joined #openstack-ansible | 00:41 | |
| lane_kong | cloudnull: thanks for reply. but how it could be a venv? | 00:41 |
| *** help is now known as Guest52792 | 00:41 | |
| *** jwitko has quit IRC | 00:41 | |
| lane_kong | there is just a directory call 'bin' in /openstack/venvs/zaqar-untagged | 00:41 |
| lane_kong | cloudnull: this is the zaqar_install.yml: http://paste.openstack.org/show/488881/ | 00:43 |
| lane_kong | cloudnull: appreciate if you could give some advise | 00:43 |
| cloudnull | lane_kong: so the venv directory doesnt have all of the other bits ? | 00:46 |
| cloudnull | lane_kong: do you have a role i can pull in and try out ? | 00:46 |
| lane_kong | cloudnull: i forked from https://github.com/stevelle/openstack-ansible-zaqar, made a few mofication internally | 00:48 |
| lane_kong | we want to deploy zaqar using ansbile in our comany | 00:48 |
| lane_kong | i am afraid the original repo didn't work | 00:49 |
| lane_kong | cloudnull: if you need , i can paste all the file of the role for you | 00:49 |
| *** phalmos has quit IRC | 00:50 | |
| cloudnull | lane_kong: can you provide a diff from the chanage you have locally | 00:50 |
| cloudnull | i can patch the original role and see if i can find something | 00:50 |
| lane_kong | cloudnull: yes, sure, wait for a moment | 00:50 |
| *** scarlisle has joined #openstack-ansible | 00:58 | |
| *** woodard has quit IRC | 01:00 | |
| *** jguy_ has joined #openstack-ansible | 01:01 | |
| *** woodard has joined #openstack-ansible | 01:01 | |
| *** sdake has joined #openstack-ansible | 01:03 | |
| *** sdake has quit IRC | 01:16 | |
| *** scarlisle has quit IRC | 01:17 | |
| *** sdake has joined #openstack-ansible | 01:31 | |
| cloudnull | anyone around to shove this through https://review.openstack.org/#/c/286923/ | 01:32 |
| cloudnull | stevelle jmccrory ^ | 01:32 |
| jmccrory | good to go | 01:33 |
| *** sdake has quit IRC | 01:33 | |
| openstackgerrit | Merged openstack/openstack-ansible-repo_build: Updated venv build process for virtualenv compat https://review.openstack.org/286923 | 01:33 |
| cloudnull | so once thats through we can move on to the next problem :) | 01:34 |
| cloudnull | whatever that is ... | 01:34 |
| openstackgerrit | Kevin Carter proposed openstack/openstack-ansible: Implements IRR for the repo_build role https://review.openstack.org/284896 | 01:34 |
| stevelle | let's roll the dice and see | 01:35 |
| *** izaakk has quit IRC | 01:37 | |
| *** Mudpuppy has joined #openstack-ansible | 01:38 | |
| lane_kong | cloudnull: appreciate your help anyway | 01:41 |
| lane_kong | you helped me find what's the problem is | 01:42 |
| cloudnull | anytime | 01:42 |
| *** fawadkhaliq has quit IRC | 01:42 | |
| stevelle | checking back in a bit on that gate. got a jam session scheduled | 01:47 |
| lane_kong | cloudnull: one more question, is there any guidance for creating pre-build venv? | 01:53 |
| *** thorst_ has quit IRC | 01:56 | |
| *** thorst_ has joined #openstack-ansible | 01:57 | |
| *** scarlisle has joined #openstack-ansible | 02:01 | |
| *** thorst_ has quit IRC | 02:05 | |
| cloudnull | lane_kong: so with your changes + what ive done here https://github.com/os-cloud/openstack-ansible-os_zaqar | 02:07 |
| cloudnull | the role should work | 02:07 |
| cloudnull | to build the venv in an isolated fashion set -e zaqar_developer_mode=true | 02:07 |
| cloudnull | that will build the zaqar source and install it in a local venv | 02:08 |
| cloudnull | without any of the additional OSA machinery | 02:08 |
| lane_kong | cloudnull: man, thanks for much! | 02:08 |
| lane_kong | so much | 02:08 |
| lane_kong | cloudnull: i will test it, hope it could work for me | 02:09 |
| *** Guest52792 has quit IRC | 02:12 | |
| *** weezS has quit IRC | 02:13 | |
| *** furlongm_ has quit IRC | 02:21 | |
| *** furlongm has joined #openstack-ansible | 02:32 | |
| *** jmckind is now known as jmckind_ | 02:40 | |
| *** daneyon has joined #openstack-ansible | 02:42 | |
| *** daneyon_ has quit IRC | 02:45 | |
| *** jmckind_ has quit IRC | 02:46 | |
| *** daneyon has quit IRC | 02:47 | |
| *** woodard has quit IRC | 03:04 | |
| *** thorst_ has joined #openstack-ansible | 03:04 | |
| *** thorst_ has quit IRC | 03:10 | |
| *** sdake has joined #openstack-ansible | 03:15 | |
| *** weshay has quit IRC | 03:22 | |
| *** scarlisle has quit IRC | 03:22 | |
| cloudnull | if any cores are around can we bang this though https://review.openstack.org/#/c/284896/ <- stevelle andymccr odyssey4me d34dh0r53 jmccrory automagically mattt hughsaunders | 03:22 |
| *** jmckind has joined #openstack-ansible | 03:22 | |
| cloudnull | which will finally unblock the gate. | 03:23 |
| * cloudnull going to eat will check back later. maybe | 03:23 | |
| stevelle | ^ done | 03:24 |
| *** jmckind is now known as jmckind_ | 03:29 | |
| prometheanfire | aren't you a helper | 03:30 |
| *** sigmavirus24 is now known as sigmavirus24_awa | 03:34 | |
| *** jmckind_ has quit IRC | 03:35 | |
| *** sdake has quit IRC | 03:39 | |
| *** jmckind has joined #openstack-ansible | 03:39 | |
| *** fawadkhaliq has joined #openstack-ansible | 03:43 | |
| *** sdake has joined #openstack-ansible | 03:53 | |
| openstackgerrit | Jimmy McCrory proposed openstack/openstack-ansible-repo_server: Use slurp to collect the repo_server ssh keys https://review.openstack.org/286042 | 03:54 |
| coolj | jmccrory: myself and others ran into the same issue even with the slurp patch https://bugs.launchpad.net/openstack-ansible/+bug/1531783 | 04:02 |
| openstack | Launchpad bug 1531783 in openstack-ansible liberty "os_nova not using all configured compute_hosts" [High,Triaged] - Assigned to Jesse Pretorius (jesse-pretorius) | 04:02 |
| jmccrory | coolj, yeah. it seems to happen if you have any host in the group involved that fails before that task | 04:03 |
| *** cemmason has joined #openstack-ansible | 04:04 | |
| jmccrory | and trying to use a limit breaks it as well | 04:04 |
| *** shausy has joined #openstack-ansible | 04:06 | |
| *** thorst_ has joined #openstack-ansible | 04:08 | |
| *** jmckind has quit IRC | 04:09 | |
| jmccrory | coolj do you still have a deployment that's failing because of that? | 04:10 |
| *** skamithi has quit IRC | 04:11 | |
| coolj | jmccrory: i was helping the deployment team troubleshoot, and i don't recall who we were adding nodes for. i can follow up with them tomorrow and see if we have any ongoing or that have hit it recently | 04:14 |
| *** thorst_ has quit IRC | 04:16 | |
| *** gfa is now known as gfa_ | 04:18 | |
| jmccrory | oh ok, also ran into it a team here a week or two ago. thinking adding a '| default("")' to the end of this line will help, will test it tomorrow https://github.com/openstack/openstack-ansible-os_nova/blob/master/tasks/nova_compute_key_distribute.yml#L19 | 04:18 |
| openstackgerrit | Jimmy McCrory proposed openstack/openstack-ansible-repo_server: Removed sshd_config file https://review.openstack.org/284410 | 04:20 |
| *** Mudpuppy has quit IRC | 04:39 | |
| *** jmckind has joined #openstack-ansible | 04:39 | |
| *** Mudpuppy has joined #openstack-ansible | 04:40 | |
| *** furlongm has quit IRC | 04:45 | |
| *** furlongm has joined #openstack-ansible | 04:48 | |
| openstackgerrit | Merged openstack/openstack-ansible-repo_server: Removed sshd_config file https://review.openstack.org/284410 | 04:50 |
| openstackgerrit | Merged openstack/openstack-ansible-repo_server: Use slurp to collect the repo_server ssh keys https://review.openstack.org/286042 | 04:56 |
| *** fawadkhaliq has quit IRC | 04:58 | |
| *** cemmason has quit IRC | 04:59 | |
| *** furlongm has quit IRC | 05:02 | |
| *** sdake has quit IRC | 05:09 | |
| *** furlongm has joined #openstack-ansible | 05:09 | |
| *** thorst_ has joined #openstack-ansible | 05:14 | |
| *** thorst_ has quit IRC | 05:21 | |
| *** furlongm has quit IRC | 05:25 | |
| *** cemmason has joined #openstack-ansible | 05:31 | |
| *** admin0 has joined #openstack-ansible | 05:33 | |
| *** pcaruana has joined #openstack-ansible | 05:36 | |
| *** admin0 has quit IRC | 05:37 | |
| *** javeriak has joined #openstack-ansible | 05:37 | |
| *** admin0 has joined #openstack-ansible | 05:37 | |
| *** furlongm has joined #openstack-ansible | 05:40 | |
| *** pcaruana has quit IRC | 05:49 | |
| *** cemmason has quit IRC | 05:54 | |
| *** cemmason has joined #openstack-ansible | 05:54 | |
| *** Mudpuppy has quit IRC | 05:56 | |
| *** jmckind has quit IRC | 05:59 | |
| *** javeriak has quit IRC | 06:07 | |
| *** furlongm has quit IRC | 06:13 | |
| *** thorst_ has joined #openstack-ansible | 06:19 | |
| *** furlongm has joined #openstack-ansible | 06:21 | |
| *** phiche has joined #openstack-ansible | 06:25 | |
| *** thorst_ has quit IRC | 06:26 | |
| *** furlongm has quit IRC | 06:26 | |
| *** furlongm has joined #openstack-ansible | 06:29 | |
| *** furlongm has quit IRC | 06:34 | |
| *** gfa_ is now known as gfa | 06:37 | |
| *** jamielennox is now known as jamielennox|away | 06:38 | |
| *** furlongm has joined #openstack-ansible | 06:38 | |
| *** furlongm has quit IRC | 06:39 | |
| *** furlongm has joined #openstack-ansible | 06:43 | |
| *** admin0 has quit IRC | 06:49 | |
| *** jamielennox|away is now known as jamielennox | 06:49 | |
| *** phiche has quit IRC | 06:57 | |
| *** furlongm has quit IRC | 06:58 | |
| *** furlongm has joined #openstack-ansible | 06:58 | |
| *** phiche has joined #openstack-ansible | 07:04 | |
| *** jamielennox is now known as jamielennox|away | 07:08 | |
| *** furlongm_ has joined #openstack-ansible | 07:16 | |
| *** furlongm has quit IRC | 07:18 | |
| *** thorst_ has joined #openstack-ansible | 07:23 | |
| openstackgerrit | Matt Thompson proposed openstack/openstack-ansible-os_aodh: Implement base configuration for independent repository https://review.openstack.org/286655 | 07:28 |
| openstackgerrit | Matt Thompson proposed openstack/openstack-ansible-os_glance: Implement base configuration for independent repository https://review.openstack.org/286667 | 07:29 |
| openstackgerrit | Matt Thompson proposed openstack/openstack-ansible-os_heat: Implement base configuration for independent repository https://review.openstack.org/286693 | 07:30 |
| *** thorst_ has quit IRC | 07:31 | |
| openstackgerrit | Matt Thompson proposed openstack/openstack-ansible-pip_install: Test version of pip being installed https://review.openstack.org/285284 | 07:32 |
| *** mikelk has joined #openstack-ansible | 07:47 | |
| *** markvoelker has quit IRC | 07:58 | |
| *** admin0 has joined #openstack-ansible | 08:00 | |
| *** pcaruana has joined #openstack-ansible | 08:02 | |
| *** raddaoui has joined #openstack-ansible | 08:15 | |
| *** jamielennox|away is now known as jamielennox | 08:25 | |
| *** thorst_ has joined #openstack-ansible | 08:29 | |
| *** raddaoui has quit IRC | 08:30 | |
| *** thorst_ has quit IRC | 08:35 | |
| *** permalac has joined #openstack-ansible | 08:42 | |
| *** permalac has quit IRC | 08:42 | |
| *** winggundamth has joined #openstack-ansible | 08:42 | |
| *** zhangjn has quit IRC | 08:43 | |
| openstackgerrit | Merged openstack/openstack-ansible-os_nova: Implement base configuration for independent repository https://review.openstack.org/286736 | 08:46 |
| *** zhangjn has joined #openstack-ansible | 08:48 | |
| *** markvoelker has joined #openstack-ansible | 08:58 | |
| *** permalac has joined #openstack-ansible | 09:00 | |
| *** javeriak has joined #openstack-ansible | 09:01 | |
| *** asettle has joined #openstack-ansible | 09:05 | |
| winggundamth | I'm using stable/liberty and have problem install repo oslo right now | 09:06 |
| winggundamth | Could not find a version that satisfies the requirement oslo.i18n<1.6.0,>=1.5.0 (from ceilometer) (from versions: 3.1.0) | 09:06 |
| winggundamth | No matching distribution found for oslo.i18n<1.6.0,>=1.5.0 (from ceilometer) | 09:06 |
| winggundamth | what is the workaround for now? | 09:06 |
| winggundamth | should I change to use master instead? I see a lot of major change version so I think it can break the installation | 09:07 |
| openstackgerrit | Merged openstack/openstack-ansible-os_cinder: Implement base configuration for independent repository https://review.openstack.org/286682 | 09:17 |
| mattt | winggundamth: i'd not recommend using master | 09:17 |
| mattt | winggundamth: fine for development, not for a live deployment | 09:17 |
| mattt | winggundamth: it sounds like we may have some dependency issues, let me build a liberty installation now and see if i can troubleshoot | 09:17 |
| mattt | winggundamth: are you on a specific liberty tag? | 09:17 |
| odyssey4me | mattt based on some of what gus was saying last night, I have the feeling that the pip/wheel/setuptools trio is causing breakage and I think we need to get the appropriate pinning back into liberty / kilo as currently we're pip<8 but wheel and setuptools are not properly pinned to ensure compatibility | 09:20 |
| odyssey4me | effectively wheel is building wheels that pip can't install because of the change in names - same as kilo (except we pinned wheel in kilo) | 09:20 |
| mattt | :( | 09:20 |
| mattt | master is moving again ? | 09:20 |
| admin0 | guys, no change was done in the current branch(liberty) right :D? | 09:21 |
| odyssey4me | mattt not sure yet - waiting for some rechecks to confirm, but it looks like the necessary changes have merged | 09:24 |
| odyssey4me | admin0 changes happen from time to time - you can see it with https://review.openstack.org/#/q/status:merged+project:openstack/openstack-ansible+branch:liberty or simply in the git history | 09:25 |
| winggundamth | mattt: yes. I'm using liberty | 09:25 |
| odyssey4me | admin0 do you have a specific question there? | 09:25 |
| admin0 | question was .. all the gate/changes of yesterday, where do they end up ? | 09:25 |
| mattt | odyssey4me: any idea why https://review.openstack.org/#/c/286694/ needed to fix a flake8 violation? why was that not being picked up before? | 09:25 |
| mattt | ^^^ jmccrory if you're awake :) | 09:25 |
| mattt | winggundamth: great -- but are you using a specific tag? | 09:26 |
| winggundamth | mattt: I'm doing git clone -b liberty https://github.com/openstack/openstack-ansible /opt/openstack-ansible. is this answer that you want? | 09:27 |
| mattt | winggundamth: yeah that helps, thanks :) i'll do a build now to test | 09:27 |
| odyssey4me | mattt I guess our choices are to ignore E501 violations, to implement a longer max line length as in https://review.openstack.org/#/c/286696/4/tox.ini or to not do '#!{{ swift_venv_enabled | bool | ternary(swift_venv_bin + "/", "/usr/bin/env ") }}python' at the top of some of the python files? | 09:28 |
| odyssey4me | or, of course, to just ignore particular files - I'mnot really in favour of this option as it leads down a dark path | 09:28 |
| winggundamth | mattt: http://eavesdrop.openstack.org/irclogs/%23openstack-ansible/%23openstack-ansible.2015-07-23.log.html I'm seeing that someone said the rackspace repo missing that oslo version | 09:29 |
| *** thorst_ has joined #openstack-ansible | 09:33 | |
| odyssey4me | winggundamth the rpc-repo doesn't have to have it - it comes from pypi if that repo doesn't have it | 09:33 |
| *** cemmason1 has joined #openstack-ansible | 09:34 | |
| *** cemmason1 has quit IRC | 09:34 | |
| winggundamth | odyssey4me: ic so I'm trying to dig into that right now | 09:34 |
| *** cemmason1 has joined #openstack-ansible | 09:34 | |
| winggundamth | I think packages version 1.5-1.6 has been missing. https://pypi.python.org/pypi/oslo.i18n | 09:35 |
| winggundamth | https://pypi.python.org/pypi/oslo.i18n/1.5.0 it still there | 09:36 |
| openstackgerrit | Merged openstack/openstack-ansible-os_neutron: Implement base configuration for independent repository https://review.openstack.org/286694 | 09:36 |
| openstackgerrit | Jesse Pretorius (odyssey4me) proposed openstack/openstack-ansible-os_horizon: [WIP] Implement base configuration for independent repository https://review.openstack.org/286650 | 09:36 |
| openstackgerrit | Jesse Pretorius (odyssey4me) proposed openstack/openstack-ansible-os_horizon: Implement base configuration for independent repository https://review.openstack.org/286650 | 09:37 |
| *** cemmason has quit IRC | 09:37 | |
| *** thorst_ has quit IRC | 09:40 | |
| odyssey4me | mattt what do you think of implementing https://review.openstack.org/286591 in all the roles? it gives us the equivalent of verbose output, but in a readable format | 09:42 |
| mattt | odyssey4me: yeah seems logical | 09:44 |
| winggundamth | mattt: this is error I got from ansible in case you want more detail http://paste.openstack.org/show/488918/ | 09:44 |
| odyssey4me | winggundamth an option you can go with is to checkout to the last tag in the liberty branch - if the head of the branch is broken then we can fix that up (we have a known issue to deal with), but the last tag should hopefully still work | 09:45 |
| *** lane_kong is now known as lynn | 09:45 | |
| *** lynn is now known as Guest29970 | 09:46 | |
| *** Guest29970 is now known as lynn_kong | 09:46 | |
| mattt | winggundamth: cool, running my build now | 09:46 |
| winggundamth | odyssey4me: but from my git clone -b liberty command. it should checkout the last tag of liberty branch right? | 09:49 |
| openstackgerrit | Jesse Pretorius (odyssey4me) proposed openstack/openstack-ansible-os_keystone: [WIP] Implement base configuration for independent repository https://review.openstack.org/286660 | 09:51 |
| odyssey4me | winggundamth the best thing to do for now is to wait for mattt's test to confirm the issue and find a solution | 09:53 |
| mattt | in gerrit, can i watch projects based on pattern rather than adding each in individually? | 09:53 |
| odyssey4me | winggundamth but if you're in a hurry you can 'git tag -l', then 'git checkout -b <tag number>' the latest 12.0.x tag, then execute the playbooks | 09:54 |
| odyssey4me | winggundamth notes that you will have to at the very least re-execute the repo-build | 09:54 |
| odyssey4me | mattt not to my knowledge, which is why I used the review dashboard in the topic :) | 09:54 |
| *** javeriak has quit IRC | 09:57 | |
| *** javeriak has joined #openstack-ansible | 09:58 | |
| *** electrofelix has joined #openstack-ansible | 10:02 | |
| openstackgerrit | Jesse Pretorius (odyssey4me) proposed openstack/openstack-ansible-os_keystone: [WIP] Implement base configuration for independent repository https://review.openstack.org/286660 | 10:03 |
| openstackgerrit | Merged openstack/openstack-ansible-os_swift: Implement base configuration for independent repository https://review.openstack.org/286692 | 10:05 |
| *** javeriak has quit IRC | 10:08 | |
| openstackgerrit | Jesse Pretorius (odyssey4me) proposed openstack/openstack-ansible-os_neutron: Implement E501 flake8 check with longer line limit https://review.openstack.org/287072 | 10:08 |
| winggundamth | odyssey4me: I just got what you talk about the branch after I check and seeing your branching system now :) | 10:08 |
| openstackgerrit | Merged openstack/openstack-ansible-os_swift_sync: Implement base configuration for independent repository https://review.openstack.org/286696 | 10:09 |
| *** lynn_kong is now known as Larrie | 10:12 | |
| *** Larrie is now known as larrie | 10:12 | |
| openstackgerrit | Jesse Pretorius (odyssey4me) proposed openstack/openstack-ansible-os_keystone: [WIP] Implement base configuration for independent repository https://review.openstack.org/286660 | 10:12 |
| *** javeriak has joined #openstack-ansible | 10:15 | |
| openstackgerrit | Matt Thompson proposed openstack/openstack-ansible-os_neutron: Set max-line-length and remove E501 ignore https://review.openstack.org/287079 | 10:16 |
| odyssey4me | hahaha mattt I already did that https://review.openstack.org/287072 | 10:22 |
| mattt | odyssey4me: ARGH :) | 10:23 |
| mattt | abandoned mine :) | 10:23 |
| odyssey4me | heh, great minds think alike :p | 10:24 |
| *** larrie is now known as larrie_kong | 10:28 | |
| winggundamth | do you know how to replace all openstack repository with user_variables.yml? I put var like requirements_git_repo but it end up clone requirements 2 times. one is main repo and another is the one I put | 10:31 |
| winggundamth | My company is lack of international bandwidth so I have to mirror repository on our gitlab and want openstack ansible clone from there instead of the main one | 10:32 |
| *** thorst_ has joined #openstack-ansible | 10:38 | |
| odyssey4me | winggundamth in your user_variably.yml in /etc/openstack_deploy you need to set the *_git_repo vars from https://github.com/openstack/openstack-ansible/blob/master/playbooks/defaults/repo_packages/*.yml to the host you want to git clone from, and ensure that your gitlab server has all those repositories | 10:39 |
| odyssey4me | mattt http://logs.openstack.org/60/286660/5/check/gate-openstack-ansible-os_keystone-dsvm-ansible-functional-trusty/1759d25/console.html#_2016-03-02_10_31_08_434 is the failure in https://review.openstack.org/286660 - I need to go afk for a while, if you have a gap you can continue to update the patch to make it work | 10:40 |
| winggundamth | odyssey4me: yes I already did that but it end up cloning 2 times. | 10:40 |
| odyssey4me | winggundamth hmm, that sounds like a bug | 10:41 |
| winggundamth | oddyssey4me: please see output here http://paste.openstack.org/show/488923/ | 10:42 |
| winggundamth | you can see it clone git.openstack.org and git.example.com twice on the same repo | 10:42 |
| winggundamth | I think maybe because of this so it cause error that mattt trying to reproduce now also | 10:43 |
| winggundamth | because it end up cloning and check out on master branch | 10:44 |
| mattt | odyssey4me: sure | 10:45 |
| *** thorst_ has quit IRC | 10:45 | |
| *** hlkv6_ is now known as hlkv6 | 10:48 | |
| winggundamth | I checkout 12.0.6 and doing setup_infrastructure.yml still error at the same task but difference error now http://paste.openstack.org/show/488929/ | 11:01 |
| *** phiche has quit IRC | 11:03 | |
| *** phiche has joined #openstack-ansible | 11:03 | |
| mattt | winggundamth: my build has almost finished | 11:04 |
| mattt | winggundamth: so let me understand something | 11:04 |
| mattt | winggundamth: you did a build off liberty, then presumably did a git pull on your openstack-ansible checkout some time later? | 11:04 |
| mattt | winggundamth: if so, you will need to re-run the repo-build.yml play to rebuild all your wheels, venvs, etc. | 11:05 |
| mattt | winggundamth: once that has been done, you can re-run the openstack plays | 11:05 |
| mattt | winggundamth: ah, i see it was your repo-build run that failed :( | 11:06 |
| winggundamth | yeah | 11:06 |
| winggundamth | mattt: please see my latest error from 12.0.6 | 11:07 |
| mattt | winggundamth: what commit is your branch on ? | 11:07 |
| winggundamth | mattt: 972b41a061be353ca7f44ddc4e3702aae3433a5d | 11:08 |
| winggundamth | I'm sorry but I'm changing to 12.0.6 now | 11:08 |
| mattt | winggundamth: ok go back to liberty branch | 11:09 |
| mattt | adn tell me what commit ? | 11:09 |
| winggundamth | 3e07fb28a649f2c3e32c8338200d4a21f63c0825 | 11:10 |
| mattt | winggundamth: yeah that's the head, which i'm on also | 11:10 |
| mattt | confused why my repo-build ran fine | 11:10 |
| winggundamth | :( | 11:11 |
| winggundamth | do you know the command for that task in ansible? | 11:11 |
| mattt | winggundamth: well what did you run? | 11:11 |
| winggundamth | so I can manually run it in container and hope I can see any useful error | 11:11 |
| mattt | winggundamth: setup-infrastrucutre? | 11:11 |
| winggundamth | yes | 11:11 |
| mattt | winggundamth: the specific play is repo-build.yml, try that | 11:11 |
| winggundamth | sure | 11:12 |
| winggundamth | running it now | 11:14 |
| winggundamth | still the same :( | 11:22 |
| winggundamth | Collecting oslo.i18n<1.6.0,>=1.5.0 (from keystone) | 11:22 |
| winggundamth | Could not find a version that satisfies the requirement oslo.i18n<1.6.0,>=1.5.0 (from keystone) (from versions: 3.1.0) | 11:22 |
| winggundamth | No matching distribution found for oslo.i18n<1.6.0,>=1.5.0 (from keystone) | 11:22 |
| winggundamth | Ignoring indexes: https://pypi.python.org/simple | 11:22 |
| winggundamth | I suspect about it said Ignoring indexes | 11:23 |
| odyssey4me | is that during the venv build? | 11:23 |
| odyssey4me | yeah, it looks like it | 11:24 |
| odyssey4me | it's supposed to ignore pypi by that stage | 11:24 |
| odyssey4me | it should already have all the packages compiled into wheels by then | 11:24 |
| winggundamth | ic | 11:24 |
| mattt | winggundamth: this is a liberty commit from today, which passed fine: http://logs.openstack.org/91/271591/5/check/gate-openstack-ansible-dsvm-commit/c7ee2a2/console.html | 11:25 |
| odyssey4me | if you cd into /opt/openstack-ansible/playbooks, then run 'ansible repo_all -m shell -a "ls -al /var/www/repo/links/oslo.*' what does it result in? | 11:25 |
| odyssey4me | winggundamth also, you mentioned that you have an internal git repo - have you implemented any other changes to cater for local isolation? | 11:26 |
| mattt | it sounds like something isn't jiving | 11:26 |
| winggundamth | odyssey4me: http://paste.openstack.org/show/488935/ | 11:27 |
| mattt | odyssey4me: just finished up looking at something for asettle, did you still want me to poke that review ? | 11:27 |
| * asettle appears | 11:27 | |
| mattt | winggundamth: your containers not syncing is also an issue, those files should be the same across the lot | 11:27 |
| mattt | winggundamth: from my deploy which just finished: http://paste.openstack.org/show/488936/ | 11:28 |
| winggundamth | mattt: should I rerun playbook from the setup-hosts.yml? | 11:29 |
| mattt | winggundamth: no, let's figure this out | 11:29 |
| mattt | why you do not have oslo.i18n | 11:30 |
| winggundamth | okay | 11:30 |
| winggundamth | this is my user_variables.yml http://paste.openstack.org/show/488937/ | 11:31 |
| winggundamth | the others is default | 11:31 |
| *** tiagogomes has quit IRC | 11:34 | |
| winggundamth | mattt: I can see from ls command that I have /var/www/repo/links/oslo.i18n-3.1.0-py2.py3-none-any.whl | 11:35 |
| mattt | winggundamth: ah right, weird wrapping on that paste | 11:35 |
| odyssey4me | ah, so you do have it - which likely means that the issue is that the repo sync hasn't happened | 11:36 |
| winggundamth | mattt: but from the error why it said oslo.i18n<1.6.0,>=1.5.0 not only oslo.i18n>=1.5.0 | 11:36 |
| odyssey4me | in the container that has the packages, restart the lsync service | 11:36 |
| odyssey4me | winggundamth also, are your git sources up to date - and are you sure that you have them all? | 11:37 |
| winggundamth | odyssey4me: what do you mean git sources up to date? | 11:37 |
| odyssey4me | winggundamth you're mirroring the git.o.o repositories for global-requirements, nova, heat, keystone, etc right? (in gitlab) - please make sure that they are all there and up to date | 11:38 |
| *** furlongm_ has quit IRC | 11:38 | |
| mattt | winggundamth: which paste has the oslo.i18n error in it ? | 11:38 |
| winggundamth | odyssey4me: oh yes. I'm sure I have it all. I'm just update it last few hours ago | 11:38 |
| odyssey4me | winggundamth you should have all the repositories in https://github.com/openstack/openstack-ansible/blob/liberty/playbooks/defaults/repo_packages/openstack_other.yml and https://github.com/openstack/openstack-ansible/blob/liberty/playbooks/defaults/repo_packages/openstack_services.yml | 11:39 |
| *** furlongm_ has joined #openstack-ansible | 11:39 | |
| winggundamth | odyssey4me: yes. I'm sure | 11:39 |
| winggundamth | mattt: http://paste.openstack.org/show/488940/ | 11:40 |
| winggundamth | mattt: here you go | 11:40 |
| mattt | maybe you didnt' update your ceilometer sources? :) | 11:40 |
| winggundamth | what I really curious is it asking for <1.6.0 that it shouldn't | 11:41 |
| odyssey4me | winggundamth it gets requirements from the service git source | 11:41 |
| winggundamth | because I have 3.1.0 in my repo container right? | 11:41 |
| winggundamth | ic | 11:41 |
| odyssey4me | winggundamth but all requirements are taken into account in the build process up-front, so that seems like you have something out of date | 11:41 |
| odyssey4me | mattt I think that https://review.openstack.org/284896 has to merge before master can be unblocked - it's busy being rechecked as it keeps hitting timeouts | 11:42 |
| winggundamth | and it is the same version that mattt successful build it too | 11:42 |
| mattt | winggundamth: https://github.com/openstack/ceilometer/blob/510aa7897a46688b9826ac8800455de03fe3dc34/requirements.txt#L18 | 11:42 |
| mattt | i have a feeling your ceilometer source may be out of date | 11:42 |
| mattt | can you verify what you have matches that? | 11:43 |
| winggundamth | sure | 11:43 |
| odyssey4me | mattt perhaps check the corresponding upper-constraints too? | 11:43 |
| *** thorst_ has joined #openstack-ansible | 11:43 | |
| winggundamth | mattt: http://paste.openstack.org/show/488942/ this is my requirements.txt from celilometer | 11:44 |
| odyssey4me | winggundamth there's a set of *.txt files in your repo server under /var/www/repo/os-releases/<tag>/ | 11:45 |
| *** tiagogomes has joined #openstack-ansible | 11:45 | |
| odyssey4me | pastebin the contents of those please | 11:45 |
| winggundamth | odyssey4me: this is from 12.0.7 tag https://www.dropbox.com/sh/knhehz5r9nt3usa/AADDJvoUgMw0HiG2ynkIbUhMa?dl=0 | 11:48 |
| openstackgerrit | Jesse Pretorius (odyssey4me) proposed openstack/openstack-ansible: IRR - Implemented for os_keystone https://review.openstack.org/279710 | 11:48 |
| *** tiagogomes has quit IRC | 11:50 | |
| odyssey4me | so global requirements has oslo.i18n>=1.5.0 and constraints has oslo.i18n<=3.1.0 | 11:50 |
| *** tiagogomes has joined #openstack-ansible | 11:50 | |
| odyssey4me | that's why you have 3.1.0 in your repo | 11:50 |
| *** thorst_ has quit IRC | 11:51 | |
| winggundamth | odyssey4me: yes same as mattt | 11:51 |
| *** Xu_Chao has joined #openstack-ansible | 11:51 | |
| winggundamth | odyssey4me: but what hit me on the wall is why the error show constraints <1.6.0 | 11:51 |
| winggundamth | odyssey4me: I couldn't find that constraint anywhere | 11:51 |
| odyssey4me | yeah, that's weird | 11:52 |
| winggundamth | odyssey4me: maybe I have to reverse this line with_dict: local_packages.results.0.item.role_packages to see where it get that packages from? | 11:54 |
| *** furlongm_ has quit IRC | 11:54 | |
| odyssey4me | wing perhaps try executing 'openstack-ansible -vv repo-build.yml' so that you can see the commands being executed and their output | 11:54 |
| *** furlongm_ has joined #openstack-ansible | 11:55 | |
| winggundamth | odyssey4me: already done that with -vvvv but nothing useful at all | 11:56 |
| odyssey4me | winggundamth check the versions of pip, wheel and setuptools across the repo_all group and the hosts group | 11:56 |
| odyssey4me | wheel will come out of 'pip freeze' | 11:56 |
| odyssey4me | pip has a version option I think | 11:56 |
| odyssey4me | setuptools you may have to find the folder on the fs | 11:56 |
| winggundamth | odyssey4me: http://paste.openstack.org/show/488944/ | 11:57 |
| winggundamth | odyssey4me: this is output from -vvvv | 11:57 |
| odyssey4me | winggundamth I see that https://review.openstack.org/284247 got added after that tag - try checking back out to the head of the liberty branch | 11:59 |
| odyssey4me | then execute the repo-install.yml playbook | 11:59 |
| winggundamth | odyssey4me: this is version http://paste.openstack.org/show/488945/ | 12:01 |
| winggundamth | odyssey4me: let me checkout that commit | 12:02 |
| odyssey4me | mattt remember this? https://github.com/openstack/openstack-ansible/commit/81f45653fa6bba4bc63fc44785b422b44f321a17 | 12:03 |
| odyssey4me | we haven't yet sorted it all out in liberty | 12:03 |
| *** sdake has joined #openstack-ansible | 12:03 | |
| mattt | sadly i do, but that's not the issue here is it? | 12:03 |
| odyssey4me | I think that the wheel is being built in a way that pip can't properly interpret | 12:04 |
| odyssey4me | it's a guess | 12:04 |
| odyssey4me | and it is weird that we're not seeing this in our gate builds | 12:04 |
| winggundamth | mattt odyssey4me do you think if I remove all the containers and rerun whole script again can fix this? | 12:05 |
| winggundamth | also I actually use deploy host as the same as one management host. is that can cause the problem? | 12:12 |
| *** admin0 has quit IRC | 12:12 | |
| logan- | Hey winggundamth it is not using your local repos from your gitlab due to the trailing .git on the url. Add something like "cinder_git_package_name: cinder" to all of your override definitions | 12:12 |
| *** sdake has quit IRC | 12:13 | |
| winggundamth | logan: I'm not sure I'm understand. so you suggest me to append '.git' to every gitlab repository? and add *_git_package_name: variable to all repository? | 12:15 |
| *** admin0 has joined #openstack-ansible | 12:16 | |
| logan- | No you already have .git at the end of the repo urls right? (I think gitlab requires this where github does not). This trailing .git in the url confuses repo build and if you look in /var/www/repo/openstack git you will see there are 2 copies of all the repos | 12:16 |
| winggundamth | logan-: exactly right | 12:17 |
| winggundamth | logan-: so what do you suggest me to do? | 12:17 |
| logan- | Specify the package name manually and it will override correctly instead of trying to clone both | 12:17 |
| openstackgerrit | Jesse Pretorius (odyssey4me) proposed openstack/openstack-ansible: Use current, but pinned versions of pip, setuptools and wheel https://review.openstack.org/287161 | 12:17 |
| logan- | So right below the line where you override with your gitlab url, put that package name var as well | 12:18 |
| winggundamth | ok | 12:18 |
| odyssey4me | logan- ah, well spotted :) you've clearly been through this | 12:18 |
| winggundamth | that's clearly | 12:19 |
| *** neilus has joined #openstack-ansible | 12:20 | |
| odyssey4me | mattt I've done https://review.openstack.org/287161 pre-emptively - it's a backport of the two patches, one of which has been voted through but hasn't merged yet thanks to shenanigans. I expect that we should get this patch in before the next tag to save a lot of people pain and suffering. | 12:21 |
| odyssey4me | but I expect it could wait for the next tag, assuming that people keep up to date :p | 12:21 |
| mattt | logan-: so is that the source of winggundamth's problems? | 12:22 |
| *** jguy_ has quit IRC | 12:22 | |
| mattt | odyssey4me: cool, i'd love to see the master patch merge first | 12:24 |
| *** asettle has quit IRC | 12:24 | |
| odyssey4me | mattt me too, of course -that's why I WIP the patch :) | 12:24 |
| mattt | who knows at this rate if it will get through :P | 12:25 |
| odyssey4me | mattt https://review.openstack.org/284896 is on a rax node, so it's got a good chance of getting through - that should unblock master | 12:25 |
| openstackgerrit | Jesse Pretorius (odyssey4me) proposed openstack/openstack-ansible: IRR - Implemented for os_keystone https://review.openstack.org/279710 | 12:29 |
| openstackgerrit | Jesse Pretorius (odyssey4me) proposed openstack/openstack-ansible: IRR - Implemented for all os_* and dep roles https://review.openstack.org/284987 | 12:29 |
| *** javeriak has quit IRC | 12:30 | |
| *** subscope has joined #openstack-ansible | 12:30 | |
| openstackgerrit | Jesse Pretorius (odyssey4me) proposed openstack/openstack-ansible: Use current, but pinned versions of pip, setuptools and wheel https://review.openstack.org/287161 | 12:34 |
| openstackgerrit | Jesse Pretorius (odyssey4me) proposed openstack/openstack-ansible: Use current, but pinned versions of pip, setuptools and wheel https://review.openstack.org/284977 | 12:35 |
| *** subscope has quit IRC | 12:36 | |
| odyssey4me | mattt that's set dependencies, so hopefully the builds will succeed and we can get them in sooner | 12:37 |
| odyssey4me | alright, now I'm going to be afk for an hour or two - bbl! | 12:37 |
| logan- | mattt: maybe... I am not sure on the broken requirement but I would think if the same shas are being used the requirements should be calculated the same regardless of there being 2 copies of all the repos. | 12:38 |
| logan- | i just saw that exact error last week when I accidentally had a neutron stable/kilo sha in my liberty install :) | 12:38 |
| *** asettle has joined #openstack-ansible | 12:39 | |
| logan- | so i would suggest also doing grep -R '<1.6.0,>=1.5.0' /var/www/repo/openstackgit | 12:39 |
| logan- | on the repo server | 12:39 |
| winggundamth | logan-: already did that and find nothing :( | 12:41 |
| logan- | winggundamth: example overrides file btw http://cdn.pasteraw.com/vt6p3tbaaii28jpmn8pfd8rfk1pevz | 12:42 |
| winggundamth | rerun repo-build again. bless me | 12:42 |
| winggundamth | logan-: thank you. will try that | 12:42 |
| logan- | you might want to rm /var/www/repo and rebuild from scratch to clean out the mess there if this repo-build run still fails | 12:43 |
| *** subscope has joined #openstack-ansible | 12:43 | |
| *** subscope has quit IRC | 12:44 | |
| winggundamth | logan-: yeah already done that ;) | 12:45 |
| *** subscope has joined #openstack-ansible | 12:46 | |
| *** thorst_ has joined #openstack-ansible | 12:48 | |
| *** markvoelker has quit IRC | 12:48 | |
| *** markvoelker has joined #openstack-ansible | 12:49 | |
| *** fawadkhaliq has joined #openstack-ansible | 12:54 | |
| winggundamth | bravo!!! | 12:58 |
| winggundamth | thank you so much logan- odyssey4me mattt. now I passed the task I have problem | 12:59 |
| *** woodard has joined #openstack-ansible | 13:00 | |
| *** woodard has quit IRC | 13:00 | |
| *** woodard has joined #openstack-ansible | 13:01 | |
| logan- | great! | 13:01 |
| winggundamth | another error T_T | 13:05 |
| *** jguy_ has joined #openstack-ansible | 13:05 | |
| winggundamth | but i'm fine. will try to find the way to fix it first | 13:05 |
| *** johnmilton has joined #openstack-ansible | 13:07 | |
| winggundamth | http://paste.openstack.org/show/488952/ :( | 13:08 |
| logan- | ls /var/www/repo/pools/mysql_python | 13:08 |
| *** johnmilton has quit IRC | 13:08 | |
| winggundamth | http://paste.openstack.org/show/488953/ | 13:08 |
| *** johnmilton has joined #openstack-ansible | 13:09 | |
| winggundamth | root@tdc-ops-mgt-01:/openstack/infra1_repo_container-5de053b6/repo/pools/mysql_python# ls | 13:09 |
| winggundamth | mysql_python-1.2.5-cp27-none-linux_x86_64.whl | 13:09 |
| logan- | make sure lsyncd is running on the primary repo container where it ran the build | 13:09 |
| mattt | winggundamth: so what was the problem? | 13:10 |
| winggundamth | mattt: it repo name that gitlab need to append with .git so it will create folder append with .git also | 13:10 |
| mattt | winggundamth: :( is there something we can do in osa to help with this situation since it burned you and logan- ? | 13:11 |
| winggundamth | mattt: I'm thinking about another session for who want to speed up deploy by caching like git, pypi, apt-cacher-ng and rackspace repo | 13:13 |
| logan- | last year when cloudnull was working on the repo build stuff i hit that issue immediately.. so he added the _package_name override. it fixes it of course but I am not sure if it is documented anywhere.. having a little doc about how to override repos would be helpful i think | 13:13 |
| winggundamth | and I'm sure that we need more document about variable here | 13:13 |
| winggundamth | maybe if I can pass all of these I can help on document for you because I need heavily using cache from the bw problem | 13:14 |
| winggundamth | logan-: http://paste.openstack.org/show/488956/ | 13:17 |
| winggundamth | logan-: they don't have 12.0.7 in os-release http://rpc-repo.rackspace.com/os-releases/ | 13:17 |
| mattt | any reason why we couldn't just strip .git off the repo name ? | 13:17 |
| logan- | that's strange. pip_lock_down should set pip to use your local repo server | 13:17 |
| mattt | feels like this should be handled internally without overriding vars | 13:17 |
| winggundamth | mattt: that's another clever way to do it! | 13:18 |
| logan- | mattt: yeah I don't disagree. as a deployer I would expect it to handle a git url with trailing .git in stride :) | 13:19 |
| mattt | cool, i'll have a look at this | 13:19 |
| winggundamth | logan-: maybe because of this var in my user_variables.yml http://paste.openstack.org/show/488961/ | 13:20 |
| winggundamth | any variable that will overrid this? | 13:20 |
| logan- | i would avoid overriding openstack_repo_url | 13:21 |
| *** fawadkhaliq has quit IRC | 13:21 | |
| logan- | openstack_repo_url: "http://{{ internal_lb_vip_address }}:{{ repo_server_port }}" | 13:22 |
| logan- | that is pointing at your repo containers by default | 13:22 |
| winggundamth | grrr ic | 13:22 |
| winggundamth | that's not clear in document also | 13:22 |
| winggundamth | http://docs.openstack.org/developer/openstack-ansible/install-guide/configure-hostlist.html | 13:23 |
| * winggundamth rerun again | 13:24 | |
| *** furlongm_ is now known as furlongm | 13:26 | |
| *** Bjoern_ has joined #openstack-ansible | 13:28 | |
| *** skamithi has joined #openstack-ansible | 13:31 | |
| *** rewriteurl has joined #openstack-ansible | 13:33 | |
| *** Bjoern_ has quit IRC | 13:34 | |
| *** cemmason1 has quit IRC | 13:38 | |
| *** cemmason has joined #openstack-ansible | 13:38 | |
| logan- | mattt odyssey4me: it does look like rpc-repo is missing 12.0.6 wheels btw | 13:41 |
| *** mathlin1 has joined #openstack-ansible | 13:52 | |
| *** retreved has joined #openstack-ansible | 13:56 | |
| mattt | cloudnull: it looks like https://github.com/openstack/openstack-ansible-os_keystone/commit/34ad63b41eb85bc947bb51e3d3895b7516374943 got unwound in your os_keystone update, was that deliberate ? | 14:00 |
| *** cemmason has quit IRC | 14:03 | |
| odyssey4me | logan- rpc-repo is manually built, and RPC is not yet using liberty - so yeah, it may or may not have built wheels on it, but it's only there to obtain wheels if they happen to be there - pypi is still accessed as well and wheels/sdists can come from there | 14:04 |
| *** cemmason has joined #openstack-ansible | 14:04 | |
| *** bapalm has quit IRC | 14:05 | |
| odyssey4me | mattt FYI https://review.openstack.org/287205 will be needed for https://review.openstack.org/284987 to pass | 14:05 |
| *** bapalm has joined #openstack-ansible | 14:06 | |
| mattt | odyssey4me: voted | 14:06 |
| *** weshay has joined #openstack-ansible | 14:07 | |
| *** cemmason1 has joined #openstack-ansible | 14:08 | |
| winggundamth | error still keep coming ;( http://paste.openstack.org/show/488976/ | 14:09 |
| *** busterswt has joined #openstack-ansible | 14:09 | |
| *** cemmason has quit IRC | 14:09 | |
| winggundamth | I remember that when I doing this with aio. I can pass it instantly :( | 14:09 |
| mattt | winggundamth: are you running memcache ? | 14:10 |
| odyssey4me | thanks mattt | 14:10 |
| winggundamth | infra1_memcached_container-0b5296e7 RUNNING 10.0.3.73, 172.27.5.224 - YES (onboot, openstack) | 14:10 |
| winggundamth | infra1_memcached_container-380ae605 RUNNING 10.0.3.245, 172.27.5.151 - YES (onboot, openstack) | 14:10 |
| winggundamth | mattt: yes I think | 14:10 |
| *** KLevenstein has joined #openstack-ansible | 14:10 | |
| odyssey4me | mattt after a good run on a rax nodepool instance, https://review.openstack.org/284896 is doing the gate check on ovh and is going far too slowly :/ | 14:16 |
| *** Xu_Chao has quit IRC | 14:16 | |
| winggundamth | this is detail from -vvvv http://paste.openstack.org/show/488979/ | 14:17 |
| *** Xu_Chao has joined #openstack-ansible | 14:19 | |
| *** subscope has quit IRC | 14:20 | |
| *** subscope has joined #openstack-ansible | 14:21 | |
| *** subscope has quit IRC | 14:21 | |
| *** subscope has joined #openstack-ansible | 14:22 | |
| mattt | winggundamth: which playbook are you running btw, and are you adding any --limits ? | 14:25 |
| mhayden | is there some gate brokenness around virtualenv? | 14:25 |
| mattt | mhayden: master presumably ? | 14:26 |
| winggundamth | mattt: no --limits. at first I'm running this openstack-ansible setup-infrastructure.yml and after that I added --tag rabbitmq-ssl | 14:26 |
| mhayden | mattt: yessir | 14:26 |
| mattt | winggundamth: try removing the tag, why are you using that out of interest? | 14:26 |
| winggundamth | mattt: I'm running without --tag first and got the error so I don't want to run the whole step again so I use --tag | 14:27 |
| *** subscope has quit IRC | 14:27 | |
| winggundamth | mattt: I'll try ansible-playbook rabbitmq-install.yml | 14:28 |
| mattt | winggundamth: you can just run the rabbitmq portion of setup-infrastrucure.yml | 14:28 |
| automagically | Should this be marked WIP until functional tests are complete? https://review.openstack.org/#/c/286650/ | 14:28 |
| mattt | winggundamth: yep | 14:28 |
| odyssey4me | mhayden master is currently quite broken - you'll need to apply https://review.openstack.org/284896 then https://review.openstack.org/279710 then https://review.openstack.org/284987 and also https://review.openstack.org/284977 to make it all go again properly | 14:28 |
| mattt | automagically: naw | 14:28 |
| mhayden | odyssey4me: haha, wow | 14:29 |
| odyssey4me | mhayden you'll notice that those patches are already chained up | 14:29 |
| mhayden | impressive | 14:29 |
| mattt | mhayden: you're welcome :) | 14:29 |
| winggundamth | mattt: no luck. still the same error :( | 14:30 |
| mhayden | i'm looking at the lbaasv2 commit -- i assume i'll need to wedge that into the os_neutron IRR ? | 14:30 |
| odyssey4me | automagically nope, first get linters passing and the gate enabled - then evaluate whether any patches went in after the code was extracted into a seperate repo - then work on implementing a working functional check | 14:30 |
| odyssey4me | automagically see https://etherpad.openstack.org/p/openstack-ansible-role-breakout :) | 14:30 |
| automagically | Thx, will start in on the repo diff | 14:30 |
| winggundamth | mattt: I curious about error "Memcache key not found" | 14:30 |
| openstackgerrit | Merged openstack/openstack-ansible-os_swift_sync: Add empty role dependency list https://review.openstack.org/287205 | 14:30 |
| odyssey4me | mattt I've checked https://review.openstack.org/286650 and it seems fine - but was hoping that you'd check it before I vote :) | 14:31 |
| mattt | winggundamth: i'm grasping at straws here, but you could try shutting down 2 memcache containers to make sure it's not something with that | 14:31 |
| mattt | odyssey4me: looking | 14:31 |
| winggundamth | mattt: and this is error from -vvvv 'The key you specified was not found within memcached. If you are load balancing memcached, attempt to connect to a single node.' | 14:31 |
| winggundamth | mattt: ok will try | 14:32 |
| *** jthorne has joined #openstack-ansible | 14:34 | |
| mattt | automagically: one small issue w/ that review, otherwise looks GTG | 14:34 |
| automagically | mattt: Thx for catching that, will amend | 14:34 |
| mattt | np | 14:34 |
| odyssey4me | automagically jmccrory cloudnull a quick review of https://review.openstack.org/287072 would be great | 14:35 |
| mattt | we'll have to go through and standardise some of the descriptions and READMEs in the roles, they're all over the place :) | 14:35 |
| mattt | but a task for another day | 14:35 |
| odyssey4me | ++ | 14:35 |
| *** shausy has quit IRC | 14:36 | |
| odyssey4me | I'd actually like each role's docs to have details of how the service is setup, what it expects and does and how to configure it all... then the install guide can focus on simple stuff and refer to the role docs for more details | 14:36 |
| automagically | odyssey4me: That sounds like a good goal, but definitely down the road | 14:37 |
| mhayden | how do we handle release notes for the IRR roles? | 14:37 |
| automagically | ^ That there is a great question. I believe there are some already in the main repo for changes in the IRRs | 14:38 |
| winggundamth | is it usual that it has 2 containers for the same role per host? | 14:38 |
| openstackgerrit | Travis Truman (automagically) proposed openstack/openstack-ansible-os_horizon: Implement base configuration for independent repository https://review.openstack.org/286650 | 14:39 |
| mhayden | automagically: okay | 14:40 |
| *** scarlisle has joined #openstack-ansible | 14:41 | |
| *** mgoddard_ has joined #openstack-ansible | 14:44 | |
| openstackgerrit | Merged openstack/openstack-ansible-os_glance: Implement base configuration for independent repository https://review.openstack.org/286667 | 14:46 |
| Bofu2U | hmm. How hard would it be to factor in the swift3 middleware into the playbooks? | 14:46 |
| openstackgerrit | Merged openstack/openstack-ansible-os_aodh: Implement base configuration for independent repository https://review.openstack.org/286655 | 14:46 |
| openstackgerrit | Merged openstack/openstack-ansible-os_heat: Implement base configuration for independent repository https://review.openstack.org/286693 | 14:46 |
| openstackgerrit | Major Hayden proposed openstack/openstack-ansible-os_neutron: Replace LBaaS v1 with v2 https://review.openstack.org/287237 | 14:47 |
| odyssey4me | mhayden the only instrumentation at this time for release notes is in the main repo - we build all docs there | 14:47 |
| *** b3rnard0 has quit IRC | 14:47 | |
| mhayden | odyssey4me: got it | 14:47 |
| openstackgerrit | Merged openstack/openstack-ansible-os_neutron: Implement E501 flake8 check with longer line limit https://review.openstack.org/287072 | 14:47 |
| odyssey4me | mhayden as we build out the documentation in the independent repositories, I expect that we'll also need to implement release notes there | 14:47 |
| *** mgoddard has quit IRC | 14:47 | |
| *** subscope has joined #openstack-ansible | 14:48 | |
| odyssey4me | mhayden but in this cycle I think it's best that we do it in the main repo | 14:48 |
| mattt | winggundamth: sorry not familiar w/ that error, i'd have to dig through the plugin and playbooks to see what is causing that | 14:48 |
| *** admin0 has quit IRC | 14:48 | |
| * odyssey4me shakes a stick at ovh nodepool instances | 14:48 | |
| *** b3rnard0 has joined #openstack-ansible | 14:48 | |
| cloudnull | morning | 14:49 |
| winggundamth | mattt: no worry. I have to go now too. almost 10pm here | 14:49 |
| Bofu2U | mornin cloudnull | 14:49 |
| mattt | winggundamth: sorry, please come back tomorrow and we can try this again :) | 14:49 |
| cloudnull | ohai Bofu2U | 14:49 |
| winggundamth | mattt: I'm sure that we will talk again tmr :) | 14:49 |
| *** alejandrito has joined #openstack-ansible | 14:49 | |
| mattt | winggundamth: great, have a good evening | 14:50 |
| cloudnull | Bofu2U: how goes the kernel experiment ? | 14:50 |
| Bofu2U | the the 4x series still up | 14:50 |
| Bofu2U | the 3.18 lasted about 3-4x as long, then died | 14:50 |
| winggundamth | thank you so much for all your help. I'm really appreciate it. | 14:50 |
| mattt | winggundamth: thank logan- , he solved your problem | 14:50 |
| cloudnull | Bofu2U: well thats good news. kinda :) | 14:50 |
| mattt | i'd never have guessed that in a million years :P | 14:51 |
| *** sdake has joined #openstack-ansible | 14:51 | |
| *** admin0 has joined #openstack-ansible | 14:51 | |
| *** Mudpuppy has joined #openstack-ansible | 14:51 | |
| Bofu2U | cloudnull yeah good enough I guess | 14:52 |
| *** Mudpuppy has quit IRC | 14:52 | |
| *** b3rnard0 has quit IRC | 14:52 | |
| *** b3rnard0 has joined #openstack-ansible | 14:52 | |
| *** Mudpuppy has joined #openstack-ansible | 14:52 | |
| *** b3rnard0 is now known as Guest11126 | 14:53 | |
| *** Guest11126 has quit IRC | 14:53 | |
| *** b3rnard0- has joined #openstack-ansible | 14:54 | |
| openstackgerrit | Major Hayden proposed openstack/openstack-ansible: Replace LBaaS v1 with v2 https://review.openstack.org/287243 | 14:55 |
| cloudnull | Bofu2U: so upgrade all the things to 4.x and all will be right with the world :) | 14:56 |
| Bofu2U | allegedly ;) | 14:56 |
| cloudnull | Bofu2U: RE: swift3 we dont have a role for it | 14:56 |
| cloudnull | so we'd have to make that | 14:56 |
| Bofu2U | ah ok | 14:56 |
| Bofu2U | yeah im just trying to get deis setup with the swift store in general, ill try to do it direct first | 14:56 |
| cloudnull | seems like https://github.com/openstack/swift3can be added to the existing swift role as it'd have to be added as middleware | 14:57 |
| cloudnull | *https://github.com/openstack/swift3 can | 14:57 |
| odyssey4me | if it's just additional middleware I expect it wouldn't be too hard | 14:57 |
| odyssey4me | what is swift3? | 14:58 |
| * odyssey4me is out of touch | 14:58 | |
| *** dmsimard is now known as dmsimard|afk | 14:58 | |
| Bofu2U | basically makes swift work with the S3 API | 14:58 |
| cloudnull | its s3 compat within swift -- it new bits that have to be installed on the proxy nodes and middleware added. so it shouldnt be too hard to do | 14:59 |
| *** mgoddard_ has quit IRC | 14:59 | |
| cloudnull | but i dont have the cycles to look at it right now. | 14:59 |
| *** mgoddard has joined #openstack-ansible | 14:59 | |
| cloudnull | Bofu2U: if you have some time to work on that i'd be happy to review it | 14:59 |
| Bofu2U | rgr | 14:59 |
| Bofu2U | gonna try to get this with swift direct, but if it doesn't work I'll try to slate some time for that | 15:00 |
| *** skamithi13 has joined #openstack-ansible | 15:00 | |
| cloudnull | sweet! | 15:00 |
| * cloudnull bbl driving in | 15:00 | |
| odyssey4me | yeah, I do think that we should add the replacement S3 and EC2 API add-ons into the stack as an optional extra | 15:01 |
| odyssey4me | the newer stuff is a lot better, I hear, and it'd be a nice feature for those who want it | 15:01 |
| *** spotz_zzz is now known as spotz | 15:03 | |
| automagically | odyssey4me: Any objections to changing the tox.ini in os_keystone IRR to run with -vvv or similar. Gate is failing and I’m seeing no useful output in http://logs.openstack.org/60/286660/5/check/gate-openstack-ansible-os_keystone-dsvm-ansible-functional-trusty/1759d25/console.html other than “failed: [keystone1] => {"attempts": 5, "failed": true, "parsed": false}” | 15:04 |
| odyssey4me | automagically I already added the human readable output - you're not going to get much better than that | 15:04 |
| automagically | ugh | 15:04 |
| odyssey4me | automagically the only way to figure that one out is going to be to look at a running build | 15:05 |
| automagically | Looks like its our keystone library blowing up | 15:05 |
| odyssey4me | automagically or that the task can't reach the service, either because the service isn't running or because the task is being given the wrong info | 15:05 |
| automagically | k, thx, will dig in a bit further, just not used to seeing such limited output in the logs | 15:06 |
| odyssey4me | automagically a suggestion - do what we've done for the other roles to make it effectively pass the functional test... let's get the base stuff merged and make the functional test work in a subsequent patch | 15:06 |
| automagically | Will do | 15:06 |
| *** sigmavirus24_awa is now known as sigmavirus24 | 15:08 | |
| openstackgerrit | Travis Truman (automagically) proposed openstack/openstack-ansible-os_keystone: Implement base configuration for independent repository https://review.openstack.org/286660 | 15:09 |
| *** ShannonM has joined #openstack-ansible | 15:11 | |
| *** javeriak has joined #openstack-ansible | 15:11 | |
| *** javeriak has quit IRC | 15:12 | |
| *** javeriak_ has joined #openstack-ansible | 15:16 | |
| *** raddaoui has joined #openstack-ansible | 15:21 | |
| *** winggundamth has quit IRC | 15:21 | |
| *** izaakk has joined #openstack-ansible | 15:23 | |
| *** skamithi13 has quit IRC | 15:25 | |
| *** skamithi13 has joined #openstack-ansible | 15:26 | |
| *** weezS has joined #openstack-ansible | 15:28 | |
| *** weezS has quit IRC | 15:28 | |
| openstackgerrit | Matt Thompson proposed openstack/openstack-ansible-plugins: Strip .git from _git_repo variables https://review.openstack.org/287266 | 15:31 |
| *** rewriteurl has quit IRC | 15:34 | |
| *** rsparule has quit IRC | 15:40 | |
| openstackgerrit | Travis Truman (automagically) proposed openstack/openstack-ansible-lxc_container_create: Reformat YAML condition to list https://review.openstack.org/287273 | 15:45 |
| openstackgerrit | Matt Thompson proposed openstack/openstack-ansible-plugins: Strip .git from _git_repo variables https://review.openstack.org/287266 | 15:47 |
| *** keedya has joined #openstack-ansible | 15:49 | |
| *** sdake_ has joined #openstack-ansible | 15:51 | |
| *** sdake has quit IRC | 15:52 | |
| *** mathlin1 has quit IRC | 15:52 | |
| Nepoc | I looked at this just at the right time... I need the strip .git from _git_repo variables today | 15:52 |
| openstackgerrit | Major Hayden proposed openstack/openstack-ansible-os_neutron: Replace LBaaS v1 with v2 https://review.openstack.org/287237 | 15:53 |
| *** Mudpuppy has quit IRC | 15:54 | |
| *** Mudpuppy has joined #openstack-ansible | 15:54 | |
| *** sdake has joined #openstack-ansible | 15:55 | |
| openstackgerrit | Nolan Brubaker proposed openstack/openstack-ansible: Update new user secrets in Liberty https://review.openstack.org/286844 | 15:56 |
| openstackgerrit | Merged openstack/openstack-ansible-pip_install: Test version of pip being installed https://review.openstack.org/285284 | 15:56 |
| *** eric_lopez has quit IRC | 15:57 | |
| *** sdake_ has quit IRC | 15:58 | |
| openstackgerrit | Nolan Brubaker proposed openstack/openstack-ansible: Update new user secrets in Liberty https://review.openstack.org/286844 | 16:00 |
| *** admin0 has quit IRC | 16:00 | |
| *** pcaruana has quit IRC | 16:05 | |
| *** rromans has quit IRC | 16:06 | |
| *** fawadkhaliq has joined #openstack-ansible | 16:06 | |
| *** rromans has joined #openstack-ansible | 16:08 | |
| *** sdake_ has joined #openstack-ansible | 16:12 | |
| *** flaviosr has joined #openstack-ansible | 16:13 | |
| *** fawadkhaliq has quit IRC | 16:14 | |
| *** alextricity25_ has joined #openstack-ansible | 16:14 | |
| *** odyssey4me_ has joined #openstack-ansible | 16:15 | |
| *** bryan_att has quit IRC | 16:15 | |
| *** larrie_kong has quit IRC | 16:15 | |
| *** alextricity25 has quit IRC | 16:15 | |
| *** larrie_kong has joined #openstack-ansible | 16:15 | |
| *** thorst_ has quit IRC | 16:15 | |
| *** maximov_ has quit IRC | 16:15 | |
| *** odyssey4me has quit IRC | 16:15 | |
| *** sdake has quit IRC | 16:15 | |
| *** git-harry has quit IRC | 16:15 | |
| *** kencjohnston has joined #openstack-ansible | 16:15 | |
| *** larrie_kong has quit IRC | 16:15 | |
| *** larrie_kong has joined #openstack-ansible | 16:15 | |
| *** git-harry has joined #openstack-ansible | 16:15 | |
| *** thorst_ has joined #openstack-ansible | 16:16 | |
| *** maximov_ has joined #openstack-ansible | 16:17 | |
| *** phalmos has joined #openstack-ansible | 16:18 | |
| *** bryan_att has joined #openstack-ansible | 16:19 | |
| *** sdake_ is now known as sdake | 16:21 | |
| openstackgerrit | Nolan Brubaker proposed openstack/openstack-ansible: Update ceilometer environment configuration https://review.openstack.org/286827 | 16:21 |
| *** rromans has quit IRC | 16:21 | |
| *** rromans has joined #openstack-ansible | 16:23 | |
| *** git-harry has left #openstack-ansible | 16:24 | |
| palendae | raddaoui, izaakk: Upgrade stuff is at https://review.openstack.org/#/q/topic:liberty_upgrade feedback on whether the docs are clear or not would be great | 16:26 |
| tiagogomes | why is there " user: root" in the playbooks? | 16:26 |
| openstackgerrit | Travis Truman (automagically) proposed openstack/openstack-ansible-galera_server: Cleanup configuration for independent repository https://review.openstack.org/287300 | 16:26 |
| *** daneyon has joined #openstack-ansible | 16:28 | |
| *** phalmos has quit IRC | 16:28 | |
| *** daneyon has quit IRC | 16:28 | |
| *** daneyon_ has joined #openstack-ansible | 16:29 | |
| izaakk | thanks palendae , will help there | 16:29 |
| *** phalmos has joined #openstack-ansible | 16:30 | |
| openstackgerrit | Nolan Brubaker proposed openstack/openstack-ansible: Add aodh, haproxy, and neutron config upgrades https://review.openstack.org/286753 | 16:32 |
| openstackgerrit | Nolan Brubaker proposed openstack/openstack-ansible: Update ceilometer environment configuration https://review.openstack.org/286827 | 16:32 |
| openstackgerrit | Nolan Brubaker proposed openstack/openstack-ansible: Update new user secrets in Liberty https://review.openstack.org/286844 | 16:32 |
| *** daneyon_ has quit IRC | 16:33 | |
| openstackgerrit | Major Hayden proposed openstack/openstack-ansible: Replace LBaaS v1 with v2 https://review.openstack.org/287243 | 16:33 |
| *** daneyon has joined #openstack-ansible | 16:33 | |
| *** odyssey4me_ is now known as odyssey4me | 16:33 | |
| *** electrofelix has quit IRC | 16:34 | |
| palendae | For anyone interested in liberty upgrades: https://review.openstack.org/#/q/topic:liberty_upgrade. https://review.openstack.org/#/c/286251/ is the base commit that everything else relies on, fwiw | 16:34 |
| *** keedya has quit IRC | 16:34 | |
| *** sdake has quit IRC | 16:35 | |
| openstackgerrit | Nolan Brubaker proposed openstack/openstack-ansible: Add missing runtime information to upgrade script https://review.openstack.org/286251 | 16:35 |
| palendae | Argh, sorry odyssey4me, just realized that commit needed a rebase after you +W :( | 16:36 |
| odyssey4me | palendae no worries, a rebase shouldn't lose the votes | 16:36 |
| palendae | Cleared the +W | 16:36 |
| palendae | And the Jenkins vote | 16:37 |
| * palendae grumbles | 16:37 | |
| openstackgerrit | Nolan Brubaker proposed openstack/openstack-ansible: Add upgrade config change tools https://review.openstack.org/285564 | 16:37 |
| openstackgerrit | Nolan Brubaker proposed openstack/openstack-ansible: Add aodh, haproxy, and neutron config upgrades https://review.openstack.org/286753 | 16:38 |
| openstackgerrit | Nolan Brubaker proposed openstack/openstack-ansible: Update ceilometer environment configuration https://review.openstack.org/286827 | 16:38 |
| openstackgerrit | Nolan Brubaker proposed openstack/openstack-ansible: Update new user secrets in Liberty https://review.openstack.org/286844 | 16:38 |
| *** eric_lopez has joined #openstack-ansible | 16:38 | |
| * palendae backs off the rebase button for a bit | 16:38 | |
| *** eric_lopez has quit IRC | 16:40 | |
| *** rcarrillocruz has left #openstack-ansible | 16:40 | |
| *** sdake has joined #openstack-ansible | 16:40 | |
| *** eric_lopez has joined #openstack-ansible | 16:40 | |
| openstackgerrit | Travis Truman (automagically) proposed openstack/openstack-ansible-openstack_openrc: No longer defining OS_TENANT_NAME in openrc https://review.openstack.org/287309 | 16:42 |
| *** cemmason1 has quit IRC | 16:43 | |
| *** cemmason has joined #openstack-ansible | 16:43 | |
| *** jwitko has joined #openstack-ansible | 16:44 | |
| *** subscope has quit IRC | 16:44 | |
| *** daneyon_ has joined #openstack-ansible | 16:47 | |
| odyssey4me | palendae I'm told by infra that the check and gate queues always rebase before testing - so we actually don't have to rebase if we depend on the head of the branch. | 16:49 |
| palendae | Ah | 16:49 |
| *** daneyon has quit IRC | 16:49 | |
| palendae | Part of that is so some of the nits that exist in parent or grandparent change for upgrades get funneled down | 16:49 |
| palendae | Cause some of these depend on each other now, rather than directly on HEAD | 16:50 |
| spotz | palensae poke if I need to re-review anything if the votes are lost | 16:50 |
| palendae | spotz: No +1s were lost, just Jenkins and +W; the rebasing thing is I'm just spamming the channel :p | 16:51 |
| spotz | hehe | 16:51 |
| palendae | But https://review.openstack.org/#/q/topic:liberty_upgrade if I'm screwing up my wordings | 16:51 |
| odyssey4me | cloudnull I'm kinda thinking of temporarily dropping affinity in master to get those IRR patches through, then we can up the affinity again. We're sitting in the crappy situation where all the gate checks go through OVH which starts the smoke test at 89mins and therefore the job times out | 17:01 |
| odyssey4me | unless you have any other ideas which we can implement in the repositories to make them go faster? | 17:01 |
| odyssey4me | automagically logan- jmccrory mattt ^ ? | 17:01 |
| *** subscope has joined #openstack-ansible | 17:02 | |
| cloudnull | odyssey4me: seems sensible | 17:03 |
| cloudnull | especially while ovh is being a pain | 17:03 |
| odyssey4me | well, the trouble is that ovh will never speed up - so I'd far rather we put some optimisations in than rely on reducing testing | 17:04 |
| odyssey4me | ultimately that's the more sustainable route | 17:04 |
| odyssey4me | cloudnull we need to get https://review.openstack.org/286660 in as a starting point for os_keystone, then try to implement the functional testing in a working state afterwards | 17:06 |
| logan- | yeah that makes sense.. what are the pain points on these ovh instances? is it network latency from mirrors? io bound? wondering what optimizations would be helpful | 17:08 |
| odyssey4me | logan- not entirely sure - we do know that we have the option of using the mirrors which infra have implemented... but they aren't signed, so we need extra instrumentation for that and it has to follow an integrated master patch anyway, so that's no good | 17:09 |
| odyssey4me | we have to figure out how to do some optimisations in the roles themselves, because those gate every time and gate quicker | 17:10 |
| odyssey4me | they then have a downstream effect on the integrated testing | 17:10 |
| odyssey4me | there may be no option except to reduce affinity in order to open the door though, let me see what I can come up with - but any ideas would be great | 17:10 |
| *** electrofelix has joined #openstack-ansible | 17:10 | |
| odyssey4me | there are logs in http://logs.openstack.org/96/284896/14/gate/gate-openstack-ansible-dsvm-commit/9f4b6d5/ which capture the state of the instance in various places during the build which may help | 17:11 |
| odyssey4me | specifically http://logs.openstack.org/96/284896/14/gate/gate-openstack-ansible-dsvm-commit/9f4b6d5/logs/instance-info/ | 17:12 |
| odyssey4me | stevelle any thoughts on options? | 17:13 |
| *** sdake has quit IRC | 17:13 | |
| stevelle | I'm fine with reduced affinity for the moment, again, remembering we did this about 6 months ago. | 17:14 |
| odyssey4me | yeah we did | 17:15 |
| logan- | i am thinking about pulling down a bunch of review logs and trying to parse out the play recap stuff and attribute averages with different providers to see % diffs between them | 17:15 |
| logan- | it would help to know which tasks are taking so much longer on ovh and see whether they are io intensive/cpu intensive, etc and look at ways to optimize from there | 17:16 |
| odyssey4me | logan- I've just checked some logs and saw that in some ovh regions we don't even get to the point where the os_nova role has completed before it times out. | 17:16 |
| *** sdake has joined #openstack-ansible | 17:17 | |
| logan- | just looking thru 1 of the random ovh logs i noticed nova venv took nearly 2 minutes to extract... seems high? | 17:17 |
| odyssey4me | but yeah, specifically check the gate-openstack-ansible-dsvm-commit logs in https://review.openstack.org/#/c/284896/ | 17:17 |
| logan- | k | 17:17 |
| *** phiche has quit IRC | 17:18 | |
| odyssey4me | you'll have to check the whole history of logs to find them - you'll see that on other providers we usually get through just fine | 17:18 |
| odyssey4me | but in the commit job specifically is where we most often land on ovh and other troublesome providers | 17:18 |
| odyssey4me | but when I last looked through the job results using logstash.openstack.org I found that 50% of failures were coming from two ovh regions - one in the us and one in europe | 17:19 |
| *** fawadkhaliq has joined #openstack-ansible | 17:21 | |
| *** subscope has quit IRC | 17:22 | |
| *** Bjoern_ has joined #openstack-ansible | 17:23 | |
| openstackgerrit | Jesse Pretorius (odyssey4me) proposed openstack/openstack-ansible: Reduce galera/rabbit/horizon/keystone/repo affinity to 1 https://review.openstack.org/287331 | 17:23 |
| palendae | spotz: Mind looking at https://review.openstack.org/#/c/286753/11/doc/source/upgrade-guide/scripts.rst, L28? Trying to decide if the verb should be 'add' or 'adds'. I think 'add' is correct now | 17:23 |
| odyssey4me | let's see if https://review.openstack.org/287331 reduces the check by enough time to matter | 17:24 |
| *** electrofelix has quit IRC | 17:25 | |
| *** Gayathri has joined #openstack-ansible | 17:26 | |
| *** spotz is now known as spotz_zzz | 17:26 | |
| *** skamithi has quit IRC | 17:27 | |
| Gayathri | Hi | 17:27 |
| openstackgerrit | Travis Truman (automagically) proposed openstack/openstack-ansible-openstack_openrc: Adding Vagrantfile for local developer testing https://review.openstack.org/287333 | 17:27 |
| Gayathri | whether the openstack-ansible community playbook support OVS? | 17:27 |
| Gayathri | for liberty | 17:27 |
| *** raddaoui has quit IRC | 17:27 | |
| *** raddaoui has joined #openstack-ansible | 17:28 | |
| *** raddaoui_ has joined #openstack-ansible | 17:28 | |
| openstackgerrit | Andy McCrae proposed openstack/openstack-ansible-os_swift: Fix Swift log directory ownership https://review.openstack.org/287337 | 17:29 |
| openstackgerrit | Andy McCrae proposed openstack/openstack-ansible-os_swift: Fix Swift log directory ownership https://review.openstack.org/287337 | 17:31 |
| *** galstrom_zzz is now known as galstrom | 17:32 | |
| *** raddaoui_ has quit IRC | 17:33 | |
| *** raddaoui has quit IRC | 17:33 | |
| *** raddaoui has joined #openstack-ansible | 17:33 | |
| odyssey4me | Gayathri not in liberty, nor in master (yet) - there is work ongoing to make that happen with the hope that it lands for Mitaka, but it may only make Newton | 17:33 |
| odyssey4me | automagically nice idea in https://review.openstack.org/287333 :) we should do that everywhere | 17:35 |
| automagically | odyssey4me: Cool, hope its easy enough to do. I’d rather not continue to crush OVH infra if it never helps us and I’d likewise like to run higher affinities and catch more bugs on all the other Zuul infra | 17:36 |
| odyssey4me | automagically also, can we add some basic docs to help developers prep their laptop in order to actually use vagrant for the testing? | 17:37 |
| palendae | Kind of related to that, I'm using this for my test AIOs now https://github.com/nrb/osad-basic-test-node. I may add a vagrantfile | 17:37 |
| automagically | odyssey4me: Whoops, thought you were referencing my comment on: https://review.openstack.org/#/c/287331/1 | 17:37 |
| palendae | Though I'm doing that repo for myself, not the general public | 17:37 |
| *** spotz_zzz is now known as spotz | 17:37 | |
| automagically | But as for the vagrant stuff, I’ll add some docs in the Contributor section | 17:37 |
| *** dmsimard|afk is now known as dmsimard | 17:38 | |
| odyssey4me | automagically yeah, it'd be great to actually have a few vagrantfiles to use on server hardware for gate tests - it'll help anyone who wants to implement a multinode test quickly | 17:38 |
| odyssey4me | but that'll be for the future :) | 17:38 |
| automagically | Believe we’ve got an outstanding review with a Vagrant based AIO | 17:38 |
| automagically | That looks pretty exciting | 17:38 |
| odyssey4me | automagically we can actually detect the provider and fail fast if we want to, but if we do that we will get nothing through the integrated gate as I think what tends to happen is that a provider is used almost exclusively for some types of tests from time to time | 17:39 |
| odyssey4me | right now we have a slight chance that it may get through - and we do get some data to try and isolate the issues | 17:40 |
| openstackgerrit | Travis Truman (automagically) proposed openstack/openstack-ansible-openstack_openrc: Correct filename of clouds.yml to clouds.yaml https://review.openstack.org/287342 | 17:40 |
| automagically | odyssey4me: Thanks for the additional gate infra context | 17:40 |
| odyssey4me | automagically yeah, I think it might be a good idea for me to share how we use OpenStack-CI at the summit... and how we're supposed to use it. :) I have discussed actually having some of the infra folk in a session like that so that they can provide some guidance with regards to how we can do better to optimise what happens in the gate and to reduce failures. | 17:42 |
| odyssey4me | At the very least pabelanger might join us, if not some others. :) | 17:42 |
| automagically | odyssey4me: I’d definitely be interested in learning more. I’m also trying to explore whether Comcast can provide some Nova compute infra for Zuul | 17:42 |
| palendae | odyssey4me: That'd be good; I get the sense that we're actually pushing infra to its limits | 17:43 |
| palendae | I'm not sure infra was meant to actually deploy an integrated openstack | 17:43 |
| odyssey4me | automagically ah nice! | 17:43 |
| *** sdake has quit IRC | 17:43 | |
| *** raddaoui has quit IRC | 17:43 | |
| odyssey4me | one of the topics that I think we also want to chat to pabelanger about is his roles for doing CI in your own infrastructure using OpenStack-CI technology... he's done a great job of pulling that infra together and I know that CI/CD is a topic many of us are interested in exploring | 17:44 |
| Gayathri | thanks for the clarification odyssey4me | 17:44 |
| logan- | are there roles for that somewhere? i had no idea | 17:45 |
| odyssey4me | logan- take a look through https://github.com/openstack?query=ansible-role and https://github.com/openstack/windmill | 17:46 |
| logan- | cool thanks | 17:46 |
| odyssey4me | I'd like to ask for him to give us an overview of how it fits together, what it does and how it can be used... it'd be great if we could adopt this as a subproject, assuming we're all interested in participating in the future development | 17:47 |
| palendae | odyssey4me: For upgrade testing that'd be cool, since I highly doubt we'll get that within a 90 minute limit in ~12 months | 17:47 |
| palendae | Deploying 1 version, then upgrading to a 2nd means roughly double the time | 17:48 |
| pabelanger | odyssey4me: Yup, have no issues explaining how it works or even demoing it | 17:48 |
| openstackgerrit | Travis Truman (automagically) proposed openstack/openstack-ansible-openstack_openrc: [DOCS] Cleanup the role docs for consistency and clarity https://review.openstack.org/287347 | 17:49 |
| *** raddaoui has joined #openstack-ansible | 17:49 | |
| odyssey4me | palendae yeah, I expect that the upgrade tests will need to be done as a periodic job rather than an on-commit job... although there is also the possibility of external CI which, if using appropriate hardware, could give fast enough feedback to make it worthwhile | 17:50 |
| palendae | odyssey4me: Yeah. I've also toyed with the idea of adding smaller tests around the scripts that do config changes | 17:50 |
| palendae | It's admittedly a small part of the overall work, but still more than we have now | 17:51 |
| odyssey4me | palendae that's a good idea | 17:52 |
| odyssey4me | palendae also note that within the roles upgrades can be tested specifically for the role, potentially, and the turnaround time there is much shorter | 17:52 |
| pabelanger | palendae: that's something I'd like to test with some of the ansible-roles I have. First deploy HEAD~1, ensure things work, then deploy HEAD and recheck. From a role POV, don't expect it to take long. | 17:52 |
| pabelanger | moving it into periodic would be a good solution too | 17:52 |
| palendae | Yeah, for roles, definitely. But the integration will still need to be tested | 17:53 |
| *** electrofelix has joined #openstack-ansible | 17:53 | |
| pabelanger | agreed | 17:53 |
| *** electrofelix has quit IRC | 17:53 | |
| jmccrory | odyssey4me logan- i put a script together the other day to parse through logs and spit out timings of per task and per playbook, i'll put that up somewhere today | 17:53 |
| *** asettle has quit IRC | 17:53 | |
| jmccrory | container creates took twice as long on ovh compared to rax | 17:54 |
| stevelle | jmccrory: nice | 17:54 |
| logan- | nice. i was just pulling a list of log files to download and do that lol | 17:54 |
| jmccrory | so affinity should help somewhat | 17:54 |
| logan- | are the flavor specs significantly different between the providers? | 17:54 |
| stevelle | I started one a couple weeks ago but never finished it | 17:54 |
| *** Gayathri has quit IRC | 17:55 | |
| *** raddaoui has quit IRC | 17:56 | |
| *** kencjohnston has quit IRC | 17:58 | |
| odyssey4me | logan- all the flavors have the same cpu/ram spec - but different disk specs | 18:01 |
| odyssey4me | the host contention and the underlying hardware (cpu, ram, disk) will be different for every provider | 18:02 |
| logan- | sure, if venv extracts and container creates are taking a long time, it makes you wonder about single threaded cpu performance and io bottlenecks | 18:02 |
| * odyssey4me is downloading his first vagrant box to test out automagically's patch :) | 18:02 | |
| *** git-harry has joined #openstack-ansible | 18:04 | |
| *** git-harry has left #openstack-ansible | 18:04 | |
| *** git-harry has joined #openstack-ansible | 18:05 | |
| *** phalmos has quit IRC | 18:07 | |
| *** skamithi has joined #openstack-ansible | 18:08 | |
| jwitko | hey odyssey4me, how are you today | 18:08 |
| openstackgerrit | Merged openstack/openstack-ansible-os_horizon: Implement base configuration for independent repository https://review.openstack.org/286650 | 18:08 |
| *** skamithi13 has quit IRC | 18:08 | |
| *** skamithi13 has joined #openstack-ansible | 18:09 | |
| *** skamithi has quit IRC | 18:09 | |
| *** mgoddard has quit IRC | 18:11 | |
| jwitko | Having an issue I'm wondering if you could help me fix... i think its a quick one. My route table on my containers is not being set with a proper gateway for the management interface. http://paste.openstack.org/raw/489018/ -- as you can see in that route table there is no default gateway set for 10.96.76.0/24 (it should be 10.96.76.1). My interface is defined in the user config | 18:12 |
| jwitko | as follows, with the gateway specified http://paste.openstack.org/raw/489019/ . Any idea what I'm doing wrong? | 18:12 |
| *** keedya has joined #openstack-ansible | 18:12 | |
| *** sigmavirus24 is now known as sigmavirus24_awa | 18:12 | |
| * odyssey4me looks | 18:12 | |
| *** willlo has joined #openstack-ansible | 18:15 | |
| *** mikelk has quit IRC | 18:17 | |
| *** cemmason has quit IRC | 18:21 | |
| *** cemmason has joined #openstack-ansible | 18:21 | |
| *** skamithi13 has quit IRC | 18:21 | |
| jwitko | odyssey4me, do I need to add the gateway at the same level as the management_netmask variable? | 18:23 |
| jwitko | it looks like the template checks if gateway is defined at that level | 18:23 |
| odyssey4me | jwitko as per http://docs.openstack.org/developer/openstack-ansible/install-guide/configure-networking.html#adding-static-routes-to-network-interfaces do you see the container having the /etc/network/interfaces.d/ with the post-up command? | 18:24 |
| jwitko | odyssey4me, yes I do. it has the correct gateway in the via | 18:24 |
| jwitko | but its not coming up with that gateway | 18:24 |
| jwitko | if I add a static route to the gateway manually, it works just fine | 18:25 |
| odyssey4me | jwitko have you tried adding it manually inside the container to see if it works? | 18:25 |
| jwitko | odyssey4me, yes and adding it manually makes the container work perfectly | 18:26 |
| jwitko | it can respond to ICMP without any issues once I add it | 18:26 |
| odyssey4me | jwitko and then if you remove it and ifdown/ifup it doesn't work? | 18:27 |
| odyssey4me | and are there any entries in syslog that may pertain to the actions? | 18:27 |
| jwitko | well I added it via the route command, not the interface file | 18:27 |
| *** subscope has joined #openstack-ansible | 18:28 | |
| odyssey4me | jwitko ok, but the lxc-container-create adds it into the interface file - so is it in the interface file? | 18:28 |
| jwitko | odyssey4me, here is the interface file | 18:28 |
| jwitko | http://paste.openstack.org/raw/489024/ | 18:29 |
| jwitko | it adds the "via gateway" on the 2nd to last line | 18:29 |
| jwitko | but that is not actually setting it as a default gateway for eth1 | 18:29 |
| jwitko | so when it recieves packets from the outside it is looking to respond via its only default gateway, the 10.0 internal one. which is of course on a different interface so it drops it | 18:30 |
| odyssey4me | jwitko the eth0 interface has a default gateway too - if you remove that, then ifdown eth1/ifup eth1 - does it work? | 18:33 |
| *** jwagner is now known as jwagner_lunch | 18:35 | |
| *** zhangjn has quit IRC | 18:37 | |
| jwitko | the other one is dhcp | 18:37 |
| odyssey4me | jwitko sure - I'm asking for that to be done as a test | 18:38 |
| jwitko | eth0, is dhcp | 18:38 |
| jwitko | you mean remove it using the route command ? | 18:38 |
| odyssey4me | yep | 18:38 |
| *** zhangjn has joined #openstack-ansible | 18:38 | |
| jwitko | sure one moment | 18:38 |
| openstackgerrit | Jesse Pretorius (odyssey4me) proposed openstack/openstack-ansible: [WIP] Gate: Set pip index and extra-index to OpenStack-CI repositories https://review.openstack.org/265867 | 18:39 |
| jwitko | odyssey4me, I have removed it but I'm getting a very odd error on eth1 | 18:40 |
| jwitko | ifdown: interface eth1 not configured | 18:40 |
| jwitko | the interface is up, it has an IP | 18:40 |
| odyssey4me | jwitko have you restarted the container since adding the extra route? | 18:41 |
| jwitko | I just restarted it before | 18:41 |
| odyssey4me | jwitko was the container created first, then the route added later - or was it all done at the same time? | 18:41 |
| *** sdake has joined #openstack-ansible | 18:41 | |
| jwitko | so, in its initial created form the route table looks as I pasted above | 18:42 |
| jwitko | http://paste.openstack.org/raw/489018/ | 18:42 |
| odyssey4me | jwitko so honestly I have no idea what could be wrong - perhaps Apsu or rackertom could assist? | 18:42 |
| odyssey4me | something appears to be a bit weird there | 18:42 |
| *** alejandrito has quit IRC | 18:43 | |
| odyssey4me | it seems that you've done the config right from an OSA standpoint, but the interface isn't doing what's expected of it | 18:43 |
| jwitko | odyssey4me, if I want to delete all the containers and rebuild them | 18:43 |
| odyssey4me | I'm really crappy at networking | 18:43 |
| jwitko | I just need to delete my inventory.json generated file and run the lxc-container-detroy right ? | 18:43 |
| jwitko | or vice versa | 18:43 |
| odyssey4me | jwitko you could try and destroy one of the containers and rebuilt it to see if it works better | 18:43 |
| odyssey4me | jwitko you can leave the entry in the inventory | 18:43 |
| odyssey4me | no need to regenerate the inventory at all | 18:43 |
| jwitko | well I'm thinking I want to regenerate the inventory with the "gateway: " variable inside the container interface config | 18:44 |
| odyssey4me | just execute the lxc-container-detroy playbook using the container name as a limit | 18:44 |
| odyssey4me | ah ok, well you can destroy the container, then remove it from the inventory using inventory-manage (in the scripts folder), then yes you can recreate the whole thing | 18:44 |
| Apsu | What's up? | 18:45 |
| Apsu | Oh, ifdown is complaining. | 18:45 |
| jwitko | Hey Apsu, here is the original issue: | 18:45 |
| jwitko | Having an issue I'm wondering if you could help me fix... i think its a quick one. My route table on my containers is not being set with a proper gateway for the management interface. http://paste.openstack.org/raw/489018/ -- as you can see in that route table there is no default gateway set for 10.96.76.0/24 (it should be 10.96.76.1). My interface is defined in the user config | 18:45 |
| jwitko | as follows, with the gateway specified http://paste.openstack.org/raw/489019/ . Any idea what I'm doing wrong? | 18:45 |
| Apsu | Yeah, ifup/ifdown are Ubuntu's way of managing interface configuration. When ifdown says that, it just means ifup wasn't used to configure it. | 18:46 |
| Apsu | Reading | 18:46 |
| openstackgerrit | Jesse Pretorius (odyssey4me) proposed openstack/openstack-ansible: [WIP] Gate: Set pip index and extra-index to OpenStack-CI repositories https://review.openstack.org/265867 | 18:46 |
| odyssey4me | I wonder if ip set dev down / ip set dev up implements the post-up commands? | 18:48 |
| *** subscope has quit IRC | 18:48 | |
| Apsu | jwitko: First of all, don't use route or ifconfig. They're abandoned and fundamentally broken. | 18:48 |
| *** jthorne has quit IRC | 18:48 | |
| *** subscope has joined #openstack-ansible | 18:48 | |
| Apsu | jwitko: Second... you can't have two (working) default gateways. The default gateway isn't supposed to be your management network | 18:48 |
| Apsu | It's supposed to be the LXC container network, on eth0. Which you have, according to the first paste. | 18:49 |
| *** subscope has quit IRC | 18:49 | |
| jwitko | Apsu, ok but then why are packets getting dropped going out ? | 18:49 |
| jwitko | to me it looks like they're trying to go out the default gateway, on eth0 after they are received on eth1. | 18:49 |
| spotz | automagically you still changing the clouds.yaml one? | 18:49 |
| Apsu | Elaborate. Which packets, generated from what, going to where, and how are you determining they're dropped? | 18:49 |
| jwitko | Apsu, so packets are coming from an F5 load balancer via icmp health check going to the container IP address. | 18:50 |
| Apsu | Container management IP? | 18:50 |
| jwitko | I am determining they're dropped by listening for all/any ICMP on the local IP and I only see the incoming ICMP requests, nothing going out | 18:50 |
| jwitko | Apsu, correct. | 18:50 |
| jwitko | sorry, container SSH IP | 18:51 |
| jwitko | I don't know if thats what manage is, or if the 10.0.3 is the mgmt | 18:51 |
| Apsu | That's not clarifying my question. According to your paste configs, 10.96.76.0/24 is your management CIDR | 18:51 |
| Apsu | On eth1 in the container. | 18:51 |
| jwitko | yes, correct | 18:51 |
| Apsu | So you're getting traffic to that IP from your F5, yes? | 18:51 |
| jwitko | then yes, container management IP. | 18:51 |
| *** retreved has quit IRC | 18:52 | |
| Apsu | Ok. What IP is that traffic coming from? | 18:52 |
| jwitko | it comes from a self-IP on the load balancer. 10.96.112.5 | 18:52 |
| Apsu | So there's your problem. It's not in your management CIDR. | 18:52 |
| Apsu | So when the container receives it, it doesn't have a route back to that CIDR, so it uses the default gateway | 18:52 |
| jwitko | exactly | 18:53 |
| Apsu | If your F5 is going to talk to your containers, you need them to be on the same CIDR, ideally. | 18:53 |
| jwitko | but the default gateway for the vlan of the mgmt network is 10.96.97.1 | 18:53 |
| jwitko | but thats not really true, if the containers could just have their default gateway for eth1 set then they can talk just fine | 18:53 |
| Apsu | That's only because you've got routing that makes it work on the host side, where your .1 lives. | 18:54 |
| Apsu | Or did you put .1 on the F5? | 18:54 |
| jwitko | no, .1 is on a cisco device | 18:54 |
| Apsu | Then that's where the routing that can see the F5's self-ip is. | 18:54 |
| Apsu | Either way, it's only due to that it will work. | 18:54 |
| Apsu | Not because what you're suggesting is more correct. | 18:54 |
| *** admin0 has joined #openstack-ansible | 18:55 | |
| Apsu | The simpler and more correct way would be to use an IP in the same range to do your health checks from :P | 18:55 |
| *** admin0 has quit IRC | 18:55 | |
| jwitko | yea.. | 18:55 |
| *** admin0 has joined #openstack-ansible | 18:55 | |
| Apsu | See, the containers need to hit the internet for various things | 18:55 |
| jwitko | that requires adding the vlan to the F5 interface, adding a self-ip to the F5 | 18:55 |
| Apsu | And we setup the host's lxc bridge to allow that, by default | 18:56 |
| Apsu | Which is plumbed into containers on eth0 | 18:56 |
| Apsu | So that's why they get the default gateway | 18:56 |
| *** subscope has joined #openstack-ansible | 18:56 | |
| Apsu | Changing that default gateway would require that your management network can hit the internet, instead | 18:56 |
| Apsu | So if that's not how you already have it setup... it's going to break other things | 18:56 |
| jwitko | well, the containers use a proxy for everything | 18:56 |
| Apsu | That's fine. | 18:57 |
| jwitko | alright. it makes sense now | 18:58 |
| jwitko | thank you for your time and explanation | 18:58 |
| Apsu | Cool. You bet. | 18:58 |
| jwitko | Apsu, one more question if you don't mind | 18:58 |
| Apsu | Sure | 18:58 |
| jwitko | In researching this I just noticed the ability to set MTUs specifically for SAN interfaces in the user config file | 18:58 |
| jwitko | my inventory.json is already generated, is there any way for me to keep all the same container IPs and stuff (they're already added to my load balancer) but get that bit added into the inventory file ? | 18:59 |
| *** galstrom is now known as galstrom_zzz | 18:59 | |
| *** Bjoern_ is now known as Bjoern_zZzZzZzZ | 18:59 | |
| Apsu | Yes. Changing things in the user config and rerunning the appropriate playbooks won't generate new IPs for the containers, it will merge in new config | 18:59 |
| Apsu | You can test this yourself if you're nervous by backing up your inventory and just running the inventory generator itself manually, and inspecting the output | 19:00 |
| *** thorst_ has quit IRC | 19:00 | |
| Apsu | It's a python file | 19:00 |
| jwitko | ah ok cool, what do I run to regenerate the json ? | 19:00 |
| *** thorst has joined #openstack-ansible | 19:01 | |
| *** thorst has quit IRC | 19:01 | |
| *** thorst has joined #openstack-ansible | 19:01 | |
| *** skamithi13 has joined #openstack-ansible | 19:01 | |
| Apsu | Under playbooks/inventory is https://github.com/openstack/openstack-ansible/blob/master/playbooks/inventory/dynamic_inventory.py | 19:01 |
| odyssey4me | executing /opt/openstack-ansible/playbooks/inventory/dynamic_inventory.py should just spit out the json | 19:01 |
| odyssey4me | to stdout | 19:02 |
| jwitko | thanks guys | 19:02 |
| Apsu | Looks like we've got a --list arg too. | 19:02 |
| Apsu | Guessing that may pretty-print :P | 19:02 |
| odyssey4me | Apsu you may wish to inspect https://review.openstack.org/#/c/286078/ and suggest alternative methods that could be used :) | 19:02 |
| *** Bjoern_zZzZzZzZ is now known as Bjoern_ | 19:03 | |
| *** phiche has joined #openstack-ansible | 19:05 | |
| *** jthorne has joined #openstack-ansible | 19:06 | |
| *** jthorne has joined #openstack-ansible | 19:06 | |
| jmccrory | logan- stevelle odyssey4me https://gist.github.com/jimmymccrory/312542d5f56d14b35f9d | 19:06 |
| *** weezS has joined #openstack-ansible | 19:06 | |
| Apsu | odyssey4me: Ugh. | 19:07 |
| *** jwagner_lunch is now known as jwagner | 19:09 | |
| Apsu | So many better ways to do it. Like using the --check feature. And || to catch the error. | 19:10 |
| odyssey4me | Apsu review away :) | 19:10 |
| odyssey4me | jmccrory AttributeError: 'NoneType' object has no attribute 'group' when run against console.html :/ | 19:11 |
| jmccrory | oh yeah...doesn't work directly against the html, needs to be the text copied out of it | 19:11 |
| odyssey4me | now if only I could point it at a URL ;) | 19:12 |
| *** kencjohnston has joined #openstack-ansible | 19:12 | |
| jmccrory | hah yeah, can work on that. more regex | 19:13 |
| logan- | thanks jmccrory that looks a lot less crappy than the one i just made | 19:13 |
| automagically | spotz: Yep, no more changes to the cloud.yaml review | 19:13 |
| *** flaviosr has quit IRC | 19:13 | |
| logan- | i just pulled down 2 gigs of the console.htmls and im regexing against that. it is pretty slow but getting some initial stuff gathered up now | 19:13 |
| spotz | okie going to look:) | 19:13 |
| automagically | Going AFK for a bit. Gotta eat some lunch | 19:15 |
| odyssey4me | logan- jmccrory FYI: http://logstash.openstack.org/#dashboard/file/logstash.json?query=project%3A%5C%22openstack%2Fopenstack-ansible%5C%22%20AND%20message%3A%5C%22Build%20timed%20out%20(after%2090%20minutes).%20Marking%20the%20build%20as%20failed.%5C%22 | 19:18 |
| *** abitha has joined #openstack-ansible | 19:20 | |
| stevelle | ovh, timeouts. Seems like a fair bit of wasted compute because of that timout. Maybe some clouds should have a longer timeout. | 19:21 |
| odyssey4me | stevelle it's unfortunately a one size fits all | 19:21 |
| odyssey4me | and infra won't let us increase the timeout because we're behaving badly | 19:22 |
| *** sigmavirus24_awa is now known as sigmavirus24 | 19:22 | |
| odyssey4me | we don't use their apt repositories, or their pip config, or any of their cached git repositories | 19:22 |
| palendae | stevelle: Yeah, odyssey4me has been saying for a while that we're not really playing by infra's rules, so they're not willing to increase that yet | 19:23 |
| odyssey4me | so basically we need to optimise, use the facilities they provide - or we need to switch to using external CI for integrated gating | 19:23 |
| Apsu | odyssey4me: Reviewed. | 19:23 |
| odyssey4me | thanks Apsu | 19:24 |
| Apsu | Yep | 19:25 |
| palendae | Seems like using the cached git repos would be low hanging fruit | 19:25 |
| palendae | Not sure that it would be the biggest time saver since I've not benchmarked, though | 19:26 |
| *** weezS has quit IRC | 19:26 | |
| odyssey4me | well, we also do other things like use external repositories for galera, a downloaded deb for rabbit, etc | 19:27 |
| odyssey4me | all these are moving parts which go outside of the cloud provider and therefore cause higher latency and potential failures | 19:28 |
| palendae | Right | 19:28 |
| stevelle | palendae: the cached git repos bit should be low hanging and if it isn't slower than it gets us closer to infra's good side which is enough | 19:28 |
| palendae | Some of those are inherent risks | 19:28 |
| odyssey4me | their apt repositories right now are only ubuntu, but I believe they're open to adding other stuff | 19:29 |
| stevelle | faster doesn't need to be the outcome in that regard | 19:29 |
| stevelle | helps if it is | 19:29 |
| odyssey4me | so the trouble is that the cached stuff is all on the host, so we'd have to figure out how to (only in the gate) bind mount that into the repo container and consume the cached stuff | 19:29 |
| palendae | Ahhh | 19:29 |
| *** eil397 has joined #openstack-ansible | 19:30 | |
| palendae | Yeah, so you'd need to bind mount it and provide a local git url | 19:30 |
| odyssey4me | all their optimisations are built with devstack in mind | 19:30 |
| palendae | Right | 19:30 |
| palendae | Hence 'DevStack VM' (dsvm) | 19:30 |
| odyssey4me | the issue I have with all this is that we deviate in the gate quite substantially from a real world deployment | 19:30 |
| palendae | Right | 19:31 |
| palendae | I was gonna say :) | 19:31 |
| palendae | I mean, an AIO is already a substantial deviation | 19:31 |
| palendae | Anyone really using that other than for sanity checks is asking for trouble | 19:31 |
| odyssey4me | but we may have to do something like that for on-commit gating and then have a periodic job (or external CI) which doesn't deviate as much | 19:31 |
| odyssey4me | yes, this is true | 19:31 |
| cloudnull | odyssey4me: https://review.openstack.org/#/c/287331/ seems to work | 19:32 |
| *** galstrom_zzz is now known as galstrom | 19:32 | |
| cloudnull | ovh env in 1 hour 20 min | 19:32 |
| odyssey4me | I'm not too much of a fan of creating code paths specifically for OpenStack-CI usage, but it seems we may have to | 19:32 |
| odyssey4me | cloudnull aha, that's a good result | 19:32 |
| odyssey4me | now the trouble is that it relies on the repo server adjustments which won't merge without that | 19:33 |
| logan- | http://cdn.pasteraw.com/hclu0giwlnb01b4ir36er4p5nr3mbwn | 19:33 |
| *** albertcard has joined #openstack-ansible | 19:33 | |
| odyssey4me | maybe we'll have to combine the patches? | 19:33 |
| odyssey4me | *repo build adjustments | 19:33 |
| *** weezS has joined #openstack-ansible | 19:33 | |
| odyssey4me | so I'm suggesting that we combine https://review.openstack.org/#/c/284896/ and https://review.openstack.org/#/c/287331/ to unblock the master gate | 19:34 |
| palendae | odyssey4me: Nor am I. I wonder what the puppet project does for their integration | 19:35 |
| odyssey4me | thoughts? stevelle cloudnull automagically jmccrory | 19:35 |
| odyssey4me | palendae they only added an integration test recently, and I see them make it non voting often | 19:35 |
| stevelle | odyssey4me: abandon 287331, merge with parent, modify commit msg of parent to reference 287331. | 19:35 |
| odyssey4me | stevelle yep, that's my thinking | 19:35 |
| palendae | odyssey4me: Gotcha. | 19:36 |
| jmccrory | sounds good to me | 19:36 |
| logan- | data points used for that: http://cdn.pasteraw.com/qmj1wpl527x93jnh3at8xp2whrwszcn | 19:36 |
| palendae | logan-: Awesome\ | 19:37 |
| odyssey4me | logan- any way we can get a reverse sorted result in reverse order of longest to shortest? | 19:37 |
| jmccrory | nice logan- | 19:37 |
| odyssey4me | even just a top 20 will do | 19:37 |
| odyssey4me | excellent extraction by the way, thanks jmccrory and logan- | 19:37 |
| stevelle | logan-: based on that I think using the apt caches and git repo caches seems highly likely to make ovh faster, ths | 19:37 |
| odyssey4me | wow: lxc_container_create | Create container: | 19:38 |
| odyssey4me | devstack-trusty-rax-iad (43 samples) - 199.56 | 19:38 |
| odyssey4me | devstack-trusty-ovh-bhs1 (58 samples) - 429.13 | 19:38 |
| *** retreved has joined #openstack-ansible | 19:38 | |
| palendae | Dang | 19:38 |
| logan- | yeah one sec @ sorted | 19:38 |
| palendae | Over twice as long | 19:38 |
| logan- | local mirror usage would help a lot but there are still things that are super slow on ovh that don't use any network like extracting things etc | 19:38 |
| odyssey4me | repo_build | Clone all upstream git repositories: | 19:39 |
| odyssey4me | devstack-trusty-rax-iad (43 samples) - 254.25 | 19:39 |
| odyssey4me | devstack-trusty-ovh-bhs1 (58 samples) - 331.85 | 19:39 |
| *** raddaoui has joined #openstack-ansible | 19:39 | |
| *** skamithi has joined #openstack-ansible | 19:39 | |
| palendae | I can't remember, does the create container task hit anything hosted by rackspace? | 19:39 |
| stevelle | I still see 5 minutes in low-hanging fruit | 19:39 |
| stevelle | even if lxc doesn't get faster | 19:39 |
| odyssey4me | palendae only this one: | 19:40 |
| odyssey4me | lxc_hosts | Download lxc cache(s): | 19:40 |
| odyssey4me | devstack-trusty-rax-iad (43 samples) - 10.91 | 19:40 |
| odyssey4me | devstack-trusty-ovh-bhs1 (58 samples) - 20.29 | 19:40 |
| jmccrory | i tested async/polling locally against the container create task over the weekend but didn't see much improvement unfortunately, not sure what can be done there | 19:40 |
| *** retreved has quit IRC | 19:40 | |
| palendae | So still twice as long, but an order of magnitude less that the create | 19:40 |
| *** retreved has joined #openstack-ansible | 19:40 | |
| logan- | yea all of the unarchive venv tasks are about the same % diff | 19:41 |
| odyssey4me | a little improvement here could do well: lxc_hosts | Prepare cached image: | 19:41 |
| odyssey4me | devstack-trusty-rax-iad (43 samples) - 44.97 | 19:41 |
| odyssey4me | devstack-trusty-ovh-bhs1 (58 samples) - 107.12 | 19:41 |
| logan- | it seems like ovh is just very slow to extract things | 19:41 |
| palendae | Is OVH still 8GB RAM? | 19:41 |
| odyssey4me | yeah, I suspect that rax is using ssd's while ovh is using spinning disks | 19:41 |
| stevelle | palendae: it is | 19:41 |
| odyssey4me | yep all flavors have the same cpu8/ram8 | 19:41 |
| palendae | Ok | 19:41 |
| logan- | if there are more cores available maybe it would help to use a threaded untar | 19:41 |
| palendae | Thought so but worth confirming | 19:41 |
| *** jwagner is now known as jwagner[away] | 19:42 | |
| stevelle | reasonable to guess ovh is i/o bound | 19:42 |
| odyssey4me | all the same - I've already done the investigation | 19:42 |
| logan- | :( | 19:42 |
| odyssey4me | and infra won't provide different flavors, even if they have access to them | 19:42 |
| palendae | Sure | 19:42 |
| odyssey4me | they don't want to set a precedent | 19:42 |
| palendae | That's reasonable | 19:42 |
| odyssey4me | also, I asked about using two nodes instead of 1 - they're open to it, once we've become a good citizen and have proved that 1 node isn't enough | 19:43 |
| odyssey4me | they also warned that the failure rate for two nodes is typically higher | 19:43 |
| *** phalmos has joined #openstack-ansible | 19:43 | |
| palendae | So some of this needs to be done even if it's not necessarily going to speed things up | 19:43 |
| odyssey4me | in our setup, I don't really see two nodes improving performance anyway | 19:43 |
| stevelle | only if we decrease the density of containers really | 19:44 |
| palendae | Yeah, unlikely for performance | 19:44 |
| stevelle | but then the added overhead | 19:44 |
| palendae | Difference between us and the rest of openstack is we're trying to emulate a full cluster | 19:44 |
| odyssey4me | the repo build still only happens in one container | 19:44 |
| palendae | Not just spinning up the bare minimum | 19:44 |
| odyssey4me | the lxc cache prep would happen in the same way on both hosts | 19:44 |
| logan- | http://cdn.pasteraw.com/7hv2xz5dopjwlf0qi0iu87lx7ohqfoe sorted | 19:46 |
| *** phalmos has quit IRC | 19:47 | |
| openstackgerrit | Jesse Pretorius (odyssey4me) proposed openstack/openstack-ansible: Implement repo_build IRR & Reduce galera/rabbit/horizon/keystone/repo affinity https://review.openstack.org/284896 | 19:49 |
| stevelle | any chance that zfs would have any benefit on ovh for the storage bottleneck | 19:49 |
| *** galstrom is now known as galstrom_zzz | 19:50 | |
| odyssey4me | stevelle cloudnull jmccrory automagically https://review.openstack.org/284896 should unblock master to give us some breathing room | 19:50 |
| odyssey4me | stevelle not sure - what're you thinking? | 19:50 |
| odyssey4me | I know that I've thought that perhaps we should change the way we do the file-based container image prep | 19:51 |
| odyssey4me | maybe use overlayfs or something | 19:51 |
| stevelle | wondering if we can do overlayfs or something on ovh-only to get speed. I know we didn't see an improvement on ssds | 19:51 |
| odyssey4me | that way we wouldn't have to extract the same data so many times | 19:51 |
| stevelle | I was thinking a copy on write might help | 19:51 |
| odyssey4me | it would be an overhaul of the way we prep the container image though | 19:52 |
| palendae | I think hughsaunders looked at that kind of thing in Juno and saw a performance decrease | 19:52 |
| odyssey4me | but hey - it could cut the time taken to create the containers fairly substantially | 19:52 |
| palendae | But maybe it was during container runs | 19:52 |
| palendae | Don't remember the details, I just know he tried some COW FS | 19:52 |
| stevelle | palendae: that was also on an ssd backend | 19:52 |
| stevelle | we weren't i/o bottlenecked | 19:52 |
| openstackgerrit | Major Hayden proposed openstack/openstack-ansible-os_neutron: Add LBaaS v2 support https://review.openstack.org/287237 | 19:52 |
| odyssey4me | palendae yeah, but when I asked all he did was have the fs be overlayfs... in our container prep we still do a tarball extract of the data into the fs of the container | 19:53 |
| odyssey4me | well, we did back then | 19:53 |
| odyssey4me | the process has changed somewhat and it may be worth the experiment again | 19:53 |
| palendae | Yeah | 19:53 |
| palendae | Ah, right | 19:53 |
| cloudnull | odyssey4me stevelle we wont be able to gain anything with the overlayfs bits if the kernel isnt 3.19+ | 19:53 |
| odyssey4me | I was working on a revised method of prepping the containers in https://review.openstack.org/272743 - I haven't had the time to get it done | 19:54 |
| palendae | He just changed the fs type in the lxc configs | 19:54 |
| stevelle | wondering if the tarball is actually faster on ovh, maybe not | 19:54 |
| stevelle | cloudnull: that's right, recalling that now | 19:54 |
| palendae | cloudnull: Also good point, but we're also moving towards 16.04 support "soon" | 19:54 |
| stevelle | not soon enough for that sadly | 19:54 |
| odyssey4me | stevelle ++ | 19:54 |
| cloudnull | ++ | 19:54 |
| cloudnull | if we convert to the download template its quite a bit faster. | 19:54 |
| palendae | Is 14.04 still < 3.19? | 19:54 |
| cloudnull | palendae: 3.13 by default | 19:55 |
| palendae | Bah | 19:55 |
| cloudnull | 4.2 is available | 19:55 |
| odyssey4me | palendae yeah, you need to add the wily kernel package to upgrade it | 19:55 |
| palendae | Yeah, so upgrade and reboot | 19:55 |
| palendae | Bleh | 19:55 |
| odyssey4me | correct | 19:55 |
| stevelle | back to liberty upgrades (boo) | 19:55 |
| openstackgerrit | Bjoern Teipel proposed openstack/openstack-ansible-os_neutron: Fix auth_url for metadata_agent.ini https://review.openstack.org/287411 | 19:55 |
| * palendae should do that too | 19:55 | |
| *** neilus has quit IRC | 19:56 | |
| stevelle | more yak shaving to get toward 3.19 | 19:56 |
| openstackgerrit | Bjoern Teipel proposed openstack/openstack-ansible-os_neutron: Fix auth_url for neutron configuration https://review.openstack.org/287411 | 19:58 |
| *** sdake_ has joined #openstack-ansible | 19:59 | |
| *** sdake has quit IRC | 20:00 | |
| *** thorst has quit IRC | 20:01 | |
| *** thorst has joined #openstack-ansible | 20:01 | |
| *** hybridpollo has joined #openstack-ansible | 20:01 | |
| *** sdake has joined #openstack-ansible | 20:03 | |
| *** thorst has quit IRC | 20:05 | |
| *** neilus has joined #openstack-ansible | 20:05 | |
| *** sdake_ has quit IRC | 20:06 | |
| openstackgerrit | Jesse Pretorius (odyssey4me) proposed openstack/openstack-ansible: Use current, but pinned versions of pip, setuptools and wheel https://review.openstack.org/284977 | 20:06 |
| openstackgerrit | Jesse Pretorius (odyssey4me) proposed openstack/openstack-ansible: [WIP] Gate: Set pip index and extra-index to OpenStack-CI repositories https://review.openstack.org/265867 | 20:06 |
| openstackgerrit | Jesse Pretorius (odyssey4me) proposed openstack/openstack-ansible: IRR - Implemented for os_keystone https://review.openstack.org/279710 | 20:06 |
| *** willlo has quit IRC | 20:07 | |
| *** thorst has joined #openstack-ansible | 20:08 | |
| *** galstrom_zzz is now known as galstrom | 20:09 | |
| *** neilus has quit IRC | 20:10 | |
| *** javeriak_ has quit IRC | 20:10 | |
| *** keedya has quit IRC | 20:19 | |
| *** unlaudable has joined #openstack-ansible | 20:27 | |
| *** permalac has quit IRC | 20:30 | |
| *** keedya has joined #openstack-ansible | 20:31 | |
| *** raddaoui has quit IRC | 20:31 | |
| openstackgerrit | Jesse Pretorius (odyssey4me) proposed openstack/openstack-ansible-openstack_openrc: No longer defining OS_TENANT_NAME in openrc https://review.openstack.org/287309 | 20:32 |
| odyssey4me | automagically ^ minor change to the commit message - using the change id allows you to click through and see the original commit and any backports | 20:33 |
| automagically | Ah cool, thanks for that odyssey4me | 20:33 |
| automagically | cloudnull: You around? | 20:39 |
| *** mathlin1 has joined #openstack-ansible | 20:42 | |
| cloudnull | i am | 20:44 |
| automagically | cloudnull: Had a qq about the intent of https://github.com/openstack/openstack-ansible-os_keystone/blob/master/tests/stand-alone.yml | 20:45 |
| automagically | I _think_ its a test for determining if the role can function outside of the typical OSA inventory patterns, but that isn’t completely clear from the file itself | 20:46 |
| cloudnull | its to test keystone without a db or messaging backend | 20:46 |
| cloudnull | basically yes to test keystone outside of a typical osa use case | 20:46 |
| automagically | Okay, the vars for rabbit and galera definitely obscure the intent | 20:47 |
| cloudnull | Like if you wanted to run keystone with a custom auth provider that didnt require access to a db | 20:47 |
| automagically | Perhaps some commentary on the top of the playbook is in order | 20:47 |
| cloudnull | yea. not all of that is finished. | 20:47 |
| automagically | Ah, gotcha | 20:47 |
| automagically | Thanks for the context, was digging into the tests there to see if I could be of help in getting the gate functional tests running for that role | 20:48 |
| odyssey4me | I quite like that idea - test.yml can be a meta-playbook that includes other playbooks - but each test playbook should have some sort of description of what it's trying to achieve | 20:48 |
| odyssey4me | although, that said, perhaps that info belongs in the docs for the role instead? | 20:48 |
| cloudnull | automagically: theres a stand-alone inventory too | 20:49 |
| stevelle | Seems odd to document test cases in docs | 20:49 |
| cloudnull | which is called by the tox functional tests . | 20:49 |
| automagically | Since I’m a TDD guy most of the time, I tend to expect the tests to communicate their intent before I go looking for docs outside of the tests | 20:49 |
| stevelle | +1 | 20:49 |
| odyssey4me | stevelle perhaps, but I'm kind-of thinking that the docs for the role should describe the use-cases, and then point at the specific test to illustrate how that use case is implemented | 20:49 |
| openstackgerrit | Travis Truman (automagically) proposed openstack/openstack-ansible-os_swift: Adding Vagrantfile for local developer testing https://review.openstack.org/287430 | 20:50 |
| automagically | odyssey4me: That combination would work well | 20:50 |
| stevelle | I'd be interested to see how that might look | 20:50 |
| odyssey4me | so sure - a short description in the playbook at the top, but the detailed background, purpose, etc in the docs | 20:50 |
| stevelle | it could work | 20:50 |
| *** dank_ has joined #openstack-ansible | 20:50 | |
| *** jwagner[away] is now known as jwagner | 20:51 | |
| odyssey4me | it would be a great way for people to understand how the role was designed to be used, what it expects as input and what it's designed to deliver | 20:51 |
| automagically | Ansible role documentation and testing is definitely an evolving art within the community | 20:51 |
| automagically | I think an ideal though is very close to what Jesse just described, which is docs that contain example playbook content that then link to the test suite for each example | 20:52 |
| odyssey4me | of course I would absolutely love it if we could use something like docstrings for the vars and have them automatically rendered for the roles in documentation | 20:52 |
| *** dank_ has left #openstack-ansible | 20:52 | |
| stevelle | odyssey4me: I'm looking at it as a way to merge your expressed interest in losing examples and replacing with docs. I'm seeing tests as the replacement, and now you're shifting how the docs play into that. | 20:53 |
| odyssey4me | automagically yep, that's pretty much what I'm suggesting - let the docs contain the long explanations of background/purpose and refer to the playbook for implementation details (which could change regularly) | 20:53 |
| odyssey4me | stevelle the trouble with the .example files is that they are documentation - but that documentation is not kept in sync with other documentation | 20:54 |
| odyssey4me | and developers update one or the other | 20:55 |
| stevelle | agreed, if they were exercised as tests, that helps | 20:55 |
| openstackgerrit | Travis Truman (automagically) proposed openstack/openstack-ansible-os_swift: Removing unneeded with_items usage for clarity https://review.openstack.org/287435 | 20:55 |
| stevelle | but the melding with a doc of a use case is interesting | 20:55 |
| *** kencjohnston has quit IRC | 20:55 | |
| stevelle | I look forward to seeing PoC | 20:55 |
| palendae | odyssey4me: fwiw, our upgrade process uses .example files | 20:55 |
| odyssey4me | ideally I see the docs as something that should contain stuff that doesn't change much - anything that does change often should ideally be automatically generated | 20:56 |
| *** b3rnard0- is now known as b3rnard0 | 20:56 | |
| palendae | Actually nevermind, the one I'm thinking of uses a straight .yml file https://review.openstack.org/#/c/286844/8/scripts/upgrade-utilities/playbooks/user-secrets-adjustment.yml | 20:56 |
| openstackgerrit | Major Hayden proposed openstack/openstack-ansible: Replace LBaaS v1 with v2 https://review.openstack.org/287243 | 20:56 |
| odyssey4me | so, for instance, the purpose of the role and the use cases it was designed to resolve don't change often - if anything you add more use cases | 20:57 |
| openstackgerrit | Major Hayden proposed openstack/openstack-ansible: Add LBaaS v2 support (OSA) https://review.openstack.org/287243 | 20:57 |
| odyssey4me | the tests are then designed against the use cases, and the code implements the use cases | 20:57 |
| mhayden | Apsu: would you mind giving me a looksee on https://review.openstack.org/#/c/287237/ and https://review.openstack.org/#/c/287243/ when you have time? | 20:57 |
| odyssey4me | so all the docs have to do is refer to the code and tests for implementation details | 20:58 |
| odyssey4me | but yeah, I need to find some time to put together some sort of PoC of how I see that all working | 20:58 |
| openstackgerrit | Bjoern Teipel proposed openstack/openstack-ansible: Adding release notes for erlang VM tuning parameters https://review.openstack.org/285506 | 20:59 |
| *** kencjohnston has joined #openstack-ansible | 20:59 | |
| logan- | more profiling from the stuff earlier.. much larger sample set. http://cdn.pasteraw.com/n2bzegt8grah7b5whijxmnyxtukc2v0 http://cdn.pasteraw.com/3l2wxvhdnqsfk6jr3q5snm17wwz5jkb http://cdn.pasteraw.com/5efufwesy1nmxnl6qis9z1zr1hbce0 | 20:59 |
| odyssey4me | wow, ovh-gra1 took over 5 mins just to clone repositories | 21:00 |
| Apsu | mhayden: Sure. | 21:01 |
| *** skamithi has quit IRC | 21:01 | |
| mhayden | Apsu: muchas gracias | 21:01 |
| automagically | logan-: That profiling is pretty crazy useful, thanks for putting that together | 21:02 |
| logan- | no prob | 21:02 |
| odyssey4me | ++ | 21:02 |
| *** neilus has joined #openstack-ansible | 21:03 | |
| odyssey4me | uh, wut: | 21:03 |
| pabelanger | odyssey4me: where did you see that? | 21:03 |
| odyssey4me | repo_build | Remove pool indexes if found: | 21:03 |
| odyssey4me | devstack-trusty-bluebox-sjc1 (50 samples) - 31.98 | 21:03 |
| odyssey4me | devstack-trusty-rax-iad (21 samples) - 40.46 | 21:03 |
| odyssey4me | repo_build | Make sure old venv build directories are clean: | 21:03 |
| odyssey4me | devstack-trusty-bluebox-sjc1 (25 samples) - 9.81 | 21:03 |
| odyssey4me | logan- are these counters the values all added together, or are they averages or aggregates? | 21:04 |
| pabelanger | odyssey4me: I think ovh has reported some network issues recently, but nothing has been confirmed | 21:04 |
| odyssey4me | pabelanger logan-'s pulled together some profiling info for our integrated test run so that we can look for optimisations - here's the set per provider: http://cdn.pasteraw.com/n2bzegt8grah7b5whijxmnyxtukc2v0 | 21:05 |
| odyssey4me | organised from longest to shortest task execution | 21:05 |
| logan- | http://cdn.pasteraw.com/j40s1vfmxmuwlm66sbijvn0yafd6m0a averages | 21:05 |
| Apsu | mhayden: In the neutron_upstart_init.yml, line 123-124 reference neutron_services['neutron-lbaas-agent'] in the when: conditional, but lines 116-117 reference neutron_services['neutron-lbaasv2-agent']. Am I missing something? | 21:06 |
| mhayden | gimme 5 to look | 21:07 |
| mhayden | jumped into another bug :/ | 21:07 |
| Apsu | np | 21:07 |
| pabelanger | odyssey4me: logan-: interesting, checking | 21:08 |
| openstackgerrit | Nolan Brubaker proposed openstack/openstack-ansible: Update user variables to Liberty names https://review.openstack.org/287438 | 21:08 |
| palendae | rromans, erikmwilson When the gate builds the docs, I'd like you guys to look at https://review.openstack.org/#/c/287438/ | 21:09 |
| Apsu | afk, grabbing some food | 21:09 |
| palendae | That commit has a dynamically build table of variable name changes | 21:09 |
| palendae | built* | 21:09 |
| pabelanger | would be interested to see which repos repo_build | Clone all upstream git repositorie does | 21:09 |
| pabelanger | likely could be optimized by using local cache | 21:10 |
| odyssey4me | pabelanger yeah, the trouble is that it's done inside a container | 21:10 |
| pabelanger | odyssey4me: have you done any updates to use the afs mirror? | 21:10 |
| odyssey4me | so we'd have to bind mount the local cache to the container | 21:10 |
| pabelanger | odyssey4me: Ya, container stuff would be interesting for git clones | 21:10 |
| odyssey4me | pabelanger I'm working on it - haven't had time to finish that up - the trouble is going to be configuring all containers to not validate the key | 21:11 |
| rromans | palendae: kk | 21:11 |
| pabelanger | odyssey4me: cool, would be interested to see that code. I haven't does anything yet myself | 21:11 |
| pabelanger | I wonder if some sort of lxc container cache could be created in the dib process | 21:12 |
| odyssey4me | pabelanger the big challenge is that for every optimisation we do for the gate, we deviate from real world deployment... and in doing so we don't properly check what a real world deployment would be like | 21:12 |
| pabelanger | so you have some sort of warm cache when jenkins runs | 21:12 |
| pabelanger | odyssey4me: right | 21:12 |
| odyssey4me | also, implementing special code paths for OpenStack-CI alone just doesn't feel right | 21:13 |
| palendae | Long shot, but does anyone know if reno releasenotes are aware of the Sphinx docs in the doc directory? I would like to add a link to detailed documentation in my release note, but would like to avoid hard linking to a URL that's not there yet | 21:13 |
| pabelanger | something like AFS mirrors might be fine, since you are just pinning to the closest one for apt, pypi and soon rpms | 21:13 |
| odyssey4me | but yeah, we need to do better - so I'm working on figuring out how to go about it without compromising the purpose of the integration test itself | 21:13 |
| pabelanger | local git repos, ya, I see what you are saying | 21:14 |
| odyssey4me | pabelanger yeah, we already had implemented 'closest mirror' stuff for apt - we just need to convert that into 'use the host's configured mirror, but allow an unsigned repo' | 21:14 |
| openstackgerrit | Major Hayden proposed openstack/openstack-ansible-os_keystone: Remove dangling Default domain cfg file https://review.openstack.org/287440 | 21:15 |
| *** unlaudable has quit IRC | 21:15 | |
| odyssey4me | using the host's default mirror is actually how our AIO is designed to work by default, but allowing an unsigned mirror is not something we would ever advocate for production and thus it has never been instrumented | 21:15 |
| pabelanger | odyssey4me: Hmm, I though gpg was working for the mirrors. Would have to test it again | 21:15 |
| odyssey4me | it is? if so, then it's changed and we can undo some of the stuff we implemented to work around that | 21:16 |
| odyssey4me | our security role, for instance, had to skip the test which checks that unsigned repositories were not enabled | 21:16 |
| pabelanger | not sure, I would double check with -infra. I don't think we are doing anything special to the packages that would require a new gpg key for them | 21:17 |
| odyssey4me | pabelanger they're not doing a straight mirror - I had a bit of a chat with monty about it asking why | 21:17 |
| odyssey4me | they're actually building their own repo | 21:17 |
| odyssey4me | so the indexes for the repo don't match the ubuntu indexes | 21:18 |
| odyssey4me | and therefore you need to allow the repo to be unsigned | 21:18 |
| openstackgerrit | Major Hayden proposed openstack/openstack-ansible: Release notes for dangling domain fix https://review.openstack.org/287444 | 21:18 |
| mhayden | Apsu: good catch on that -- fixing now | 21:19 |
| odyssey4me | pabelanger hence https://github.com/openstack/openstack-ansible/commit/b4bb5105b62404b4f9c5e6f1b0384e5f60b65506 | 21:20 |
| openstackgerrit | Major Hayden proposed openstack/openstack-ansible-os_neutron: Add LBaaS v2 support https://review.openstack.org/287237 | 21:20 |
| mhayden | automagically: dang, thanks for the fast +1 :P | 21:20 |
| *** cemmason has quit IRC | 21:20 | |
| *** unlaudable has joined #openstack-ansible | 21:20 | |
| *** cemmason has joined #openstack-ansible | 21:20 | |
| openstackgerrit | Nolan Brubaker proposed openstack/openstack-ansible: Update user variables to Liberty names https://review.openstack.org/287438 | 21:20 |
| automagically | mhayden: I remembered the change from OSA, so its a simple review to see you move it to the IRR | 21:21 |
| mhayden | woot | 21:21 |
| mhayden | thanks anyways, automagically :) | 21:21 |
| * mhayden needs one more core on openstack-manuals :/ | 21:21 | |
| *** unlaudable has quit IRC | 21:22 | |
| *** dnt has joined #openstack-ansible | 21:23 | |
| openstackgerrit | Nolan Brubaker proposed openstack/openstack-ansible: Update user variables to Liberty names https://review.openstack.org/287438 | 21:23 |
| pabelanger | odyssey4me: Ah, that is right. Forgot about that | 21:23 |
| *** cemmason1 has joined #openstack-ansible | 21:23 | |
| *** unlaudable has joined #openstack-ansible | 21:24 | |
| mhayden | Apsu: i updated that commit | 21:25 |
| *** cemmason has quit IRC | 21:25 | |
| odyssey4me | mhayden feedback on https://review.openstack.org/287440 - I'm not sure that I'm right, but suspect that I am :) | 21:26 |
| odyssey4me | you should possibly test | 21:26 |
| mhayden | you might be right | 21:27 |
| mhayden | your method is a bit cleaner | 21:27 |
| odyssey4me | mhayden another comment | 21:27 |
| odyssey4me | sudden realisation :) | 21:27 |
| mhayden | i see your point, but i might check for it anyway | 21:27 |
| mhayden | just in case someone removed it | 21:28 |
| mhayden | :P | 21:28 |
| *** unlaudable has quit IRC | 21:28 | |
| odyssey4me | mhayden then a lot more would blow up | 21:28 |
| mhayden | true | 21:28 |
| mhayden | but my task wouldn't! | 21:28 |
| odyssey4me | the template conditionals expect it to be defined | 21:28 |
| * mhayden chuckles | 21:28 | |
| * mhayden serious'd | 21:28 | |
| odyssey4me | :p | 21:28 |
| mhayden | odyssey4me: relevant -> https://s-media-cache-ak0.pinimg.com/736x/c4/21/3e/c4213e6c108dbf313801958d4cd67b16.jpg | 21:29 |
| *** unlaudable has joined #openstack-ansible | 21:29 | |
| odyssey4me | haha | 21:29 |
| odyssey4me | anyone for Rocket League on XBox? | 21:29 |
| odyssey4me | it's time to stop computering | 21:29 |
| palendae | odyssey4me: pft, xbox | 21:30 |
| openstackgerrit | Major Hayden proposed openstack/openstack-ansible-os_keystone: Remove dangling Default domain cfg file https://review.openstack.org/287440 | 21:30 |
| palendae | Also Rocket League is like typing with oven mitts :p | 21:30 |
| mhayden | XBOX? more like http://i.imgur.com/TlqWzC8.jpg | 21:31 |
| * mhayden waits for redditors to come to my rescue | 21:31 | |
| mhayden | ah, i messed up pronouns -- i guess that makes it coffee time | 21:32 |
| openstackgerrit | Nolan Brubaker proposed openstack/openstack-ansible: Update user variables to Liberty names https://review.openstack.org/287438 | 21:33 |
| *** jguy_ has quit IRC | 21:34 | |
| *** unlaudable has quit IRC | 21:35 | |
| jwitko | hey, has anyone every received this error before during the pip install of keystone to the keystone container ? | 21:38 |
| jwitko | Collecting testtools<2.0.0,>=0.9.22 (from fixtures<1.3.0,>=0.3.14->oslo.concurrency<1.9.0,>=1.8.2->keystone) | 21:38 |
| jwitko | Could not find a version that satisfies the requirement testtools<2.0.0,>=0.9.22 (from fixtures<1.3.0,>=0.3.14->oslo.concurrency<1.9.0,>=1.8.2->keystone) (from versions: 2.0.0) | 21:38 |
| jwitko | No matching distribution found for testtools<2.0.0,>=0.9.22 (from fixtures<1.3.0,>=0.3.14->oslo.concurrency<1.9.0,>=1.8.2->keystone) | 21:39 |
| jwitko | Attachment | 21:39 |
| openstackgerrit | Jesse Pretorius (odyssey4me) proposed openstack/openstack-ansible: [WIP] Gate: Set pip index and extra-index to OpenStack-CI repositories https://review.openstack.org/265867 | 21:39 |
| *** dnt has quit IRC | 21:41 | |
| *** willlo has joined #openstack-ansible | 21:41 | |
| mhayden | palendae: i'm reviewing https://review.openstack.org/#/c/286753/11/scripts/upgrade-utilities/scripts/add_new_neutron_env.py -- what's the flag file? | 21:41 |
| jwitko | it looks like my repo is missing a package ? | 21:41 |
| jwitko | the repo was just built in the previous steps though. and i built it locally, not with mirrors | 21:41 |
| palendae | mhayden: In the same review, https://review.openstack.org/#/c/286753/11/doc/source/upgrade-guide/overview.rst | 21:41 |
| mhayden | palendae: ah, just to keep track of what's been moved from kilo > liberty so far? that makes sense | 21:42 |
| palendae | mhayden: And the creates line in https://review.openstack.org/#/c/286753/11/doc/source/upgrade-guide/overview.rst | 21:42 |
| palendae | Er, https://review.openstack.org/#/c/286753/11/scripts/upgrade-utilities/playbooks/deploy-config-changes.yml | 21:42 |
| palendae | mhayden: It's so that the calling Ansible task doesn't re-run | 21:42 |
| mhayden | got it | 21:42 |
| *** johnmilton has quit IRC | 21:42 | |
| palendae | The scripts themselves *should* be harmless | 21:43 |
| mhayden | the logic makes sense now -- thakns | 21:43 |
| palendae | If run 2+ times | 21:43 |
| openstackgerrit | Jesse Pretorius (odyssey4me) proposed openstack/openstack-ansible: [WIP] Gate: Use apt sources configured on the host https://review.openstack.org/279633 | 21:44 |
| odyssey4me | jwitko did you execute repo-build? which branch/tag? | 21:45 |
| jwitko | odyssey4me, yes I did. keystone. | 21:45 |
| jwitko | a lot of other packages make it in just fine | 21:45 |
| odyssey4me | jwitko which branch/tag are you building from? | 21:45 |
| odyssey4me | jwitko also, have you verified that the repo servers all have the same content? | 21:45 |
| jwitko | http://10.96.112.34:8181/os-releases/11.2.9/ | 21:45 |
| automagically | Later all | 21:45 |
| jwitko | odyssey4me, oh is it possible they're not sync'd yet ? | 21:46 |
| odyssey4me | jwitko it is | 21:46 |
| odyssey4me | it's also possible that wheel has built wheels that pip can't install | 21:47 |
| odyssey4me | jwitko https://github.com/openstack/openstack-ansible/commit/81f45653fa6bba4bc63fc44785b422b44f321a17 | 21:47 |
| jwitko | odyssey4me, so i see the package | 21:47 |
| odyssey4me | if you're building your own repo then that's a possible cause | 21:47 |
| jwitko | http://10.96.112.34:8181/os-releases/11.2.9/testtools-2.0.0-py2.py3-none-any.whl | 21:47 |
| jwitko | the requirements are for <2.0.0 as per the error message above | 21:47 |
| jwitko | and the package built into 11.2.9 repo seems to be 2.0.0 ? | 21:48 |
| *** unlaudable has joined #openstack-ansible | 21:48 | |
| Apsu | mhayden: +1 | 21:48 |
| odyssey4me | jwitko are you using your own mirror/fork of the git repositories? | 21:48 |
| jwitko | yes, i built my own | 21:49 |
| odyssey4me | jwitko do your git url's have '.git' at the end of them? | 21:49 |
| jwitko | sorry, i don't know what you mean by git urls ? | 21:49 |
| jwitko | where do I see htose? | 21:49 |
| jwitko | those*? | 21:49 |
| *** b3rnard0 is now known as b3rnard0_away | 21:50 | |
| odyssey4me | jwitko eg: https://github.com/openstack/openstack-ansible/blob/kilo/playbooks/defaults/repo_packages/openstack_services.yml#L33 | 21:50 |
| jwitko | ## Global Requirements | 21:51 |
| jwitko | requirements_git_repo: https://git.openstack.org/openstack/requirements | 21:51 |
| jwitko | requirements_git_install_branch: 0517298926fa413c3aa03d7e93d5a21bdb9d6ca9 # HEAD of "stable/kilo" as of 22.01.2016 | 21:51 |
| jwitko | requirements_git_dest: "/opt/requirements_{{ requirements_git_install_branch | replace('/', '_') }}" | 21:51 |
| odyssey4me | jwitko ok, so you haven't changed those vars to a local git repo | 21:51 |
| jwitko | and I need to? wouldn't the head of stable/kilo be fine ? | 21:51 |
| openstackgerrit | Nate Potter proposed openstack/openstack-ansible: Add haproxy-install to setup-infrastructure.yml https://review.openstack.org/287468 | 21:52 |
| odyssey4me | jwitko you should have a report that looks like this - can you pastebin it? http://rpc-repo.rackspace.com/reports/11.2.9.json | 21:52 |
| odyssey4me | jwitko you don't need to - that's just something that sometimes causes issue like you're experiencing - I'm just eliminating possible root causes | 21:53 |
| jwitko | odyssey4me, http://paste.openstack.org/raw/489058/ | 21:53 |
| admin0 | guys .. new setup, new install .. what could cause this ? http://pastebin.com/2vY83q0J - KeyError: 'children' | 21:58 |
| jwitko | children are subsets of host groups, check your keystone hosts in your user config? | 21:59 |
| *** thorst has quit IRC | 22:01 | |
| odyssey4me | jwitko ok, that looks fine - are you able to share your useR_variables file? | 22:02 |
| jwitko | sure, any specific part? | 22:02 |
| *** thorst has joined #openstack-ansible | 22:02 | |
| odyssey4me | jwitko anything that's not a default | 22:02 |
| openstackgerrit | Nate Potter proposed openstack/openstack-ansible: Clarify HAProxy configuration necessity https://review.openstack.org/287471 | 22:04 |
| jwitko | odyssey4me, here is the entire file minus comments: http://paste.openstack.org/raw/lFlpChevBV5FKLTmG9Bw/ | 22:04 |
| jwitko | nothing specific to repo | 22:04 |
| openstackgerrit | Bjoern Teipel proposed openstack/openstack-ansible: Adding release notes for erlang VM tuning parameters https://review.openstack.org/285506 | 22:05 |
| *** neilus has quit IRC | 22:06 | |
| *** thorst has quit IRC | 22:06 | |
| openstackgerrit | Jesse Pretorius (odyssey4me) proposed openstack/openstack-ansible: [WIP] Only include common overrides in user_variables.yml https://review.openstack.org/287474 | 22:08 |
| *** mathlin1 has quit IRC | 22:09 | |
| *** albertcard has quit IRC | 22:09 | |
| *** woodard has quit IRC | 22:09 | |
| jwitko | also, odyssey4me, i went into a repo container and checked the wheel version | 22:10 |
| jwitko | it is 0.26.0 | 22:10 |
| jwitko | which is the version that fix you linked me to would set it at | 22:11 |
| jwitko | so i think we're good on that end | 22:11 |
| odyssey4me | jwitko ok - I need to stop computering and can't really think much any more (it's 10:15pm)... hopefully someone else can help you get to the bottom of it | 22:11 |
| jwitko | oh sorry didn't realize your time different. thanks for your help, have a good night | 22:12 |
| palendae | I'm off for a while, so no changes to https://review.openstack.org/#/q/topic:liberty_upgrade if people are looking at them. I'll probably check in to poke the gates later if it's necessary, though | 22:15 |
| *** sdake_ has joined #openstack-ansible | 22:15 | |
| *** sdake has quit IRC | 22:18 | |
| *** Mudpuppy has quit IRC | 22:18 | |
| *** spotz is now known as spotz_zzz | 22:18 | |
| *** weshay has quit IRC | 22:19 | |
| admin0 | the deploy hsot need to be able to reach the management network because its using the management network for containers ssh also right ? | 22:19 |
| *** b3rnard0_away is now known as b3rnard0 | 22:20 | |
| jwitko | admin0, yes correct | 22:20 |
| admin0 | ok | 22:20 |
| jwitko | or the deploy host can be inside of the management network | 22:21 |
| *** thorst has joined #openstack-ansible | 22:23 | |
| *** thorst has quit IRC | 22:27 | |
| admin0 | how many times for different cloud installations ( rough figures ) is this ansible playbook used inside rackspace ? | 22:29 |
| admin0 | i need to sell this to mgmt tomorrow as why i am a big fan of this and recommend this as the next method | 22:29 |
| admin0 | 1000+ ? | 22:30 |
| *** elopez has quit IRC | 22:31 | |
| bsv | admin0: being able to reproduce your entire setup with a playbook, should be reason enough :) | 22:35 |
| bsv | disaster recovery +++ | 22:36 |
| admin0 | :D | 22:36 |
| bsv | compliance ++++ | 22:36 |
| bsv | documentation ++ | 22:36 |
| bsv | ;) | 22:36 |
| admin0 | it must have taken 1000s of tries to get this playook right | 22:36 |
| *** neilus has joined #openstack-ansible | 22:37 | |
| admin0 | how do I force-redo a container creation ? | 22:44 |
| bgmccollum | just a single container? | 22:44 |
| admin0 | something happened . on the start of setup-hosts, it says c11_xxx_container failed | 22:44 |
| admin0 | and its not even trying to create it | 22:45 |
| bgmccollum | you can `lxc-stop -n $CONTAINER_NAME` then `lxc-destroy -n $CONTAINER_NAME`...then rerun playbooks and it will get recreated | 22:45 |
| admin0 | yes, but that container is not there in the first place | 22:46 |
| admin0 | it says | 22:46 |
| admin0 | SSH Error: data could not be sent to the remote host. Make sure this host can be reached over ssh | 22:46 |
| bgmccollum | o_O | 22:46 |
| admin0 | so i think it assumes its there, | 22:46 |
| admin0 | and not tries to recreate | 22:46 |
| bgmccollum | which task? | 22:46 |
| admin0 | lxc-ls is blank | 22:46 |
| bsv | i wasted 3-4 hours today, running playbooks, mysql-container failed all the time... until we discovered that someone had not run the ha-proxy playbook >.< | 22:46 |
| admin0 | setup-hosts | 22:46 |
| bgmccollum | right, but which task inside that playbook? | 22:47 |
| admin0 | bgmccollum: http://pastebin.com/N4HVWbDJ | 22:48 |
| admin0 | its trying to gather facts of a non-existent container | 22:48 |
| bgmccollum | well, it might be related to the `--limit=c11_galera_container-f0a631d3` in your command... | 22:49 |
| admin0 | i did that iwthout it also .. same | 22:49 |
| admin0 | somehow its skipping assuming a container that is not even there in the first place | 22:49 |
| admin0 | maybe delete from inventory and it might try to create again | 22:49 |
| bgmccollum | admin0 how vanilla is this setup? | 22:50 |
| admin0 | brand new | 22:50 |
| bgmccollum | i mean, you're not doing any customer env.d stuff, right? | 22:50 |
| bgmccollum | custom* | 22:50 |
| admin0 | i will redo | 22:50 |
| *** yarkot_ has joined #openstack-ansible | 22:53 | |
| bgmccollum | admin0 did you try to move galera to `is_metal: true` or something? | 22:56 |
| admin0 | i removed the is_metal | 22:56 |
| bgmccollum | ah... | 22:56 |
| admin0 | figured out except neutro and haproxy, everything else is good in containers :D) | 22:57 |
| *** yarkot_ has quit IRC | 22:57 | |
| *** galstrom is now known as galstrom_zzz | 22:57 | |
| *** busterswt has quit IRC | 23:00 | |
| *** eil397 has quit IRC | 23:01 | |
| admin0 | everythign back on track again :D | 23:07 |
| admin0 | there was one ip mapping file | 23:07 |
| admin0 | that was causing this | 23:07 |
| *** izaakk has quit IRC | 23:07 | |
| admin0 | maybe from a previous run control C in the middle stuff | 23:07 |
| * admin0 notes to self: never do a control C in mid, change user_file and expect it to be flawless :D | 23:09 | |
| *** dnt has joined #openstack-ansible | 23:12 | |
| *** dnt has left #openstack-ansible | 23:12 | |
| *** retreved has quit IRC | 23:13 | |
| *** sigmavirus24 is now known as sigmavirus24_awa | 23:15 | |
| *** albertcard has joined #openstack-ansible | 23:17 | |
| jwitko | hey bgmccollum, can you help me with a repo issue? I built a local repo during a fresh kilo install with openstack-ansible repo release 11.2.9, and I'm seeing the following error when trying to do the pip install of keystone onto the keystone container. There are tons of other keystone related pip libraries that install before without any issue. Seems to be a very mismatch problem | 23:24 |
| jwitko | via a dependency? http://paste.openstack.org/raw/zTlWg2aZY32xSb1krJx7/ | 23:24 |
| jwitko | or cloudnull, if you're around | 23:24 |
| bgmccollum | jwitko, let me kick off a fresh kilo install and see what happens. are you using a tag, or the head of the kilo branch? | 23:27 |
| jwitko | bgmccollum, i'm using the 11.2.9 release tag | 23:28 |
| bgmccollum | thanks | 23:28 |
| *** KLevenstein has quit IRC | 23:30 | |
| jwitko | bgmccollum, earlier odyssey4me pointed out https://github.com/openstack/openstack-ansible/commit/81f45653fa6bba4bc63fc44785b422b44f321a17 | 23:31 |
| jwitko | but it doesn't look like I'm having that issue | 23:32 |
| jwitko | root@oss-ctrl02_keystone_container-259c49f7:~# pip list | grep wheel | 23:32 |
| jwitko | wheel (0.26.0) | 23:32 |
| bgmccollum | jwitko can you look at the version of wheel in your repo container? | 23:32 |
| jwitko | root@oss-ctrl02_repo_container-cba2bb4c:~# pip list | grep wheel | 23:33 |
| jwitko | wheel (0.26.0) | 23:33 |
| *** johnmilton has joined #openstack-ansible | 23:35 | |
| *** subscope has quit IRC | 23:37 | |
| *** admin0 has quit IRC | 23:38 | |
| bgmccollum | jwitko ill let you know what happens... | 23:38 |
| jwitko | thanks | 23:38 |
| *** jmckind has joined #openstack-ansible | 23:41 | |
| *** jmckind has quit IRC | 23:42 | |
| *** jguy_ has joined #openstack-ansible | 23:42 | |
| *** jmckind has joined #openstack-ansible | 23:42 | |
| jwitko | bgmccollum, it looks like the error lies in the fixtures package, the inheritance is "keystone --> oslo.concurrency --> fixtures | 23:50 |
| jwitko | hey bgmccollum, i'm looking at the repo for rpc-repo.rackspace.com, http://rpc-repo.rackspace.com/os-releases/11.2.9/ | 23:53 |
| jwitko | it has testtools=1.9.0 | 23:53 |
| jwitko | why is my repo different? | 23:53 |
| jwitko | did I just clone it at a bad time ? | 23:53 |
| bgmccollum | jwitko did you run the clone, or build playbook? | 23:53 |
| *** kencjohnston has quit IRC | 23:53 | |
| jwitko | build | 23:53 |
| *** jmckind is now known as jmckind_ | 23:54 | |
| bgmccollum | i think build might result it something different, as uncapped requirement might drift into breakage as you've discovered. might need to pin something new. if you do repo clone, you will get exactly whats in the RPC repo. | 23:55 |
| bgmccollum | i didnt know you were building your own repo instead of cloning, so my build is likely to work. | 23:55 |
| jwitko | bgmccollum, so do you have a suggested course of action ? | 23:56 |
| jwitko | can I just replace the rpm ? | 23:56 |
| jwitko | err, lol. .whl | 23:56 |
| bgmccollum | clone the RPC repo instead of building from scratch | 23:56 |
| bgmccollum | there are likely other nuanced differences in versions | 23:56 |
| jwitko | and so in repo-install.yml i should replace "repo-build.yml" with "repo-clone-mirror.yml" ? | 23:56 |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!
