| cloudnull | hughsaunders palendae commit https://review.openstack.org/#/c/163006/ and https://review.openstack.org/#/c/163194/ seem to be competing. | 00:07 |
|---|---|---|
| cloudnull | post refactor the install branch for tempest was the tag "3" | 00:08 |
| cloudnull | it looks like palendae is changing it to master. | 00:08 |
| cloudnull | and hughsaunders is changing it to a sha | 00:08 |
| openstackgerrit | Steve Lewis proposed stackforge/os-ansible-deployment: Test for fixe of Bug: 1425953 https://review.openstack.org/163238 | 00:17 |
| openstack | bug 1425953 in openstack-ansible "Unknown var external_vip_address referenced in os-horizon" [High,New] https://launchpad.net/bugs/1425953 - Assigned to Steve Lewis (steve-lewis) | 00:17 |
| *** sdake_ has joined #openstack-ansible | 00:19 | |
| openstackgerrit | Steve Lewis proposed stackforge/os-ansible-deployment: Remove unused external_vip_address reference https://review.openstack.org/163238 | 00:20 |
| *** BjoernT has quit IRC | 00:42 | |
| *** vmtrooper has joined #openstack-ansible | 01:05 | |
| *** vmtrooper has quit IRC | 01:11 | |
| *** devlaps has joined #openstack-ansible | 01:31 | |
| palendae | cloudnull: I need to go look to see where that SHA was - I think it should be master, but I haven't reviewed tempest's git log | 01:50 |
| *** devlaps has quit IRC | 01:51 | |
| cloudnull | yea palendae , +2'd your commit because it looks good however when I saw hughsaunders commit i -2 both so that we can nail down what version we want to use. | 01:53 |
| palendae | Yep yep, saw that | 01:54 |
| palendae | I don't have a strong opinion without looking that stuff up | 01:54 |
| cloudnull | yea, im sitting on my mobile or i'd go look things up too. | 01:56 |
| palendae | The dependency changes seem to apply to master, 3, and the SHA | 01:57 |
| cloudnull | so they changed deps on the stable tag ? | 01:58 |
| palendae | Seemed so; I might have hit other things | 02:00 |
| palendae | `git branch -a --contains 7c73dd5b18d29f0f3f3be0088514121dade53c8f` in tempest says master contains that commit | 02:01 |
| *** sdake_ has quit IRC | 02:04 | |
| *** sdake_ has joined #openstack-ansible | 02:05 | |
| *** sdake_ has quit IRC | 02:20 | |
| *** sdake_ has joined #openstack-ansible | 02:41 | |
| *** sdake_ has quit IRC | 02:45 | |
| *** vmtrooper has joined #openstack-ansible | 02:54 | |
| *** vmtrooper has quit IRC | 02:59 | |
| *** stevemar has quit IRC | 03:34 | |
| *** galstrom_zzz is now known as galstrom | 03:57 | |
| *** jmccrory has joined #openstack-ansible | 03:59 | |
| *** galstrom is now known as galstrom_zzz | 04:27 | |
| *** vmtrooper has joined #openstack-ansible | 04:43 | |
| *** vmtrooper has quit IRC | 04:48 | |
| *** jmccrory has quit IRC | 06:20 | |
| *** vmtrooper has joined #openstack-ansible | 06:32 | |
| *** vmtrooper has quit IRC | 06:37 | |
| *** prometheanfire has quit IRC | 08:04 | |
| *** mrodden has quit IRC | 08:18 | |
| *** vmtrooper has joined #openstack-ansible | 08:21 | |
| *** mrodden has joined #openstack-ansible | 08:24 | |
| *** vmtrooper has quit IRC | 08:26 | |
| *** vmtrooper has joined #openstack-ansible | 10:10 | |
| *** vmtrooper has quit IRC | 10:15 | |
| odyssey4me | so mattt hughsaunders cloudnull palendae andymccr there's a review to switch tempest to master | 10:39 |
| odyssey4me | however, I think that'll present us with too much of a moving target | 10:39 |
| odyssey4me | master currently uses the tempest tag 3 | 10:40 |
| odyssey4me | but that tag doesn't include the needful's from the sha that we're using in all the other gates | 10:40 |
| odyssey4me | we could try pinning to the same sha, or we could just pick the current sha and pin to that | 10:41 |
| odyssey4me | thoughts? | 10:41 |
| odyssey4me | the sha we're pinned to in the other branches is dated jan 20 | 10:42 |
| mattt | odyssey4me: 3 was tagged back in 2014, so what exactly changed that prevents that from installing? | 10:42 |
| mattt | (installing tempest now to see) | 10:44 |
| odyssey4me | bah, no info in the logs - http://logs.openstack.org/84/162984/1/check/os-ansible-deployment-dsvm-check-commit/ebac74e/console.html#_2015-03-10_13_28_40_592 - will have to deploy something myself to see | 10:45 |
| mattt | odyssey4me: i'm running tempest on a deploy from yesterday, gimme 5 | 10:46 |
| mattt | (although this play may take longer than 5) | 10:46 |
| odyssey4me | I would think that it'd be better for us to target the same sha for now, at least, and to include whatever we need to make that happen | 10:46 |
| odyssey4me | we can move to a later sha when we have this working | 10:47 |
| mattt | odyssey4me: https://gist.github.com/mattt416/22f8f6193fa6d10b07fd | 10:47 |
| odyssey4me | ah, so oslo.log-1.0.0 is what changed | 10:50 |
| odyssey4me | it was tagged 3 days ago | 10:50 |
| mattt | odyssey4me: i'm guessing we need to pin tempest-lib | 10:52 |
| mattt | that's what brought in the oslo.log change | 10:52 |
| odyssey4me | yup, feb 9 they switched to using oslo.log | 10:52 |
| odyssey4me | and tempest doesn't pin tempest-lib | 10:52 |
| odyssey4me | odd though, feb 9 is ages ago | 10:53 |
| mattt | odyssey4me: i'm guessing the tag 0.3 was just cut tho | 10:54 |
| odyssey4me | mattt nope, also feb 9 | 10:54 |
| odyssey4me | oh hang on - latest commit if feb 9 | 10:55 |
| odyssey4me | *is | 10:55 |
| odyssey4me | it may have been cut more recently | 10:55 |
| mattt | odyssey4me: i installed http://rpc-slushee.rackspace.com/python_packages/icehouse/tempest_lib-0.2.1-py2-none-any.whl, tempest still doesn't install | 10:57 |
| mattt | so pinning that will be useless i guess | 11:00 |
| odyssey4me | what's it error out on then? | 11:01 |
| mattt | same error actually | 11:01 |
| odyssey4me | did you try to remove oslo.log as well | 11:02 |
| mattt | it doesn't install oslo.log | 11:04 |
| mattt | odyssey4me: https://github.com/openstack/oslo.log/commit/21424056763cc72168d5e69a8a0b22eebcb52638 | 11:07 |
| mattt | (i think you pointed that out above) | 11:07 |
| mattt | odyssey4me: i'm a bit confused, so shouldn't the repo builder stuff take into account the changed requirement? | 11:08 |
| odyssey4me | sure, but it doesn't make sense - if it isn't using oslo.log then why is this a problem? | 11:08 |
| odyssey4me | yeah, I'm confused too | 11:08 |
| mattt | odyssey4me: https://github.com/openstack/tempest-lib/blob/master/requirements.txt#L11 | 11:08 |
| mattt | odyssey4me: so back to understanding why tempest-lib 0.2.1 doesn't install | 11:09 |
| odyssey4me | mattt but you tried downgrading to tempest-lib 0.2.1 and it was still failing, right? | 11:09 |
| mattt | (or rather why tempest still doesn't install w/ tempest-lib 0.2.1) | 11:09 |
| odyssey4me | https://gist.github.com/mattt416/22f8f6193fa6d10b07fd#file-gistfile1-txt-L37 | 11:10 |
| cloudnull | Morning. | 11:11 |
| mattt | howdy cloudnull | 11:11 |
| odyssey4me | cloudnull morning? you're up early | 11:11 |
| cloudnull | Flying home | 11:12 |
| cloudnull | Presently in iad | 11:12 |
| cloudnull | How's it going? | 11:12 |
| mattt | odyssey4me: https://gist.githubusercontent.com/mattt416/cfc71492c88daee9f08d/raw/06c6f1914b60f62a1fdf3e3869baef885d6cffdc/gistfile1.txt | 11:12 |
| odyssey4me | mattt cloudnull so I'm thinking of trying this out - just pinning tempest-lib for now to 0.2.1 in playbooks/vars/repo_packages/openstack_other.yml | 11:12 |
| mattt | that's w/ tempest-lib 0.2.1 installed | 11:12 |
| mattt | odyssey4me: but actually, i see the prob there | 11:13 |
| mattt | it's still trying to resolve tempest-lib from http://172.29.236.3:8181/os-releases/master/tempest_lib-0.3.0-py2-none-any.whl | 11:13 |
| odyssey4me | ah, so if we pin it down a level then all will be well with the world again | 11:13 |
| mattt | let me put the tempest_lib in the repo container | 11:13 |
| mattt | and see if that helps | 11:13 |
| mattt | odyssey4me: yeah i think so! | 11:13 |
| cloudnull | So tempest isn't lining things and the recent release doesn't work with anything? | 11:15 |
| odyssey4me | cloudnull the master branch of tempest seemed to work ok | 11:16 |
| odyssey4me | but the branch '3' and the sha we're using in other branches didn't work | 11:16 |
| cloudnull | Oh. Can we pin to a more modern sha? | 11:17 |
| odyssey4me | the concern here is that if we pin to master then we open up another can of worms and spend another few months trying to get tempest working across all the tests | 11:17 |
| odyssey4me | so I'm thinking we pin to the same, later-than-3 sha we're using in the other branches | 11:17 |
| odyssey4me | but we also pin the offending update | 11:17 |
| cloudnull | This sound sensible. | 11:17 |
| odyssey4me | once we have a fully working set of tempest tests again, we can try upping the tempest sha to something more recent | 11:18 |
| mattt | ok i think we have further issues beyond tempest-lib | 11:18 |
| odyssey4me | mattt :( | 11:18 |
| cloudnull | I have access to a computer and the interwebs for an hour or so. If I can help let me know. | 11:20 |
| openstackgerrit | Jesse Pretorius proposed stackforge/os-ansible-deployment: Restore tempest install branch to pre-refactor SHA https://review.openstack.org/163006 | 11:21 |
| odyssey4me | cloudnull did I do the package pin for the wheel repo right? | 11:21 |
| cloudnull | But it sounds like you guys are hot on the trail. So I might just stay out of the way ;p | 11:22 |
| odyssey4me | the actual name of the lib is 'tempest-lib', so the git_dest might or might not be right | 11:22 |
| odyssey4me | bah, broken - let me fix the naming | 11:22 |
| mattt | cloudnull: if i manually downgrade a package on my repo container, do i need to do anything for that to become visible? | 11:22 |
| mattt | it's still trying to pull down the old package for whatever reason | 11:23 |
| openstackgerrit | Jesse Pretorius proposed stackforge/os-ansible-deployment: Restore tempest install branch to pre-refactor SHA https://review.openstack.org/163006 | 11:23 |
| odyssey4me | mattt I think there's an index of some sort? a json file on the repo server? | 11:23 |
| mattt | odyssey4me: ah that could be it | 11:23 |
| cloudnull | If you could add an inline comment stating why the sha it'll be nice for future supportability. | 11:23 |
| cloudnull | Is comments in the top of the file. | 11:24 |
| cloudnull | Matt it'll replicate out if you did it on the first container. | 11:24 |
| cloudnull | Run the repo-build play with the index tag to regent the indexes. | 11:25 |
| mattt | odyssey4me: i think pinning to 0.2.1 will work | 11:26 |
| mattt | it's just my repo servers are in a bad state now :P | 11:26 |
| cloudnull | Mattt: openstack-ansible repo-build.yml --tags repo-html-indexes | 11:27 |
| openstackgerrit | Jesse Pretorius proposed stackforge/os-ansible-deployment: Restore tempest install branch to pre-refactor SHA https://review.openstack.org/163006 | 11:27 |
| mattt | odyssey4me: i think we should stay on the tag | 11:27 |
| mattt | it's much cleaner than that random sha | 11:27 |
| mattt | odyssey4me: if we pin tempest-lib then we should be ok | 11:28 |
| mattt | cloudnull: trying now | 11:28 |
| odyssey4me | mattt but all other branches are on that sha | 11:28 |
| mattt | cloudnull: trying now! | 11:28 |
| mattt | cloudnull: i think the idea was to have master point to a bit more current tempest, without being on master itself | 11:30 |
| mattt | ^ odyssey4me instead | 11:30 |
| odyssey4me | mattt that sha is later that the tag 3 | 11:30 |
| odyssey4me | so my thinking is we pin to the same sha as the other branches for now, so that we can deal with the issues across all branches with the same method | 11:30 |
| odyssey4me | once we have all the desired testing working at that sha, we can pin to a later sha in master - we may have to do this for kilo anyway | 11:31 |
| mattt | ok fair enough | 11:32 |
| mattt | cloudnull: yep that worked thanks | 11:32 |
| mattt | and now tempest installs with a downgraded tempest_lib 0.2.1 | 11:32 |
| cloudnull | In kilo for defcore were likely going to need an even layer release of tempest. | 11:32 |
| cloudnull | But I agree functional should be the target for sure. | 11:33 |
| odyssey4me | yeah, so for kilo we'll need to pin to a later sha and possibly remove some other pins... but we cross that bridge when we get there | 11:34 |
| mattt | yeah | 11:34 |
| cloudnull | +1 | 11:34 |
| odyssey4me | aha, here's where the sha was set: https://github.com/stackforge/os-ansible-deployment/commit/278ec800c4ad14534ed4406c0cb6900a6a6f779b | 11:35 |
| cloudnull | I made a few issues against the kilo milestone for that very thing already. | 11:35 |
| odyssey4me | cloudnull yeah, we noticed int he triage yesterday | 11:36 |
| odyssey4me | all the kilo bugs have been marked as medium as a placeholder | 11:36 |
| cloudnull | Ah nice. Sorry I could be present | 11:36 |
| odyssey4me | we'll have to properly go through them when we start to focus on Kilo | 11:36 |
| cloudnull | Couldn't | 11:36 |
| odyssey4me | heh, yeah - no worries - it looks like the operators mid-cycle meet was good | 11:37 |
| cloudnull | Yup yup. It was. | 11:37 |
| openstackgerrit | Jesse Pretorius proposed stackforge/os-ansible-deployment: Revert "Trim ansible's output to console.log" https://review.openstack.org/162984 | 11:42 |
| odyssey4me | hmm, it would seem that perhaps nodepool is broken again | 11:48 |
| cloudnull | Ha. Just can't catch a break :p | 11:51 |
| odyssey4me | no worries - I'll test it manually at least to get some confirmation that it's working | 11:52 |
| * cloudnull boarding plane see you all soonish. | 11:53 | |
| andymccr | ok so | 11:54 |
| andymccr | i think we have galera depency issues also :/ | 11:54 |
| andymccr | dependency | 11:54 |
| mattt | hahaha | 11:54 |
| mattt | cloudnull: have a good flight | 11:54 |
| andymccr | like the gate jobs are failing before tempest even gets run | 11:54 |
| andymccr | The following packages have unmet dependencies: | 11:54 |
| andymccr | mariadb-galera-server-5.5 : Depends: galera-3 (>= 25.3) but it is not going to be installed | 11:54 |
| andymccr | i thought maybe id screwed something up in my swift only deploy | 11:55 |
| andymccr | but turns out its not just me | 11:55 |
| andymccr | so.... | 11:55 |
| odyssey4me | hmm, master doesn't have a pinned galera any more :( | 11:55 |
| andymccr | if i dont respond for the next while its totally because i threw my laptop against a wall | 11:55 |
| odyssey4me | but the repo it's using should have everything it needs | 11:55 |
| mattt | andymccr: did you say something, i was afk banging my head against the wall | 11:55 |
| mattt | :P | 11:55 |
| andymccr | srsly... k | 11:56 |
| andymccr | update my daily thing - blocked on all the dependency things | 11:56 |
| andymccr | holidaying. | 11:56 |
| odyssey4me | odd, http://mirror.rackspace.com/mariadb/repo/5.5/ubuntu/pool/main/g/galera-3/ | 11:59 |
| *** vmtrooper has joined #openstack-ansible | 11:59 | |
| odyssey4me | the package is there and if it depends on it, surely it should fine it? | 11:59 |
| odyssey4me | *find | 11:59 |
| odyssey4me | I have an AIO building from scratch right now - will let you know if it bombs. | 12:00 |
| mattt | also have a build going through right now | 12:00 |
| mattt | yep it bombed | 12:01 |
| odyssey4me | mattt same dep missed? | 12:01 |
| mattt | odyssey4me yeah | 12:02 |
| mattt | odyssey4me: http://134.213.53.6:8080/job/dev-heat-ansible/28/console | 12:02 |
| odyssey4me | E: Unable to correct problems, you have held broken packages. | 12:04 |
| *** vmtrooper has quit IRC | 12:04 | |
| odyssey4me | I have seen this before when the apt cache wasn't updated after the repo was added - essentially it ends up with packages from the trusty repo instead of the mariadb repo, then at a point it can't install the mariadb packages because the right bits aren't there. | 12:05 |
| mattt | odyssey4me: that may be it, becuase if i manually install mariadb-galera-server-5.5 it goes fine | 12:06 |
| odyssey4me | mattt is that after an 'apt-get update', or did you just 'apt-get install' and it worked? | 12:06 |
| mattt | odyssey4me: i actually just did an apt-get install, no update | 12:07 |
| cloudnull | Just checked back are the rax mirrors jacked up? Should we move master back to upstream marria repos instead of using our supposed pinned ones? | 12:11 |
| odyssey4me | odd mattt I have exactly the same thing - apt-get install mariadb-galera-server-5.5 just works | 12:11 |
| andymccr | ok well i'll ask for a tl;dr on how to fix that after i get back from having some lunch :D | 12:11 |
| odyssey4me | but for ansible it does not | 12:12 |
| andymccr | team team team. | 12:12 |
| odyssey4me | cloudnull in master we have no pinning at all right now for mariadb | 12:12 |
| odyssey4me | that's part of the whole discussion around whether pinning belongs inside the project or not | 12:12 |
| odyssey4me | this problem seems to be a little different | 12:13 |
| cloudnull | But the repo is the rax repo which is supposed to be a maria mirror | 12:13 |
| odyssey4me | cloudnull it is, and it works | 12:13 |
| odyssey4me | just for some reason it's not working in the ansible play | 12:13 |
| cloudnull | Ah. | 12:13 |
| cloudnull | If you rerun the play does it work ? | 12:14 |
| odyssey4me | trying that now | 12:14 |
| *** markvoelker has joined #openstack-ansible | 12:14 | |
| cloudnull | Or only by hand does it succeed? | 12:14 |
| cloudnull | Ok | 12:14 |
| odyssey4me | I checked the apt-cache policy first, and the right repo was there and apt knew about it | 12:15 |
| odyssey4me | nope, not working - let me try a few things here | 12:16 |
| *** britthouser has quit IRC | 12:18 | |
| cloudnull | Welp that sux | 12:19 |
| odyssey4me | ok, easy fix and root cause is amusing | 12:19 |
| odyssey4me | the package galera is now galera-3 | 12:19 |
| cloudnull | And the dep is not getting the correct package name ? | 12:20 |
| odyssey4me | 'galera' is coming from somewhere else | 12:20 |
| cloudnull | Or were calling it by the transition package ? | 12:20 |
| odyssey4me | and that's breaking it somehow | 12:20 |
| odyssey4me | waiting for my play to finish to look | 12:21 |
| mattt | we must pass in galera | 12:21 |
| odyssey4me | galera only has the debian builds: http://mirror.stshosting.co.uk/mariadb/repo/5.5/ubuntu/pool/main/g/galera/ | 12:21 |
| mattt | we specify 'galera' in teh galera_apt_packages list | 12:21 |
| odyssey4me | it seems that the packages got renamed | 12:21 |
| mattt | so we need to update that | 12:21 |
| odyssey4me | yeah mattt so that's what I updated to 'galera-3' to make it wor | 12:22 |
| odyssey4me | *work | 12:22 |
| cloudnull | Well this is all good. Its better to know this now and to know it in master ;) | 12:22 |
| mattt | yeah but we should go back to pinning in master :x | 12:22 |
| odyssey4me | so here's the question - how do we get two upstream breaks past the gate to merge the fixes? | 12:22 |
| cloudnull | Not in some stable release in a customer deployment. | 12:22 |
| mattt | odyssey4me: haha | 12:23 |
| cloudnull | Mattt I disagree on pinning apt packages in master. | 12:23 |
| mattt | cloudnull: really | 12:23 |
| mattt | but then we're chasing this sort of stuff all day long | 12:23 |
| mattt | odyssey4me: i imagine we have to double up a commit to fix both :( dirty but what other options? | 12:23 |
| odyssey4me | perhaps I could do a patch to fix galera, but also disable tempest temporarily - then a subsequent patch to re-enable tempest and apply the fixes there? | 12:23 |
| cloudnull | It should be relatively unstable and our pins will likely not be someone else's pins in productions. | 12:23 |
| odyssey4me | at least then the patches are still focused on one issue at a time | 12:24 |
| cloudnull | Rax should pin , others that use this system should pin. But to what they pin should be up to them. | 12:24 |
| mattt | odyssey4me: seems long-winded, i'd rather just fix both and not disable gating | 12:25 |
| cloudnull | +1 single patch commit message "fixes gating..." And be done with it. | 12:26 |
| odyssey4me | ok, will sort that out now then | 12:26 |
| andymccr | unfortunately, i think we should do that. in general though probably not the best, but srsly what are the odds of this shit happening at the same time | 12:26 |
| odyssey4me | I guess juno and icehouse aren't affected, as they're still pinned | 12:26 |
| andymccr | is tempest pinned odyssey4me? | 12:26 |
| andymccr | like the packages for tempest. | 12:26 |
| andymccr | in juno/icehouse | 12:26 |
| andymccr | galera should be ok | 12:27 |
| mattt | andymccr: yes but not tempest_lib | 12:27 |
| mattt | that change will need to get backported | 12:27 |
| odyssey4me | ^ what mattt said | 12:27 |
| andymccr | cool yeh thought that was the case | 12:27 |
| odyssey4me | but we can do a modified cherry-pick I guess | 12:27 |
| odyssey4me | or a fresh patch for juno and cherry pick that back to icehouse | 12:27 |
| mattt | afk for a bit | 12:30 |
| cloudnull | I'd say fresh patch to Juno cherry pick to icehouse. At this point were only Pretending that we can cherrypick from master to Juno. | 12:30 |
| cloudnull | OK so it seems that the plane has sat on the tarmac for the requisite amount of time and the captain has been sufficiently convinced to fly the thing. So now I'm going to go. Afk for a few hours. | 12:31 |
| openstackgerrit | Jesse Pretorius proposed stackforge/os-ansible-deployment: Fix gate: tempest and galera updates https://review.openstack.org/163006 | 12:35 |
| odyssey4me | while I wait for that build - time for lunch... afk for a bit | 12:38 |
| *** KLevenstein has joined #openstack-ansible | 12:39 | |
| *** galstrom_zzz is now known as galstrom | 12:43 | |
| *** KLevenstein has quit IRC | 12:43 | |
| *** KLevenstein has joined #openstack-ansible | 12:47 | |
| *** galstrom is now known as galstrom_zzz | 13:11 | |
| mattt | odyssey4me: will cloudnull need to remove his -1 before we can merge https://review.openstack.org/163006? | 13:13 |
| mattt | maybe he'll have arrived before gate jobs have run :P | 13:13 |
| *** sdake_ has joined #openstack-ansible | 13:15 | |
| sdake_ | morning | 13:16 |
| odyssey4me | mattt yeah, he'll have to remove the -2 before it can merge | 13:20 |
| mattt | sdake_: good morning to you | 13:20 |
| *** vmtrooper has joined #openstack-ansible | 13:48 | |
| *** vmtrooper has quit IRC | 13:53 | |
| *** Mudpuppy has joined #openstack-ansible | 14:19 | |
| palendae | odyssey4me: Just back. Your patch duplicates https://review.openstack.org/#/c/163194/ | 14:21 |
| palendae | Or not | 14:21 |
| palendae | You pin lib, I pin oslo.context | 14:22 |
| *** Mudpuppy has quit IRC | 14:22 | |
| palendae | Heh, and I emailed hugh and andy hoping this wouldn't get duplicated again today | 14:22 |
| *** Mudpuppy has joined #openstack-ansible | 14:23 | |
| odyssey4me | palendae yeah, so mattt and I have been looking into it today.. | 14:24 |
| palendae | I noticed | 14:24 |
| odyssey4me | you'll notice that there was also a galera issue, which we've resolved | 14:24 |
| palendae | I didn't see that manifesting yesterday, but yeah | 14:25 |
| odyssey4me | also, I've aimed to pin tempest at the same sha as we do in the other branches - that way we are able to get tempest to the right level of completeness in all branches before we go adventurating to newer sha's for tempest. | 14:25 |
| palendae | So did you see the patch and it was insufficient, or? | 14:25 |
| mattt | palendae: yeah we saw both | 14:25 |
| palendae | Ok, just making sure | 14:26 |
| odyssey4me | your method is good if we're choosing to move to using the moving target of master and changing the other dependancies with it | 14:26 |
| palendae | odyssey4me: Not the tempest pin | 14:26 |
| palendae | Everything else | 14:27 |
| odyssey4me | personally I don't think we should be doing that just yet | 14:27 |
| palendae | Ignoring the tempest sha | 14:27 |
| palendae | The requirements fixes | 14:27 |
| mattt | palendae: we figured it didn't make sense to pin an oslo library because it was getting pulled in by a non-pinned tempest_lib | 14:27 |
| palendae | Ok | 14:27 |
| *** sigmavirus24_awa is now known as sigmavirus24 | 14:27 | |
| palendae | So we have competing patches | 14:27 |
| odyssey4me | yeah, everything else was primarily there because of the tempest switch to master, or because of tempest-lib having no cap in the requirements | 14:28 |
| palendae | What I was trhying to avoid was one side of the ocean knowing something and the other having to wait | 14:28 |
| palendae | Which is where I was stuck yesterday afternoon | 14:28 |
| palendae | I guess I'll toss those few hours of work then | 14:28 |
| odyssey4me | well, I was kinda thinking it'd be good to get to a point of having two approaches to discuss when y'all came online :) | 14:29 |
| palendae | I guess that's fair | 14:29 |
| odyssey4me | two working approaches :) | 14:29 |
| palendae | It is reasonable to pin tempest-lib | 14:30 |
| palendae | I was seeing requirements failures on both 'the SHA' and master | 14:30 |
| palendae | I get what you're saying about master being a moving target; either's better than '3' IMO | 14:30 |
| palendae | Since '3' doesn't have the neutron tests that we need | 14:30 |
| mattt | palendae: i think not addressing tempest_lib doesn't solve the problem | 14:30 |
| palendae | Ok | 14:31 |
| palendae | Would you mind commenting on the review so it's documented? | 14:31 |
| mattt | palendae: rather, i think it may not solve the problem long-term | 14:32 |
| palendae | Same request :) | 14:32 |
| mattt | because it's oslo.context <- oslo.log <- tempest_lib | 14:32 |
| mattt | so without controlling tempest_lib you don't control the other two | 14:33 |
| palendae | Ok | 14:33 |
| palendae | Would you mind putting that on my review> | 14:33 |
| mattt | palendae: sure | 14:33 |
| odyssey4me | good call :) | 14:34 |
| odyssey4me | palendae agreed on not using 3, and in fact for this project I would recommend that we always target something specific at least on a per sprint basis | 14:34 |
| palendae | That's reasonable | 14:34 |
| mattt | i was less concerned about using 3 :P | 14:35 |
| palendae | Better to have a known quantity | 14:35 |
| palendae | mattt: You weren't writing neutron tests then :) | 14:35 |
| palendae | Our master and juno weren't on parity for tests | 14:35 |
| odyssey4me | so I would certainly advocate targeting a newer sha if we need to, but for now we don't know whether we need to | 14:35 |
| palendae | Newer than the 7.... ? | 14:35 |
| odyssey4me | palendae yep | 14:37 |
| palendae | Gotcha | 14:37 |
| palendae | I'm for 'the SHA' | 14:37 |
| odyssey4me | if you say you do need newer than that for neutron to work, then I'm down with that - for all branches though, which will be fun (not!) | 14:37 |
| palendae | Though left a comment about the SHA being in the role | 14:37 |
| palendae | No, the 7 one works for juno at least | 14:37 |
| palendae | I'm less concerned about icehouse right now | 14:38 |
| palendae | Agreed that maintaining pins for all branches will be a pain, but it's also kind of the nature of the beast | 14:38 |
| odyssey4me | ok palendae mattt let me ensure I understand you correctly - you're suggesting that in the role we default to something like master, but that in the user_config we override that for the gate? | 14:39 |
| palendae | odyssey4me: It's being pinned here https://review.openstack.org/#/c/163006/5/playbooks/vars/repo_packages/openstack_services.yml | 14:40 |
| mattt | it just looks cleaner not having the sha as a default value | 14:40 |
| mattt | and as palendae says we are already overriding it | 14:40 |
| mattt | so from a gate perspective nothing needs changing | 14:41 |
| odyssey4me | ah fair enough - the wheel repo only provides one version to install anyway | 14:41 |
| palendae | Yeah | 14:41 |
| odyssey4me | the stuff in the defaults is only used if the frozen repo isn't used | 14:41 |
| odyssey4me | (I guess) | 14:41 |
| palendae | I actually don't know if they've done a tag newer than 3 | 14:42 |
| odyssey4me | palendae they haven't | 14:42 |
| palendae | =\ | 14:42 |
| openstackgerrit | Jesse Pretorius proposed stackforge/os-ansible-deployment: Fix gate: tempest and galera updates https://review.openstack.org/163006 | 14:48 |
| odyssey4me | I've updated the commit message to try to explain everything more clearly. | 14:48 |
| odyssey4me | and also removed the role default pin target change, so it stays at master | 14:49 |
| mattt | sweet | 14:51 |
| palendae | odyssey4me: I +1'd the review. You guys changed my mind, I think your patch is the better approach :) | 14:58 |
| *** stevemar has joined #openstack-ansible | 15:02 | |
| odyssey4me | palendae :) | 15:02 |
| odyssey4me | mattt and hughsaunders have wasted many hours on tempest | 15:03 |
| odyssey4me | don't fall into the trap :p | 15:03 |
| odyssey4me | but it'd be nice to see networking working ;) | 15:03 |
| palendae | It would | 15:04 |
| palendae | I also got bitten because '-s' is not '--serial' | 15:04 |
| palendae | So I think a bunch of failures I was seeing were invalid | 15:04 |
| openstackgerrit | Jesse Pretorius proposed stackforge/os-ansible-deployment: Revert "Trim ansible's output to console.log" https://review.openstack.org/162984 | 15:04 |
| *** galstrom_zzz is now known as galstrom | 15:25 | |
| *** vmtrooper has joined #openstack-ansible | 15:36 | |
| Apsu | palendae: For this I blame you :) | 15:38 |
| palendae | yeah, that was my fault | 15:39 |
| palendae | I will take the blame for that one | 15:39 |
| Apsu | hehe | 15:39 |
| *** vmtrooper has quit IRC | 15:42 | |
| *** jwagner_away is now known as jwagner | 15:42 | |
| *** britthouser has joined #openstack-ansible | 15:48 | |
| openstackgerrit | Evan Callicoat proposed stackforge/os-ansible-deployment: Enable offline compression for resource files https://review.openstack.org/163183 | 15:56 |
| mattt | Apsu: the only blame we do in here is a git-blame | 16:01 |
| Apsu | mattt: Uh oh. Time to cover my digital tracks | 16:02 |
| palendae | git config --global user.name "Not It" | 16:03 |
| Apsu | ^ | 16:03 |
| cloudnull | Just landed and removed my -2 | 16:04 |
| cloudnull | Hopefully gate completes sometime today ;) | 16:04 |
| mattt | cloudnull: :) | 16:05 |
| palendae | cloudnull: Did you give a +2? | 16:05 |
| palendae | Or just a 0? :) | 16:05 |
| cloudnull | Nope | 16:06 |
| cloudnull | +1 | 16:06 |
| odyssey4me | +1 until gate succeeds | 16:06 |
| cloudnull | Waiting on gate | 16:06 |
| palendae | Ok | 16:06 |
| cloudnull | Are we abandoning https://review.openstack.org/#/c/163194/ | 16:12 |
| palendae | cloudnull: yes | 16:12 |
| cloudnull | OK. So I'll leave my -2 as is. | 16:13 |
| palendae | cloudnull: Writing the abandon message now | 16:13 |
| cloudnull | Kk | 16:13 |
| *** britthouser has quit IRC | 16:13 | |
| *** erikmwilson has quit IRC | 16:13 | |
| *** d34dh0r53 has quit IRC | 16:13 | |
| *** sdake__ has joined #openstack-ansible | 16:13 | |
| *** matt__ has joined #openstack-ansible | 16:14 | |
| *** darrenc_ has joined #openstack-ansible | 16:14 | |
| *** bgmccollum has quit IRC | 16:14 | |
| *** lbragstad has quit IRC | 16:14 | |
| *** jmeridth has quit IRC | 16:14 | |
| *** b3rnard0 has quit IRC | 16:14 | |
| *** odyssey4me has quit IRC | 16:14 | |
| *** jroll has quit IRC | 16:14 | |
| *** darrenc has quit IRC | 16:14 | |
| *** mattoliverau has quit IRC | 16:14 | |
| *** dolphm has quit IRC | 16:14 | |
| *** erikmwilson has joined #openstack-ansible | 16:15 | |
| *** b3rnard0 has joined #openstack-ansible | 16:15 | |
| *** d34dh0r53 has joined #openstack-ansible | 16:15 | |
| *** bgmccollum has joined #openstack-ansible | 16:15 | |
| *** jroll has joined #openstack-ansible | 16:15 | |
| *** dolphm has joined #openstack-ansible | 16:15 | |
| *** lbragstad has joined #openstack-ansible | 16:15 | |
| *** jmeridth has joined #openstack-ansible | 16:15 | |
| *** britthouser has joined #openstack-ansible | 16:16 | |
| *** odyssey4me has joined #openstack-ansible | 16:16 | |
| *** jroll has quit IRC | 16:16 | |
| *** jroll has joined #openstack-ansible | 16:16 | |
| *** sdake_ has quit IRC | 16:17 | |
| *** jmccrory has joined #openstack-ansible | 16:19 | |
| *** jwagner is now known as jwagner_lunch | 16:21 | |
| *** jwagner_lunch is now known as jwagner_away | 16:22 | |
| daneyon | If I am running my network_host on the same server as my compute_host, can I still follow the instructions for adding a compute host: http://docs.rackspace.com/rpc/api/v10/bk-rpc-installation/content/sec-operations-add-compute-host.html | 16:24 |
| daneyon | Just add the neutron-all playbook run? | 16:24 |
| Apsu | daneyon: Should be able to mix and match host 'roles' at will. | 16:27 |
| Apsu | That's why AIOs work. There's some minor caveats in a full AIO (which is why we have scripts for ours, such as for gating), but for your purpose probably will Just Work⢠| 16:28 |
| Apsu | Let us know if not | 16:28 |
| *** galstrom is now known as galstrom_zzz | 16:32 | |
| *** sdake_ has joined #openstack-ansible | 16:33 | |
| *** sdake__ has quit IRC | 16:37 | |
| daneyon | Apsu: thanks! | 16:38 |
| Apsu | Sure thing! | 16:38 |
| *** britthouser has quit IRC | 16:39 | |
| odyssey4me | we have a winnner: https://review.openstack.org/163006 :) | 16:40 |
| Apsu | yey | 16:41 |
| *** openstack has joined #openstack-ansible | 16:56 | |
| odyssey4me | oh dear, several requirements updates going on in openstack - expect more breakage :/ | 17:03 |
| *** jwagner_away is now known as jwagner | 17:14 | |
| *** vmtrooper has joined #openstack-ansible | 17:25 | |
| *** vmtrooper has quit IRC | 17:31 | |
| openstackgerrit | Andy McCrae proposed stackforge/os-ansible-deployment: Dont look for used_ips on containerless hosts https://review.openstack.org/163083 | 17:38 |
| *** nosleep77 has joined #openstack-ansible | 17:38 | |
| *** devlaps has joined #openstack-ansible | 17:42 | |
| *** jmccrory has quit IRC | 17:42 | |
| openstackgerrit | Nolan Brubaker proposed stackforge/os-ansible-deployment: Enable network basic ops tests https://review.openstack.org/163544 | 17:48 |
| *** stevemar has quit IRC | 18:04 | |
| *** stevemar has joined #openstack-ansible | 18:05 | |
| *** jmccrory has joined #openstack-ansible | 18:08 | |
| *** galstrom_zzz is now known as galstrom | 18:27 | |
| *** jaypipes is now known as jaypipes-afk | 18:34 | |
| *** galstrom is now known as galstrom_zzz | 18:36 | |
| *** jmccrory has quit IRC | 18:39 | |
| *** jmccrory has joined #openstack-ansible | 18:39 | |
| jwagner | i stopped all the services on the cimputes | 18:40 |
| jwagner | oops wrong channel | 18:41 |
| palendae | reviews plz https://review.openstack.org/#/c/163544/ | 18:45 |
| *** galstrom_zzz is now known as galstrom | 18:50 | |
| *** jmccrory has quit IRC | 19:02 | |
| *** sdake__ has joined #openstack-ansible | 19:08 | |
| *** sdake_ has quit IRC | 19:12 | |
| *** vmtrooper has joined #openstack-ansible | 19:14 | |
| *** vmtrooper has quit IRC | 19:19 | |
| *** galstrom is now known as galstrom_zzz | 19:20 | |
| *** jmccrory has joined #openstack-ansible | 19:31 | |
| palendae | Is there a place I can check my review's place in the zuul line? | 19:33 |
| stevelle | maybe at http://status.openstack.org/zuul/ | 19:38 |
| palendae | stevelle: Thanks! | 19:38 |
| *** galstrom_zzz is now known as galstrom | 19:41 | |
| *** BjoernT has joined #openstack-ansible | 19:52 | |
| BjoernT | quick questions, if I want to rewrite the openrc in all containers with ansible, what would be the best way ? a custom playbook with just few common roles? | 19:53 |
| stevelle | sounds good, targeting should be easy enough if you just want all containers | 19:56 |
| *** galstrom is now known as galstrom_zzz | 19:56 | |
| bgmccollum | BjoernT: is this a one off? | 20:00 |
| jwagner | odyssey4me are u around? | 20:05 |
| jwagner | ls | 20:05 |
| openstackgerrit | Merged stackforge/os-ansible-deployment: Fix gate: tempest and galera updates https://review.openstack.org/163006 | 20:10 |
| *** galstrom_zzz is now known as galstrom | 20:13 | |
| palendae | https://jenkins07.openstack.org/job/os-ansible-deployment-dsvm-check-commit/72/ !!! neutron tests passing on the gate | 20:36 |
| palendae | Hasn't updated gerrit yet though | 20:37 |
| *** sdake__ has quit IRC | 20:51 | |
| *** sdake_ has joined #openstack-ansible | 20:53 | |
| daneyon | I deployed v10. The initial deployment was 3 controller nodes clustered. These nodes were not running the neutron api. I added 2 combined network/compute hosts using the add-host, nova-all, neutron-all playbooks. The plays all ran successfully, but the nova-api containers never were updated to communicate with the neutron-api containers. SO, I ran the openstack-setup play again, but still the nova-api containers were n | 21:02 |
| daneyon | ever updated with the neutron container info. Am I missing something? | 21:02 |
| *** vmtrooper has joined #openstack-ansible | 21:03 | |
| *** vmtrooper has quit IRC | 21:08 | |
| daneyon | cloudnull ^ | 21:08 |
| palendae | daneyon: They were adding to the user_config.yml file? | 21:11 |
| palendae | added* | 21:11 |
| daneyon | palendae: yes: network_hosts: compute-server71: ip: 192.168.220.71 The compute hosts have the neutron containers running. That seems to work. But updtaing the existing 3 control nodes (separate hardware) seems to be missing. The nova api containers running on my 3 controllers do not know how to speak with the neutron api (running on compute 71 and 73) because nova.conf was never updated with the req info and the necess | 21:14 |
| daneyon | ary services restarted. | 21:14 |
| daneyon | palendae: So, I was wondering of I missed a step, but I have successfully ran all the plays- add-host, nova-all, neutron-all. The I reran openstack-setup and still don;t see the nova-api containers updated with the info needed to talk to the neutron api container | 21:16 |
| andymccr | daneyon: are you using the haproxy loadbalancer? | 21:17 |
| daneyon | andymccr: how funny, I just thought about that. I think i need to update the haproxy node. let me try that | 21:18 |
| andymccr | let me know if thats not it | 21:19 |
| daneyon | andymccr: that was it. I keep forgetting to update my haproxy host... urgh! | 21:20 |
| andymccr | i do that pretty much all the time. i think its probably because we never really made the haproxy as a "permanent" solution but more of a "do this so we can see it work and dev on it" | 21:20 |
| andymccr | but it should probably be fixed/updated to be a bit better | 21:20 |
| daneyon | andymccr: If I wanted to run the neutron api on my controller cluster and run the agents on compute nodes, do the playbooks support this? | 21:22 |
| daneyon | andymccr: I can del with it for now. If all goes well with my eval, I should be able to make the load-balancer a 1st class citizen in the project. | 21:23 |
| palendae | yeah, haproxy isn't really supported. It's there and kinda works | 21:23 |
| daneyon | deal | 21:23 |
| palendae | I think it's mostly for AIOs really | 21:23 |
| daneyon | I like some of the new service discovery stuff that works well with haproxy and I need to have a good open source server load-balancing option. I can deal with that stuff after getting through the basics. | 21:25 |
| *** matt__ is now known as mattoliverau | 21:25 | |
| daneyon | I come from a Puppet background, so I'm trying to bring muyself up to speed with Ansible and understand all the awesome work y'all have accomplished. | 21:25 |
| *** darrenc_ is now known as darrenc | 21:26 | |
| *** galstrom is now known as galstrom_zzz | 21:30 | |
| *** jmccrory has quit IRC | 21:31 | |
| *** jmccrory1 has joined #openstack-ansible | 21:31 | |
| andymccr | daneyon: re: the neutron question, i don't think its possible from what i can see - cloudnull will probably correct me on that :) | 21:34 |
| *** sc has quit IRC | 21:40 | |
| *** Mudpuppy has quit IRC | 21:45 | |
| *** sdake_ has quit IRC | 21:46 | |
| *** sdake_ has joined #openstack-ansible | 21:48 | |
| *** sdake_ has quit IRC | 21:48 | |
| *** sdake_ has joined #openstack-ansible | 21:48 | |
| daneyon | andymccr: I'm reviewing the neutron configs and I see that physical_interface_mappings in ml2 conf. I thought eth10 is used within the agents container and I figured that eth10 would be in the conf file. | 21:48 |
| daneyon | andymccr: I'm a little fuzzy on where the neutron public network fits into this. If the vxlan network is used to interconnect the instances between hosts, where do I specify a network to use for floating ips. I'm used to bringing a network to eth1 on compute hosts running the neutron agents used for floating ips | 21:50 |
| *** KLevenstein has quit IRC | 21:56 | |
| daneyon | palendae ^ | 22:00 |
| *** nosleep77 has quit IRC | 22:00 | |
| *** sdake_ has quit IRC | 22:00 | |
| daneyon | palendae andymccr nm I figured out my question | 22:01 |
| *** jwagner is now known as jwagner_away | 22:09 | |
| *** nosleep77 has joined #openstack-ansible | 22:22 | |
| *** movielovers has joined #openstack-ansible | 22:23 | |
| *** movielovers has left #openstack-ansible | 22:23 | |
| *** jwagner_away is now known as jwagner | 22:31 | |
| *** jaypipes-afk is now known as jaypipes | 22:31 | |
| *** BjoernT has quit IRC | 22:31 | |
| *** vmtrooper has joined #openstack-ansible | 22:52 | |
| *** sdake_ has joined #openstack-ansible | 22:55 | |
| *** vmtrooper has quit IRC | 22:57 | |
| *** sdake__ has joined #openstack-ansible | 22:57 | |
| *** sdake_ has quit IRC | 23:00 | |
| *** jwagner is now known as jwagner_away | 23:02 | |
| *** sdake__ has quit IRC | 23:06 | |
| bgmccollum | any reason why the swift *-server.conf files don't have a [*-replicator] section? the replicator services are not starting... | 23:08 |
| *** sdake_ has joined #openstack-ansible | 23:09 | |
| *** sdake_ has quit IRC | 23:14 | |
| bgmccollum | or maybe something else is amuck...either way replicators (and a few other) services arent running | 23:15 |
| andymccr | bgmccollum: are you running master? | 23:18 |
| bgmccollum | 10.1.2 | 23:18 |
| andymccr | is replication-network specified in your user_config or swift config? | 23:18 |
| bgmccollum | not explicitly... | 23:18 |
| bgmccollum | let me check | 23:18 |
| bgmccollum | storage_network: 'br-storage' | 23:19 |
| bgmccollum | replication_network: 'br-storage' | 23:19 |
| andymccr | ok cool. so there is a bug that has already been fixed but wasnt backported | 23:20 |
| andymccr | if you specify replication_network to be the same as storage_network it will treat it like a dedicated replication network but since it tries to start 2 services on the same IP/Port the second (replicator) fails | 23:20 |
| bgmccollum | makes sense | 23:21 |
| andymccr | if you had a dedicated replication network the replicator conf would be split into a second conf (which ahs happened here) but would be on a separate ip | 23:21 |
| andymccr | so it wouldn't conflict and would start fine | 23:21 |
| andymccr | its been fixed in that it will now check and if they are the same will not split out the conf | 23:21 |
| bgmccollum | so nix the repl network | 23:21 |
| andymccr | yeh | 23:21 |
| bgmccollum | cool | 23:21 |
| andymccr | you "might" have to do some manual remova;l of the dedicated replicator confs | 23:21 |
| andymccr | which other services arent started? | 23:22 |
| bgmccollum | as is...i dont see a single conf file for object / account / container servers | 23:22 |
| bgmccollum | no replicator specific confs | 23:22 |
| bgmccollum | rather I *only* see a single conf for ... | 23:23 |
| andymccr | oh weird... | 23:23 |
| andymccr | i mean it should either dump down the conf file incl replicator or not at all in which case replicator goes into the server specific confs | 23:23 |
| andymccr | hmm wonder if the fix didnt work properly for 10 branch | 23:23 |
| bgmccollum | i see what it should be doing, but isn't haha | 23:24 |
| andymccr | if the conf still has a conditional that is incorrect it wont drop the new conf or the replicator section | 23:24 |
| bgmccollum | i have to run, ill poke it some more this evening | 23:24 |
| andymccr | lemme check | 23:24 |
| andymccr | but yeh drop the replication_network and it should work | 23:24 |
| bgmccollum | ok...thanks | 23:24 |
| *** sdake_ has joined #openstack-ansible | 23:26 | |
| *** jmccrory1 has quit IRC | 23:28 | |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!