| *** sarob_ has joined #openstack-infra | 00:00 | |
| *** penguinRaider__ has joined #openstack-infra | 00:01 | |
| *** luqas__ has quit IRC | 00:01 | |
| dougwig | hi infra, we need a second core devstack-gate review to re-enable services in the neutron jobs. just passed jenkins: https://review.openstack.org/#/c/140864/ | 00:01 |
|---|---|---|
| *** luqas__ has joined #openstack-infra | 00:02 | |
| *** sarob has quit IRC | 00:02 | |
| fungi | dougwig: revert approved | 00:03 |
| dougwig | fungi: ty | 00:03 |
| fungi | dougwig: and congrats! | 00:03 |
| *** markmcclain has joined #openstack-infra | 00:03 | |
| dougwig | :) | 00:03 |
| mordred | mtreinish: https://www.itelegram.com/ | 00:03 |
| *** lhcheng has left #openstack-infra | 00:03 | |
| dougwig | thank you. :) | 00:04 |
| openstackgerrit | Steven Weston proposed openstack-infra/system-config: Adds puppet system-config module split script https://review.openstack.org/137991 | 00:04 |
| *** baoli has quit IRC | 00:04 | |
| *** penguinRaider has quit IRC | 00:05 | |
| fungi | mordred: "MailGram $18.95 includes up to 100 words: Delivered in 3 to 5 business days. An exceptional value for sending non-urgent messages." | 00:05 |
| fungi | wow | 00:05 |
| mtreinish | fungi: heh, I was jsut about to comment on that too | 00:05 |
| mtreinish | fungi: I like this more: http://www.phonegram.com/ | 00:06 |
| mtreinish | "Send 30 words for just $9.95!" | 00:06 |
| fungi | touché | 00:06 |
| mtreinish | I would have expected that to be cheaper because they just have to dial the phone instead of finding someone | 00:07 |
| mordred | of course, I'm pretty sure that they're just printing an email on old looking paper and then sending it through the mail | 00:07 |
| fungi | mordred: drat. you've sussed out my genius business model | 00:08 |
| fungi | (how can it fail?!?) | 00:08 |
| mtreinish | fungi: take a look at the examples for personal phonegrams on that site... | 00:08 |
| mtreinish | it's quite the list | 00:09 |
| *** signed8bit is now known as signed8bit_ZZZzz | 00:09 | |
| fungi | they're missing "impersonal break-up message" | 00:09 |
| fungi | that's where i'll corner the market | 00:10 |
| *** Ryan_Lane has quit IRC | 00:11 | |
| *** changbl has quit IRC | 00:11 | |
| *** KurtMartin is now known as kmartin | 00:12 | |
| mtreinish | yeah, I guess they have the market on complaints, contacting long lost relatives, and getting through to important people | 00:13 |
| *** marcusvrn has quit IRC | 00:14 | |
| *** penguinRaider has joined #openstack-infra | 00:14 | |
| *** penguinRaider__ has quit IRC | 00:15 | |
| *** dims__ has quit IRC | 00:15 | |
| *** shashankhegde has quit IRC | 00:15 | |
| *** marcusvrn has joined #openstack-infra | 00:15 | |
| *** dims__ has joined #openstack-infra | 00:15 | |
| mordred | fungi, mtreinish: I would like to also submit this: http://s3-ec.buzzfed.com/static/2014-12/9/9/enhanced/webdr10/enhanced-30804-1418136239-17.jpg | 00:16 |
| mordred | it is believed that that is a bicycle | 00:17 |
| *** bradjones has quit IRC | 00:18 | |
| *** dims__ has quit IRC | 00:18 | |
| *** dims__ has joined #openstack-infra | 00:18 | |
| *** marcusvrn1 has joined #openstack-infra | 00:19 | |
| *** marcusvrn has quit IRC | 00:20 | |
| *** _nadya_ has joined #openstack-infra | 00:21 | |
| *** david-lyle is now known as david-lyle_afk | 00:23 | |
| *** Masahiro has joined #openstack-infra | 00:23 | |
| *** marcusvrn1 has quit IRC | 00:24 | |
| grue_pm | mordred: that guy is open to stacking his bike ;-p | 00:24 |
| *** tjones1 has joined #openstack-infra | 00:24 | |
| *** palar has joined #openstack-infra | 00:25 | |
| *** _nadya_ has quit IRC | 00:25 | |
| *** MarkAtwood has quit IRC | 00:26 | |
| *** marcusvrn has joined #openstack-infra | 00:27 | |
| *** palar has quit IRC | 00:27 | |
| *** Masahiro has quit IRC | 00:28 | |
| *** ZZelle has quit IRC | 00:28 | |
| *** palar has joined #openstack-infra | 00:31 | |
| *** penguinRaider__ has joined #openstack-infra | 00:33 | |
| *** penguinRaider has quit IRC | 00:36 | |
| *** palar has quit IRC | 00:39 | |
| *** zz_dimtruck is now known as dimtruck | 00:39 | |
| *** shashankhegde has joined #openstack-infra | 00:42 | |
| openstackgerrit | Merged openstack-infra/devstack-gate: Revert "Temporarily disable testing lbaas, fwaas, and vpnaas" https://review.openstack.org/140864 | 00:43 |
| waynr | zaro: hey i tried comparing 1.0.0-SNAPSHOT against 1.0.0 using the tuple returned by pkg_resources.parse_version and found that ordering does not occur as with MAVEN; 1.0.0-SNAPSHOT is considered greater than 1.0.0 | 00:43 |
| waynr | so i'm thinking about implementing a MavenVersion class in a module called jenkins_jobs/jenkins/plugin.py | 00:44 |
| waynr | was wondering what you think of that | 00:44 |
| waynr | (thoughts from anyone else familiar with jenkins job builder would be apprecated :) | 00:44 |
| *** dims__ has quit IRC | 00:45 | |
| *** Ryan_Lane has joined #openstack-infra | 00:45 | |
| *** unicell has joined #openstack-infra | 00:46 | |
| *** penguinRaider__ has quit IRC | 00:46 | |
| *** penguinRaider has joined #openstack-infra | 00:46 | |
| *** dims__ has joined #openstack-infra | 00:46 | |
| *** andreykurilin__ has quit IRC | 00:47 | |
| zaro | waynr: would that be just to special case the compare when 'SNAPSHOT' is found in the version?? | 00:50 |
| *** dims__ has quit IRC | 00:50 | |
| waynr | pretty much, one other option i am considering is to just mutate "([0-9](.[0-9]){0,2}?-SNAPSHOT" into " | 00:52 |
| waynr | pretty much, one other option i am considering is to just mutate "([0-9])(.[0-9]){0,2}?-SNAPSHOT" into "{0}{1}{2}".format(re.group(1), re.group(1), ".preview") | 00:52 |
| *** dimtruck is now known as zz_dimtruck | 00:53 | |
| waynr | or something like that i don't remember exactly how the regex match object works | 00:53 |
| jogo | fungi: experimental jobs are backed up | 00:53 |
| jogo | 25 hr 7 min | 00:53 |
| jogo | experimental-tempest-dsvm-neutron-full-dibtest" | 00:53 |
| jogo | is still queued | 00:53 |
| waynr | the only problem then is that it would become impossible to match directly against "0.0.1-SNAPSHOT" | 00:54 |
| *** Masahiro has joined #openstack-infra | 00:54 | |
| waynr | because of the mutation, the version available when working on the module for a particular plugin would then be "0.0.1.preview" which does sort the way we want it to for Maven version | 00:54 |
| waynr | if we document this--the need to do match against ".preview" instead of "-SNAPSHOT"--then we wouldn't have to create our own MavenVersion class | 00:55 |
| waynr | or we could just always assume that folks working on a module won't ever try to match directly against -SNAPSHOT | 00:56 |
| waynr | well, assume that, document it, and be rigorous in reviewing module code that might attempt to do something like that | 00:57 |
| *** dannywilson has quit IRC | 00:58 | |
| *** dannywilson has joined #openstack-infra | 00:59 | |
| *** luqas__ has quit IRC | 01:01 | |
| *** luqas has joined #openstack-infra | 01:01 | |
| fungi | jogo: i'm betting we lack any workers for that. checking now | 01:02 |
| zaro | waynr: for simplicity i'm leaning towards documenting and not caring about SNAPSHOT. my thought is that users should not be using JJB with a snapshot version of a jenkins plugin. | 01:03 |
| fungi | needs devstack-trusty-dib slaves | 01:03 |
| fungi | and there are none | 01:03 |
| fungi | checking for images now | 01:03 |
| *** dannywilson has quit IRC | 01:04 | |
| mordred | mtreinish: explode updated | 01:05 |
| fungi | aha, we have devstack-trusty-dib images but no nodes. looks like they're in an endless build loop | 01:05 |
| zaro | notmyname: i'm not aware of anyone else requesting a link shortener service in gerrit so i don't think it's even on the radar. it might get on the radar if you enter a feature request thought :) https://code.google.com/p/gerrit/issues/list | 01:06 |
| zaro | fungi: ^ | 01:07 |
| notmyname | zaro: thanks :-) | 01:07 |
| jogo | fungi: thanks | 01:07 |
| *** bswartz has joined #openstack-infra | 01:08 | |
| fungi | jogo: i'm trying to build a replacement diskimage now. will upload it to hpcloud if/when it succeeds and see if that helps | 01:09 |
| *** otter768 has joined #openstack-infra | 01:09 | |
| jogo | fungi: cool beans | 01:12 |
| waynr | zaro: okay so are you saying don't even worry about making the -SNAPSHOT changes you suggested in your review? | 01:13 |
| waynr | i mean with respect to the test cases | 01:13 |
| * anteaya is stuck in chicago for the night, will be lucky to be home by noon tomorrow | 01:14 | |
| notmyname | zaro: https://code.google.com/p/gerrit/issues/detail?id=3057 | 01:14 |
| *** otter768 has quit IRC | 01:14 | |
| pleia2 | anteaya: sorry to hear :\ | 01:16 |
| *** markmcclain has quit IRC | 01:16 | |
| anteaya | I was none to pleased myself | 01:16 |
| anteaya | I want to pet my cat | 01:16 |
| heyongli | jogo, hope you could have time to take a look the test case and bp:https://review.openstack.org/#/c/139000/ | 01:17 |
| heyongli | jogo, https://review.openstack.org/#/c/139927/ | 01:18 |
| *** shayneburgess has quit IRC | 01:18 | |
| *** yjiang5 is now known as yjiang5_away | 01:20 | |
| * jogo looks | 01:20 | |
| *** yaguang has joined #openstack-infra | 01:20 | |
| *** gyee has joined #openstack-infra | 01:21 | |
| *** yaguang has quit IRC | 01:21 | |
| *** yaguang has joined #openstack-infra | 01:22 | |
| heyongli | jogo, very thanks. | 01:22 |
| *** zhiwei has joined #openstack-infra | 01:22 | |
| jogo | heyongli: so getting Daniel Berrange's feedback is very useful | 01:23 |
| *** ChuckC has quit IRC | 01:23 | |
| jogo | I propose changing the scope of https://review.openstack.org/#/c/139927 | 01:23 |
| *** markmcclain has joined #openstack-infra | 01:23 | |
| jogo | to discuss how you plan on setting up the third party CI | 01:23 |
| jogo | mainly what it will test, where the tests will live etc. | 01:23 |
| heyongli | jogo, which part? bp? or the testcases? | 01:23 |
| *** penguinRaider__ has joined #openstack-infra | 01:24 | |
| jogo | and why a fake libvirt driver isn't sufficient (what Daniel said) | 01:24 |
| heyongli | jogo, Daniel Berrange and I prefer the CI, refer to https://review.openstack.org/#/c/139927/1/specs/kilo/approved/fake-libvirt-hw-testing-driver.rst | 01:24 |
| jogo | heyongli: and I an fine with that | 01:24 |
| jogo | heyongli: but we need to sort out what it will test (shouldn't be an issue), and where the code will live | 01:24 |
| jogo | (which may be an issue) | 01:24 |
| heyongli | so, i want to further make the testcases to be good enough. | 01:24 |
| jogo | heyongli: because we want to make sure others can run these tests as well | 01:25 |
| jogo | if they have different hardware to test | 01:25 |
| jogo | etc, | 01:25 |
| *** zhiwei has quit IRC | 01:25 | |
| heyongli | jogo, sure, i also want this wonder happen. | 01:25 |
| *** zhiwei has joined #openstack-infra | 01:25 | |
| heyongli | so, i will try to ping Daniel Berrange on the testcases, to get feedback. | 01:26 |
| mtreinish | mordred: cool, thanks | 01:27 |
| jogo | heyongli: that would be great, hopefully there won't be too many differences in opinions on that | 01:27 |
| *** penguinRaider has quit IRC | 01:27 | |
| *** dims__ has joined #openstack-infra | 01:27 | |
| jogo | heyongli: I am more concerned about getting consensus on finding a home for the tests | 01:27 |
| jogo | IMHO nova functional testing may be a good place | 01:27 |
| heyongli | jogo, do you mean sevices/compute? | 01:29 |
| *** sarob_ has quit IRC | 01:30 | |
| jogo | heyongli: the directory nova/tests/functional/ | 01:30 |
| jogo | heyongli: but we can sort that detail out in the spec | 01:30 |
| jogo | heyongli: so I think the spec should discuss what it will test | 01:31 |
| jogo | heyongli: and how to make the tests runnable by others, and where the tests will live | 01:31 |
| *** arosen1 has joined #openstack-infra | 01:31 | |
| jogo | heyongli: AFAIK this will be the first set of tests that we will have that will be run by only 3rd party CI | 01:32 |
| jogo | so this is new territory | 01:32 |
| *** ChuckC has joined #openstack-infra | 01:32 | |
| anteaya | jogo: can you just advocate for nova/tests/functional? | 01:33 |
| anteaya | since that seems like it will make sean happy | 01:33 |
| heyongli | jogo, good suggestion, but that bp is currently for the fakelibvit, is it good place to discuss these? | 01:33 |
| anteaya | and will be one less thing for heyongli to try to figure out when he writes the spec | 01:33 |
| *** arosen has quit IRC | 01:34 | |
| jogo | anteaya: well I think we still need to sort it out, but we should sort out how they will fit in nova/tests/functional | 01:34 |
| jogo | heyongli: maybe create a new blueprint and abandon the fakelibvirt one? | 01:34 |
| anteaya | okay and I understand | 01:34 |
| anteaya | but I wonder to what degree that point is confusing for heyongli | 01:35 |
| heyongli | jogo, ok, this bp is for tempest | 01:35 |
| jogo | heyongli: very excited to see this work being done btw | 01:35 |
| jogo | heyongli: BP for nova | 01:35 |
| anteaya | my read is taht seems to be a point of difficulty for him, in terms of understanding | 01:35 |
| anteaya | unless I am reading things incorrectly | 01:35 |
| jogo | anteaya: hmm yeah, want to help out | 01:35 |
| heyongli | jogo, nova , we had miss the bp propalsal time, hope this is ok. | 01:35 |
| anteaya | you are doing a great job | 01:35 |
| anteaya | I just know how fast your mind works | 01:36 |
| anteaya | and I know heyongli is not a native english speaker | 01:36 |
| mtreinish | jogo: for comparison on tempest we've made it clear nothing can land unless it's run in a ci system (3rd party or infra) and nothign gets approved until it's shown running | 01:36 |
| jogo | heyongli: it shouldn't be hard to make an exception for this bp/spec | 01:36 |
| anteaya | native english speakers experienced with openstack have difficulty keeping up with how fast your brain works jogo | 01:36 |
| anteaya | :D | 01:36 |
| mtreinish | jogo: as far as I know no one's ever done that for tests that only work on a 3rd party ci | 01:36 |
| *** ChuckC has quit IRC | 01:36 | |
| jogo | mtreinish: right, that is part of the issue here | 01:37 |
| jogo | anteaya: heh | 01:37 |
| *** ChuckC has joined #openstack-infra | 01:37 | |
| heyongli | jogo, ok, good news. | 01:37 |
| jogo | heyongli: so this nova BP/spec should cover: | 01:38 |
| jogo | * What tests (try to sync with Daniel on this) | 01:38 |
| jogo | * How the tests will be runnable for other peoples hardware | 01:38 |
| *** shakamunyi has joined #openstack-infra | 01:38 | |
| jogo | * The mechanics of running the tests (assuming they will live in nova/tests/functional). We will need a way to just run these tests and nothing else | 01:39 |
| anteaya | heyongli: do you know what sync means? | 01:39 |
| jogo | mtreinish: would tempest-lib be a good base for this? | 01:39 |
| heyongli | anteaya, ,yeah, i know | 01:39 |
| anteaya | heyongli: okay great | 01:40 |
| *** markmcclain has quit IRC | 01:40 | |
| heyongli | anteaya, jogo , before sync with Daniel, i'd like get bp sent out first, then ping Daniel, the we got something to discuss, not just sync on fly. | 01:41 |
| jogo | heyongli: ++ | 01:41 |
| mtreinish | jogo: what do you want to put there? | 01:41 |
| jogo | mtreinish: https://review.openstack.org/#/c/139000/ | 01:41 |
| mtreinish | jogo: tempest-lib has no test cases | 01:42 |
| jogo | mtreinish: no, as in can we build those test cases in nova repo on tempest-lib | 01:43 |
| mtreinish | jogo: yeah, probably pretty easily | 01:44 |
| jogo | mtreinish: excellent, as those tests don't make a ton of sense in tempest proper | 01:44 |
| mtreinish | they don't make any sense, it's a layer violation. Tempest shouldn't care what the underlying hardware is | 01:45 |
| mtreinish | and that couples it to the hardware | 01:45 |
| *** gyee has quit IRC | 01:47 | |
| heyongli | jogo, the nova functional testing seems all based on some fake facility, right? | 01:48 |
| mtreinish | jogo: the one thing is the tempest-lib stuff you'd need to build those tests isn't there yet, it's next on the todo list | 01:48 |
| heyongli | mtreinish, build tempest cases from nova functional test, do you mean this? sorry i'm not very familiar with this. | 01:50 |
| mtreinish | heyongli: in tempest there is a high prio effort to split out common functionality (like rest clients, ssh utils, base test cases, etc) into a library tempest-lib | 01:51 |
| mtreinish | that way you can construct tempest-like tests suites inside other projects | 01:51 |
| fungi | jogo: even with updated images, the devstack-trusty-dib nodes don't boot successfully. until someone has a chance to debug these (likely using an out-of-band console if hpcloud has such things), that job should probably be removed | 01:51 |
| mtreinish | heyongli: jogo's proposal was to have the similar (if not the same) test cases that you wrote for tempest (which doesn't really fit in tempest) live in the nova tree and use tempest-lib | 01:52 |
| mtreinish | because the testing is more appropriate in the nova functional testing tree. | 01:53 |
| heyongli | mtreinish, thank, such a good feature, but not ready, this is a problem. jogo, do you have any suggestion on this? | 01:54 |
| *** yamamoto_ has joined #openstack-infra | 01:54 | |
| jogo | heyongli: I can help with the tempest-lib aspects if you want | 01:56 |
| jogo | fungi: who added the dib tests anyway | 01:57 |
| heyongli | jogo, very thanks. i just worry we build a feature on a todo feature. | 01:57 |
| *** yaguang has quit IRC | 01:58 | |
| jogo | heyongli: yeah I understand that concern I am happy to help close any gaps there | 01:58 |
| fungi | jogo: git blame zuul/layout.yaml? | 01:58 |
| jogo | ahh clarkb ^ | 01:59 |
| heyongli | jogo, great thanks, that's a base point for a sort of work could start on. i'm going to construct the bp. | 01:59 |
| * jogo used pickaxe instead of git blame | 02:00 | |
| fungi | jogo: you daredevil | 02:00 |
| jogo | heyongli: great! | 02:00 |
| fungi | jogo: anyway, i think experimental jobs are fair game to stop running without heavy discussion if they aren't supportable. we can always add it back later | 02:01 |
| jogo | fungi: should I just remove it from zuul/layout.yaml and leave it elsewhere? | 02:02 |
| fungi | jogo: yeah | 02:02 |
| *** boris-42 has quit IRC | 02:03 | |
| fungi | we'll probably bring it back next week, just for now i don't know how to get at the hpcloud oob console to see why the images we're creating won't boot | 02:03 |
| *** asettle is now known as asettle-BRB | 02:03 | |
| anteaya | jogo: +1 to what fungi said | 02:03 |
| anteaya | that is the point of experimental jobs | 02:03 |
| fungi | it was basically a job to test that the nodes booted from these images work, so it's pretty useless anyway for now when we can't boot the nodes it tests | 02:05 |
| *** marun has quit IRC | 02:05 | |
| openstackgerrit | Joe Gordon proposed openstack-infra/project-config: Remove dibtest from experimental queue https://review.openstack.org/141245 | 02:06 |
| anteaya | ha ha ha | 02:06 |
| clarkb | I added dib tests? | 02:07 |
| *** baoli has joined #openstack-infra | 02:07 | |
| jogo | clarkb: 2b5fedd95162ab9203e2dc3acdc91c2bb514d67f | 02:07 |
| jogo | If69f18a61529fe886a836c6c284103babbcfc2cb | 02:07 |
| jogo | pickaxe knows all | 02:07 |
| clarkb | I cant look those up on my phone | 02:08 |
| clarkb | do you mean the tests that run on dib built slaves? | 02:08 |
| clarkb | and if so why remove those? | 02:08 |
| clarkb | we are finally able to move that stuff forward again | 02:08 |
| fungi | clarkb: because we can't boot them, so check experimental hangs indefinitely | 02:09 |
| clarkb | huh | 02:09 |
| clarkb | that sucks | 02:09 |
| jogo | yup | 02:09 |
| * mordred is getting closer to having all of that in-hand | 02:09 | |
| fungi | clarkb: i even tried refreshing the image with a new image-build and image-upload to the 5 hpcloud environments | 02:09 |
| * mordred apologizes for the time - turns out the debug cycle is SLOW | 02:09 | |
| clarkb | any idea why they dont boot? | 02:09 |
| clarkb | and should I become a farmer? | 02:10 |
| *** hdd has joined #openstack-infra | 02:10 | |
| mordred | clarkb: I will farm the things with you | 02:10 |
| fungi | clarkb: nodepool is building and then shunting them to delete within a few minutes | 02:10 |
| fungi | clarkb: i wasn't sure how to get at the instance console in hpcloud to try booting one myself and watching what it's doing | 02:10 |
| clarkb | fungi any errors in the log? that said there are about 5 nodepool changes we need | 02:11 |
| clarkb | fungi you can gui it or use the novaclient | 02:11 |
| jeblair | (also we need zuul job launch timeouts) | 02:12 |
| fungi | clarkb: LaunchStatusException: Server 194d88ea-ed26-4a00-830c-664a6d96745c for node id: 68325 status: ERROR | 02:13 |
| fungi | clarkb: they all do that after a few minutes | 02:13 |
| jogo | what did I do wrong? https://review.openstack.org/#/c/141245/ | 02:14 |
| jogo | puppet-apply-centos6 doesn't like me | 02:14 |
| clarkb | jogo puppet doesnt like anyone today | 02:14 |
| clarkb | its why I may become a faer | 02:14 |
| clarkb | *farmer | 02:15 |
| *** luqas has quit IRC | 02:16 | |
| *** luqas has joined #openstack-infra | 02:16 | |
| mordred | clarkb: puppet stab stab | 02:16 |
| clarkb | speaking of I think we may consider dropping that test but thats not great eother | 02:17 |
| clarkb | we could docker a trusty master and puppet agent against that in those trsts | 02:17 |
| clarkb | this is where I let the crazy flow | 02:17 |
| fungi | just add beer | 02:18 |
| clarkb | its already there. burnside and gigantic do a good job | 02:18 |
| mordred | clarkb: gimeme a few more days and Ill have all your dockers for you | 02:18 |
| clarkb | seriously though spinning up a master might not be terrible i those tests | 02:18 |
| clarkb | since its more like what we actually do | 02:19 |
| anteaya | jogo: your patch failed jenkins | 02:20 |
| clarkb | except we puppet apply for image builds :( | 02:21 |
| clarkb | I really dont think we have any good options. we could apply that patch ourselves | 02:21 |
| clarkb | fungi jeblair mordred ^ do you hate that as much as I do? | 02:22 |
| fungi | clarkb: it does sound questionable to need to backport patches from puppet 4 to the stable released and supported puppet we run | 02:23 |
| fungi | clarkb: is there a channel through which a stable backport could be requested? | 02:23 |
| fungi | "this is broken, please don't expect people to just wait for puppet 4" | 02:24 |
| mordred | I do not like that | 02:24 |
| mordred | well, hang on | 02:24 |
| mordred | I do not mind spinning up a master to do our tests | 02:24 |
| mordred | I _want_ to get to masterless | 02:24 |
| mordred | but at the moment testing apply and running agent _is_ strangeish | 02:24 |
| jogo | so lots of broken things, oh joy | 02:24 |
| mordred | so as a stop-gap until I can kill our master, I'm game with dockering one (or something) for what we're doing with apply tests today | 02:25 |
| clarkb | fungi I requested a backport. it is apparently being discussed | 02:26 |
| mordred | clarkb: what's the ETA on 4? | 02:26 |
| mordred | 6 months? a year? 3? | 02:26 |
| clarkb | who knows but its not backward compat | 02:26 |
| mordred | no, I know that | 02:26 |
| nibalizer | so whats the problem with running without directory enviornments for the apply test? | 02:26 |
| mordred | I'm wondering how much time we have before armegeddon happens again | 02:27 |
| clarkb | we would have to redo our master | 02:27 |
| *** shashankhegde has quit IRC | 02:27 | |
| nibalizer | its actually just some config changes | 02:27 |
| nibalizer | using $environment like it was before isn't a problem | 02:28 |
| nibalizer | mordred: 4 lands feburaryish | 02:28 |
| nibalizer | 3.7 won't go end of life for a long time tho | 02:28 |
| fungi | and when 3.x does go end of life they'll announce it a good month or two ahead of time | 02:29 |
| mordred | I thnk we should a) redo our master to old-style environment b) make a test job that spins up a master and does agent instead of apply, but keep apply tests too c) move more quickly to kill master with prejudice | 02:30 |
| nibalizer | mordred: why do you hate the master? | 02:30 |
| mordred | nibalizer: because it's part of why the puppetlabs "we hate upgrades" sucks so mad | 02:30 |
| mordred | bad | 02:31 |
| nibalizer | its sortof orthogonal to the current problem | 02:31 |
| mordred | it is | 02:31 |
| mordred | it's just on my hitlist | 02:31 |
| nibalizer | okay | 02:31 |
| nibalizer | well there is a new puppet server that runs in java | 02:31 |
| nibalizer | you could hate that too :P | 02:31 |
| mordred | yeah. that's not gonna get deployed | 02:31 |
| nibalizer | i agree with a and b, i dont care about c | 02:31 |
| nibalizer | with multinode nodepool we could easily get a master and agent up super fast | 02:32 |
| clarkb | no just docker it | 02:32 |
| nibalizer | we could also use the beaker testing stuff that I've been working on with the stackforge folks | 02:32 |
| *** adalbas has quit IRC | 02:32 | |
| nibalizer | mk | 02:32 |
| clarkb | we dont need 2 nodes | 02:32 |
| *** baoli has quit IRC | 02:33 | |
| mordred | yah | 02:34 |
| *** baoli has joined #openstack-infra | 02:34 | |
| mordred | I can make a docker with the master in it - and we just docker that on the apply nodes and point the config at it etc. | 02:34 |
| arosen1 | fungi: how do you get the patch revision number from gerrit to use with zuul client? It looks like gerrit doesn't return that via it's jsonrpc api? | 02:34 |
| mordred | I'll take a quick stab at that tomorrow | 02:34 |
| fungi | arosen1: there's a patchsets option to the query call i think. you probably want the one listed as latest | 02:35 |
| nibalizer | and i should prep a patch to go back to non-directory environments? | 02:36 |
| clarkb | ya lets prep it | 02:36 |
| clarkb | I think its the least bad option | 02:36 |
| nibalizer | kk | 02:36 |
| clarkb | nibalizer do we have to update anything other than puppet.conf on the master? | 02:36 |
| nibalizer | yea i'll come up with a list of steps | 02:37 |
| fungi | i'm good with that as the smallest possible modification for now to get the job back on track | 02:37 |
| clarkb | nibalizer: thanks | 02:37 |
| arosen1 | fungi: ah thanks i see had to include --patch-sets | 02:37 |
| *** baoli has quit IRC | 02:42 | |
| *** spzala has quit IRC | 02:42 | |
| *** shashankhegde has joined #openstack-infra | 02:44 | |
| jesusaurus | clarkb: whats the problem with directory environments? | 02:44 |
| clarkb | broken on centos6 | 02:45 |
| clarkb | will be fixed in puppet 4 maybe fix will be backported to 3.7 | 02:46 |
| *** oomichi has joined #openstack-infra | 02:46 | |
| jesusaurus | ugh | 02:46 |
| openstackgerrit | Joshua Hesketh proposed openstack-infra/project-config: Dogfood swift logs with infra jobs https://review.openstack.org/141250 | 02:47 |
| *** tjones1 has quit IRC | 02:48 | |
| *** tjones1 has joined #openstack-infra | 02:49 | |
| *** dims__ has quit IRC | 02:49 | |
| *** tjones1 has quit IRC | 02:49 | |
| *** dims__ has joined #openstack-infra | 02:50 | |
| *** KanagarajM has joined #openstack-infra | 02:52 | |
| *** dims__ has quit IRC | 02:54 | |
| *** koolhead17 has joined #openstack-infra | 02:56 | |
| *** Ryan_Lane has quit IRC | 02:56 | |
| *** shakamunyi has quit IRC | 02:57 | |
| *** shakamunyi has joined #openstack-infra | 02:57 | |
| *** tylerdurden has joined #openstack-infra | 02:57 | |
| *** luqas has quit IRC | 02:59 | |
| *** luqas has joined #openstack-infra | 02:59 | |
| jhesketh | clarkb: ping | 03:01 |
| *** yaguang has joined #openstack-infra | 03:01 | |
| clarkb | pong | 03:01 |
| clarkb | should I review that change? | 03:02 |
| clarkb | jhesketh +2 | 03:04 |
| *** rmcall has quit IRC | 03:09 | |
| *** sabari is now known as zz_sabari | 03:10 | |
| *** otter768 has joined #openstack-infra | 03:10 | |
| *** otter768 has quit IRC | 03:15 | |
| jhesketh | clarkb: ah thanks, was going to ask about this log though: http://logs.openstack.org/79/133179/1/experimental/experimental-swift-logs-system-config-pep8-centos6/cc75c20 | 03:16 |
| jhesketh | clarkb: that's from https://review.openstack.org/#/c/133179/ | 03:16 |
| jhesketh | it's the experimental log that was working earlier this week | 03:16 |
| jhesketh | and now it's not | 03:16 |
| *** ivar-lazzaro has quit IRC | 03:16 | |
| jhesketh | clarkb: are you able to check if that object exists in swift please? | 03:16 |
| *** jerryz has joined #openstack-infra | 03:16 | |
| clarkb | not right now | 03:21 |
| clarkb | but maybe in a bit. fence fell over \o/ | 03:21 |
| *** shashankhegde has quit IRC | 03:21 | |
| *** erikmwilson has quit IRC | 03:23 | |
| *** ddieterly has joined #openstack-infra | 03:27 | |
| *** penguinRaider__ has quit IRC | 03:27 | |
| *** penguinRaider has joined #openstack-infra | 03:28 | |
| *** shashankhegde has joined #openstack-infra | 03:28 | |
| openstackgerrit | Joshua Hesketh proposed openstack-infra/os-loganalyze: Assume trailing slashes want to load indexes https://review.openstack.org/141260 | 03:29 |
| jhesketh | clarkb: all good, no hurry :-) | 03:29 |
| *** fungi has quit IRC | 03:31 | |
| *** fungi has joined #openstack-infra | 03:31 | |
| clarkb | meh fence will have to wait. its cold and winy | 03:32 |
| jhesketh | lol | 03:34 |
| *** weshay has quit IRC | 03:36 | |
| clarkb | jhesketh: ya the index.html, console.html and logs/ are there | 03:36 |
| clarkb | I guess I should check logs.o.o for errors? | 03:36 |
| jhesketh | clarkb: that'd be handy please :-) | 03:36 |
| *** harlowja is now known as harlowja_away | 03:36 | |
| clarkb | jhesketh: 403 Forbidden | 03:39 |
| *** zz_sabari is now known as sabari | 03:39 | |
| clarkb | let me log in as the ro user and see (I checked initially with rw user) | 03:40 |
| jhesketh | hmm | 03:40 |
| clarkb | "error loading content. Try refreshing page | 03:41 |
| clarkb | so something with that user went sideways | 03:41 |
| *** asettle-BRB is now known as asettle | 03:42 | |
| jogo | docker just crashed my box | 03:42 |
| clarkb | jogo: fun | 03:42 |
| jogo | kernel panic | 03:42 |
| clarkb | jhesketh: I think I am just going to have to submit a ticket with rax for cloud files | 03:43 |
| clarkb | jhesketh: don't really have the bw to debug further tonight but I will submit tha before I afk | 03:43 |
| clarkb | jhesketh: anything else you think I should do? | 03:43 |
| *** sabari is now known as zz_sabari | 03:47 | |
| *** zz_sabari is now known as sabari | 03:49 | |
| jhesketh | clarkb: nope, that seems reasonable | 03:51 |
| jhesketh | you could check the permissions on the user(s) are still set | 03:51 |
| jhesketh | but it doesn't sound like that's it | 03:51 |
| clarkb | how do I do that? log in as the non special user? | 03:51 |
| clarkb | I can try that | 03:51 |
| jhesketh | that'd be a good test | 03:51 |
| jhesketh | there are some headers that define which users have access to what, but I can't remember them so a master-account check is a better idea | 03:52 |
| clarkb | the rw user definitely works | 03:52 |
| clarkb | but will check via master account | 03:52 |
| *** unicell has quit IRC | 03:53 | |
| *** baoli has joined #openstack-infra | 03:55 | |
| openstackgerrit | Joshua Hesketh proposed openstack-infra/project-config: zuul's log path should always be a folder https://review.openstack.org/141263 | 03:56 |
| jhesketh | oh okay | 03:56 |
| jhesketh | so it's possibly a permissions thing | 03:57 |
| clarkb | jhesketh: I have confirmed that infra-files-ro has observer view only access on files | 03:57 |
| clarkb | jhesketh: so if it is a permissions thing then its not something that we seem to be able to toggle. Going to submit a ticket now | 03:57 |
| jhesketh | okay, seems reasonable | 03:58 |
| *** dannywilson has joined #openstack-infra | 04:00 | |
| clarkb | jhesketh: 141212-ord-0000211 is the ticket id for reference | 04:00 |
| jhesketh | thanks | 04:01 |
| *** hdd has quit IRC | 04:02 | |
| *** patrickeast has quit IRC | 04:03 | |
| *** dannywilson has quit IRC | 04:04 | |
| *** penguinRaider__ has joined #openstack-infra | 04:08 | |
| *** shakamunyi has quit IRC | 04:08 | |
| *** penguinRaider has quit IRC | 04:11 | |
| *** oomichi has quit IRC | 04:12 | |
| *** Ryan_Lane has joined #openstack-infra | 04:12 | |
| *** baoli has quit IRC | 04:13 | |
| *** penguinRaider has joined #openstack-infra | 04:13 | |
| *** penguinRaider__ has quit IRC | 04:14 | |
| *** shashankhegde has quit IRC | 04:15 | |
| *** sabari is now known as zz_sabari | 04:16 | |
| *** Ryan_Lane has quit IRC | 04:17 | |
| *** zz_sabari is now known as sabari | 04:17 | |
| *** r-daneel has quit IRC | 04:19 | |
| *** penguinRaider__ has joined #openstack-infra | 04:21 | |
| *** penguinRaider has quit IRC | 04:24 | |
| *** stevemar has joined #openstack-infra | 04:30 | |
| *** gondoi has quit IRC | 04:32 | |
| *** roaet- has quit IRC | 04:32 | |
| *** jbryce has quit IRC | 04:32 | |
| *** lbragstad has quit IRC | 04:32 | |
| *** cyeoh has quit IRC | 04:33 | |
| *** zz_dimtruck has quit IRC | 04:33 | |
| *** lbragstad has joined #openstack-infra | 04:33 | |
| *** zz_gondoi has joined #openstack-infra | 04:33 | |
| *** jbryce has joined #openstack-infra | 04:33 | |
| *** zz_gondoi is now known as gondoi | 04:33 | |
| *** roaet_ has joined #openstack-infra | 04:33 | |
| *** talluri has joined #openstack-infra | 04:33 | |
| *** zz_dimtruck has joined #openstack-infra | 04:34 | |
| *** cyeoh has joined #openstack-infra | 04:34 | |
| *** zz_dimtruck is now known as dimtruck | 04:34 | |
| *** unicell has joined #openstack-infra | 04:38 | |
| *** luqas__ has joined #openstack-infra | 04:38 | |
| armax | can anyone help with: https://review.openstack.org/#/c/141256/ as without it, Neutron is broken | 04:38 |
| armax | ? | 04:38 |
| armax | thanks! | 04:38 |
| *** luqas has quit IRC | 04:39 | |
| *** asettle has quit IRC | 04:40 | |
| *** sabari is now known as zz_sabari | 04:44 | |
| *** penguinRaider has joined #openstack-infra | 04:44 | |
| *** SumitNaiksatam has quit IRC | 04:44 | |
| *** penguinRaider__ has quit IRC | 04:44 | |
| *** SumitNaiksatam has joined #openstack-infra | 04:44 | |
| *** markvan has quit IRC | 04:45 | |
| *** luqas__ has quit IRC | 04:45 | |
| *** markvan has joined #openstack-infra | 04:46 | |
| *** luqas__ has joined #openstack-infra | 04:46 | |
| *** mmaglana has quit IRC | 04:46 | |
| *** SumitNaiksatam has quit IRC | 04:51 | |
| *** penguinRaider__ has joined #openstack-infra | 04:57 | |
| *** zz_sabari is now known as sabari | 05:00 | |
| *** penguinRaider has quit IRC | 05:01 | |
| *** penguinRaider__ has quit IRC | 05:09 | |
| *** penguinRaider has joined #openstack-infra | 05:09 | |
| *** otter768 has joined #openstack-infra | 05:11 | |
| *** otter768 has quit IRC | 05:16 | |
| *** penguinRaider__ has joined #openstack-infra | 05:19 | |
| *** penguinRaider has quit IRC | 05:22 | |
| heyongli | anteaya, Matthew Treinish post comments on https://review.openstack.org/#/c/139000/, which is very valuable. but i'm confused by some information there, could you please check it and get me some ideas about that? thanks. | 05:29 |
| heyongli | anteaya, mtreinish, tomorrow is weekend, i will check any comment on that next week, and thanks for your feed back. | 05:31 |
| *** sabari is now known as zz_sabari | 05:32 | |
| *** penguinRaider__ has quit IRC | 05:37 | |
| *** penguinRaider has joined #openstack-infra | 05:37 | |
| *** zz_sabari is now known as sabari | 05:37 | |
| grue_pm | heyongli: how are you finding it all so far? | 05:38 |
| *** wenlock has joined #openstack-infra | 05:39 | |
| heyongli | grue_pm, some confuse, do first CI or first test cases, or both can parallel? | 05:39 |
| grue_pm | heyongli: I'm new so I totally have no idea <-: I understand the confusion - you probably have more idea than I do. | 05:40 |
| grue_pm | heyongli: but folk around here are really nice and helpful and patient | 05:40 |
| heyongli | grue_pm, no that more, i want to know if there some thrid party CI running some test cases not merged to tempest or anywhere, and is it possible. but based mtreinish comments, it does possible. so my question, how could i setting our CI like that. we need make CI work first, them we had possible to share the 'qualified' test cases to tempest. | 05:42 |
| *** stevemar has quit IRC | 05:43 | |
| grue_pm | heyongli: I'm from a sysadmin background and I'm learning all of this from scratch (except IRC) pretty much - it's my 2nd week so like I said I have no idea but there should be some folk around to ask in the morning for them | 05:45 |
| heyongli | grue_pm, don't worry, we discuss here, nice guy will show up and guide us -:) | 05:46 |
| grue_pm | heyongli: I know some of Europe will be up and onboard soon | 05:46 |
| grue_pm | best bit about openstack and IRC is that it's worldwide and there are people on most of the time | 05:47 |
| grue_pm | except perhaps the weekends. | 05:47 |
| *** penguinRaider__ has joined #openstack-infra | 05:48 | |
| *** penguinRaider has quit IRC | 05:52 | |
| nibalizer | clarkb: hrm im starting to wonder if turning of directory environments is acutally a fix | 05:54 |
| nibalizer | since im having a hard time reproducing the error :/ | 05:54 |
| clarkb | install 3.7.3 on centos6 from puppet repo. install vcsrepo the use it | 05:55 |
| clarkb | nibalizer remember its self testing | 05:56 |
| clarkb | heyongli when you run third party ci the test environments and tests you run are completely up to you | 05:59 |
| clarkb | that is kind of the whole point | 05:59 |
| clarkb | nibalizer what aspect of reproducing isnt working? | 05:59 |
| openstackgerrit | Joshua Hesketh proposed openstack-infra/project-config: Add in parent links to zuul log indexes https://review.openstack.org/141277 | 06:02 |
| *** penguinRaider__ has quit IRC | 06:02 | |
| *** penguinRaider has joined #openstack-infra | 06:02 | |
| *** nikil22_ has joined #openstack-infra | 06:02 | |
| *** amitgandhinz has quit IRC | 06:05 | |
| *** KanagarajM2 has joined #openstack-infra | 06:10 | |
| *** KanagarajM has quit IRC | 06:11 | |
| *** harlowja_at_home has joined #openstack-infra | 06:12 | |
| *** luqas__ has quit IRC | 06:12 | |
| *** luqas__ has joined #openstack-infra | 06:12 | |
| *** changbl has joined #openstack-infra | 06:12 | |
| *** jyuso has quit IRC | 06:13 | |
| *** KanagarajM2 has quit IRC | 06:13 | |
| *** penguinRaider__ has joined #openstack-infra | 06:14 | |
| openstackgerrit | Stefano Maffulli 'reed' proposed openstack-infra/system-config: Getting activity board under infra https://review.openstack.org/141282 | 06:16 |
| nibalizer | clarkb: i have centos, puppet 3.7.3, ruby 1.8, ... oh wait | 06:16 |
| nibalizer | i have plabs vcsrepo not openstack vcsrepo | 06:16 |
| nibalizer | maybe thats it? | 06:16 |
| clarkb | maybe | 06:16 |
| *** penguinRaider has quit IRC | 06:17 | |
| nibalizer | ya no dice :/ | 06:18 |
| nibalizer | my worry is that its a transient failure so maybe the 'working without directory environments' wasn't actually a fix | 06:18 |
| nibalizer | booooooo | 06:18 |
| clarkb | I dont think its transient | 06:18 |
| clarkb | I turned them back on and it failed | 06:18 |
| clarkb | then dow graded through 3.7.2 .1 and .0 with the same result | 06:19 |
| nibalizer | hrm then there is some complexity that makes it hard to reproduce ... :( | 06:19 |
| *** arosen1 has quit IRC | 06:20 | |
| *** luqas__ has quit IRC | 06:20 | |
| *** luqas__ has joined #openstack-infra | 06:21 | |
| openstackgerrit | Stefano Maffulli 'reed' proposed openstack-infra/system-config: Getting activity board under infra https://review.openstack.org/141282 | 06:22 |
| clarkb | I dunno we seem to trip it consistently. how'd you make your test machine? | 06:22 |
| clarkb | I would start on new base centos6 image run install_puppet.sh then the nodepool script | 06:23 |
| nikil22_ | hi I have read like there are lots of jenkins jobs for each projects. When i look into "https://git.openstack.org/cgit/openstack-infra/project-config/tree/jenkins/jobs" this i see only few jobs. I just wanted to execute the tempest jobs related to neutron , where can i find these jobs | 06:23 |
| *** doude has quit IRC | 06:23 | |
| clarkb | then test as that will set up puppet in the way it fails for us. keep in mind it does run to set itself up so something puppet does to itself may be related | 06:23 |
| clarkb | nikil22_ they are in that dir in the devstack-gate.yaml file | 06:24 |
| *** reed has quit IRC | 06:28 | |
| crinkle | clarkb: yeah i can't reproduce it either, this is what i'm trying http://paste.openstack.org/show/149714/ | 06:28 |
| nikil22_ | clarkb: so if i use this in "devstack-gate.yaml" and build then all jobs will be uploaded to my jenkiins server? . All these jobs are tempest or unit test also included in this | 06:29 |
| clarkb | crinkle I am glad centos vim hates you too | 06:29 |
| heyongli | clarkb, thanks , good to know , what process should be take to make sure the CI quality? and get it online? | 06:30 |
| nibalizer | clarkb: i used hp centos node, installed puppet, pulled down vcsrepo from git | 06:30 |
| clarkb | crinkle you are skipping the initial puppet run | 06:30 |
| *** achanda has joined #openstack-infra | 06:31 | |
| clarkb | we know it works first time because thats how images are built. but that process installs puppet.conf among other things and is likely the missing piecein reproducing | 06:31 |
| clarkb | nibalizer^ | 06:31 |
| crinkle | clarkb: well this is masterless, my understanding was that it was a problem with puppet apply | 06:31 |
| clarkb | nikil22_ they are not unittests and yes you can use those templates to get the neutron tempest jobs | 06:32 |
| * crinkle also not paying that much attention | 06:32 | |
| clarkb | crinkle ya its puppet apply in both cases | 06:32 |
| clarkb | crinkle but there is an initial image build run that works then a run post image build that fails | 06:32 |
| clarkb | so likely that the initial run sets up puppet to fail | 06:33 |
| crinkle | oh that sounds strange | 06:33 |
| *** penguinRaider has joined #openstack-infra | 06:33 | |
| *** penguinRaider__ has quit IRC | 06:33 | |
| clarkb | heyongli thats a very nebulous question | 06:34 |
| clarkb | heyongli I would recommend starting small and getting something going then working from there | 06:34 |
| *** luqas__ has quit IRC | 06:34 | |
| clarkb | crinkle let me link you script to run | 06:35 |
| *** luqas has joined #openstack-infra | 06:35 | |
| *** pcrews has quit IRC | 06:35 | |
| nibalizer | i have the puppet.conf from the git repo | 06:36 |
| clarkb | https://git.openstack.org/cgit/openstack-infra/project-config/tree/nodepool/scripts/prepare_node_bare.sh | 06:36 |
| clarkb | it will install puppet and all that | 06:36 |
| *** MarkAtwood has joined #openstack-infra | 06:36 | |
| crinkle | clarkb: without looking at it i think it is going to fail on ./prepare_node.sh for me | 06:37 |
| clarkb | you nees that repo | 06:37 |
| heyongli | clarkb, yeah, i try to run first cases on our CI, to online CI, ie. post some comments to patches. our server setting up and some test cases is running but not post any comments. before we can post comments there should some procedure i can take, now i'm working to enable our CI can post comments, i think this action should approved by infra, but how to get approved not clear to me. | 06:37 |
| crinkle | but i can clone project-config and run it out of that | 06:37 |
| clarkb | and run in that dir | 06:37 |
| crinkle | :) | 06:37 |
| *** jyuso has joined #openstack-infra | 06:38 | |
| clarkb | heyongli any account can comment. you dont need approval bht we recommend you start with the sandbox reposo that you can work out any bugs there first | 06:39 |
| *** vigneshvar has joined #openstack-infra | 06:39 | |
| heyongli | clarkb, anteaya give some feedback to our log server, we are addressing that, and will bring it back soon. anteaya suggest post testcases out to get reviewd, this is what i'm doing now. | 06:39 |
| *** jamielennox is now known as jamielennox|away | 06:39 | |
| *** shakamunyi has joined #openstack-infra | 06:41 | |
| heyongli | clarkb, ok, i get the basic idea, very thanks. this is just what i want to know. so then, task try to merge our test cases to tempest or nova and task to make our CI env better can be done parallel, it's so cool. | 06:41 |
| *** rmcall has joined #openstack-infra | 06:42 | |
| *** fabo has quit IRC | 06:42 | |
| *** fabo has joined #openstack-infra | 06:43 | |
| heyongli | clarkb, actually, our test cases, even though not fit the tempest well, running about several weeks already, and addressed most of problem we encounter. to get our CI can vote, there is some way to go, but we does need make the first step, get it stable an feed back to patches. | 06:45 |
| *** harlowja_at_home has quit IRC | 06:46 | |
| crinkle | clarkb: so that script installs puppet 2.7 for me | 06:47 |
| crinkle | not entirely sure why | 06:47 |
| clarkb | wah | 06:47 |
| crinkle | it ends by failing trying to install lvm2 from updates | 06:48 |
| clarkb | oh puppet version 2 | 06:48 |
| crinkle | yes | 06:48 |
| clarkb | nibalizer ^ thats all you | 06:48 |
| * nibalizer denies all knowlege | 06:48 | |
| nibalizer | it should be running in '3' mode | 06:48 |
| nibalizer | by default | 06:48 |
| nibalizer | and any puppet 2 logic should be sent to /dev/null | 06:48 |
| clarkb | default is 2 | 06:48 |
| nibalizer | :( | 06:48 |
| clarkb | but then how are we getting 3.7.3 | 06:49 |
| clarkb | this makes no sense | 06:49 |
| crinkle | huh running just install_puppet.sh gets the right version | 06:51 |
| crinkle | still upset about lvm2 | 06:51 |
| clarkb | ya the bad default is in prepare-node.sh | 06:53 |
| crinkle | I don't see where in this repo environments are being configured | 06:53 |
| clarkb | so theory time. we install puppet2.7 which runs and installs puppet 3 | 06:53 |
| crinkle | you're managing your puppet with puppet? | 06:54 |
| clarkb | again nibalizer ^ | 06:54 |
| crinkle | lol | 06:54 |
| clarkb | but yes I think so | 06:54 |
| crinkle | all blame nibalizer | 06:54 |
| clarkb | well he wrote all that | 06:54 |
| clarkb | as part of transition to puppet 3 | 06:55 |
| nibalizer | what | 06:55 |
| nibalizer | we install puppet 2.7 that configures puppet 3? | 06:55 |
| nibalizer | i dont think so, bob | 06:55 |
| clarkb | yes you made it do the 3 now | 06:56 |
| clarkb | but prepare node is like but 2 | 06:56 |
| clarkb | so we should fix prepare node | 06:56 |
| clarkb | and I dont know what lvm2 is doing. maybe our images come with a package you need ? | 06:56 |
| *** mrunge has joined #openstack-infra | 06:57 | |
| crinkle | seems like it | 06:57 |
| crinkle | it also hiccups here https://git.openstack.org/cgit/openstack-infra/project-config/tree/nodepool/scripts/prepare_node.sh#n45 because epel isn't installed | 06:57 |
| clarkb | ya our images epel | 06:57 |
| *** mrunge has quit IRC | 06:57 | |
| clarkb | welcome to cloyd | 06:58 |
| crinkle | okay well is that ideal? if we had the script install epel this would be a little more user-reproduceable | 06:58 |
| clarkb | its not ideal but its also something we have never had to do | 06:59 |
| clarkb | so it doesnt get tested | 06:59 |
| *** zul has quit IRC | 07:00 | |
| clarkb | note this is why we want to build from scratch | 07:00 |
| clarkb | then its all consistent for everyonw | 07:01 |
| *** achanda has quit IRC | 07:04 | |
| *** rmcall has quit IRC | 07:04 | |
| *** achanda has joined #openstack-infra | 07:05 | |
| *** mrmartin has joined #openstack-infra | 07:05 | |
| *** achanda_ has joined #openstack-infra | 07:07 | |
| *** tnovacik has joined #openstack-infra | 07:07 | |
| *** yolanda has quit IRC | 07:08 | |
| *** achanda has quit IRC | 07:09 | |
| *** oomichi has joined #openstack-infra | 07:11 | |
| *** otter768 has joined #openstack-infra | 07:12 | |
| *** yolanda has joined #openstack-infra | 07:16 | |
| *** achanda_ has quit IRC | 07:17 | |
| *** otter768 has quit IRC | 07:17 | |
| openstackgerrit | Joshua Hesketh proposed openstack-infra/project-config: Add in parent links to zuul log indexes https://review.openstack.org/141277 | 07:19 |
| openstackgerrit | Joshua Hesketh proposed openstack-infra/project-config: Add extra detail to swift indexes https://review.openstack.org/141286 | 07:19 |
| openstackgerrit | Joshua Hesketh proposed openstack-infra/project-config: Add extra detail to swift indexes https://review.openstack.org/141286 | 07:20 |
| *** oomichi has quit IRC | 07:21 | |
| *** jgallard_ has joined #openstack-infra | 07:23 | |
| *** xyang1 has quit IRC | 07:24 | |
| openstackgerrit | Colleen Murphy proposed openstack-infra/project-config: Only guard yum update against epel if enabled https://review.openstack.org/141288 | 07:28 |
| *** mrunge has joined #openstack-infra | 07:28 | |
| *** wenlock has quit IRC | 07:33 | |
| *** lifeless has quit IRC | 07:36 | |
| *** luqas has quit IRC | 07:36 | |
| *** luqas has joined #openstack-infra | 07:36 | |
| *** HeOS has quit IRC | 07:36 | |
| *** penguinRaider__ has joined #openstack-infra | 07:37 | |
| *** liam_ has joined #openstack-infra | 07:37 | |
| *** liam_ is now known as Guest77493 | 07:38 | |
| *** luqas__ has joined #openstack-infra | 07:40 | |
| *** luqas has quit IRC | 07:40 | |
| *** penguinRaider has quit IRC | 07:40 | |
| *** jyuso has quit IRC | 07:40 | |
| *** belmoreira has joined #openstack-infra | 07:41 | |
| *** viglesias has quit IRC | 07:41 | |
| *** lifeless has joined #openstack-infra | 07:43 | |
| *** viglesias has joined #openstack-infra | 07:43 | |
| *** mrunge has quit IRC | 07:45 | |
| *** _nadya_ has joined #openstack-infra | 07:45 | |
| *** achanda has joined #openstack-infra | 07:46 | |
| *** luqas__ has quit IRC | 07:47 | |
| *** luqas has joined #openstack-infra | 07:47 | |
| *** _nadya_ has quit IRC | 07:49 | |
| *** amitgandhinz has joined #openstack-infra | 07:49 | |
| *** viglesias has quit IRC | 07:50 | |
| *** viglesias has joined #openstack-infra | 07:51 | |
| *** Longgeek has joined #openstack-infra | 07:58 | |
| *** Guest77493 has quit IRC | 07:59 | |
| *** jamespage_ has joined #openstack-infra | 08:02 | |
| *** ala_ has joined #openstack-infra | 08:04 | |
| *** jamespage_ has quit IRC | 08:04 | |
| *** doude has joined #openstack-infra | 08:05 | |
| *** dtantsur|afk is now known as dtantsur | 08:05 | |
| *** mrunge has joined #openstack-infra | 08:06 | |
| *** sabari is now known as zz_sabari | 08:06 | |
| *** unicell1 has joined #openstack-infra | 08:07 | |
| *** rediskin has joined #openstack-infra | 08:08 | |
| *** yjiang5 has joined #openstack-infra | 08:08 | |
| *** jpich has joined #openstack-infra | 08:08 | |
| *** vponomaryov1 has joined #openstack-infra | 08:08 | |
| openstackgerrit | Travis Tripp proposed openstack/requirements: Updating to the 0.15.0 relese https://review.openstack.org/141298 | 08:08 |
| *** jcoufal has joined #openstack-infra | 08:09 | |
| *** jaypipes_ has joined #openstack-infra | 08:10 | |
| *** achanda has quit IRC | 08:11 | |
| *** zz_sabari is now known as sabari | 08:11 | |
| *** achanda has joined #openstack-infra | 08:12 | |
| *** maurosr- has joined #openstack-infra | 08:12 | |
| *** zz_avozza is now known as avozza | 08:13 | |
| *** mrodden_ has joined #openstack-infra | 08:14 | |
| *** unicell has quit IRC | 08:14 | |
| *** jaypipes has quit IRC | 08:14 | |
| *** vponomaryov has quit IRC | 08:14 | |
| *** maurosr has quit IRC | 08:14 | |
| *** funzo has quit IRC | 08:14 | |
| *** yjiang5_away has quit IRC | 08:14 | |
| *** redixin has quit IRC | 08:14 | |
| *** mrodden has quit IRC | 08:14 | |
| *** maurosr- is now known as maurosr | 08:14 | |
| *** mrodden_ is now known as mrodden | 08:14 | |
| *** avozza is now known as zz_avozza | 08:15 | |
| *** achanda has quit IRC | 08:16 | |
| *** achanda has joined #openstack-infra | 08:18 | |
| *** ociuhandu has joined #openstack-infra | 08:18 | |
| *** cody-somerville has quit IRC | 08:18 | |
| *** funzo has joined #openstack-infra | 08:26 | |
| *** e0ne has joined #openstack-infra | 08:28 | |
| *** jcoufal has quit IRC | 08:30 | |
| openstackgerrit | Andreas Jaeger proposed openstack/requirements: Add yasfb to global-requirements https://review.openstack.org/141303 | 08:32 |
| *** mpaolino has joined #openstack-infra | 08:33 | |
| *** markus_z has joined #openstack-infra | 08:36 | |
| *** luqas has quit IRC | 08:36 | |
| *** luqas has joined #openstack-infra | 08:37 | |
| *** jyuso has joined #openstack-infra | 08:38 | |
| gilliard | Hello -infra. I'm having some trouble with the -puppet-apply- jobs on this patch https://review.openstack.org/#/c/125279/ - is anyone around who can help me understand what's going wrong? | 08:39 |
| *** andreaf has joined #openstack-infra | 08:41 | |
| *** penguinRaider__ has quit IRC | 08:41 | |
| *** penguinRaider has joined #openstack-infra | 08:42 | |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config: Fix api-wg job names and placement https://review.openstack.org/141306 | 08:42 |
| *** Hal_ has joined #openstack-infra | 08:44 | |
| gilliard | I've raised the problem I described above at https://storyboard.openstack.org/#!/story/2000081 - never used sb before, so let me know if I did it wrong. | 08:45 |
| gilliard | s/if/how/ | 08:46 |
| *** Dafna has quit IRC | 08:48 | |
| *** MaxV has joined #openstack-infra | 08:49 | |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config: Fix api-wg job names and placement https://review.openstack.org/141306 | 08:50 |
| *** tnovacik has quit IRC | 08:50 | |
| *** penguinRaider__ has joined #openstack-infra | 08:52 | |
| *** penguinRaider has quit IRC | 08:56 | |
| *** HeOS has joined #openstack-infra | 08:57 | |
| *** MarkAtwood has quit IRC | 08:59 | |
| *** dtantsur is now known as dtantsur|bbl | 08:59 | |
| *** jcoufal has joined #openstack-infra | 09:00 | |
| *** jedimike has joined #openstack-infra | 09:02 | |
| *** penguinRaider__ has quit IRC | 09:05 | |
| *** yamamoto_ has quit IRC | 09:05 | |
| *** sabari is now known as zz_sabari | 09:05 | |
| *** penguinRaider has joined #openstack-infra | 09:07 | |
| *** penguinRaider__ has joined #openstack-infra | 09:09 | |
| openstackgerrit | Armando Migliaccio proposed openstack-infra/project-config: Check that Gerrit ACL files are normalized https://review.openstack.org/140821 | 09:12 |
| *** penguinRaider has quit IRC | 09:13 | |
| *** otter768 has joined #openstack-infra | 09:13 | |
| *** mfink has quit IRC | 09:15 | |
| *** jlibosva has joined #openstack-infra | 09:16 | |
| *** penguinRaider__ has quit IRC | 09:17 | |
| *** penguinRaider has joined #openstack-infra | 09:17 | |
| *** armax has quit IRC | 09:17 | |
| *** otter768 has quit IRC | 09:18 | |
| *** penguinRaider has quit IRC | 09:19 | |
| *** jistr has joined #openstack-infra | 09:19 | |
| *** penguinRaider has joined #openstack-infra | 09:19 | |
| *** _nadya_ has joined #openstack-infra | 09:23 | |
| *** derekh has joined #openstack-infra | 09:25 | |
| *** skolekonov has joined #openstack-infra | 09:27 | |
| *** SotK has joined #openstack-infra | 09:28 | |
| *** e0ne has quit IRC | 09:28 | |
| *** ZZelle_ is now known as ZZelle | 09:28 | |
| *** ihrachyshka has joined #openstack-infra | 09:32 | |
| *** ihrachyshka_ has joined #openstack-infra | 09:34 | |
| *** penguinRaider__ has joined #openstack-infra | 09:35 | |
| *** pblaho has joined #openstack-infra | 09:36 | |
| *** ihrachyshka has quit IRC | 09:37 | |
| *** mpaolino has quit IRC | 09:37 | |
| *** _nadya_ has quit IRC | 09:38 | |
| *** penguinRaider has quit IRC | 09:39 | |
| *** ociuhandu has quit IRC | 09:42 | |
| *** luqas has quit IRC | 09:50 | |
| *** luqas__ has joined #openstack-infra | 09:50 | |
| *** mpaolino has joined #openstack-infra | 09:50 | |
| *** penguinRaider__ has quit IRC | 09:52 | |
| *** penguinRaider has joined #openstack-infra | 09:53 | |
| *** Masahiro has quit IRC | 09:53 | |
| *** BobBall_AWOL is now known as BobBall | 09:57 | |
| *** doude has quit IRC | 10:06 | |
| *** doude has joined #openstack-infra | 10:06 | |
| *** alexpilotti has joined #openstack-infra | 10:08 | |
| *** rlandy has joined #openstack-infra | 10:08 | |
| *** isviridov_away is now known as isviridov | 10:10 | |
| *** ociuhandu has joined #openstack-infra | 10:12 | |
| *** nfedotov has joined #openstack-infra | 10:14 | |
| nikil22_ | hi I have set up the CI system. Now say i want the job "/jobs/neutron-functional.yaml" to be imported to my jenkins. How or what command i suould use so that the jenkins job builder will create a job. Or if i just keep the yamal file in "/etc/jenkins_jobs/config" and do "jenkins-jobs --flush-cache update /etc/jenkins_jobs/config/" will it work? | 10:17 |
| *** penguinRaider__ has joined #openstack-infra | 10:19 | |
| *** penguinRaider has quit IRC | 10:23 | |
| *** e0ne has joined #openstack-infra | 10:25 | |
| *** achanda has quit IRC | 10:28 | |
| *** ociuhandu has quit IRC | 10:35 | |
| *** andreykurilin_ has joined #openstack-infra | 10:36 | |
| *** yfried_ has quit IRC | 10:41 | |
| *** ihrachyshka_ has quit IRC | 10:43 | |
| nikil22_ | Like i want to create a new job from the yaml file using JJB how can i do it? | 10:45 |
| *** ociuhandu has joined #openstack-infra | 10:50 | |
| *** yaguang has quit IRC | 10:54 | |
| *** dims__ has joined #openstack-infra | 10:56 | |
| *** ZZelle has quit IRC | 10:57 | |
| *** ZZelle has joined #openstack-infra | 10:58 | |
| *** dims__ has quit IRC | 11:00 | |
| *** penguinRaider__ has quit IRC | 11:01 | |
| *** penguinRaider has joined #openstack-infra | 11:01 | |
| *** ociuhandu has quit IRC | 11:01 | |
| *** penguinRaider has quit IRC | 11:03 | |
| *** penguinRaider has joined #openstack-infra | 11:03 | |
| *** _nadya_ has joined #openstack-infra | 11:06 | |
| *** adalbas has joined #openstack-infra | 11:06 | |
| *** cnesa2 has joined #openstack-infra | 11:06 | |
| *** cnesa has quit IRC | 11:07 | |
| *** Hal_ has quit IRC | 11:08 | |
| *** cnesa2 has quit IRC | 11:11 | |
| *** jgallard_ has quit IRC | 11:11 | |
| *** zhiwei has quit IRC | 11:14 | |
| *** otter768 has joined #openstack-infra | 11:14 | |
| *** otter768 has quit IRC | 11:18 | |
| *** aysyd has joined #openstack-infra | 11:20 | |
| *** yfried_ has joined #openstack-infra | 11:21 | |
| *** _nadya_ has quit IRC | 11:27 | |
| *** penguinRaider__ has joined #openstack-infra | 11:33 | |
| *** cdent has joined #openstack-infra | 11:34 | |
| *** ihrachyshka_ has joined #openstack-infra | 11:35 | |
| *** penguinRaider has quit IRC | 11:36 | |
| *** yfried_ is now known as yfried|afk | 11:37 | |
| *** ihrachyshka__ has joined #openstack-infra | 11:38 | |
| *** dizquierdo has joined #openstack-infra | 11:38 | |
| *** penguinRaider has joined #openstack-infra | 11:39 | |
| *** penguinRaider__ has quit IRC | 11:39 | |
| *** yamamoto has joined #openstack-infra | 11:40 | |
| *** penguinRaider__ has joined #openstack-infra | 11:40 | |
| *** ihrachyshka_ has quit IRC | 11:41 | |
| *** Masahiro has joined #openstack-infra | 11:42 | |
| openstackgerrit | Ivan Udovichenko proposed openstack-infra/system-config: Remove hardcoded commands https://review.openstack.org/141073 | 11:43 |
| *** penguinRaider has quit IRC | 11:45 | |
| *** dims__ has joined #openstack-infra | 11:45 | |
| *** baoli has joined #openstack-infra | 11:45 | |
| *** baoli has quit IRC | 11:46 | |
| *** baoli has joined #openstack-infra | 11:46 | |
| *** Masahiro has quit IRC | 11:46 | |
| *** penguinRaider has joined #openstack-infra | 11:48 | |
| *** dims__ has quit IRC | 11:49 | |
| *** penguinRaider__ has quit IRC | 11:49 | |
| *** dims__ has joined #openstack-infra | 11:49 | |
| *** dims__ has quit IRC | 11:54 | |
| *** penguinRaider__ has joined #openstack-infra | 11:55 | |
| *** boris-42 has joined #openstack-infra | 11:55 | |
| *** dims__ has joined #openstack-infra | 11:56 | |
| *** yamamoto has quit IRC | 11:56 | |
| *** yfried|afk is now known as yfried_ | 11:57 | |
| *** yamamoto has joined #openstack-infra | 11:57 | |
| *** penguinRaider has quit IRC | 11:58 | |
| *** pc_m has joined #openstack-infra | 12:01 | |
| social | Hi, can someone help me? someone who has forcepush rights to stackforge/packstack so he can remove tags 2014.2.dev1359.g40f3b9d and 2014.2.dev1375.g0a45ca7 ? | 12:02 |
| *** pc_m has quit IRC | 12:02 | |
| *** skolekonov has quit IRC | 12:03 | |
| *** pc_m has joined #openstack-infra | 12:03 | |
| *** pc_m has quit IRC | 12:04 | |
| *** yamamoto has quit IRC | 12:04 | |
| *** pc_m has joined #openstack-infra | 12:05 | |
| *** penguinRaider__ has quit IRC | 12:06 | |
| *** penguinRaider has joined #openstack-infra | 12:06 | |
| *** skolekonov has joined #openstack-infra | 12:07 | |
| *** yfried_ is now known as yfried|afk | 12:07 | |
| *** mpaolino has quit IRC | 12:07 | |
| openstackgerrit | Merged openstack-infra/nose-html-output: Workflow documentation is now in infra-manual https://review.openstack.org/139426 | 12:10 |
| *** dtantsur|bbl is now known as dtantsur | 12:10 | |
| openstackgerrit | Merged openstack-infra/jeepyb: Add a git clean command before updating local copy https://review.openstack.org/126567 | 12:10 |
| openstackgerrit | Merged openstack-infra/puppet-apparmor: Add Rakefile file and update puppet coding style https://review.openstack.org/102794 | 12:10 |
| *** yfried|afk is now known as yfried_ | 12:10 | |
| openstackgerrit | Merged openstack-infra/gear: Workflow documentation is now in infra-manual https://review.openstack.org/139416 | 12:10 |
| openstackgerrit | Merged openstack-infra/gerritlib: Workflow documentation is now in infra-manual https://review.openstack.org/139419 | 12:11 |
| openstackgerrit | Merged openstack-infra/os-loganalyze: Workflow documentation is now in infra-manual https://review.openstack.org/139427 | 12:11 |
| openstackgerrit | Merged openstack-infra/gearman-plugin: Workflow documentation is now in infra-manual https://review.openstack.org/139417 | 12:12 |
| openstackgerrit | Merged openstack-infra/zuul: Doc: ref links for "approval" requirement https://review.openstack.org/130202 | 12:15 |
| *** tnovacik has joined #openstack-infra | 12:16 | |
| openstackgerrit | Andrea Frittoli proposed openstack-infra/devstack-gate: Fix branch fallback logic https://review.openstack.org/130499 | 12:18 |
| openstackgerrit | Merged openstack-infra/zuul: Document that the "Stream Events" permission is needed. https://review.openstack.org/132645 | 12:18 |
| openstackgerrit | Merged openstack-infra/zuul: Document that the "Stream Events" permission is needed. https://review.openstack.org/132645 | 12:18 |
| *** yfried_ is now known as yfried|afk | 12:20 | |
| openstackgerrit | Merged openstack-infra/infra-manual: Explain Zuul setup without gating https://review.openstack.org/139678 | 12:21 |
| *** ihrachyshka__ is now known as ihrachyshka | 12:23 | |
| *** timrc has quit IRC | 12:25 | |
| *** amuller has joined #openstack-infra | 12:26 | |
| *** penguinRaider__ has joined #openstack-infra | 12:26 | |
| *** penguinRaider has quit IRC | 12:29 | |
| *** yfried|afk is now known as yfried_ | 12:29 | |
| *** tnovacik has quit IRC | 12:30 | |
| openstackgerrit | Merged openstack-infra/zuul: Remove "properly" from error message for clarity https://review.openstack.org/134122 | 12:30 |
| *** zul has joined #openstack-infra | 12:32 | |
| nikil22_ | Hi in the CI setup. Always the devstack-gate install the devstack with master branch. how can i point to use the neutron as particular branch instead of the standard one | 12:38 |
| *** jgjl has joined #openstack-infra | 12:38 | |
| *** achanda has joined #openstack-infra | 12:39 | |
| nfedotov | Hello . You may set branch in local.conf | 12:39 |
| *** e0ne_ has joined #openstack-infra | 12:40 | |
| nfedotov | You may set bash environment varibale to name of a branch/repo you need. To find out what env variables you need to set you may look into stackrc file (devstack root folder) | 12:41 |
| *** hashar has joined #openstack-infra | 12:41 | |
| *** penguinRaider__ has quit IRC | 12:41 | |
| nikil22_ | nfedotov: I see that local.conf of devstack also created by devstack gate itself. where should i configure in devstack gate or the job. So that it will enter that repo path in loca.conf | 12:41 |
| *** e0ne has quit IRC | 12:42 | |
| nikil22_ | exactly how the local.conf is generated by devstack-gate | 12:42 |
| *** baoli has quit IRC | 12:43 | |
| *** baoli has joined #openstack-infra | 12:44 | |
| *** dprince has joined #openstack-infra | 12:45 | |
| *** _nadya_ has joined #openstack-infra | 12:46 | |
| *** vigneshvar has quit IRC | 12:47 | |
| *** dims__ has quit IRC | 12:48 | |
| *** dims__ has joined #openstack-infra | 12:48 | |
| *** dims__ has quit IRC | 12:52 | |
| *** yfried_ is now known as yfried|afk | 12:54 | |
| *** dkliban_afk is now known as dkliban | 12:54 | |
| *** dims__ has joined #openstack-infra | 13:01 | |
| *** tnovacik has joined #openstack-infra | 13:02 | |
| *** baoli has quit IRC | 13:04 | |
| *** yamamoto has joined #openstack-infra | 13:05 | |
| *** lttrl has joined #openstack-infra | 13:05 | |
| *** Hal_ has joined #openstack-infra | 13:06 | |
| *** markus_z has quit IRC | 13:07 | |
| dims__ | mrmartin: any news on the gate-infra-puppet-apply-centos6 breakage? | 13:09 |
| mrmartin | hi dims_, my latest info was that clarkb tried to replace module install with direct git fetch | 13:10 |
| *** yamamoto has quit IRC | 13:10 | |
| mrmartin | maybe this patch: https://review.openstack.org/#/c/141182/ | 13:11 |
| *** luqas__ has quit IRC | 13:12 | |
| *** luqas has joined #openstack-infra | 13:13 | |
| mrmartin | SergeyLukjanov: this patch is failing on gate-infra-puppet-apply-centos6 too. | 13:13 |
| *** yfried|afk is now known as yfried_ | 13:14 | |
| *** otter768 has joined #openstack-infra | 13:15 | |
| *** weshay has joined #openstack-infra | 13:15 | |
| *** mwagner_lap has quit IRC | 13:17 | |
| *** spzala has joined #openstack-infra | 13:18 | |
| *** MarkAtwood has joined #openstack-infra | 13:18 | |
| *** otter768 has quit IRC | 13:20 | |
| pc_m | fungi: ping - question on changing email address for gerrit/launchpad | 13:20 |
| openstackgerrit | Merged openstack-infra/nodepool: libffi-dev/libssl-dev/qemu-utils are required https://review.openstack.org/125515 | 13:21 |
| *** markus_z has joined #openstack-infra | 13:22 | |
| *** palar has joined #openstack-infra | 13:23 | |
| *** yfried_ is now known as yfried|afk | 13:24 | |
| *** dims__ has quit IRC | 13:26 | |
| *** vigneshvar has joined #openstack-infra | 13:27 | |
| *** dims__ has joined #openstack-infra | 13:27 | |
| *** amitgandhinz has quit IRC | 13:29 | |
| *** Masahiro has joined #openstack-infra | 13:30 | |
| *** yaguang has joined #openstack-infra | 13:31 | |
| *** dims__ has quit IRC | 13:31 | |
| *** signed8bit_ZZZzz is now known as signed8bit | 13:33 | |
| *** signed8bit has quit IRC | 13:34 | |
| *** koolhead17 has quit IRC | 13:35 | |
| *** mattfarina has joined #openstack-infra | 13:35 | |
| *** mattfarina has quit IRC | 13:35 | |
| *** Masahiro has quit IRC | 13:35 | |
| *** andymaier has joined #openstack-infra | 13:36 | |
| *** bswartz has quit IRC | 13:38 | |
| *** jpich has quit IRC | 13:40 | |
| tnovacik | hi, there are horizon selenium integration tests that are sometimes failing randomly - it would be really helpful if such test provided in a log a screen shot of browser(only on failure) - does anyone know how could this be achieved? | 13:41 |
| *** jpich has joined #openstack-infra | 13:41 | |
| tnovacik | i am mean not the screen shot part - but rather how to make the logs available on the gates | 13:41 |
| *** dkranz has joined #openstack-infra | 13:42 | |
| *** penguinRaider__ has joined #openstack-infra | 13:46 | |
| *** mpaolino has joined #openstack-infra | 13:47 | |
| *** cnesa has joined #openstack-infra | 13:47 | |
| *** zhiwei has joined #openstack-infra | 13:48 | |
| *** ddieterly has quit IRC | 13:49 | |
| *** ayoung has joined #openstack-infra | 13:49 | |
| *** ayoung has quit IRC | 13:49 | |
| *** yfried|afk is now known as yfried_ | 13:50 | |
| *** penguinRaider has joined #openstack-infra | 13:51 | |
| *** penguinRaider__ has quit IRC | 13:51 | |
| *** cnesa has quit IRC | 13:52 | |
| openstackgerrit | Merged openstack-infra/devstack-gate: Add oslo.context to devstack-vm-gate-wrap.sh https://review.openstack.org/135093 | 13:54 |
| *** zz_avozza is now known as avozza | 13:57 | |
| *** hdd has joined #openstack-infra | 13:58 | |
| *** yfried_ has quit IRC | 13:58 | |
| *** nikil22_ has quit IRC | 13:58 | |
| *** pnascimento has joined #openstack-infra | 13:59 | |
| pnascimento | hello, any #askbot maintainers for Openstack online? | 14:00 |
| *** dustins has joined #openstack-infra | 14:01 | |
| *** dims__ has joined #openstack-infra | 14:02 | |
| *** penguinRaider__ has joined #openstack-infra | 14:02 | |
| *** xyang1 has joined #openstack-infra | 14:03 | |
| dims__ | mrmartin: thanks!! | 14:03 |
| openstackgerrit | Ivan Udovichenko proposed openstack-infra/system-config: Remove hardcoded variables from openstack_project module (Wiki) https://review.openstack.org/141374 | 14:03 |
| *** cnesa has joined #openstack-infra | 14:03 | |
| openstackgerrit | Jan Klare proposed openstack-infra/project-config: Added gate-{name}-chef-rake job-template https://review.openstack.org/137134 | 14:04 |
| *** penguinRaider has quit IRC | 14:06 | |
| openstackgerrit | Merged openstack-infra/devstack-gate: Add tooz to devstack-vm-gate-wrap.sh https://review.openstack.org/141151 | 14:06 |
| *** bswartz has joined #openstack-infra | 14:07 | |
| openstackgerrit | Jan Klare proposed openstack-infra/project-config: Added gate-{name}-chef-rake job-template https://review.openstack.org/137134 | 14:09 |
| mrmartin | pnascimento: #askbot as irc channel or askbot as website? | 14:09 |
| *** achanda has quit IRC | 14:11 | |
| *** penguinRaider__ has quit IRC | 14:11 | |
| *** penguinRaider has joined #openstack-infra | 14:11 | |
| *** ryanpetrello has joined #openstack-infra | 14:11 | |
| *** achanda has joined #openstack-infra | 14:12 | |
| *** ddieterly has joined #openstack-infra | 14:15 | |
| *** k4n0 has quit IRC | 14:17 | |
| *** doug-fish has joined #openstack-infra | 14:17 | |
| openstackgerrit | Ivan Udovichenko proposed openstack-infra/system-config: Remove hardcoded variables from openstack_project module (zuul_dev) https://review.openstack.org/141381 | 14:21 |
| *** salv-orlando has joined #openstack-infra | 14:22 | |
| *** penguinRaider__ has joined #openstack-infra | 14:22 | |
| *** signed8bit has joined #openstack-infra | 14:24 | |
| *** BobBall is now known as BobBall_AWOL | 14:24 | |
| *** timcline has joined #openstack-infra | 14:25 | |
| *** penguinRaider has quit IRC | 14:25 | |
| *** salv-orlando has quit IRC | 14:26 | |
| *** yamamoto has joined #openstack-infra | 14:27 | |
| *** salv-orlando has joined #openstack-infra | 14:27 | |
| *** yapeng_ is now known as yapeng | 14:29 | |
| openstackgerrit | Ivan Udovichenko proposed openstack-infra/system-config: Remove hardcoded variables from openstack_project module (zuul_merger) https://review.openstack.org/141385 | 14:29 |
| *** timcline has quit IRC | 14:29 | |
| *** cnesa has quit IRC | 14:30 | |
| *** Sincler has joined #openstack-infra | 14:31 | |
| *** scheuran has joined #openstack-infra | 14:32 | |
| openstackgerrit | Ivan Udovichenko proposed openstack-infra/system-config: Remove hardcoded variables from openstack_project module (zuul_prod) https://review.openstack.org/141388 | 14:33 |
| *** rmcall has joined #openstack-infra | 14:34 | |
| *** baoli has joined #openstack-infra | 14:35 | |
| fungi | okay, back | 14:36 |
| *** unicell1 has quit IRC | 14:38 | |
| *** rlandy has quit IRC | 14:40 | |
| *** markus_z has quit IRC | 14:41 | |
| *** penguinRaider__ has quit IRC | 14:41 | |
| *** penguinRaider has joined #openstack-infra | 14:41 | |
| *** yapeng has quit IRC | 14:42 | |
| fungi | pc_m: how have you tried to change your launchpad and gerrit e-mail addresses? | 14:43 |
| fungi | mrmartin: dims__: according to scrollback, clarkb/nibalizer/crinkle were all digging into it more last night. the prevailing theory is that directory environments in puppet 3.7 are broken on centos 6. puppet has a fix slated for 4.x and is looking at a backport. in the interim, it looks like we may end up backing out the changes which added that for our systems | 14:45 |
| fungi | i'm hunting now to see if there's anything i can pick up and continue iterating on | 14:46 |
| mrmartin | fungi: and what will be the short-term solution? Most of my patches stucked due this bug. | 14:46 |
| social | fungi: how can I get someone who has forcepush rights to stackforge/packstack so he can remove tags 2014.2.dev1359.g40f3b9d and 2014.2.dev1375.g0a45ca7 ? | 14:47 |
| fungi | mrmartin: i think that is the short term solution | 14:47 |
| mrmartin | and who is doing the backport, puppetlabs? | 14:47 |
| mrmartin | or will we use a patched puppet? | 14:47 |
| fungi | mrmartin: the backport is the longer-term fix, though they were still trying to confirm this was the cause | 14:47 |
| *** rmcall has quit IRC | 14:48 | |
| *** penguinRaider__ has joined #openstack-infra | 14:48 | |
| mrmartin | and moving from modules to git fetch not solves this issue? | 14:48 |
| fungi | mrmartin: nope--we got deploy from git to where it would work for ubuntu but it continued failing the same way as non-git deployment on centos | 14:49 |
| mrmartin | ok. | 14:49 |
| *** ayoung has joined #openstack-infra | 14:50 | |
| mestery | fungi: What we did for the services repos earlier this week, I want to do the same for my stackforge/networking-odl repo, I have all the commits ready with history, do I need special permissions to do that? | 14:51 |
| fungi | mestery: if we haven't approved the change to add the project yet, then yes. that's what the import at project creation is for | 14:51 |
| *** penguinRaider has quit IRC | 14:51 | |
| dims__ | fungi: thanks! | 14:51 |
| *** Hal_ has quit IRC | 14:51 | |
| fungi | social: i'm having trouble matching you to anyone in the https://review.openstack.org/#/admin/groups/456,members list | 14:51 |
| mestery | fungi: Crap, it's already there. I think I messed up then, it's not possible post creation? | 14:52 |
| social | fungi: oh yes not work nick - lbezdick | 14:52 |
| social | fungi: I could kick in mmagr if you want him to show up | 14:52 |
| openstackgerrit | Antoine Musso proposed openstack-infra/zuul: Soften statsd requirement to bring 3.x https://review.openstack.org/141391 | 14:52 |
| fungi | social: it looks like those tags must be intentional because they were pushed more than a week apart | 14:52 |
| social | fungi: yes they were | 14:52 |
| social | fungi: but we didn't know they will break our build process :) | 14:53 |
| *** jpich has quit IRC | 14:53 | |
| social | fungi: corrct tag should be something like 2014.2.1 | 14:53 |
| fungi | mestery: it's possible, but it's a lot of effort for a stackforge project. we cleaned up that mess for the official neutron projects because there wasn't much choice | 14:53 |
| *** vigneshvar has quit IRC | 14:53 | |
| *** tonytan4ever has joined #openstack-infra | 14:54 | |
| *** lttrl has quit IRC | 14:54 | |
| social | fungi: should I open bug somewhere? | 14:54 |
| pc_m | fungi: So I've created a second email address in gerrit and that's it so far. | 14:54 |
| mestery | fungi: OK, makes total sense. Any other options, or did I screw this up badly enough that getting the history will be too difficult? | 14:54 |
| fungi | this is turning into "stackforge projects make mistakes and want the openstack infra team to spend time fixing them" day unfortunately, when i have openstack things to work on | 14:54 |
| pc_m | fungi: Was going to make it primary and then change launchpad, but Anita suggested I check with you to make sure there aren't any issues with doing that. | 14:55 |
| fungi | no offense | 14:55 |
| mestery | fungi: Ack, please ignore then, apologies :) | 14:55 |
| fungi | but it's a priority issue | 14:55 |
| mestery | fungi: None taken, I've had those days too | 14:55 |
| mestery | fungi: I'll play around a bit and figure things out, thanks, I do always appreciate your help! | 14:55 |
| fungi | mestery: you're welcome | 14:55 |
| social | fungi: for us we can live with that for while but we have to build from local repo where we remove the tags manually :/ | 14:56 |
| fungi | pc_m: there is a chance that when you change your launchpad account's primary e-mail address and then log in with that, then authrnticating to gerrit may create a new gerrit account conflicting with your original one (because launchpad likes to change the openid url for your account in those situations). if that happens, let me know | 14:56 |
| *** lttrl has joined #openstack-infra | 14:57 | |
| pc_m | fungi: OK thanks! | 14:57 |
| *** wenlock has joined #openstack-infra | 14:57 | |
| fungi | social: mestery: i may be able to get to your stackforge help requests later, but at the moment want to focus on getting our infra puppet jobs working again so we can dislodge a bunch of pending reviews | 14:57 |
| social | fungi: sure no problem | 14:58 |
| mestery | fungi: Thanks, appreciate it! | 14:58 |
| *** r-daneel has joined #openstack-infra | 15:00 | |
| *** andymaier has quit IRC | 15:01 | |
| *** dmsimard_away is now known as dmsimard | 15:01 | |
| *** andymaier has joined #openstack-infra | 15:01 | |
| fungi | thanks for understanding! | 15:01 |
| *** erikmwilson has joined #openstack-infra | 15:03 | |
| *** cnesa has joined #openstack-infra | 15:04 | |
| openstackgerrit | Jeremy Stanley proposed openstack-infra/system-config: Revert "Remove logic from puppet.conf" https://review.openstack.org/141395 | 15:05 |
| *** ociuhandu has joined #openstack-infra | 15:05 | |
| *** salv-orlando has quit IRC | 15:05 | |
| *** penguinRaider__ has quit IRC | 15:06 | |
| *** penguinRaider has joined #openstack-infra | 15:06 | |
| openstackgerrit | Jeremy Stanley proposed openstack-infra/system-config: Revert "Add puppet-httpd to modules.env" https://review.openstack.org/141396 | 15:06 |
| *** salv-orlando has joined #openstack-infra | 15:06 | |
| *** penguinRaider has quit IRC | 15:07 | |
| *** penguinRaider has joined #openstack-infra | 15:08 | |
| *** amitgandhinz has joined #openstack-infra | 15:09 | |
| *** yamamoto has quit IRC | 15:12 | |
| *** hdd has quit IRC | 15:16 | |
| *** otter768 has joined #openstack-infra | 15:16 | |
| clarkb | fungi I am not really here yet but in prepare_node.sh we default to installing puppet2. I think puppet 2 then installs puppet3. This may be why image builds work at all or it could be what is breaking us | 15:16 |
| clarkb | fungi probabaly worth booting rax/hpcloud centos6 and running with it set to install puppet 3 by default | 15:17 |
| clarkb | and checking if behavior at image build time and after changes | 15:17 |
| fungi | clarkb: i'll see if i can poke at that as a possibility. also i do find a lot of references to "Invalid parameter provider on Vcsrepo" in puppet bug reports and discussions... was sifting through them for any insights | 15:17 |
| *** spzala has quit IRC | 15:17 | |
| fungi | some dating back to the era where we forked vcsrepo | 15:18 |
| clarkb | fungi oh and I still have a held node in a broken state if you want to poke at it | 15:18 |
| isviridov | Hi there, can anybody look at devstack gate log output. Looks like verify_tempest_config.py causes gate build fails. Any hint is very appreciated http://logs.openstack.org/40/140740/11/check/gate-magnetodb-devstack-dsvm/665926a/logs/devstacklog.txt.gz#_2014-12-12_07_27_47_599 | 15:18 |
| clarkb | ends in 434 iirc | 15:18 |
| fungi | k | 15:18 |
| fungi | thanks | 15:18 |
| mordred | clarkb: are you sure we don't pass the "3" to prepare_node? | 15:18 |
| *** timcline has joined #openstack-infra | 15:19 | |
| clarkb | mordred yul | 15:19 |
| clarkb | *yup | 15:19 |
| *** dpaterson has joined #openstack-infra | 15:19 | |
| * mordred stabs things | 15:19 | |
| clarkb | mordred nodepool only passes in NODEPOOL_* args iircs | 15:19 |
| fungi | also this discussion looks very current... https://tickets.puppetlabs.com/browse/PUP-1515 | 15:19 |
| *** Masahiro has joined #openstack-infra | 15:19 | |
| *** MarkAtwood has quit IRC | 15:19 | |
| mordred | install_puppet's default is 3 | 15:19 |
| mordred | PUPPET_VERSION=${PUPPET_VERSION:-3} | 15:20 |
| mordred | if [ "$PUPPET_VERSION" = '3' ]; then | 15:20 |
| mordred | that's at the top | 15:20 |
| clarkb | ya but prepare node is 2 | 15:20 |
| *** otter768 has quit IRC | 15:20 | |
| clarkb | and it calls install puppet | 15:21 |
| mordred | AH - it's explicitly | 15:21 |
| clarkb | ya ... | 15:21 |
| mordred | wow | 15:21 |
| mordred | ok | 15:21 |
| *** lttrl has quit IRC | 15:21 | |
| mordred | have we pushed up a patch for that yet? | 15:21 |
| fungi | i was about to, but if you want feel free. unfortunately we need new images after it merges | 15:21 |
| *** timcline has quit IRC | 15:22 | |
| mordred | fungi: go ahead. just making sure | 15:22 |
| *** MarkAtwood has joined #openstack-infra | 15:22 | |
| fungi | and we'll likely have to force it to merge | 15:22 |
| *** lttrl has joined #openstack-infra | 15:22 | |
| clarkb | well I didnt want to merge it because it may be how we have centos6 nodes at all | 15:23 |
| *** timcline has joined #openstack-infra | 15:23 | |
| clarkb | so needed more testing than I could do at 11pm | 15:23 |
| mordred | yah. need to build a node with that fixed by hand and see if it works | 15:23 |
| *** Masahiro has quit IRC | 15:24 | |
| pnascimento | mrmartin: as a website, I used the hashtag to gain attention (sorry if it was missused) | 15:24 |
| openstackgerrit | Jeremy Stanley proposed openstack-infra/project-config: Stop defaulting node builds to Puppet 2 https://review.openstack.org/141403 | 15:25 |
| pnascimento | mrmartin: do you now any? | 15:25 |
| fungi | pnascimento: what do you need done with ask.openstack.org? | 15:25 |
| fungi | (our askbot server) | 15:26 |
| *** penguinRaider__ has joined #openstack-infra | 15:26 | |
| pnascimento | fungi: with ask.openstack.org itself? nothing. I need the maintainers for helping me with an issue on askbot (for a instance of mine) | 15:27 |
| *** andreaf has quit IRC | 15:27 | |
| *** andreaf has joined #openstack-infra | 15:28 | |
| *** yamamoto has joined #openstack-infra | 15:28 | |
| fungi | pnascimento: oh, evgeny is usually our go-to with askbot questions... http://askbot.org/en/users/1/evgeny/ | 15:28 |
| *** penguinRaider__ has quit IRC | 15:28 | |
| pnascimento | fungi: I have installed from Github an instance of askbot, and I want to implement OAuth login on it, but I had some trouble with it... after enabling "custom" OAuth the login system crashes :/ even the database user (default) | 15:28 |
| *** penguinRaider__ has joined #openstack-infra | 15:29 | |
| *** yamamoto has quit IRC | 15:29 | |
| fungi | pnascimento: we didn't actually install the version of askbot we're running. evgeny did, and he's been sort of maintaining it for us | 15:29 |
| clarkb | mordred are you able to test 141403? | 15:29 |
| pnascimento | fungi: I have created an question the Q&A forum of Askbot, but no one responded :( | 15:29 |
| fungi | pnascimento: though there is an effort underway to reinstall and maintain it ourself | 15:29 |
| *** wenlock has left #openstack-infra | 15:30 | |
| clarkb | I have to afk for the morning shortly so unlikely that I will | 15:30 |
| *** penguinRaider has quit IRC | 15:30 | |
| pnascimento | fungi: Evgeny helped me already :) gave me tips how to create the OAuth Login Module ;) | 15:30 |
| pnascimento | fungi: but, away from QA I don't know how to contact him (correction: I think he gave is email address on profile page, but I don't wanna be pushy on that level :( ) | 15:31 |
| fungi | pnascimento: oh, well sorry we're not more help. mrmartin may have played around with it a little bit (he started some work building a couple puppet modules to install and manage it), but not sure if he tried oauth2 or just openid | 15:32 |
| *** avozza is now known as zz_avozza | 15:32 | |
| *** lttrl has quit IRC | 15:32 | |
| pnascimento | fungi: You helped in some way :) | 15:33 |
| *** lttrl has joined #openstack-infra | 15:33 | |
| *** kgiusti has joined #openstack-infra | 15:33 | |
| *** tnovacik has quit IRC | 15:36 | |
| *** wenlock has joined #openstack-infra | 15:36 | |
| nelsnelson | Greetings, channel. May I please ask one of the core reviewers to take a look at https://review.openstack.org/#/c/134348/? It's been idle since November 24. | 15:37 |
| *** _nadya_ has quit IRC | 15:37 | |
| *** penguinRaider__ has quit IRC | 15:37 | |
| *** penguinRaider has joined #openstack-infra | 15:37 | |
| *** esker has joined #openstack-infra | 15:39 | |
| pnascimento | fungi: I will try to engage Evgeny without being pushy | 15:39 |
| *** mriedem has joined #openstack-infra | 15:39 | |
| *** nfedotov has quit IRC | 15:40 | |
| *** marun has joined #openstack-infra | 15:40 | |
| *** timcline has quit IRC | 15:41 | |
| *** e0ne_ is now known as e0ne | 15:41 | |
| *** timcline has joined #openstack-infra | 15:41 | |
| *** talluri has quit IRC | 15:42 | |
| *** markus_z has joined #openstack-infra | 15:42 | |
| *** mfink has joined #openstack-infra | 15:42 | |
| *** jaypipes_ is now known as jaypipes | 15:44 | |
| *** mrunge has quit IRC | 15:44 | |
| *** unicell has joined #openstack-infra | 15:45 | |
| mordred | clarkb: I'll give it a try | 15:45 |
| *** markmcclain has joined #openstack-infra | 15:46 | |
| *** mwagner_lap has joined #openstack-infra | 15:47 | |
| jeblair | clarkb, fungi, mordred: morning; i'm caught up on your exciting scrollback | 15:47 |
| mordred | jeblair: woot! | 15:47 |
| clarkb | fungi https://bugs.launchpad.net/openstack-ci/+bug/1401895 shouldnt filing that have been impossible? | 15:47 |
| uvirtbot | Launchpad bug 1401895 in openstack-ci "check-grenade-dsvm-neutron is failing mostly because of neutron services split" [Undecided,New] | 15:47 |
| jeblair | hrm, it came in via "also affects" | 15:48 |
| jeblair | so not a direct filing | 15:48 |
| clarkb | oh huh | 15:48 |
| clarkb | anyways I made it affect neutron and grenade and am removing infra | 15:49 |
| fungi | yeah, looks like there's no good way to prevent that | 15:49 |
| openstackgerrit | Travis Tripp proposed openstack/requirements: Updating python-glanceclient to 0.15.0 release https://review.openstack.org/141298 | 15:49 |
| mordred | only actually removing the project, I beleive | 15:49 |
| *** penguinRaider__ has joined #openstack-infra | 15:50 | |
| *** yaguang has quit IRC | 15:50 | |
| fungi | clarkb: how have you been recreating the centos6 failure? is there an easy way to fake out zuul-cloner or do i need to export dummy values for all the envvars it wants? | 15:50 |
| mtreinish | mordred: heh, that might not even be a guarantee | 15:50 |
| mordred | fungi: oh, are you working on testing the 2->3 change? | 15:51 |
| *** andymaier has quit IRC | 15:52 | |
| fungi | mordred: i'm trying to come up to speed by first confirming i can recreate the failure | 15:52 |
| clarkb | fungi just puppet apply -e 'vcsrepo { "/home/clarkb/derp": provider => git, source => "repourlhere" }' | 15:52 |
| *** andymaier has joined #openstack-infra | 15:53 | |
| clarkb | fungi you may have to give that --modulepath=/etc/puppet/modules | 15:53 |
| *** xyang1 has quit IRC | 15:53 | |
| *** penguinRaider has quit IRC | 15:53 | |
| fungi | clarkb: oh, i'll look at your held node then. i was trying to make sure i had teh from-scratch reproduction identified | 15:53 |
| clarkb | on the node I held see history on my user | 15:53 |
| fungi | yep | 15:53 |
| clarkb | I use a test.pp instead of -e though wasnt sure how small reproduction case would be when I started | 15:54 |
| *** rmcall has joined #openstack-infra | 15:55 | |
| *** andymaier has quit IRC | 15:55 | |
| *** andymaier has joined #openstack-infra | 15:56 | |
| *** _nadya_ has joined #openstack-infra | 15:56 | |
| pleia2 | morning fungi and clarkb! if either of you could have a look at the zanata spec, finally having that finished today would make my week https://review.openstack.org/#/c/133222/ | 15:57 |
| mordred | so - just to confirm with folks - to test the end to end, I want to spin up a centos6 node, copy the modified nodepool scripts in, run prepare_node then run clarks' test above | 15:57 |
| clarkb | mordred yup | 15:58 |
| mordred | kk | 15:58 |
| clarkb | pleia2 that is now very high on my list | 15:58 |
| *** pnascimento has quit IRC | 15:58 | |
| pleia2 | clarkb: thank you | 15:58 |
| fungi | clarkb: one thing i notice is that environmentpath is /etc/puppet/environments in /etc/puppet/puppet.conf, but that directory is nonexistent... | 15:58 |
| clarkb | mordred wait. run prepare node bare | 15:59 |
| mordred | yah | 15:59 |
| clarkb | mordred since that will set appropriate flags | 15:59 |
| clarkb | fungi oh | 15:59 |
| *** bdpayne has joined #openstack-infra | 15:59 | |
| clarkb | maybe that is why disabling dir envs fixes this but its also fixed by that puppet patch | 16:00 |
| mordred | ++ | 16:00 |
| mordred | makes sense | 16:00 |
| *** hdd has joined #openstack-infra | 16:00 | |
| fungi | clarkb: i'll play around with this hypothesis for a few minutes and see how far i get | 16:00 |
| *** yjiang5 is now known as yjiang5_away | 16:00 | |
| *** carl_baldwin has joined #openstack-infra | 16:02 | |
| clarkb | also fyi swift ro access broke at some point. I filed a bug with rax last night | 16:03 |
| clarkb | s/bug/ticket/ | 16:03 |
| jeblair | i'll check on that | 16:04 |
| fungi | clarkb: `mkdir -p /etc/puppet/environments/production && touch /etc/puppet/environments/production/environment.conf` seems to fix this? | 16:04 |
| fungi | clarkb: i'll propose a patch to the test script and see what happens | 16:04 |
| clarkb | jeblair ticket number is in sb if you need it | 16:04 |
| jeblair | clarkb: which account? | 16:05 |
| clarkb | fungi wow. ok so our base.pp that sets up puppet dir envs should make sure that stuff exists | 16:05 |
| clarkb | jeblair infra-files-ro | 16:05 |
| *** pcrews has joined #openstack-infra | 16:05 | |
| jeblair | clarkb: i mean which cloud account? | 16:05 |
| jeblair | ah jenkins | 16:05 |
| clarkb | thats the one I filed witj | 16:05 |
| clarkb | but its a user under openstackjenkins | 16:06 |
| mordred | clarkb: should it? I think it should only do that on the master | 16:06 |
| jeblair | clarkb: yep | 16:06 |
| *** tylerdurden has quit IRC | 16:06 | |
| clarkb | mordred I think this is a case of dir envs break puppet apply | 16:06 |
| mordred | clarkb: yah - confirmed, we only set up environment dirs on puppetmaster | 16:06 |
| clarkb | mordred unless you set up dir envs | 16:06 |
| clarkb | so maybe we need that not in puppet but around any puppet apply? | 16:07 |
| * mordred is becomming more a fan of reverting the new-style environment setup | 16:07 | |
| mordred | I really don't like the need to set up some directories on a host before we can run puppet apply on it | 16:07 |
| clarkb | +1 | 16:08 |
| mordred | especially if we're not intending on using them | 16:08 |
| mordred | otoh - we could put more smarts around our puppet.conf | 16:08 |
| mordred | and only install the one that refers to environments on the puppetmaster | 16:08 |
| mordred | (or only install those settings if puppetmaster is true) | 16:08 |
| *** tnovacik has joined #openstack-infra | 16:08 | |
| mordred | because this is also beacuse we install identical puppet.conf files everywhere | 16:08 |
| openstackgerrit | Jeremy Stanley proposed openstack-infra/system-config: Puppet apply test creates puppet directory env https://review.openstack.org/141413 | 16:08 |
| *** prad has joined #openstack-infra | 16:08 | |
| *** sabeen has joined #openstack-infra | 16:09 | |
| mordred | fungi: ^^ brilliant | 16:09 |
| fungi | seeing if that ^ does what i think it will | 16:09 |
| fungi | seemed to solve clarkb's trivial repro anyway | 16:09 |
| fungi | https://jenkins05.openstack.org/job/gate-infra-puppet-apply-centos6/425/console | 16:10 |
| *** penguinRaider__ has quit IRC | 16:11 | |
| mordred | oh - it's separate anyway | 16:11 |
| *** penguinRaider has joined #openstack-infra | 16:11 | |
| *** achanda has quit IRC | 16:11 | |
| mordred | we have different settings for the master | 16:11 |
| jeblair | I was able to take a look at the user in question and did notice that the user does not have a API-KEY generated. Can you possibly generate an API-KEY for the user and then attempt to view Cloud Files container? | 16:11 |
| *** shakamunyi has quit IRC | 16:11 | |
| jeblair | clarkb: ^ :/ | 16:11 |
| jeblair | i don't think we should need an api key | 16:11 |
| jeblair | we pass in the password... | 16:11 |
| *** achanda has joined #openstack-infra | 16:11 | |
| clarkb | jeblair agreed and it worked just the other day | 16:11 |
| jeblair | unless they just broke openstack compatibility or something | 16:11 |
| * mordred will be unhappy if they broke compat | 16:12 | |
| jeblair | i will say that we are unable to use rax extensions with our app which is written to openstack specs, and also it worked the other day :) | 16:12 |
| mordred | ++ | 16:12 |
| fungi | bam. 141413 will unblock us for now if we want, while people iterate on better solutions | 16:12 |
| mtreinish | fungi: \o/ | 16:13 |
| fungi | assuming this doesn't have follow-on side effects in the other jobs | 16:13 |
| fungi | but the centos6 job passed anyway | 16:13 |
| openstackgerrit | Monty Taylor proposed openstack-infra/system-config: Only use environments on the master https://review.openstack.org/141415 | 16:14 |
| mordred | fungi, clarkb: ^^ what about that? | 16:14 |
| jeblair | clarkb: oh, i also can not see cloud files content in the webui | 16:14 |
| *** ala_ has quit IRC | 16:14 | |
| fungi | mordred: i am fine with that as a fix too | 16:14 |
| *** andymaier has quit IRC | 16:14 | |
| mordred | fungi: I thinik we need 141413 too | 16:14 |
| fungi | mordred: arguably more correct, but will need new images | 16:14 |
| mordred | to get us there | 16:14 |
| jeblair | clarkb: i should expect to see the container and its contents in the dfw region; wouldn't you expect that? | 16:15 |
| fungi | yeah, we can revert 141413 once 141415 merges and we have all new images | 16:15 |
| mordred | so I say we land 141413 to unstick, then 141415 and let new images build, then we can revert 141413 | 16:15 |
| *** belmoreira has quit IRC | 16:15 | |
| mordred | yah | 16:15 |
| jeblair | clarkb: (when logged in as infra-files-ro) | 16:15 |
| mordred | fungi: can you verify that deleting the environment dir and then moving that in puppet.conf works on the held node? | 16:15 |
| mordred | jeblair: yes | 16:16 |
| fungi | mordred: doing | 16:16 |
| mordred | jeblair: especially if you're logged in to the web console | 16:16 |
| *** carl_baldwin has quit IRC | 16:16 | |
| *** achanda has quit IRC | 16:16 | |
| clarkb | jeblair ya I think its dfw | 16:16 |
| *** markus_z has quit IRC | 16:16 | |
| clarkb | jeblair when you login as the -rw user they just show up | 16:16 |
| fungi | mordred: that does indeed work | 16:17 |
| clarkb | when you log in as ro user its an error in the gui | 16:17 |
| clarkb | jeblair so ya its webui and client affected. the original ticket should explain both of the errors I saw | 16:17 |
| *** pnascimento has joined #openstack-infra | 16:18 | |
| jeblair | okay, replied with 'we really mean it' message | 16:18 |
| *** jerryz has quit IRC | 16:18 | |
| mordred | fungi: woot! | 16:18 |
| fungi | mordred: so i think we have a plan there | 16:18 |
| *** achanda has joined #openstack-infra | 16:18 | |
| mordred | fungi: ok, so, at this point, I think we have a workaround, a fix, AND we actually understand what was broken in the first palce | 16:18 |
| clarkb | and puppet may properly fix at some point | 16:19 |
| fungi | mordred: let's approve 141413, then once it merges we should be able to merge 141415, then after the weekend revert 141413 | 16:19 |
| clarkb | +1 | 16:19 |
| mordred | ++ | 16:19 |
| pnascimento | fungi: one more question, Evgeny usually engange or login in this channels? | 16:19 |
| *** reed has joined #openstack-infra | 16:19 | |
| fungi | and in the meantime recheck everything in affected projects with a -1 or -2 from jenkins | 16:19 |
| *** yjiang5_away is now known as yjiang5 | 16:19 | |
| *** tjones1 has joined #openstack-infra | 16:20 | |
| fungi | pnascimento: we usually e-mail him i think | 16:20 |
| fungi | pnascimento: reed may know more | 16:20 |
| reed | i know nothing :) | 16:20 |
| fungi | heh | 16:20 |
| clarkb | pleia2 +2 | 16:20 |
| *** bdpayne has quit IRC | 16:20 | |
| clarkb | fungi approved | 16:20 |
| reed | i learn things and forget them at an incredible pace :) | 16:20 |
| fungi | reed: how do we usually get in touch with evgeny? | 16:20 |
| reed | pnascimento, fungi: usually email him, why? | 16:21 |
| *** e0ne_ has joined #openstack-infra | 16:21 | |
| pleia2 | clarkb: thanks | 16:21 |
| fungi | reed: pnascimento is working on another askbot installation and wondering how we usually reach evgeny with questions/requests | 16:21 |
| fungi | something about trying to develop an oauth2 plugin for it i think | 16:22 |
| *** tnovacik has quit IRC | 16:23 | |
| *** penguinRaider__ has joined #openstack-infra | 16:23 | |
| reed | pnascimento, are you having troubles reaching him? | 16:23 |
| *** thedodd has joined #openstack-infra | 16:23 | |
| *** e0ne has quit IRC | 16:24 | |
| *** salv-orlando has quit IRC | 16:25 | |
| *** carl_baldwin has joined #openstack-infra | 16:25 | |
| pnascimento | reed: not yet, I must say I only contacted askbot community as whole through Askbot Q&A site | 16:25 |
| mrmartin | I'm here | 16:26 |
| pnascimento | reed: but fungi told me that Evgeny is the man to contact, so I was wondering which would be the correct (ans less evasive) way to contact him :) | 16:26 |
| pnascimento | s/ans/and/ | 16:26 |
| *** penguinRaider has quit IRC | 16:27 | |
| *** jgjl has quit IRC | 16:27 | |
| mrmartin | pnascimento: so what you want to ask regarding askbot? | 16:27 |
| clarkb | mordred for going back to not dir envs keep in mind we need to move things around on the master iirc | 16:27 |
| reed | pnascimento, https://github.com/ASKBOT/, askbot forum ... I have his personal email and the Foundation has a support contract | 16:27 |
| fungi | evgeny also has his e-mail address listed on his askbot forum profile welcoming people to contact him there, so presumably he doesn't mind | 16:30 |
| pnascimento | reed: I have submitted a question on askbot forum (Q&A site) mid-late of November :) and already read a from post from Evgeny | 16:30 |
| mrmartin | he is a bit busy so it take a several weeks to get some answer | 16:30 |
| pnascimento | mrmartin: on "my" askbot I'm trying to login ONLY with OAuth (custom), after I activated as Evgeny suggested (http://askbot.org/en/question/9819/how-can-i-add-an-oauth2-custom-login-provider-to-askbot/) I encountered a problem which disables every login system/plugin :/ | 16:31 |
| mrmartin | but the good news, that it is open-source and python, so if you have some reverse-engineering skills it is a great fün | 16:31 |
| *** zz_avozza is now known as avozza | 16:32 | |
| pnascimento | mrmartin: already started with reverse-engineering if you can call it that way... | 16:32 |
| pnascimento | and I have found that none of the login actions works :/ | 16:32 |
| *** boris-42 has quit IRC | 16:32 | |
| mrmartin | even after restore local login and clear redis / whatever cache you are using? | 16:33 |
| *** bdpayne has joined #openstack-infra | 16:33 | |
| *** annegent_ has joined #openstack-infra | 16:33 | |
| *** jcoufal has quit IRC | 16:33 | |
| *** dims__ has quit IRC | 16:33 | |
| *** bdpayne has quit IRC | 16:34 | |
| *** dims__ has joined #openstack-infra | 16:34 | |
| mrmartin | anyway an oauth2 login should be useful for us also, currently using openid | 16:34 |
| *** penguinRaider__ has quit IRC | 16:34 | |
| *** penguinRaider has joined #openstack-infra | 16:34 | |
| *** armax has joined #openstack-infra | 16:34 | |
| openstackgerrit | Merged openstack-infra/system-config: Puppet apply test creates puppet directory env https://review.openstack.org/141413 | 16:34 |
| pnascimento | clear cache? not yet... restore local login? tried, not unsuccessfull | 16:34 |
| mrmartin | what type of cache are you using? in-memory or you deployed redis or memcached? | 16:35 |
| *** zhiwei has quit IRC | 16:35 | |
| fungi | i'll get started rechecking changes now | 16:36 |
| pnascimento | python manage.py runserver points this out: "WARNING!!! You are using a 'locmem' (local memory) caching backend" | 16:36 |
| mrmartin | yeap, it means you not configured a real caching backend. | 16:36 |
| reed | doesn't google auth mechanism use OAuth2? | 16:36 |
| mrmartin | reed, yes it is. | 16:37 |
| mrmartin | but usually google oauth2 modules have hard-coded google service urls. | 16:37 |
| pnascimento | mrmartin: yes, I though so. But, using locmem after chaging login could be the problem? | 16:38 |
| *** penguinRaider__ has joined #openstack-infra | 16:38 | |
| sbadia | hi infra o/ | 16:38 |
| sbadia | I wondering if there was a procedure to deeply change/update a GIT repo in gerrit | 16:38 |
| *** dims__ has quit IRC | 16:38 | |
| mrmartin | no, you can go into db, enable the local login, remove the cache / restart your wsgi container | 16:38 |
| sbadia | It's for puppet-sahara (stackforge), the actual module was rewrite from ground, and we (puppet-openstack team) have decided to replace the actual by the new. | 16:38 |
| sbadia | Refs, https://groups.google.com/a/puppetlabs.com/forum/#!topic/puppet-openstack/0BETdNvrd70 | 16:38 |
| sbadia | I made a non-atomic big patch on the current repo, or is it possible to make a kind of reset with gerrit? | 16:38 |
| *** bdpayne has joined #openstack-infra | 16:39 | |
| pnascimento | reed: I more interested on OAuth 1, like Twitter or LinkedIn: https://github.com/ASKBOT/askbot-devel/blob/master/askbot/deps/django_authopenid/util.py#L477 | 16:40 |
| pnascimento | I have looked into this page in order to get some tipos how to solve my problem :/ | 16:41 |
| *** penguinRaider has quit IRC | 16:41 | |
| *** skolekonov has quit IRC | 16:41 | |
| clarkb | nibalizer: crinkle thank you for helping to look into this | 16:42 |
| *** penguinRaider has joined #openstack-infra | 16:42 | |
| *** penguinRaider__ has quit IRC | 16:43 | |
| *** dims__ has joined #openstack-infra | 16:44 | |
| *** luqas has quit IRC | 16:45 | |
| clarkb | sbadia: typically the best way to make changesl ike that is via a feature branch. you make changes to the side of master when everything is ready to go you merge it back into master | 16:46 |
| clarkb | sbadia: that way everyone can review the rewrite as it happens without needing to worry about the master tree | 16:46 |
| *** Hal_ has joined #openstack-infra | 16:47 | |
| clarkb | sbadia: my suggestion would be to create a feature branch, push your changes up into it, get review and when people are happy with it merge it into master | 16:47 |
| sbadia | make sense indeed, but… if we import the master of this new module will generate tones of review/mails | 16:48 |
| sbadia | I can maybe stash revelent changes and import thems to the feature branch | 16:49 |
| *** cody-somerville has joined #openstack-infra | 16:49 | |
| *** david-lyle_afk is now known as david-lyle | 16:49 | |
| sbadia | clarkb: thanks! | 16:50 |
| sbadia | no black magic solution then :) | 16:50 |
| *** luqas has joined #openstack-infra | 16:50 | |
| *** achanda has quit IRC | 16:50 | |
| clarkb | sbadia: no. the problem with black magic is it bypasses review and can potentially involve rewriting history which is bad | 16:51 |
| *** achanda has joined #openstack-infra | 16:51 | |
| *** Hal_ has quit IRC | 16:51 | |
| sbadia | I agree indeed | 16:51 |
| dmsimard | You guys probably know that paste.openstack.org is returning 500's on submitting ? | 16:52 |
| openstackgerrit | Merged openstack-infra/system-config: Add mysql-proxy to enable read-only access to a db https://review.openstack.org/122311 | 16:52 |
| mtreinish | dmsimard: try refreshing the page before you submit, it normally fixes the 500s for me... | 16:53 |
| clarkb | fungi: speaking of ^ I wonder if we should try adding that same reconnect to paste that you added to jeepyb | 16:53 |
| mtreinish | although I'm not sure why it does it | 16:53 |
| clarkb | dmsimard: mtreinish it has to do with paste losing its DB connection | 16:53 |
| fungi | clarkb: lodgeit uses sqlalchemy as an orm, and its supposed to do that itself already, so the paste issue may be unrelated | 16:53 |
| mtreinish | clarkb: ah, that would explain it | 16:54 |
| clarkb | fungi: oh hrm | 16:54 |
| *** penguinRaider__ has joined #openstack-infra | 16:54 | |
| openstackgerrit | Merged openstack-infra/system-config: Disable cgit's dumb HTTP protocol endpoint support https://review.openstack.org/140805 | 16:54 |
| openstackgerrit | Merged openstack-infra/system-config: Add Gerrit tracking id for storyboard https://review.openstack.org/140492 | 16:54 |
| mordred | clarkb: yeah - I'm fine with leaving the status quo for now | 16:55 |
| *** achanda has quit IRC | 16:56 | |
| *** penguinRaider has quit IRC | 16:57 | |
| *** dkranz has quit IRC | 16:57 | |
| *** andreykurilin_ has quit IRC | 16:58 | |
| openstackgerrit | Merged openstack-infra/system-config: Correct keystone log links https://review.openstack.org/139805 | 16:58 |
| openstackgerrit | Merged openstack-infra/system-config: Remove third-party-requests mailing list https://review.openstack.org/139123 | 16:58 |
| *** andreykurilin_ has joined #openstack-infra | 16:58 | |
| openstackgerrit | Merged openstack-infra/system-config: Turn off color in module install https://review.openstack.org/134872 | 16:59 |
| jeblair | clarkb: can you explain the bandersnatch deleted file problem? | 16:59 |
| *** shashankhegde has joined #openstack-infra | 17:00 | |
| *** penguinRaider has joined #openstack-infra | 17:00 | |
| *** penguinRaider__ has quit IRC | 17:00 | |
| jeblair | fungi: or maybe you? :) | 17:01 |
| *** achanda has joined #openstack-infra | 17:03 | |
| fungi | jeblair: the one where bandersnatch redeletes a file every time it runs for ~1 day even when we manually add it? | 17:03 |
| *** miqui has quit IRC | 17:03 | |
| jeblair | fungi: that sounds like it, yes | 17:03 |
| *** rkukura has joined #openstack-infra | 17:03 | |
| jeblair | fungi: i'm just trying to understand the mechanics of it | 17:04 |
| *** miqui_ has quit IRC | 17:04 | |
| *** miqui has joined #openstack-infra | 17:04 | |
| *** miqui_ has joined #openstack-infra | 17:04 | |
| fungi | jeblair: the way i understood it, there was a contributing race in pypi where it was possible to skip a file addition because the next serial was reported on a json blob without the file listed, and fastly would cache that the first time it was requested through the cdn | 17:05 |
| fungi | jeblair: and so bandersnatch would then thing the file's absence in the json implied a deletion, and would remove it if it existed | 17:05 |
| fungi | s/thing/think/ | 17:05 |
| fungi | jeblair: and would continue to find that incorrect json for the one-day cache duration on the cdn | 17:06 |
| *** mfink has quit IRC | 17:06 | |
| *** annegent_ has quit IRC | 17:07 | |
| fungi | ajjaegar seems to have gone through and rechecked a ton of project-config changes already | 17:07 |
| clarkb | fungi yup thats it | 17:07 |
| *** amuller has quit IRC | 17:07 | |
| sdague | clarkb: why aren't there aio jobs in experimental here - https://review.openstack.org/#/c/136504/7 ? | 17:07 |
| fungi | er, ajaeger | 17:07 |
| fungi | but i guess he's not in irc right now | 17:07 |
| clarkb | and the race in pypi is upload sdist and whl back to back first file wins second loses | 17:08 |
| *** Masahiro has joined #openstack-infra | 17:08 | |
| clarkb | sdague I dont know | 17:08 |
| dstufft | oh | 17:08 |
| dstufft | I forgot about that | 17:08 |
| clarkb | need to afk now. back after lunch | 17:09 |
| * dstufft goes to fix the race condition in pypi | 17:09 | |
| openstackgerrit | Merged openstack-infra/system-config: Adding more details about the server and link to wiki page https://review.openstack.org/135671 | 17:09 |
| fungi | dstufft: appreciated! ;) | 17:09 |
| mtreinish | sdague: it's because the dib experimental job has been stuck for >40hrs | 17:10 |
| openstackgerrit | Merged openstack-infra/system-config: Make localuser gid and uid required params https://review.openstack.org/140415 | 17:10 |
| mtreinish | so it hasn't reported back to gerrit yet | 17:10 |
| sdague | mtreinish: oh... fun | 17:10 |
| sdague | 40 hrs? | 17:10 |
| *** avozza is now known as zz_avozza | 17:10 | |
| sdague | it hasn't timed out yet? | 17:10 |
| mtreinish | sdague: it's waiting for a node | 17:10 |
| mtreinish | that's the count on status.o.o | 17:10 |
| openstackgerrit | Merged openstack-infra/system-config: Add system backups to storyboard https://review.openstack.org/138447 | 17:11 |
| fungi | sdague: zuul doesn't (yet) have a feature to time out pending jobs which never run | 17:11 |
| openstackgerrit | Merged openstack-infra/system-config: Enable SSL for groups.openstack.org https://review.openstack.org/138011 | 17:11 |
| *** mriedem has quit IRC | 17:11 | |
| openstackgerrit | Merged openstack-infra/system-config: Install lsof on the git mirror nodes https://review.openstack.org/136168 | 17:11 |
| openstackgerrit | Merged openstack-infra/system-config: Set 0 nodes for devstack-f20-virt-preview https://review.openstack.org/132977 | 17:11 |
| openstackgerrit | Merged openstack-infra/system-config: Add AFS docs https://review.openstack.org/129748 | 17:11 |
| openstackgerrit | Colleen Murphy proposed openstack-infra/project-config: Only guard yum update against epel if enabled https://review.openstack.org/141288 | 17:11 |
| openstackgerrit | Merged openstack-infra/system-config: Run puppet even if git servers fail https://review.openstack.org/129722 | 17:11 |
| *** penguinRaider__ has joined #openstack-infra | 17:12 | |
| fungi | sdague: jogo has a change proposed to temporarily stop running that job until someone has a chance to look into why we can't boot those experimental nodes. even with a fresh dib-generated image uploaded, nodepool reports that nova is booting them into error state | 17:12 |
| sdague | fungi: ok, where's that change? | 17:12 |
| jeblair | EOPENSTACK | 17:12 |
| *** Masahiro has quit IRC | 17:13 | |
| jeblair | sdague: 141245 | 17:13 |
| mtreinish | fungi: it looks like the mysql-proxy change got applied: http://puppetboard.openstack.org/report/logstash.openstack.org/5b9d528ab004a761abd6828bdfc45aa367532b76 | 17:14 |
| sdague | yeh, I found it, centos6 job wacked it | 17:14 |
| mtreinish | but I'm not able to connect to it, did the service fail to start? | 17:14 |
| mestery | infra folks: Quick question: I had some issues with my DNS around MX records which resulted in my MX record disappearing for hte past 12 hours (now fixed). Would I need to re-subcribe to the mailing lists due to some auto unsubscribe thing perhaps? | 17:14 |
| openstackgerrit | Merged openstack-infra/system-config: Logstash: Modifying rewrite rules to allow kibana 3 https://review.openstack.org/93167 | 17:15 |
| fungi | mtreinish: i'll check in a bit | 17:15 |
| *** dtantsur is now known as dtantsur|afk | 17:15 | |
| openstackgerrit | Merged openstack-infra/project-config: Add docs to python-barbicanclient gates https://review.openstack.org/139259 | 17:15 |
| *** penguinRaider has quit IRC | 17:15 | |
| jeblair | mestery: probably not, if you give me your address i can check the mail queue on the list server | 17:15 |
| mtreinish | mestery: for only 12hours? I've had my mail server crash for longer and the ml eventually trickles back in | 17:15 |
| mtreinish | fungi: cool, thanks | 17:16 |
| mestery | mtreinish: Cool, thanks jeblair, PM'd you my email | 17:16 |
| *** salv-orlando has joined #openstack-infra | 17:17 | |
| *** otter768 has joined #openstack-infra | 17:17 | |
| fungi | mestery: also if you log into the mailman webui it should show you if your subscription has been suspended due to excessive bounces | 17:17 |
| mestery | fungi: Ack, thanks sir! | 17:18 |
| *** otter768 has quit IRC | 17:21 | |
| *** talluri has joined #openstack-infra | 17:22 | |
| *** penguinRaider__ has quit IRC | 17:22 | |
| *** penguinRaider has joined #openstack-infra | 17:22 | |
| *** gyee has joined #openstack-infra | 17:22 | |
| *** signed8bit has quit IRC | 17:23 | |
| *** doug-fish has quit IRC | 17:23 | |
| *** _nadya_ has quit IRC | 17:23 | |
| *** eharney has joined #openstack-infra | 17:24 | |
| *** jogo is now known as flashgordon | 17:24 | |
| jeblair | mestery: whatever the issue was, it was such that exim considered the address unroutable and bounced messages to it, so i don't think you will recieve messages sent during the broken period. but it seems to be delivering mail to you now. | 17:24 |
| mestery | jeblair: Cool, thanks for verifying! | 17:24 |
| jeblair | np | 17:24 |
| *** jaypipes is now known as leakypipes | 17:25 | |
| openstackgerrit | Merged openstack-infra/project-config: Add compass-install project to stackforge https://review.openstack.org/140465 | 17:25 |
| openstackgerrit | Michael Krotscheck proposed openstack-infra/storyboard: Plugins may now register cron workers. https://review.openstack.org/129609 | 17:25 |
| *** dansmith is now known as superdan | 17:26 | |
| *** penguinRaider__ has joined #openstack-infra | 17:26 | |
| *** hashar has quit IRC | 17:26 | |
| jeblair | fungi, dstufft, clarkb: if the json data of the file list is missing a file, how does it get downloaded? | 17:27 |
| dstufft | it doesn't, other than I think y'all manually drop the file into place | 17:28 |
| fungi | jeblair: dstufft: or refreshing the entire mirror from scratch also seems to resolve it, once the broken json is no longer cached in the cdn | 17:29 |
| *** shakamunyi has joined #openstack-infra | 17:29 | |
| *** ryanpetrello_ has joined #openstack-infra | 17:29 | |
| jeblair | fungi: oh, why do we do that? because openstack is broken and needs a new release of something to fix it? | 17:29 |
| *** tylerdurden has joined #openstack-infra | 17:29 | |
| jeblair | and we can't wait 24h for cache expiration? | 17:29 |
| fungi | jeblair: no, because bandersnatch retrieves the file index, it doesn't generate the file index, so it's often linking the most recent version of some package to a nonexistent file on our mirror | 17:30 |
| *** penguinRaider has quit IRC | 17:30 | |
| *** fandi has joined #openstack-infra | 17:30 | |
| fungi | so some job says install the most recent versions of listed requirements, one of which is a 404 on the mirror, job fails | 17:30 |
| *** doug-fish has joined #openstack-infra | 17:30 | |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config: Introduce openstack-server-release-jobs template https://review.openstack.org/140326 | 17:31 |
| fungi | and because the bandersnatch sync breakage is deterministic, when it happens it tends to happen across all our mirrors, so the jobs all fail on a 404 no matter where they run | 17:31 |
| *** ryanpetrello has quit IRC | 17:31 | |
| *** ryanpetrello_ is now known as ryanpetrello | 17:31 | |
| *** shakamunyi has quit IRC | 17:31 | |
| *** shashankhegde has quit IRC | 17:32 | |
| *** yjiang5 is now known as gingeryjiang | 17:32 | |
| jeblair | fungi: so this is two bugs -- that race, which alone can cause an issue, but is correctable in the next run; but then if that happens to a package with a dual-upload where only one file is recorded in the json, then it's stuck for 24h and we can't just drop the file in because it will be deleted? | 17:32 |
| *** scheuran has quit IRC | 17:33 | |
| *** gingeryjiang is now known as gingerjiang | 17:33 | |
| fungi | jeblair: pretty much, unless we manually add the file and stop running bandersnatch until the cache is better | 17:33 |
| jeblair | the second bug could also happen without the first, and we would only notice it if we were expecting a file and it didn't show up | 17:33 |
| jeblair | right? | 17:34 |
| *** openstackgerrit has quit IRC | 17:34 | |
| jeblair | or, actually, is it always going to show up with the first because of the index page? | 17:34 |
| *** openstackgerrit has joined #openstack-infra | 17:34 | |
| *** mfink has joined #openstack-infra | 17:34 | |
| openstackgerrit | Merged openstack-infra/project-config: Add requirements check for tooz https://review.openstack.org/140923 | 17:35 |
| fungi | jeblair: there are other failure modes which can also expose that second issue | 17:35 |
| openstackgerrit | Merged openstack-infra/project-config: sort oslo channels https://review.openstack.org/140933 | 17:35 |
| openstackgerrit | Merged openstack-infra/project-config: Check requirements for taskflow https://review.openstack.org/140764 | 17:35 |
| fungi | jeblair: for example, bandersnatch will proceed, under some circumstances, to cache an updated index pointing to a file it failed to retrieve | 17:35 |
| openstackgerrit | Merged openstack-infra/project-config: Make check-heat-dsvm-functional-mysql voting https://review.openstack.org/138940 | 17:36 |
| zaro | waynr: about plugin versions.. | 17:36 |
| openstackgerrit | Merged openstack-infra/system-config: Update comment in logs.vhost https://review.openstack.org/129861 | 17:36 |
| fungi | assuming i understand what you're referring to as the second issue | 17:36 |
| openstackgerrit | Merged openstack-infra/system-config: Prevents double-execution of ci_page_loaded https://review.openstack.org/125279 | 17:37 |
| openstackgerrit | Elizabeth K. Joseph proposed openstack-infra/system-config: Remove comment about old 00-puppet.pref file https://review.openstack.org/141439 | 17:37 |
| jeblair | dstufft: why does bandersnatch copy the index instead of generate it? | 17:37 |
| waynr | zaro: i'm listening | 17:37 |
| zaro | waynr: how about you update change 132927 with your comments from irc then i'll update with mine so we can get feedback from others? | 17:37 |
| *** annegent_ has joined #openstack-infra | 17:38 | |
| waynr | okay | 17:38 |
| *** andreykurilin_ has quit IRC | 17:38 | |
| dstufft | jeblair: previously because it was mandated to do so. Part of the mirroring spec used to be that the index file was signed by PyPI and those signatures were mirrored too, and things like pip could verify the mirror wasn't malicous by verifying the signature of the index | 17:38 |
| zaro | waynr: does the suggestion about shortname make sense? | 17:38 |
| dstufft | we deprecated that though because nobody actually did anything with it | 17:38 |
| jeblair | got it, thx | 17:39 |
| *** andreykurilin_ has joined #openstack-infra | 17:39 | |
| waynr | zaro: yeah, the implementation i am currently working on is adding a key to a given plugin's dict for whatever is available among longName and shortName | 17:40 |
| *** barra204 has joined #openstack-infra | 17:40 | |
| fungi | jeblair: clarkb: SergeyLukjanov: jhesketh: we should probably remember to approve 141415 sometime today, so that we have a good chance of updated images next week and can revert the workaround | 17:40 |
| SergeyLukjanov | fungi, mornng | 17:41 |
| waynr | erh, what i said isn't quite accurate | 17:41 |
| SergeyLukjanov | fungi, looking on the patch | 17:41 |
| dstufft | fungi: clarkb jeblair https://bitbucket.org/pypa/pypi/commits/6c974b11ac3f850f8830a682703edb57a3f8d0e4 | 17:41 |
| waynr | so the overall dictionary of plugins will potentially have 1 or 2 keys referring to each plugin: one key for each of shortName and longName | 17:41 |
| *** pc_m is now known as pc_m_away | 17:41 | |
| *** penguinRaider has joined #openstack-infra | 17:42 | |
| *** penguinRaider__ has quit IRC | 17:42 | |
| *** annegent_ has quit IRC | 17:42 | |
| mtreinish | fungi: hmm, now I'm able to connect to the proxy, but it errors out with: "ERROR 1043 (08S01): Bad handshake" | 17:43 |
| mtreinish | googling yielded: http://bugs.mysql.com/bug.php?id=69432 which looks similar | 17:43 |
| fungi | mtreinish: sounds like maybe an ssl issue? | 17:43 |
| SergeyLukjanov | fungi, +2 from me | 17:43 |
| dstufft | fungi: I just deployed that and purged everything from Fastly | 17:44 |
| *** lttrl has quit IRC | 17:44 | |
| dstufft | fungi: y'all might want to do a full sync on bandersnatch | 17:44 |
| fungi | dstufft: that was quick--thanks! | 17:44 |
| mtreinish | fungi: yeah, it might be. Although, now I can't connect anymore... | 17:44 |
| openstackgerrit | Michael Krotscheck proposed openstack-infra/storyboard: Plugins may now register cron workers. https://review.openstack.org/129609 | 17:44 |
| fungi | dstufft: not a bad idea. it's a slow day anyway so i don't expect any of our projects are probably releasing things they need mirrored asap | 17:44 |
| fungi | mtreinish: i'll pop onto the server now and take a peek | 17:45 |
| dstufft | fungi: turns out we were already using transactions, it's just the default transaction level in postgresql is READ COMMITTED which means that subsequent queries in the same transaction will see data that was commited to the DB in other tranasactions after the start of the current transaction | 17:45 |
| fungi | dstufft: indeed, that was unfortunate | 17:46 |
| dstufft | fungi: so I just switched it so the stuff that bandersnatch uses for deteremining the mirror state was using a stricter transaction isolation level | 17:46 |
| fungi | simple and straighthforward | 17:46 |
| jeblair | fungi: aprvd | 17:47 |
| dstufft | assuming this fixes the pypi race condition (which it should), that should eliminate the persistent broken mirror | 17:47 |
| fungi | dstufft: thanks again as always | 17:47 |
| fungi | mtreinish: i see the problem | 17:48 |
| *** e0ne_ has quit IRC | 17:48 | |
| *** pnascimento has quit IRC | 17:48 | |
| fungi | mtreinish: 2014-12-12 17:22:26: (message) added read/write backend: 127.0.0.1:3306 | 17:48 |
| dstufft | if someone uploads a file in between when you fetch the JSON and when you fetch the index you'll still be broken, but that'll resolve itself the next time you sync, (or should anyways), and that can be solved by https://bitbucket.org/pypa/bandersnatch/issue/54/reverse-the-race-condition-to-make-it-less | 17:48 |
| fungi | mtreinish: 2014-12-12 17:35:49: (message) proxy-plugin.c.1503: connect(127.0.0.1:3306) failed: Connection refused. | 17:48 |
| mtreinish | fungi: oh, is mysql running on logstash.o.o? | 17:49 |
| *** ivar-lazzaro has joined #openstack-infra | 17:49 | |
| fungi | mtreinish: nope | 17:49 |
| *** openstackgerrit has quit IRC | 17:49 | |
| *** openstackgerrit has joined #openstack-infra | 17:49 | |
| *** ihrachyshka has quit IRC | 17:49 | |
| fungi | mtreinish: i think mysql-proxy assumes a default rw backend on localhost if you don't tell it otherwise, and then goes toes-up when it can't access that | 17:49 |
| fungi | mtreinish: it's also connecting a read-only backend to the trove instance, which looks fine | 17:50 |
| mtreinish | fungi: hmm, I think I remember that on my local tests and it still worked ok, let me spin it up again | 17:50 |
| *** ivar-laz_ has joined #openstack-infra | 17:51 | |
| *** bdpayne has quit IRC | 17:51 | |
| *** jlibosva has quit IRC | 17:51 | |
| mtreinish | fungi: is it running though? because I'm still not able to connect | 17:52 |
| *** derekh has quit IRC | 17:52 | |
| fungi | mtreinish: no, it looks like it starts up, tries to reach its nonexistent rw backend, times out on that after a bit, then exits. and then puppet starts it again on the next run | 17:52 |
| openstackgerrit | Marton Kiss proposed openstack-infra/system-config: Initial commit of Ask website https://review.openstack.org/140043 | 17:53 |
| mtreinish | hmm, I wonder why I was able to get a response for a bit then? | 17:53 |
| fungi | mtreinish: it binds to the listening sockets right away | 17:53 |
| fungi | mtreinish: even though it's apparently not done initializing its backends | 17:53 |
| *** sandywalsh has quit IRC | 17:54 | |
| *** ivar-lazzaro has quit IRC | 17:54 | |
| *** andreykurilin_ has quit IRC | 17:54 | |
| *** penguinRaider__ has joined #openstack-infra | 17:54 | |
| *** bdpayne has joined #openstack-infra | 17:55 | |
| *** sandywalsh has joined #openstack-infra | 17:55 | |
| *** shashankhegde has joined #openstack-infra | 17:56 | |
| mtreinish | fungi: ok, I'm able to connect again, and it seems to be staying up | 17:56 |
| mordred | fungi, jeblair: where is the code we're using to upload to swift? | 17:57 |
| *** dims__ is now known as dimsum__ | 17:57 | |
| *** sarob has joined #openstack-infra | 17:57 | |
| mtreinish | fungi: I think I might have just crashed it before when I tried the telnet bug reproduce... :) | 17:58 |
| waynr | zaro: comment added | 17:58 |
| fungi | mtreinish: oh | 17:58 |
| *** penguinRaider has quit IRC | 17:58 | |
| fungi | mtreinish: yeah because so far i see it starting, per the log, at 17:22:26, 17:47:41 and 17:52:55 | 17:59 |
| fungi | mtreinish: maybe it's just fragile? | 17:59 |
| *** annegent_ has joined #openstack-infra | 17:59 | |
| * mordred is now learning about python-swiftclient | 17:59 | |
| mtreinish | fungi: yeah probably | 17:59 |
| mtreinish | fungi: it's up now, but still throwing bad handshake | 17:59 |
| mordred | and the swift commandline client is too well written, so I cannot use it for examples of how to use python-swiftclient to do things | 17:59 |
| openstackgerrit | Merged openstack-infra/jenkins-job-builder: adding support for the custom tools plugin https://review.openstack.org/134011 | 18:00 |
| fungi | mordred: http://git.openstack.org/cgit/openstack-infra/project-config/tree/jenkins/scripts/zuul_swift_upload.py | 18:00 |
| fungi | mordred: it does not appear to use swiftclient though, uses http api | 18:01 |
| fungi | mordred: which i think is the recommended way of uploading things to swift anyway | 18:02 |
| fungi | (at least from within other software) | 18:02 |
| fungi | hopefully someone will correct me if that's not true after all ;) | 18:03 |
| *** thedodd has quit IRC | 18:03 | |
| *** salv-orlando has quit IRC | 18:03 | |
| mtreinish | fungi: I just spun up the policy on a rax node to connect to a test db I have running on hp cloud and it has the same 127.0.0.1 connection refused error in the logs. But, the proxy works fine. | 18:04 |
| fungi | mtreinish: okay, in that case it just silently dies on some bad inputs i guess and gets restarted the next time puppet runs and notices it's not up | 18:04 |
| fungi | mtreinish: if you figure out how to crash it, that would probably be good info | 18:05 |
| fungi | though it sounded like this software may be unmaintained at this point? | 18:05 |
| mtreinish | fungi: telnet logstash.openstack.org 4040 seems to do the trick :) | 18:05 |
| fungi | ouch | 18:05 |
| mtreinish | fungi: there was a big gap between a couple of releases | 18:05 |
| fungi | doesn't sound like something we want to rely on until that's better | 18:06 |
| *** shashankhegde has quit IRC | 18:06 | |
| mtreinish | but the most recent release was from sept | 18:06 |
| mtreinish | http://dev.mysql.com/doc/relnotes/mysql-proxy/en/mysql-proxy-news-0-8-5.html | 18:06 |
| *** doug-fish has quit IRC | 18:06 | |
| mtreinish | I think the package in ubuntu is old though | 18:06 |
| fungi | oh, then maybe they're receptive to bug reports/patches | 18:06 |
| fungi | aha, could be | 18:06 |
| *** doug-fish has joined #openstack-infra | 18:06 | |
| fungi | 0.8.1 is what we're running | 18:06 |
| mordred | fungi: wait- you're saying that "not using python-swiftclient" is the recommended way to do it? | 18:06 |
| mtreinish | yeah I think the telnet crash got fixed in 0.8.3 | 18:07 |
| *** bdpayne has quit IRC | 18:07 | |
| mtreinish | which is from 2012 | 18:07 |
| fungi | mordred: that was what i gleaned from the discussions a while back on using formpost to handle uploads | 18:07 |
| * mordred sobs like a baby | 18:07 | |
| fungi | mtreinish: so maybe file an ubuntu bug in lp and ask for a working version or at least a backport of that fix | 18:08 |
| mtreinish | fungi: ok I can do that | 18:08 |
| *** ivar-laz_ has quit IRC | 18:08 | |
| mtreinish | but I think the handshake issue might be unrelated | 18:08 |
| *** penguinRaider__ has quit IRC | 18:08 | |
| fungi | mtreinish: it's in universe rather than main, so we might have to convince someone to do that work (or do it ourselves) | 18:09 |
| *** ivar-lazzaro has joined #openstack-infra | 18:09 | |
| *** penguinRaider has joined #openstack-infra | 18:09 | |
| openstackgerrit | Merged openstack-infra/system-config: Puppet the puppetmaster apache vhost https://review.openstack.org/128707 | 18:09 |
| *** bdpayne has joined #openstack-infra | 18:09 | |
| openstackgerrit | Merged openstack-infra/project-config: Change os-ansible-aio-build check settings https://review.openstack.org/141072 | 18:10 |
| fungi | mtreinish: i can connect to it | 18:10 |
| openstackgerrit | Merged openstack-infra/project-config: Remove dibtest from experimental queue https://review.openstack.org/141245 | 18:11 |
| fungi | mtreinish: mysql -u query -p -h logstash.openstack.org -P 4040 subunit2sql | 18:11 |
| openstackgerrit | Merged openstack-infra/project-config: Add neutron jobs to service repos, until they get their own tests https://review.openstack.org/141122 | 18:11 |
| mtreinish | oh, so it must be on my end... sigh | 18:11 |
| mtreinish | oh now I get a different error: ERROR 1105 (HY000): (proxy) all backends are down | 18:11 |
| *** shashankhegde has joined #openstack-infra | 18:12 | |
| fungi | mtreinish: yep, that's what i'm getting too | 18:12 |
| fungi | checking the log for anything obvious | 18:12 |
| fungi | mtreinish: http://paste.openstack.org/show/150019/ | 18:13 |
| *** jistr has quit IRC | 18:14 | |
| fungi | mtreinish: i'll double-check socket connectivity to the trove instance from the server | 18:14 |
| *** mfink__ has joined #openstack-infra | 18:14 | |
| openstackgerrit | Michael Krotscheck proposed openstack-infra/storyboard: Added cron plugin to clean old access tokens. https://review.openstack.org/140536 | 18:14 |
| mtreinish | fungi: ok, thanks. you can also check the registered backends by connecting the proxy with mysql -u admin -p localhost --port 4041 | 18:15 |
| *** mmaglana has joined #openstack-infra | 18:16 | |
| *** pblaho has quit IRC | 18:16 | |
| fungi | mtreinish: connection to the configured trove instance checks out at least | 18:16 |
| *** penguinRaider has quit IRC | 18:17 | |
| *** harlowja_away is now known as harlowja | 18:17 | |
| mtreinish | err, nm it doesn't look like the admin interface works on localhost, only remotely. Which is kinda weird... | 18:17 |
| nibalizer | i just -1'd https://review.openstack.org/#/c/141415/ ... if someone wants to try to -2 it before it goes | 18:17 |
| nibalizer | we need to move basemodulepath as well im pretty sure | 18:18 |
| fungi | mtreinish: yeah, i got to it via 127.0.0.1 | 18:18 |
| fungi | mtreinish: had to be tcp not unix socket | 18:18 |
| *** mfink has quit IRC | 18:18 | |
| mtreinish | ah, ok | 18:18 |
| fungi | mtreinish: it shows two backends. i'll put it on paste.o.o | 18:18 |
| mtreinish | yeah 2 is what would be expected, locahost which is down and trove which should be up | 18:19 |
| fungi | mtreinish: http://paste.openstack.org/show/150023/ | 18:20 |
| *** HeOS has quit IRC | 18:20 | |
| fungi | mtreinish: seems it wants to connect all requests to the downed rw backend? | 18:20 |
| *** hdd has quit IRC | 18:20 | |
| *** mmaglana_ has joined #openstack-infra | 18:20 | |
| annegent_ | jeblair: mtreinish: do either/both of you have time for a cross-project-ptl meeting next week? I want to talk out loud about support levels and how you currently set expectations with teams. | 18:20 |
| mtreinish | yeah, that's what it looks like, which is weird because it's status is down... | 18:20 |
| openstackgerrit | Michael Krotscheck proposed openstack-infra/storyboard: Plugins may now register cron workers. https://review.openstack.org/129609 | 18:21 |
| mtreinish | annegent_: yeah, I should have time for that | 18:21 |
| *** patrickeast has joined #openstack-infra | 18:21 | |
| annegent_ | I'll send a meeting invite when I hear if jeblair is around | 18:21 |
| annegent_ | who else might be good to talk to? Thierry I suppose. | 18:22 |
| mtreinish | annegent_: yeah, ttx would probably have good insight | 18:22 |
| *** amitgandhinz has quit IRC | 18:22 | |
| mtreinish | fungi: hmm, I wonder if something is weird with the lua script? | 18:22 |
| *** mmaglana has quit IRC | 18:23 | |
| fungi | yep... that's the "big three" (infra, qa, release management) | 18:23 |
| *** leakypipes is now known as leaky-tennis | 18:23 | |
| fungi | well, "big four" (+ docs) | 18:23 |
| annegent_ | fungi: HMMMMMMmmmmm now I know what you think! | 18:23 |
| mtreinish | fungi: I did do my tests on 14.10, although I don't think the package changed between releases | 18:24 |
| fungi | mtreinish: yeah, the version on trusty seems to be the same version on later ubuntu too | 18:25 |
| fungi | mtreinish: oh, precise... comparing | 18:25 |
| *** rmcall has quit IRC | 18:25 | |
| annegent_ | horizontal teams named in the governance post now also include stable maintenance, vulnerability management, translators | 18:25 |
| *** hdd has joined #openstack-infra | 18:26 | |
| annegent_ | is vulnerability management the same as security team? | 18:26 |
| fungi | annegent_: vulnerability management and stable maintenance are part of release cycle management | 18:26 |
| mtreinish | fungi: they're all the same | 18:26 |
| mtreinish | fungi: https://bugs.launchpad.net/ubuntu/+source/mysql-proxy/+bug/1402011 | 18:26 |
| fungi | mtreinish: yep, confirmed | 18:26 |
| uvirtbot | Launchpad bug 1402011 in mysql-proxy "telnet crashes mysql-proxy" [Undecided,New] | 18:27 |
| *** bdpayne has quit IRC | 18:27 | |
| fungi | annegent_: but yeah, daisy might be good to include in a horizontal effort ptls discussion | 18:27 |
| *** rmcall has joined #openstack-infra | 18:28 | |
| *** russellb is now known as rustlebee | 18:29 | |
| *** zz_sabari is now known as sabari | 18:29 | |
| mtreinish | fungi: how hard would it be to just use tcpproxy from logstash.o.o to connect to the trove instance? I'm wondering if we should just abandon the mysql-proxy | 18:30 |
| *** rmcall has quit IRC | 18:30 | |
| *** jedimike has quit IRC | 18:30 | |
| *** tonytan4ever has quit IRC | 18:30 | |
| *** Ryan_Lane has joined #openstack-infra | 18:30 | |
| fungi | mtreinish: probably not hard at all. the query account is already only granted select on that one database, tested fairly rigorously | 18:31 |
| *** rmcall has joined #openstack-infra | 18:31 | |
| *** atiwari has joined #openstack-infra | 18:31 | |
| mtreinish | fungi: ok, I'll push out some policy for that. Are there any examples of using it in system-config? | 18:32 |
| openstackgerrit | Michael Krotscheck proposed openstack-infra/storyboard: Added cron plugin to clean old access tokens. https://review.openstack.org/140536 | 18:32 |
| mtreinish | fungi: because even if we fix the backend issue, there's still the telnet dos to worry about | 18:32 |
| fungi | mtreinish: i don't think so | 18:32 |
| *** talluri has quit IRC | 18:33 | |
| *** mpaolino has quit IRC | 18:34 | |
| jeblair | annegent_: absolutely, thanks! | 18:39 |
| jeblair | mordred, fungi: did you see nibalizer's comment on 141415? | 18:40 |
| fungi | jeblair: nibalizer: thanks, i'll amend it | 18:41 |
| nibalizer | cool | 18:41 |
| jeblair | i revoked my vote; it was about to merge | 18:41 |
| *** tonytan4ever has joined #openstack-infra | 18:42 | |
| openstackgerrit | Jeremy Stanley proposed openstack-infra/system-config: Only use environments on the master https://review.openstack.org/141415 | 18:42 |
| fungi | jeblair: nibalizer: mordred: ^ | 18:42 |
| nibalizer | woot +1 | 18:43 |
| fungi | worth noting, now that the job is passing most of the time, this run hit a 30min timeout... http://logs.openstack.org/79/139879/1/check/gate-infra-puppet-apply-centos6/83b3358/console.html | 18:45 |
| fungi | this one too... http://logs.openstack.org/86/141286/2/check/gate-infra-puppet-apply-centos6/0954fa2/console.html | 18:46 |
| fungi | different changes | 18:46 |
| openstackgerrit | Merged openstack-infra/project-config: Fix neutron jobs for tempest on stable branches https://review.openstack.org/140503 | 18:46 |
| fungi | wow--this one didn't even make it past cloning? http://logs.openstack.org/73/141173/1/check/gate-infra-puppet-apply-centos6/83f59cc/console.html | 18:47 |
| nibalizer | fungi: that first one looks like maybe it timed out trying to use a git resource | 18:48 |
| nibalizer | it would be super verbose, but we could add --evaltrace to the puppet apply command in apply-test.sh to see the 'starting resoruce X, finished resource X, X took Y seconds' | 18:49 |
| *** annegent_ has quit IRC | 18:49 | |
| *** annegent_ has joined #openstack-infra | 18:51 | |
| openstackgerrit | Merged openstack-infra/project-config: ACL update for group-based-policy proj to create branches https://review.openstack.org/139879 | 18:51 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul: Set gearman timeout to 300 https://review.openstack.org/141460 | 18:52 |
| *** palecrow has joined #openstack-infra | 18:53 | |
| jeblair | fungi, nibalizer: do we want to _remove_ basemodulepath from main? | 18:53 |
| jeblair | come to think of it... | 18:53 |
| *** mikedillion has joined #openstack-infra | 18:53 | |
| jeblair | nibalizer: why wasn't the original patch correct? shouldn't it be safe to have basemodulepath in main, so that it applies everywhere? | 18:53 |
| *** palecrow has quit IRC | 18:54 | |
| fungi | jeblair: spotted a typo in 141460 | 18:55 |
| nibalizer | basemodulepath, environmentpath, and environmenttimeout are all directory environment config options | 18:55 |
| jeblair | oops | 18:55 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul: Set gearman timeout to 300 https://review.openstack.org/141460 | 18:56 |
| nibalizer | modulepath is the thing that maybe we want everywhere... but we specify the modulepath on the command line in apply-test.sh so that isn't an issue | 18:56 |
| jeblair | nibalizer: ok. confusingly named option then. :) should be more like environmentmodulepath. | 18:57 |
| *** aysyd has quit IRC | 18:57 | |
| *** Masahiro has joined #openstack-infra | 18:57 | |
| nibalizer | yep | 18:57 |
| jeblair | +3 | 18:57 |
| *** aysyd has joined #openstack-infra | 18:58 | |
| openstackgerrit | Matthew Treinish proposed openstack-infra/system-config: Switch mysql-proxy to simpleproxy https://review.openstack.org/141462 | 18:59 |
| mtreinish | fungi: ^^^ I think that should work | 18:59 |
| jeblair | mtreinish: would you mind throwing some "ensure => absent" lines in there for the bits you're getting rid of? | 19:00 |
| *** dprince has quit IRC | 19:00 | |
| *** sarob has quit IRC | 19:00 | |
| mtreinish | jeblair: ok, sure one sec | 19:00 |
| *** ivar-laz_ has joined #openstack-infra | 19:01 | |
| jeblair | mtreinish: (usually it's sufficient to leave the resource definition, remove all the attributes other than ensure, then change ensure to absent) | 19:01 |
| *** ivar-laz_ has quit IRC | 19:01 | |
| *** dizquierdo has quit IRC | 19:01 | |
| *** Masahiro has quit IRC | 19:01 | |
| *** lttrl has joined #openstack-infra | 19:01 | |
| jeblair | so like: file { '/etc/mysql-proxy/mysql-proxy.conf': \n ensure => absent \n } | 19:01 |
| *** ivar-laz_ has joined #openstack-infra | 19:02 | |
| fungi | mtreinish: also, the puppet module may be a bit misnamed now? | 19:02 |
| mtreinish | fungi: well it's still setting up a proxy to mysql :) | 19:03 |
| *** boris-42 has joined #openstack-infra | 19:03 | |
| mtreinish | I can rename it to something else though | 19:03 |
| mtreinish | any suggestions? | 19:03 |
| openstackgerrit | Merged openstack-infra/system-config: Only use environments on the master https://review.openstack.org/141415 | 19:03 |
| fungi | mtreinish: meh, it is pretty single-purpose for the moment | 19:03 |
| openstackgerrit | Matthew Treinish proposed openstack-infra/system-config: Switch mysql-proxy to simpleproxy https://review.openstack.org/141462 | 19:03 |
| fungi | mtreinish: if you parameterized it further then maybe socket_proxy or something | 19:03 |
| mtreinish | jeblair: ^^^ like that? | 19:03 |
| *** salv-orlando has joined #openstack-infra | 19:04 | |
| *** MaxV has quit IRC | 19:04 | |
| *** ivar-lazzaro has quit IRC | 19:04 | |
| jeblair | yep, though i'm about to leave further comments | 19:04 |
| *** palecrow has joined #openstack-infra | 19:05 | |
| mtreinish | heh, I didn't expect to get it right on my first shot :) | 19:05 |
| *** sarob has joined #openstack-infra | 19:06 | |
| *** salv-orlando has quit IRC | 19:09 | |
| openstackgerrit | Jeremy Stanley proposed openstack-infra/system-config: Revert "Puppet apply test creates puppet directory env" https://review.openstack.org/141468 | 19:10 |
| *** vigneshvar has joined #openstack-infra | 19:10 | |
| palecrow | Not sure if this is the right venue...but I was running tox -e py34 on horizon...is that supposed to succeed? | 19:11 |
| fungi | palecrow: probably eventually, but i wouldn't expect it to succeed any time soon | 19:11 |
| palecrow | ok, thanks! | 19:11 |
| fungi | palecrow: we don't currently gate horizon on python 3 support, as i think it has quite a few dependencies which still don't support python 3 | 19:12 |
| *** sarob has quit IRC | 19:12 | |
| *** dpaterson has quit IRC | 19:14 | |
| *** dpaterson has joined #openstack-infra | 19:15 | |
| *** amitgandhinz has joined #openstack-infra | 19:16 | |
| *** esker has quit IRC | 19:16 | |
| *** otter768 has joined #openstack-infra | 19:18 | |
| *** salv-orlando has joined #openstack-infra | 19:18 | |
| *** salv-orlando has quit IRC | 19:18 | |
| *** unicell has quit IRC | 19:18 | |
| *** talluri has joined #openstack-infra | 19:19 | |
| *** dannywilson has joined #openstack-infra | 19:20 | |
| *** dannywil_ has joined #openstack-infra | 19:20 | |
| *** dannywilson has quit IRC | 19:20 | |
| *** palecrow has quit IRC | 19:21 | |
| *** otter768 has quit IRC | 19:22 | |
| *** hdd has quit IRC | 19:23 | |
| *** rmcall has quit IRC | 19:25 | |
| *** esker has joined #openstack-infra | 19:25 | |
| openstackgerrit | Matthew Treinish proposed openstack-infra/system-config: Switch mysql-proxy to simpleproxy https://review.openstack.org/141462 | 19:25 |
| mtreinish | jeblair: ^^^ I think I got the init file right there | 19:25 |
| *** rmcall has joined #openstack-infra | 19:27 | |
| *** ChuckC has quit IRC | 19:27 | |
| mordred | spectacular | 19:28 |
| *** salv-orlando has joined #openstack-infra | 19:28 | |
| *** salv-orlando has quit IRC | 19:29 | |
| *** talluri has quit IRC | 19:29 | |
| fungi | mrmartin: if you're around, groups.openstack.org is successfully redirecting to and serving via https now | 19:29 |
| openstackgerrit | Matthew Treinish proposed openstack-infra/system-config: Switch mysql-proxy to simpleproxy https://review.openstack.org/141462 | 19:29 |
| mrmartin | I'm here, checking now | 19:29 |
| openstackgerrit | Merged openstack-infra/project-config: Support name change in devstack to debs https://review.openstack.org/141177 | 19:30 |
| *** pc_m_away is now known as pc_m | 19:30 | |
| mtreinish | mordred: what is? | 19:30 |
| mrmartin | fungi: thank you, I'll rolling out the related patches on monday | 19:30 |
| mordred | mtreinish: was just reading the simpleproxy patch :) | 19:30 |
| *** _shaps__ has quit IRC | 19:30 | |
| mtreinish | mordred: heh, that's what I though :) The DOS by telnet...? | 19:31 |
| *** hdd has joined #openstack-infra | 19:32 | |
| *** salv-orlando has joined #openstack-infra | 19:33 | |
| mtreinish | fungi: about your comment on the simpleproxy, that dir only has one file which was set to absent in server.pp. Do I need to assert an order between the 2 somehow? | 19:34 |
| *** sabari is now known as zz_sabari | 19:35 | |
| openstackgerrit | Merged openstack-infra/project-config: Introduce project-specific stable-maint teams https://review.openstack.org/140738 | 19:36 |
| *** barra204 has quit IRC | 19:36 | |
| mordred | notmyname: you aroudn? | 19:36 |
| notmyname | mordred: hi | 19:37 |
| *** salv-orlando has quit IRC | 19:37 | |
| *** zz_sabari is now known as sabari | 19:38 | |
| mordred | notmyname: can you point me to any good resources on how to do simple swift operations in python either using python-swiftclient or not? all of the existing code I can find is written to handle all of the various complex things you might want to do - all I want to do is "upload one file" | 19:38 |
| *** mikedillion has quit IRC | 19:38 | |
| notmyname | mordred: maybe https://github.com/notmyname/directory_uploader | 19:38 |
| mordred | notmyname: THANK YOU | 19:39 |
| *** mrmartin has quit IRC | 19:39 | |
| notmyname | mordred: that was "all I want to do is put my kid's soccer team pictures in the cloud why is this so hard" | 19:39 |
| mordred | notmyname: yes. we have the same problem :) | 19:39 |
| mordred | notmyname: that's quite literally the most helpful file of code I've seen in years | 19:40 |
| notmyname | mordred: as I saw on twitter this morning: https://twitter.com/craig_tracey/status/543477105582166016 | 19:40 |
| *** thedodd has joined #openstack-infra | 19:40 | |
| notmyname | (ie keystone trusts) | 19:40 |
| *** mrmartin has joined #openstack-infra | 19:41 | |
| mordred | notmyname: well, lucky for me I don't need trusts yet | 19:41 |
| notmyname | lucky you :-) | 19:41 |
| notmyname | mordred: let me know if you need anything else | 19:41 |
| mordred | notmyname: I'll show you what I wind up with when I've got it | 19:41 |
| *** andreaf has quit IRC | 19:41 | |
| notmyname | cool | 19:41 |
| clarkb | dstufft you rock | 19:42 |
| clarkb | anything else I should read up on before food? | 19:42 |
| fungi | Execution of '/usr/bin/git fetch --tags origin' returned 128: fatal: https://git.openstack.org/cgit/openstack-infra/storyboard/info/refs not found: did you run git update-server-info on the server? http://puppetboard.openstack.org/report/storyboard.openstack.org/b2f2ec0423c8de7aa546c883894e2afbdff1fd60 | 19:43 |
| fungi | wtf | 19:43 |
| fungi | broken git server? | 19:43 |
| clarkb | try running that against all the backends? | 19:44 |
| mordred | notmyname: oh, one more question - is there a "more" efficient to answer "does this container exist" than "if container in get_container('')" ? | 19:44 |
| *** dannywil_ has quit IRC | 19:44 | |
| fungi | mtreinish: i think just ensure=>absent on a directory doesn't work, even if it's empty. i can't remember which options you have to add to make it delete the dir | 19:44 |
| fungi | clarkb: yep | 19:44 |
| fungi | clarkb: d'oh! i see why ;) | 19:45 |
| fungi | "cgit" | 19:45 |
| jeblair | fungi: why is cgit in there? | 19:45 |
| jeblair | heh :) | 19:45 |
| notmyname | mordred: head_container() | 19:45 |
| fungi | jeblair: because presumably the storyboard module has that. patch on the way | 19:45 |
| mordred | notmyname: cool. thanks | 19:45 |
| mtreinish | fungi: hmm ok, I'll look around on how to do that | 19:46 |
| fungi | and then i can remove the clone and let vcsrepo replace it | 19:46 |
| jeblair | fungi: neat! turning that off helped us find a bug! | 19:46 |
| *** hdd has quit IRC | 19:46 | |
| fungi | nibalizer: what's the magic incantation to make puppet remove an empty directory resource? ensure=>absent alone doesn't cut it, right? | 19:46 |
| fungi | jeblair: yah, right? | 19:46 |
| notmyname | mordred: alternatively, you can get_account() and see if it's in the response (ie the list of containers). but that is subject to eventual consistency, so if you're checking right after you create it it might not be there | 19:47 |
| mtreinish | fungi: it looks like recurse => true, force => true | 19:47 |
| mtreinish | at least from a 30sec google search | 19:47 |
| fungi | mtreinish: that sounds right | 19:47 |
| *** ildikov has quit IRC | 19:47 | |
| *** tnurlygayanov has quit IRC | 19:47 | |
| mtreinish | I'll respin the patch one sec | 19:47 |
| openstackgerrit | Matthew Treinish proposed openstack-infra/system-config: Switch mysql-proxy to simpleproxy https://review.openstack.org/141462 | 19:47 |
| notmyname | mordred: depends if you want to check more than one container. if so, do an account listing [get_account()] and check the response. if not, then head_container() | 19:47 |
| mtreinish | fungi: ^^^ | 19:47 |
| *** hdd has joined #openstack-infra | 19:48 | |
| fungi | jeblair: oh, actually, it's correct in the puppet module, so presumably it was wrong at some point in the past and got corrected, but the old clone was never reset to the new origin | 19:49 |
| fungi | easy fix | 19:49 |
| openstackgerrit | Merged openstack-infra/project-config: zuul's log path should always be a folder https://review.openstack.org/141263 | 19:49 |
| anteaya | I am finally home | 19:50 |
| mordred | notmyname: thanks. super helpful. that is working for me very well | 19:50 |
| anteaya | and the horrible noise is not noising | 19:50 |
| *** ryanpetrello has quit IRC | 19:50 | |
| anteaya | silence finally, my favourite thing | 19:50 |
| *** dannywilson has joined #openstack-infra | 19:50 | |
| *** HeOS has joined #openstack-infra | 19:51 | |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config: Add link for Trove RSS feed https://review.openstack.org/140549 | 19:53 |
| nibalizer | fungi: purge => true i think | 19:54 |
| nibalizer | or force => true maybe | 19:54 |
| *** talluri has joined #openstack-infra | 19:54 | |
| *** ryanpetrello has joined #openstack-infra | 19:54 | |
| nibalizer | im on phone so kinda useleww | 19:54 |
| fungi | nibalizer: no worries, and thanks | 19:55 |
| *** ddieterly has quit IRC | 19:58 | |
| HenryG | Is anyone browsing gerrit reviews not seeing the checks table? And the <Toggle CI> button missing? | 20:00 |
| fungi | HenryG: ooh, i know what happened. just a sec--fixing | 20:00 |
| *** imcsk8 has quit IRC | 20:00 | |
| *** salv-orl_ has joined #openstack-infra | 20:00 | |
| *** imcsk8 has joined #openstack-infra | 20:00 | |
| HenryG | fungi: it seems to happen only if coming in on IPv6 | 20:01 |
| fungi | HenryG: it should fix itself in the next couple minutes | 20:01 |
| *** ZZelle_ has joined #openstack-infra | 20:01 | |
| HenryG | cool, thanks fungi, I'll check again soon | 20:02 |
| fungi | HenryG: i think a change to the hideci.js merged and so gerrit stopped serving it, but i've triggered gerrit to notice the new version and rebundle it | 20:02 |
| HenryG | you da man | 20:02 |
| *** e0ne has joined #openstack-infra | 20:02 | |
| openstackgerrit | Nels proposed openstack-infra/devstack-gate: Adding support for DEVSTACK_GATE_LIBVIRT_TYPE https://review.openstack.org/134348 | 20:03 |
| fungi | HenryG: let me know if you don't see it again within the next few minutes | 20:03 |
| HenryG | fungi: it's already fixed, it seems! thanks! | 20:03 |
| *** barra204 has joined #openstack-infra | 20:03 | |
| fungi | HenryG: thanks for bringing it to our attention, and apologies for leaving it broken and not noticing | 20:04 |
| HenryG | I refreshed a review page and bingo | 20:04 |
| openstackgerrit | Matthew Treinish proposed openstack-infra/system-config: Switch mysql-proxy to simpleproxy https://review.openstack.org/141462 | 20:04 |
| HenryG | fungi: no worries, small thing | 20:04 |
| *** otter768 has joined #openstack-infra | 20:05 | |
| *** salv-orl_ has quit IRC | 20:05 | |
| *** Sincler has quit IRC | 20:06 | |
| *** thedodd has quit IRC | 20:06 | |
| *** baoli has quit IRC | 20:06 | |
| *** cdent has quit IRC | 20:07 | |
| *** ivar-laz_ has quit IRC | 20:07 | |
| *** ivar-lazzaro has joined #openstack-infra | 20:07 | |
| *** talluri has quit IRC | 20:08 | |
| *** ryanpetrello has quit IRC | 20:08 | |
| *** talluri has joined #openstack-infra | 20:08 | |
| *** salv-orlando has joined #openstack-infra | 20:09 | |
| openstackgerrit | Merged openstack-infra/project-config: Add link for Trove RSS feed https://review.openstack.org/140549 | 20:09 |
| *** ivar-lazzaro has quit IRC | 20:09 | |
| *** rmcall has quit IRC | 20:09 | |
| *** ivar-lazzaro has joined #openstack-infra | 20:10 | |
| *** alexpilotti has quit IRC | 20:10 | |
| *** salv-orlando has quit IRC | 20:11 | |
| jeblair | mtreinish: a suggestion for the init script | 20:11 |
| *** dtantsur|afk has quit IRC | 20:11 | |
| *** alexpilotti has joined #openstack-infra | 20:12 | |
| *** KurtMartin has joined #openstack-infra | 20:13 | |
| openstackgerrit | Jeremy Stanley proposed openstack-infra/system-config: Documentation reflects new stable-maint workflow https://review.openstack.org/141480 | 20:14 |
| fungi | ttx: ^ it's in your hands now | 20:14 |
| fungi | ttx: https://review.openstack.org/#/admin/groups/?filter=-stable-maint | 20:14 |
| *** dkliban is now known as dkliban_brb | 20:14 | |
| mtreinish | jeblair: oh, good call. Yeah I don't think it will make a pid file. I'll respin it in a bit | 20:14 |
| *** rmcall has joined #openstack-infra | 20:15 | |
| *** e0ne has quit IRC | 20:15 | |
| anteaya | fungi: any reason I should not +A https://review.openstack.org/#/c/141403/1 ? | 20:16 |
| *** sabari is now known as zz_sabari | 20:16 | |
| *** thedodd has joined #openstack-infra | 20:16 | |
| fungi | anteaya: not to my knowledge | 20:17 |
| *** dprince has joined #openstack-infra | 20:17 | |
| anteaya | very good | 20:17 |
| *** salv-orlando has joined #openstack-infra | 20:17 | |
| *** kmartin has quit IRC | 20:17 | |
| anteaya | I haven't read all the backscroll | 20:17 |
| anteaya | didn't want to just start shoving stuff in if it had been discussed and I missed it | 20:18 |
| pleia2 | our centos slaves broke a bit yesterday | 20:18 |
| pleia2 | so all the tests failed, fungi did rechecks | 20:18 |
| anteaya | the recheck came back clean | 20:18 |
| anteaya | and thanks | 20:18 |
| fungi | a few still failed that job on rechecks, but now because we seem to be hovering near the 30-minute timeout for that job | 20:18 |
| *** shashankhegde has quit IRC | 20:19 | |
| openstackgerrit | Doug Hellmann proposed openstack-infra/system-config: Generate specs.o.o root page from template https://review.openstack.org/141175 | 20:21 |
| anteaya | did we track down why centos slaves were/are unhappy? | 20:21 |
| *** salv-orlando has quit IRC | 20:21 | |
| anteaya | or did I just stir a pot? | 20:21 |
| *** markmcclain has quit IRC | 20:22 | |
| *** alexpilotti has quit IRC | 20:22 | |
| openstackgerrit | Merged openstack-infra/project-config: Stop defaulting node builds to Puppet 2 https://review.openstack.org/141403 | 20:23 |
| fungi | anteaya: see https://storyboard.openstack.org/#!/story/2000071 for links | 20:23 |
| *** alexpilotti has joined #openstack-infra | 20:23 | |
| anteaya | fungi: thanks | 20:23 |
| *** salv-orlando has joined #openstack-infra | 20:23 | |
| *** vigneshvar has quit IRC | 20:23 | |
| *** tonytan4ever has quit IRC | 20:24 | |
| *** dtantsur has joined #openstack-infra | 20:24 | |
| fungi | anteaya: sorry not actually that one | 20:24 |
| fungi | anteaya: https://storyboard.openstack.org/#!/story/2000081 is the one i meant | 20:25 |
| *** rmcall has quit IRC | 20:26 | |
| *** talluri has quit IRC | 20:26 | |
| anteaya | the first one was interesting too | 20:27 |
| *** alexpilotti has quit IRC | 20:27 | |
| *** salv-orlando has quit IRC | 20:28 | |
| *** mjturek has quit IRC | 20:28 | |
| *** mpaolino has joined #openstack-infra | 20:28 | |
| *** rmcall has joined #openstack-infra | 20:28 | |
| *** tonytan4ever has joined #openstack-infra | 20:29 | |
| anteaya | fungi: so this one is ready to be un-wiped and reviewed? https://review.openstack.org/#/c/141468/ | 20:29 |
| fungi | anteaya: not until next week probably | 20:30 |
| anteaya | okay | 20:30 |
| fungi | anteaya: it's just cleanup, once the long-term fix finds its way onto our nodepool images over the weekend | 20:30 |
| *** salv-orlando has joined #openstack-infra | 20:30 | |
| anteaya | ah right yes | 20:31 |
| fungi | anteaya: reverting my short-term workaround | 20:31 |
| fungi | basically | 20:31 |
| anteaya | and why create havoc for the weekend | 20:31 |
| anteaya | great | 20:31 |
| anteaya | thanks for the link to the story | 20:31 |
| fungi | story time is always fun | 20:31 |
| fungi | "gather 'round, pull up a chair and let me tell you a story..." | 20:32 |
| anteaya | will there be a fire? | 20:34 |
| anteaya | I like a fire | 20:34 |
| *** rmcall has quit IRC | 20:34 | |
| fungi | you've come to the right project then | 20:35 |
| anteaya | ha ha ha | 20:35 |
| fungi | we've always got fires _everywhere_ | 20:35 |
| anteaya | glad to be home | 20:35 |
| Mithrandir | it's traditional to put the storyteller in the middle, isn't it? | 20:35 |
| *** esker has quit IRC | 20:35 | |
| pleia2 | haha | 20:35 |
| fungi | in the middle of the fire, yes | 20:35 |
| *** salv-orl_ has joined #openstack-infra | 20:36 | |
| * fungi wears flame-retardant pants every day | 20:36 | |
| *** alexpilotti has joined #openstack-infra | 20:36 | |
| *** sarob has joined #openstack-infra | 20:36 | |
| fungi | ugh, now to figure out why our kdcs have stopped reporting into puppet since yesterday | 20:36 |
| *** achanda has quit IRC | 20:37 | |
| *** vigneshvar has joined #openstack-infra | 20:37 | |
| *** salv-orlando has quit IRC | 20:37 | |
| jeblair | zaro: the css is really weird on review-dev, especially for access controls... know anything about that? | 20:39 |
| *** mjturek has joined #openstack-infra | 20:39 | |
| *** MarkAtwood has quit IRC | 20:40 | |
| *** ddieterly has joined #openstack-infra | 20:40 | |
| openstackgerrit | Matthew Treinish proposed openstack-infra/system-config: Switch mysql-proxy to simpleproxy https://review.openstack.org/141462 | 20:41 |
| mtreinish | jeblair: ^^^ | 20:41 |
| *** dustins has quit IRC | 20:41 | |
| *** sweston1 has joined #openstack-infra | 20:43 | |
| anteaya | quite the chorus line here on the dogfood swift logs in infra jobs patch: https://review.openstack.org/#/c/141250/1 was about to +A | 20:43 |
| anteaya | looks like the only reason someone else hasn't beat me to it was waiting on the recheck | 20:43 |
| dteselkin | Hi, I need some advice about gerrit / gearman | 20:43 |
| *** julim has quit IRC | 20:43 | |
| anteaya | any reason to hold off? | 20:43 |
| *** esker has joined #openstack-infra | 20:44 | |
| anteaya | dteselkin: ask away, we will do our best | 20:44 |
| dteselkin | For some reason jenkins doesn't send messages to register jobs in gearman | 20:44 |
| anteaya | do you have a paste of any output or config files you think might be relevant | 20:45 |
| dteselkin | zuul host (where gearman server installed) is reachable from jenkins | 20:45 |
| anteaya | to help us get some insight into what you are doing on your system? | 20:45 |
| * anteaya +A's the swift logs patch | 20:45 | |
| dteselkin | I can do 'status' via telnet both from localhost and jenkins | 20:45 |
| *** Masahiro has joined #openstack-infra | 20:46 | |
| dteselkin | I run tcpdump on port 4730 to check if there are any messages | 20:46 |
| dteselkin | Some activity was detected for a moment when I did 'service jenkins stop' | 20:47 |
| dteselkin | But after 'service jenkins start' there was nothing | 20:47 |
| dteselkin | So I'm trying to figure out how can I force jenkins to register jobs | 20:47 |
| fungi | okay, figured out the puppetry with the kdcs... combination of missing puppet comment on the crontab entry on kdc01 causing misleading errors, and neither kdc having been added to the known_hosts on the puppetmaster yet | 20:48 |
| *** rmcall has joined #openstack-infra | 20:49 | |
| dteselkin | 'test connection' from gearman plugin on jenkins config page passes | 20:49 |
| dteselkin | And I see packages in tcpdump in the same time | 20:50 |
| openstackgerrit | Merged openstack-infra/project-config: Introduce openstack-server-release-jobs template https://review.openstack.org/140326 | 20:50 |
| *** ChuckC has joined #openstack-infra | 20:50 | |
| *** Masahiro has quit IRC | 20:50 | |
| openstackgerrit | Merged openstack-infra/project-config: Dogfood swift logs with infra jobs https://review.openstack.org/141250 | 20:51 |
| dteselkin | Hmm | 20:52 |
| openstackgerrit | Doug Hellmann proposed openstack-infra/project-config: Generate specs.o.o root page from template https://review.openstack.org/141173 | 20:52 |
| dteselkin | I disable the plugin and enabled it again | 20:52 |
| dteselkin | That helped | 20:52 |
| clarkb | ok /me is back | 20:52 |
| asselin | dteselkin, make sure you don't have "too many" jenkins running | 20:53 |
| *** dimtruck is now known as zz_dimtruck | 20:53 | |
| anteaya | clarkb: welcome back | 20:54 |
| dteselkin | I have only one node | 20:54 |
| *** zz_sabari is now known as sabari | 20:54 | |
| *** sweston1 has quit IRC | 20:54 | |
| dteselkin | And see two processes on the node | 20:55 |
| asselin | dteselkin, I mean the jenkins process: ps -ef | grep jenkins | 20:55 |
| asselin | dteselkin, ok that's correct | 20:55 |
| dteselkin | one process bin/daemon, another bin/java | 20:55 |
| clarkb | to confirm puppet is happy now? should I clean up my patch to install everything from git anyways? I think it is something we should consider doing. Also, did we merge the change to use puppet 3 in image builds? | 20:56 |
| clarkb | fungi: mordred ^ | 20:56 |
| *** otter768 has quit IRC | 20:56 | |
| *** hdd has quit IRC | 20:56 | |
| nibalizer | insatll everything from git anyways! | 20:56 |
| *** dkliban_brb is now known as dkliban | 20:56 | |
| clarkb | nibalizer: ya I think i twould be a good thing to do | 20:56 |
| *** alexpilotti has quit IRC | 20:57 | |
| mtreinish | clarkb: you missed some fun: https://bugs.launchpad.net/ubuntu/+source/mysql-proxy/+bug/1402011 and https://review.openstack.org/141462 | 20:57 |
| uvirtbot | Launchpad bug 1402011 in mysql-proxy "telnet crashes mysql-proxy" [Undecided,New] | 20:57 |
| fungi | clarkb: yeah, that's why i didn't -2 it | 20:57 |
| *** ddieterly has quit IRC | 20:57 | |
| *** annegent_ has quit IRC | 20:57 | |
| fungi | seems reasonable to do anyway | 20:57 |
| pleia2 | mtreinish: hah, ouch | 20:58 |
| *** annegent_ has joined #openstack-infra | 20:58 | |
| clarkb | mtreinish: wow | 20:58 |
| clarkb | "open tcp connection" -> crash | 20:58 |
| *** alexpilotti has joined #openstack-infra | 20:58 | |
| clarkb | mtreinish: fungi out of curiousity why not haproxy which we already have deployed elsewhere? | 20:59 |
| clarkb | I guess we don't need any load balancing functionality so it is probably overkill | 20:59 |
| *** alexpilotti has quit IRC | 20:59 | |
| *** otter768 has joined #openstack-infra | 20:59 | |
| fungi | clarkb: very good point. i guess even though we don't need this to be "ha" it's still a reasonable minimism | 20:59 |
| mordred | clarkb: yeah, I think we shoudl install eveything from git anyway | 20:59 |
| *** alexpilotti has joined #openstack-infra | 21:00 | |
| fungi | i could go either way on it honestly | 21:00 |
| clarkb | fungi: I am a fan of using the tool I already know is broken :) | 21:00 |
| fungi | the devil you know | 21:00 |
| mtreinish | clarkb: heh, so rewrite the patch again? :) | 21:00 |
| mordred | god point | 21:00 |
| *** rmcall has quit IRC | 21:00 | |
| *** alexpilotti has quit IRC | 21:01 | |
| clarkb | mtreinish: no I guess not this should work. Was just curious if there was a specific reason to not do that or if we just did a thing because simpleproxy | 21:01 |
| *** alexpilotti has joined #openstack-infra | 21:02 | |
| *** alexpilotti has quit IRC | 21:02 | |
| openstackgerrit | Doug Hellmann proposed openstack-infra/infra-manual: Expand more of the repo verification steps https://review.openstack.org/140828 | 21:03 |
| mtreinish | clarkb: nah, it's more me not knowing anything. fungi said tcp proxy I googled it, and simpleproxy was the first hit | 21:03 |
| *** mfink__ has quit IRC | 21:03 | |
| clarkb | mtreinish: works for m | 21:03 |
| clarkb | e | 21:03 |
| *** alexpilotti has joined #openstack-infra | 21:03 | |
| *** prad has quit IRC | 21:04 | |
| *** rmcall has joined #openstack-infra | 21:05 | |
| *** alexpilotti has quit IRC | 21:05 | |
| *** ryanpetrello has joined #openstack-infra | 21:06 | |
| *** dprince has quit IRC | 21:07 | |
| *** sweston1 has joined #openstack-infra | 21:07 | |
| *** alexpilotti has joined #openstack-infra | 21:07 | |
| *** sweston1 has quit IRC | 21:09 | |
| *** otter768 has quit IRC | 21:09 | |
| *** alexpilotti has quit IRC | 21:09 | |
| clarkb | mtreinish: +2 on that change | 21:11 |
| mtreinish | clarkb: cool, thanks | 21:11 |
| fungi | mestery: where's the corrected git repository you want to use for networking-odl? | 21:11 |
| *** alexpilotti has joined #openstack-infra | 21:11 | |
| fungi | mtreinish: lgtm too, +3 | 21:11 |
| fungi | mtreinish: i'll check back in on it once it hits the server | 21:12 |
| mtreinish | fungi: thanks, I'll keep an eye out too | 21:12 |
| *** tjones1 has quit IRC | 21:12 | |
| mtreinish | I'm anxious to look at the data | 21:12 |
| *** alexpilotti has quit IRC | 21:13 | |
| fungi | mtreinish: we should have quite a bit there now | 21:13 |
| clarkb | fungi: mordred looks like a couple changes to the way we build images got in (use puppet 3 and only use env dirs on the master) | 21:13 |
| *** sweston is now known as _sweston | 21:13 | |
| *** alexpilotti has joined #openstack-infra | 21:13 | |
| fungi | clarkb: yep | 21:14 |
| clarkb | I am going to try building a centos6 image by hand on new nodepool now just to make sure that goes smoothly and if it doens't that we don't wait all weekend to fix it | 21:14 |
| fungi | that's a good idea. i simply planned to check back in on the image updates tomorrow and see if they went okay (and maybe troubleshoot if not) | 21:14 |
| *** otter768 has joined #openstack-infra | 21:14 | |
| openstackgerrit | Colleen Murphy proposed openstack-infra/project-config: Only guard yum update against epel if enabled https://review.openstack.org/141288 | 21:15 |
| clarkb | that is running now | 21:15 |
| clarkb | the other thing I wanted to do this afternoon is kill the old git.o.o servers | 21:15 |
| *** alexpilotti has quit IRC | 21:15 | |
| clarkb | I think they have a couple things still hitting them but at this point quickest way to fix that is the turn them off :) | 21:15 |
| *** melwitt has joined #openstack-infra | 21:16 | |
| *** mpaolino has quit IRC | 21:16 | |
| openstackgerrit | Merged openstack-infra/system-config: Switch mysql-proxy to simpleproxy https://review.openstack.org/141462 | 21:17 |
| *** bdpayne has joined #openstack-infra | 21:17 | |
| fungi | i'm going to go grab an early dinner, then come back to continue seeing if i can get a little more done | 21:17 |
| clarkb | kk | 21:17 |
| * fungi will bbl | 21:17 | |
| *** leaky-tennis is now known as leakypipes | 21:17 | |
| *** sweston has joined #openstack-infra | 21:19 | |
| openstackgerrit | Clark Boylan proposed openstack-infra/system-config: Install all puppet modules from source https://review.openstack.org/141182 | 21:20 |
| *** atiwari has quit IRC | 21:20 | |
| *** bdpayne has quit IRC | 21:21 | |
| *** obondarev has joined #openstack-infra | 21:22 | |
| *** dmsimard is now known as dmsimard_away | 21:24 | |
| *** shashankhegde has joined #openstack-infra | 21:25 | |
| *** alexpilotti has joined #openstack-infra | 21:25 | |
| *** aysyd has quit IRC | 21:25 | |
| *** mrmartin has quit IRC | 21:25 | |
| *** mrmartin has joined #openstack-infra | 21:26 | |
| *** bdpayne has joined #openstack-infra | 21:26 | |
| *** alexpilotti has quit IRC | 21:27 | |
| *** jedimike has joined #openstack-infra | 21:27 | |
| *** timrc has joined #openstack-infra | 21:28 | |
| *** palar has quit IRC | 21:28 | |
| *** dpaterson has quit IRC | 21:30 | |
| clarkb | sdague: did the lack of aiopcpu experimental testing get sorted? | 21:30 |
| *** dpaterson has joined #openstack-infra | 21:30 | |
| clarkb | sdague: I am just about to where I can look at that if it is still an issue | 21:30 |
| *** bdpayne_ has joined #openstack-infra | 21:30 | |
| sdague | so it was hung on stalled dib tests | 21:30 |
| sdague | which are now removed | 21:31 |
| sdague | but aio tests weren't working anyway, cpu mismatch for live migration | 21:31 |
| clarkb | well that should only be a problem in the last patch | 21:31 |
| clarkb | which doesn't work for other reasons too | 21:31 |
| *** hashar has joined #openstack-infra | 21:31 | |
| clarkb | and ya rax apparently doesn't give consistent cpus out like hpcloud | 21:32 |
| *** ddieterly has joined #openstack-infra | 21:32 | |
| clarkb | haproxy is stopped on both old git.o.o's. If no screaming happens asap I will delete them by end of today | 21:32 |
| *** boris-42 has quit IRC | 21:33 | |
| *** e0ne has joined #openstack-infra | 21:33 | |
| *** tjones1 has joined #openstack-infra | 21:33 | |
| jeblair | clarkb: oh, is that something we need to do some engineering on? | 21:33 |
| *** bdpayne_ has quit IRC | 21:33 | |
| clarkb | jeblair: (assuming its the live migration thing) I think jogo was going to try setting the cpu type in nova/qemu across the jobs | 21:34 |
| *** bdpayne has quit IRC | 21:34 | |
| clarkb | jeblair: and if we do that with the cpu hpcloud exposes I expect it to just work everywhere since that is an ancient cpu that qemu on these VMs should be happy with | 21:34 |
| *** baoli has joined #openstack-infra | 21:34 | |
| *** hdd has joined #openstack-infra | 21:34 | |
| jeblair | ok. i'll leave it there for now. thx. | 21:34 |
| sdague | yeh, I think that's basically why hp cloud tells you that their guests are core2duo | 21:35 |
| sdague | so all the cpus are compatible for live migration | 21:35 |
| *** obondarev has quit IRC | 21:35 | |
| clarkb | all of the patches before the tempest full change should be ready for review though | 21:35 |
| clarkb | I think I should already be +2 on the ones I did not write | 21:36 |
| clarkb | flashgordon: ^ is that still the plan? | 21:38 |
| flashgordon | clarkb: yes that is the plan, was going to do it yesterday but tests weren't running | 21:39 |
| clarkb | it does look like bare-centos6 in rax-dfw is building ok. It is now trying to snapshot | 21:39 |
| flashgordon | and doing specs today, but yes that is still the plan | 21:39 |
| clarkb | flashgordon: thanks | 21:39 |
| flashgordon | clarkb: I added some links in a comment of what needs to be set from what I understand | 21:40 |
| sdague | do we have a run there? | 21:41 |
| openstackgerrit | Merged openstack-infra/system-config: Add iptables rule to log dropped packets https://review.openstack.org/135694 | 21:41 |
| jeblair | clarkb, fungi: fyi if you had a screen session running during the re-uid, you may run into You are not the owner of /var/run/screen/S-corvus. fix is obvious/trivial. | 21:42 |
| mtreinish | clarkb: d'oh, the firewall. We punched open 4040 before for the proxy, but now it's 3306... | 21:42 |
| clarkb | jeblair: I shouldn't (I tr to be good about cleaning those up) | 21:42 |
| clarkb | mtreinish: lol, thats a simple patch if you want ot push it up I can just approve it | 21:43 |
| *** kmartin has joined #openstack-infra | 21:45 | |
| openstackgerrit | Matthew Treinish proposed openstack-infra/system-config: Open port 3306 on logstash.o.o https://review.openstack.org/141498 | 21:45 |
| mtreinish | clarkb: ^^^ | 21:45 |
| *** KurtMartin has quit IRC | 21:47 | |
| *** timcline_ has joined #openstack-infra | 21:48 | |
| *** timcline has quit IRC | 21:48 | |
| clarkb | we have a new image in rax-dfw for centos6 now. Looks like the build went well | 21:51 |
| *** kgiusti has quit IRC | 21:54 | |
| *** annegent_ has quit IRC | 21:54 | |
| *** baoli has quit IRC | 21:55 | |
| openstackgerrit | Nels proposed openstack-infra/devstack-gate: Adding support for DEVSTACK_GATE_LIBVIRT_TYPE https://review.openstack.org/134348 | 21:55 |
| *** boris-42 has joined #openstack-infra | 21:55 | |
| *** baoli has joined #openstack-infra | 21:55 | |
| mordred | clarkb: woot | 21:55 |
| *** melwitt1 has joined #openstack-infra | 21:55 | |
| *** melwitt has quit IRC | 21:56 | |
| *** melwitt has joined #openstack-infra | 21:57 | |
| mmedvede | clarkb: in the light of https://review.openstack.org/#/c/140480/, which git entry point third party CIs are we supposed to switch to? | 21:58 |
| clarkb | did rax fix our swift ro account? looks like jhesketh's dogfood logs in swift change made it in and I can see logs so account is working | 21:58 |
| anteaya | yay | 21:58 |
| clarkb | mmedvede: git.openstack.org is still the place to get your git | 21:58 |
| anteaya | yes I approved that patch | 21:58 |
| openstackgerrit | Merged openstack-infra/system-config: Open port 3306 on logstash.o.o https://review.openstack.org/141498 | 21:58 |
| openstackgerrit | Merged openstack-infra/system-config: Open port 3306 on logstash.o.o https://review.openstack.org/141498 | 21:59 |
| clarkb | mmedvede: its a round robin between the back end load balancers | 21:59 |
| anteaya | I did give the channel the chance to stop me before I did, but noone spoke up | 21:59 |
| clarkb | anteaya: ya its fine. I am mostly wondering if rax said anything because it is working | 21:59 |
| anteaya | why did the bot report the same merged patch twice? | 21:59 |
| anteaya | cool | 21:59 |
| mmedvede | clarkb: weird, I am getting connection refused on git.o.o | 21:59 |
| *** melwitt1 has quit IRC | 21:59 | |
| clarkb | mmedvede: you may be one of the folks that was hitting the old git server for whatever reason | 22:00 |
| clarkb | mmedvede: make sure you haven't cached old dns records | 22:00 |
| mmedvede | clarkb: probably is the case. Would investigate. Thanks | 22:01 |
| openstackgerrit | Nels proposed openstack-infra/devstack-gate: Adding support for DEVSTACK_GATE_LIBVIRT_TYPE https://review.openstack.org/134348 | 22:03 |
| *** mrmartin has quit IRC | 22:04 | |
| *** doug-fish1 has joined #openstack-infra | 22:05 | |
| clarkb | nibalizer: any idea why in http://logs.openstack.org/82/141182/6/check//gate-infra-puppet-apply-precise/3ebcfa2/console.html puppet module list lists a bunch of modules before we run tests then after we run tests it says no modules? | 22:05 |
| openstackgerrit | Merged openstack/requirements: Updating python-glanceclient to 0.15.0 release https://review.openstack.org/141298 | 22:05 |
| openstackgerrit | Merged openstack/requirements: Upgrade oslo.vmware>=0.8.0 https://review.openstack.org/141099 | 22:05 |
| nibalizer | mumble mumble modulepath? | 22:06 |
| nibalizer | clarkb: no i dont have any idea | 22:07 |
| clarkb | I think I am going to remove the trailing module listing from my change as its logically independent and it doesn't work as is | 22:07 |
| *** doug-fish has quit IRC | 22:08 | |
| *** baoli has quit IRC | 22:08 | |
| *** bswartz has quit IRC | 22:09 | |
| mestery | fungi: Hope you got your puppet stuff done today! Wanted to pester you with questions on the historical import into a stackforge project. I have the stuff ready, just need to push it. Is this something I could get permissions for just to push out? | 22:11 |
| mestery | fungi: I realize I screwed this up and it's all my fault here, looking if there is a way forward here or not :) | 22:11 |
| mestery | fungi: If you're still busy, let me know and I'll stop pestering ;) | 22:11 |
| anteaya | fungi: in the backscroll I believe fungi requested the name of the repo under discussion | 22:12 |
| anteaya | just to ensure he had the name correct | 22:12 |
| *** tjones2 has joined #openstack-infra | 22:12 | |
| anteaya | mestery: sorry the above was to you not fungi ^ | 22:13 |
| anteaya | it was about fungi | 22:13 |
| mestery | anteaya: Ah, cool! | 22:13 |
| mestery | fungi anteaya: The repo is "stackforge/networking-odl" | 22:13 |
| mestery | fungi anteaya: I have the split done thanks to dougwig's script, all commits are local, but I can push to github easily enough as well. | 22:14 |
| mestery | anteaya: Thanks for the help! | 22:14 |
| dougwig | mestery: for the github push, remember to use "git push --all && git push --tags" | 22:14 |
| *** tjones1 has quit IRC | 22:14 | |
| mestery | dougwig: Ack, it's in your instructions, but thanks for the reminder! | 22:15 |
| *** tonytan4ever has quit IRC | 22:15 | |
| anteaya | mestery: do the push to github, it has to be a public repo | 22:15 |
| mestery | anteaya: Will do so now. | 22:15 |
| openstackgerrit | Clark Boylan proposed openstack-infra/system-config: Install all puppet modules from source https://review.openstack.org/141182 | 22:15 |
| clarkb | mordred: nibalizer ^ that is all cleaned up and ready for welcoming our new git overlords | 22:15 |
| mordred | clarkb: can we do it with mercurial instead? | 22:16 |
| clarkb | mordred: only if I never have to touch it | 22:16 |
| jesusaurus | clarkb: oooh | 22:16 |
| nibalizer | clarkb: noice | 22:18 |
| nibalizer | clarkb: after that lands ill submit a change to bump postgres, puppetdb, inifile, and stdlib i think | 22:19 |
| *** dkliban is now known as dkliban_afk | 22:19 | |
| clarkb | nibalizer: inifile doesn't need a bump | 22:19 |
| clarkb | we already have the version that we need (1.1.3) | 22:19 |
| *** tylerdurden has quit IRC | 22:20 | |
| *** shakamunyi has joined #openstack-infra | 22:20 | |
| mtreinish | fungi, clarkb: can you take a look at logstash.o.o when you get a sec, the port is open but I'm getting connection refused (with the ip) and Network is unreachable (with the hostname) trying to connect to the db | 22:21 |
| jesusaurus | clarkb: why is vcsrepo being installed from openstack-infra, is it being forked? | 22:21 |
| *** MaxV has joined #openstack-infra | 22:21 | |
| *** dpaterson has quit IRC | 22:22 | |
| jesusaurus | oh it used to be openstackci-vcsrepo, ok | 22:22 |
| clarkb | jesusaurus: we have installed it thee for years. tl;dr is that upstream vcsrepo didn't work beacuse it tries to do all the vcs. mordred said meh and fixed git in ways that don't really work for svn or some such | 22:22 |
| mtreinish | err, nm I misread nmap it says the port is closed... | 22:22 |
| mordred | yah | 22:22 |
| mordred | I tried to care once about upstreaming the fixes, but then didn't | 22:22 |
| clarkb | because svn | 22:22 |
| mestery | anteaya fungi dougwig: https://github.com/mestery/x-stackforge-odl | 22:23 |
| clarkb | I seem to recall specific complaining about svn | 22:23 |
| *** nelsnelson has quit IRC | 22:23 | |
| mordred | well, that, and also because upstream is VERY careful not to step on any local state that's been checked out | 22:23 |
| mordred | except the whole point of puppet is to express a desired state | 22:23 |
| *** timcline_ has quit IRC | 22:23 | |
| *** nelsnelson has joined #openstack-infra | 22:23 | |
| mordred | so why puppet shuld be careful around my manual actions is beyond me | 22:23 |
| clarkb | mtreinish: looks like firewall is open but proxy isn't running | 22:24 |
| mtreinish | clarkb: hmm, I tested the policy on a rax vm before and it seemed to work there | 22:24 |
| clarkb | mtreinish: try now. I manually ran init script for proxy | 22:24 |
| mtreinish | clarkb: yep, thanks works now | 22:25 |
| clarkb | mtreinish: could it be you didn't make the service => running requires => initscript? | 22:25 |
| *** e0ne is now known as e0ne_ | 22:25 | |
| mtreinish | clarkb: yep, that must be it: just enable => true | 22:26 |
| clarkb | mtreinish: ok thats fine, just means we have to manually start it (which is common for our services) | 22:26 |
| mtreinish | http://git.openstack.org/cgit/openstack-infra/system-config/tree/modules/mysql_proxy/manifests/server.pp#n35 | 22:26 |
| mtreinish | clarkb: ok | 22:26 |
| mtreinish | I copy and pasted it from somewhere, so that would explain it | 22:26 |
| *** hdd has quit IRC | 22:27 | |
| *** hdd has joined #openstack-infra | 22:27 | |
| clarkb | hrm looks like it forks to accept connections | 22:27 |
| clarkb | whatever I guess that is easy mode | 22:27 |
| jeblair | there's a break in the rain so i'm going to go run around outside while i can | 22:27 |
| clarkb | mtreinish: you have access now ya? | 22:27 |
| mtreinish | clarkb: yep, it's working fine | 22:28 |
| mtreinish | thanks | 22:28 |
| nibalizer | mordred: hrm you give me ideas | 22:30 |
| nibalizer | ideas of a dangerous, friday kind... | 22:30 |
| *** e0ne_ has quit IRC | 22:31 | |
| asselin | anyone know if this is supported on ubuntu 14.04? Error: Execution of '/usr/sbin/a2enmod mem_cache' returned 1: ERROR: Module mem_cache does not exist! | 22:31 |
| asselin | Error: /Stage[main]/Zuul/A2mod[mem_cache]/ensure: change from absent to present failed: Execution of '/usr/sbin/a2enmod mem_cache' returned 1: ERROR: Module mem_cache does not exist! | 22:31 |
| openstackgerrit | Matthew Treinish proposed openstack-infra/project-config: Add nodepool script to preseed testrepository from subunit2sql https://review.openstack.org/136234 | 22:31 |
| nibalizer | maybe they renamed the module for 2.4/14.04? | 22:32 |
| clarkb | asselin: trusty has apache 2.4 | 22:32 |
| clarkb | precise is 2.2. likely a renamed module as nibalizer suggestes | 22:32 |
| mordred | nibalizer: uhoh. are you going to fix the vcsrepo module? | 22:32 |
| asselin | clarkb, nibalizer ok, so best to stick with precise 12.04 | 22:32 |
| clarkb | asselin: or just fix puppet... | 22:33 |
| *** mattfarina has joined #openstack-infra | 22:34 | |
| nibalizer | asselin: ya | 22:34 |
| nibalizer | is there an $apache_version fact we can if on? | 22:34 |
| *** carl_baldwin has quit IRC | 22:34 | |
| *** Masahiro has joined #openstack-infra | 22:34 | |
| nibalizer | or maybe just know that 14.04 = 2.4 | 22:35 |
| *** r-daneel has quit IRC | 22:35 | |
| *** stevemar has joined #openstack-infra | 22:36 | |
| * asselin looking | 22:37 | |
| *** ryanpetrello has quit IRC | 22:37 | |
| mtreinish | jeblair: qq, what do you want to do about: https://review.openstack.org/106584 | 22:39 |
| *** Masahiro has quit IRC | 22:39 | |
| *** MaxV has quit IRC | 22:39 | |
| jeblair | mtreinish: i think i was wating for clarkb to review it | 22:40 |
| clarkb | me? ok I will review | 22:40 |
| *** mfink has joined #openstack-infra | 22:41 | |
| mtreinish | jeblair: well I mean we kinda went ahead and implemented it waiting for clarkb. | 22:41 |
| mtreinish | it also doesn't reflect reality 100% anymore, but I guess it's close enough | 22:42 |
| jeblair | mtreinish: if there's something significant that's changed, could you update the spec with a followup patch? | 22:42 |
| mtreinish | jeblair: sure, it's mostly s/mysql-proxy/simpleproxy | 22:43 |
| mtreinish | I think, it's been a while since I looked at it | 22:43 |
| jeblair | mtreinish: it should also be moved to implemented/ (in a followup patch) | 22:43 |
| clarkb | well the dashboard is still wip | 22:43 |
| mtreinish | yeah, as is the testrepository integration (although I won't be using that on the test slaves like in the spec, its too slow) | 22:44 |
| clarkb | I am +2 on that version of the doc. I think follow up with a new change to address the changes? | 22:45 |
| clarkb | jeblair: ^ should I go ahead and approve that change as is or wait? | 22:45 |
| *** sarob has quit IRC | 22:45 | |
| mtreinish | clarkb: yeah, I can do that now while I'm waiting for my testr preseed script to finish running | 22:46 |
| fungi | mestery: i'll take a stab at updating from that in a bit | 22:46 |
| mestery | fungi: Thanks! After all your awesome help, I'm moving on from owing you beer to owing you expensive bourbon ;) | 22:47 |
| *** kmartin has quit IRC | 22:48 | |
| *** asselin has quit IRC | 22:48 | |
| *** jedimike has quit IRC | 22:48 | |
| *** asselin has joined #openstack-infra | 22:49 | |
| *** kmartin has joined #openstack-infra | 22:49 | |
| *** kmartin has quit IRC | 22:49 | |
| *** asselin has quit IRC | 22:50 | |
| *** kmartin has joined #openstack-infra | 22:50 | |
| *** asselin has joined #openstack-infra | 22:50 | |
| *** kmartin has quit IRC | 22:51 | |
| *** asselin has quit IRC | 22:51 | |
| *** kmartin has joined #openstack-infra | 22:51 | |
| *** asselin has joined #openstack-infra | 22:51 | |
| *** dimsum__ has quit IRC | 22:51 | |
| *** kmartin has quit IRC | 22:51 | |
| *** asselin has quit IRC | 22:52 | |
| *** mattfarina has quit IRC | 22:52 | |
| *** dimsum__ has joined #openstack-infra | 22:52 | |
| *** asselin has joined #openstack-infra | 22:52 | |
| *** kmartin has joined #openstack-infra | 22:52 | |
| *** kmartin has quit IRC | 22:53 | |
| *** kmartin has joined #openstack-infra | 22:53 | |
| asselin | clarkb, nibalizer fyi, the name changed to mod_cache_disk. ref: http://httpd.apache.org/docs/trunk/upgrading.html | 22:54 |
| clarkb | asselin: noted, thanks | 22:55 |
| *** esker has quit IRC | 22:56 | |
| *** otter768 has quit IRC | 22:56 | |
| *** dimsum__ has quit IRC | 22:57 | |
| *** annegent_ has joined #openstack-infra | 22:57 | |
| *** talluri has joined #openstack-infra | 22:58 | |
| fungi | mestery: i do enjoy a good bourbon now and then. or we can just call it scratch when i eventually need some neutron help | 22:58 |
| mestery | fungi: Deal either way ;) | 22:59 |
| *** MarkAtwood has joined #openstack-infra | 22:59 | |
| *** amitgandhinz has quit IRC | 23:00 | |
| *** vigneshvar has quit IRC | 23:00 | |
| zaro | jeblair: css weirdness is probably from this change: https://review.openstack.org/#/c/134238/ | 23:03 |
| zaro | jeblair: i think that change was applied to review-dev, but not review.o.o | 23:03 |
| fungi | my evaluation of how well it worked out on review-dev is apparently lacking | 23:03 |
| clarkb | zaro: correct | 23:04 |
| *** rmcall has quit IRC | 23:05 | |
| *** annegent_ has quit IRC | 23:06 | |
| *** andreykurilin_ has joined #openstack-infra | 23:07 | |
| mtreinish | fungi: so I ran that nodepool script locally to test it, it took ~30min to finish and that was only for 3 full size runs (the other 7 were large-ops) | 23:07 |
| mtreinish | so it's about 10min to generate a full subunit stream from the db | 23:07 |
| *** hdd has quit IRC | 23:07 | |
| *** talluri has quit IRC | 23:08 | |
| fungi | mtreinish: so the question that arises... is adding 10 minutes to image updates recouped in job run-time savings? | 23:08 |
| fungi | (multiplied by however many jobs) | 23:08 |
| mtreinish | hmm, that's hard to say. | 23:09 |
| clarkb | I doubt it, but worth a test probably | 23:09 |
| clarkb | also you could just make it, cache it, and use it once a day | 23:09 |
| mtreinish | clarkb: do you remember when we tested the preseeding, I don't think it really made a difference | 23:09 |
| clarkb | something our image builds could do | 23:10 |
| mtreinish | clarkb: that's what we're talking about doing | 23:10 |
| clarkb | mtreinish: ya iirc preseeding differences were minimal | 23:10 |
| mtreinish | what I really want to use it for is shaking up the test order on the workers | 23:10 |
| mtreinish | it should cause more bugs to be exposed | 23:10 |
| clarkb | mtreinish: it should already be shaken up | 23:10 |
| clarkb | oh wait no lifeless changed that | 23:11 |
| mtreinish | the testr scheduler just uses a sort before it groups things | 23:11 |
| mtreinish | unless it knows about time data | 23:11 |
| *** otter768 has joined #openstack-infra | 23:11 | |
| fungi | i expect we can "shake up" the test order without a 10-minute database query | 23:11 |
| clarkb | fungi: mtreinish: except we can run that query once a day then the total cpu used to get it is negligible | 23:12 |
| fungi | do like the python hash seed in tox and just report the shuffle randomization seed you used | 23:12 |
| fungi | so that it can be reused when you need to recreate the run exactly | 23:13 |
| mtreinish | fungi: it's inside testr, I can find the line, but without timing data the algorithm right now is deterministic | 23:13 |
| *** carl_baldwin has joined #openstack-infra | 23:13 | |
| fungi | right, we could randomize the timing data i mean | 23:14 |
| fungi | if that's the goal | 23:14 |
| fungi | just playing devil's advocate | 23:14 |
| fungi | i can imagine all sorts of ways the subunit2sql data would be useful to us, just not convinced that testr pre-seeding necessarily sees significant gains (worth trying out, i don't disagree there) | 23:15 |
| *** salv-orl_ has quit IRC | 23:16 | |
| mtreinish | ah, ok. I didn't think about that, yeah that should be doable. I just assumed use real data to do it :) | 23:16 |
| fungi | use real data sounds fine to me unless it takes so long to get real data that we don't actually recoup the time spent. but again, i'd rather not judge without empirical data, so not opposed to getting some | 23:17 |
| *** doug-fish1 has left #openstack-infra | 23:18 | |
| *** carl_baldwin has quit IRC | 23:18 | |
| openstackgerrit | Ramy Asselin proposed openstack-infra/system-config: Add support for running Zuul on Trusty https://review.openstack.org/141518 | 23:19 |
| asselin | clarkb, nibalizer still to to test a bit more, but at least now it installs. ^^ | 23:20 |
| asselin | clarkb, thanks for the encouragement | 23:20 |
| clarkb | asselin: cool. that looks sane. You may need to update the vhost file too though | 23:21 |
| clarkb | asselin: but I am not sure on that | 23:21 |
| asselin | clarkb, thanks for the heads up | 23:21 |
| *** jklare has quit IRC | 23:22 | |
| *** zz_dimtruck is now known as dimtruck | 23:23 | |
| anteaya | asselin: you are doing great work | 23:23 |
| clarkb | asselin: it depends on whether or not just the mod name changed or if the vhost directives changed too | 23:23 |
| *** jklare has joined #openstack-infra | 23:24 | |
| *** andreykurilin_ has quit IRC | 23:24 | |
| *** andreykurilin_ has joined #openstack-infra | 23:25 | |
| *** hdd has joined #openstack-infra | 23:25 | |
| *** skata_ has joined #openstack-infra | 23:26 | |
| *** sabeen has quit IRC | 23:27 | |
| fungi | mestery: it looks like i've got the filtered history pushed in now (branches, tags and all). let me know if you spot anything awry | 23:30 |
| jeblair | zaro: thanks, left a comment. | 23:30 |
| fungi | jeblair: yep, those are both things i didn't think to test | 23:32 |
| fungi | good catch | 23:32 |
| openstackgerrit | Merged openstack-infra/infra-specs: Add spec for creating a test metric database https://review.openstack.org/106584 | 23:34 |
| *** Longgeek has quit IRC | 23:34 | |
| jeblair | mtreinish: ^ you are clear to start implementation! ;) | 23:34 |
| mtreinish | jeblair: heh, cool :) | 23:35 |
| *** Masahiro has joined #openstack-infra | 23:35 | |
| fungi | implementastic | 23:37 |
| openstackgerrit | Ramy Asselin proposed openstack-infra/system-config: Add support for running Zuul on Trusty https://review.openstack.org/141518 | 23:37 |
| mtreinish | fungi: oh, I just tested the script with https://review.openstack.org/#/c/132119/ applied (which is how it should be done) and it took like 2 secs | 23:37 |
| fungi | mtreinish: okay, that's... like... many orders of magnitude faster | 23:39 |
| nibalizer | anteaya: looks good! | 23:39 |
| notmyname | in our gerrit deploy, I'm guessing that pushing a branch to refs/meta/dashboards/* is pretty restricted, right? | 23:40 |
| *** Masahiro has quit IRC | 23:40 | |
| mtreinish | fungi: yeah, because it doesn't touch the test_runs table just tests. | 23:40 |
| fungi | mtreinish: i can't imagine the aggregate savings across thousands of runs a day wouldn't add up to more than 2 seconds spent in the image build | 23:40 |
| mtreinish | so >24M rows vs like 2 | 23:40 |
| fungi | notmyname: currently it's needed infra assistance iirc | 23:41 |
| clarkb | mtreinish: that sounds like an appropriate optimization? | 23:41 |
| notmyname | fungi: thanks. | 23:41 |
| anteaya | nibalizer: I'm glad to hear it but I lack context, what is the thing we are discussing? asselin's patch? | 23:41 |
| fungi | notmyname: there was, i think, talk about coming up with a way to config-manage those and automate their application/updating but i don't believe anyone's hacked out a solution for it | 23:41 |
| mtreinish | clarkb: yeah, it's also a better data set, because it uses the moving avg run_time from all the runs instead of 10 samples | 23:41 |
| *** hashar has quit IRC | 23:42 | |
| fungi | mtreinish: clarkb: sounds good to me | 23:42 |
| nibalizer | anteaya: yea i just tab completed wrong my bad | 23:43 |
| clarkb | ok I am going to go delete those git servers now | 23:43 |
| mtreinish | fungi: heh, but the sql2subunit bug around timestamps needs to get fixed first | 23:44 |
| anteaya | nibalizer: cool, so asselin that one above was for you | 23:44 |
| *** sabari is now known as zz_sabari | 23:46 | |
| fungi | clarkb: douns good | 23:47 |
| fungi | sounds too | 23:47 |
| clarkb | they are gone | 23:48 |
| fungi | social: whenever you're around again, those tags are deleted now. note that tag deletion tends not to propagate (clones will generally keep them in their refs list until they're locally deleted or the clone is redone from scratch) | 23:48 |
| *** MarkAtwood has quit IRC | 23:57 | |
| *** ryanpetrello has joined #openstack-infra | 23:57 | |
| *** jklare has quit IRC | 23:59 | |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!
