Friday, 2016-03-11

*** Sukhdev has joined #openstack-infra00:00
*** Sukhdev_ has quit IRC00:00
fungiianw: yeah, getting clarkb's input would be good since i think he was behind the design of the db usage in those tests, so knowing whether that's really intentional or whether something like the mysql root account should be used to do those additional account grants instead would be helpful00:00
clarkbfungi: ianw yes grant is required00:01
jeblairanteaya: a quick survey of mainstream us online news sites suggests that if the purpose is to make the us aware of canada (or even the existence of other countries at all), it's not working.  :|00:01
anteayajeblair: colour me surprised00:01
anteayajeblair: but thanks for looking00:01
*** sridhar_ram1 has joined #openstack-infra00:01
anteayajeblair: I appreciate that at the very least, you are now aware00:01
anteayamordred: sorry if I was obtuse earlier00:02
fungiwatchers/readers of mainstream news in the usa likely don't realize canada has a government00:02
clarkbok 15-20 will be puppeted this next pass00:02
clarkb18-20 should alrady be done00:02
anteayamordred: I had eroniously believed you might have known what I was talking about00:02
pleia2fungi: it's a really big, unusual state, with moose in it00:02
anteayafungi: most I think aren't aware canada has an atomsphere00:02
*** rbrndt__ has quit IRC00:02
clarkbanteaya: don't worry half the country is threatening to move to canada after novemeber00:03
mtreinishfungi: wait, canada has a government? That's not possible :)00:03
clarkbanteaya: which half depends on who gets eleected00:03
anteayaif you look at all the us weather reports, the weather stops at the 49th parallel00:03
* fungi imagines the stalwart mounties logging firs and riding their faithful moose steeds00:03
anteayaclarkb: I know it00:03
*** sridhar_ram has quit IRC00:03
anteayathey have been contacting me00:03
pleia2fungi: I imagine that too00:03
anteayaI'm not joking00:03
openstackgerritIan Wienand proposed openstack-infra/project-config: Add ability to grant to openstack_citest mysql user
ianwfungi / clarkb: ^00:03
anteayamtreinish: it is a fable00:03
anteayafungi: you can't ride moose00:04
anteayathe legs are too thin00:04
anteayacan't hold a load00:04
anteayathey tried00:04
fungisomehow i'm not surprised that they tried00:04
jeblairfungi: not for nothin' but i think i had something to do with
anteayafungi: ha ha ha00:04
fungijeblair: ah, yep, that was your handiwork!00:05
clarkbanteaya: you can too ride a moose. Let me introduce you to the most MURICAN of presidents, teddy roosevelt00:05
jeblairi at least banged on my keyboard at it a bit00:05
fungijeblair: so anyway, adding grant option for openstack_citest user in jobs sound like a good idea then?00:05
*** Sukhdev has quit IRC00:05
jeblairyep! :)00:05
*** Swami has quit IRC00:06
clarkb(its apparently a fake but the picture is famous)00:06
fungijeblair: thanks!!!00:06
*** Sukhdev has joined #openstack-infra00:06
anteayaclarkb: tell me more00:06
fungiclarkb: the same could be said for ted himself00:06
jeblairalso, i'm happy to continue to defer to clarkb on any future responsibility that comes from that patch00:06
anteayaha ha ha00:06
anteayain the water00:07
jeblair(but in all seriousness, it was kind of a team idea we all batted around for a few months, as i recall)00:07
anteayaI'm impressed so many americans can pronounce the word Canada00:07
anteayawarms my heart it does00:07
fungijeblair: in fairness, i was only going to blame clarkb for the bad parts of it anyway00:07
clarkbI just know about the grant stuff because it is one of the things that got tricky with the puppet-ymsql upgrade00:08
*** Qiming has quit IRC00:08
jrollanteaya: it's pronounced kuh-nay-dee-ah, right? :P00:08
anteayajroll: you've studied00:08
anteayawell done00:08
*** Sukhdev has quit IRC00:08
jeblairanteaya: sometimes in california, i get really confused about how to pronounce canada:,_California00:08
anteayaha ha ha00:08
anteayajeblair: nice00:08
fungiclarkb: now i want to make a version of that photo with teddy riding a bearodactyl00:08
jrolljeblair: there's also
anteayathe obama's pose for photos like art models00:09
jeblairjroll: they even have their own canadian cody00:10
pleia2love the vintage photoshopping00:10
anteayafungi: what is a bearodactyl?00:10
jrolljeblair: heh, weird00:10
*** kzaitsev_mb has quit IRC00:12
jeblairmordred: congrats on being a ptl!  where DO you find the time!00:13
ianwfungi: ok, you can take back that host too; i think i have a legitimate bug in my image-size tracking code00:13
anteayajeblair mordred wat?00:14
clarkb10-14 will be this round00:14
*** akshai has quit IRC00:14
fungianteaya: a farcical hybrid animal invented for an online comic strip called "the oatmeal"00:15
jeblairanteaya: i mean, it's not official yet, but it sure looks like he has the race locked up.00:15
*** tphummel has quit IRC00:15
mordredjeblair: do I need to announce my candidacy any more explicity do you think?00:15
*** doug-fish has joined #openstack-infra00:16
anteayaha ha ha00:16
anteayamordred: I don't think anyone can challenge you00:16
anteayafungi: ah, good comic, though it has been a while since I've read00:16
funginor can anyone else vote for him!00:16
mordredanteaya: I suppose I might have to start producing some git repos00:16
openstackgerritRyan Hallisey proposed openstack-infra/tripleo-ci: Allow the continer job to run again
anteayafungi: ha ha ha00:17
*** doug-fish has quit IRC00:17
anteayamordred: I guess?00:17
jeblairmordred: i guess you'd have to push something to soon to make it official00:17
openstackgerritIan Wienand proposed openstack-infra/project-config: CentOS AFS build : use homedir
*** doug-fish has joined #openstack-infra00:17
ianwfungi / jeblair : mind having a quick poke at ^ --> failing logs
jeblairianw: already on it ;)00:18
fungimordred: yes, after tristanC's question i was doing a quick peek through the electoral roll equivalent files created by my most recent summit invite batch run, and was surprised when the shortest list was just you. you are the lowest common denominator contributor across our projects, i think00:18
anteayaha ha ha00:18
jeblairfungi: nice turn of phrase00:18
anteayathat needs a t-shirt00:18
mordredfungi: if I haven't contributed to your project, you're not really OpenStack00:18
anteayathe new criteria00:19
fungithat goes on the big tent application checklist00:19
anteayaha ha ha00:19
fungiwait, no commits from mordred? are you _sure_ this is a real project?00:19
anteayaeasy enough benchmark to evaluate00:20
jeblairianw: actually jenkins runs scripts with a CWD of $WORKSPACE which is something like ~jenkins/workspace/<JOBNAME>00:20
jeblairianw: having said that, i think the fix is correct :)00:20
fungigah, trying to load that change just crashed gertty00:21
fungiAttributeError: 'NoneType' object has no attribute 'topic'00:21
jeblairnuh uh00:21
*** austin81 has quit IRC00:21
jeblairfungi: have a full traceback?00:22
*** doug-fish has quit IRC00:22
*** doug-fis_ has joined #openstack-infra00:22
fungijeblair: i do, collecting it now00:22
*** openstackgerrit_ has quit IRC00:23
openstackgerritJames E. Blair proposed openstack-infra/zuul: Make all configuration in-repo configuration
openstackgerritJames E. Blair proposed openstack-infra/zuul: Add job tags
openstackgerritJames E. Blair proposed openstack-infra/zuul: Fix race in tests
*** yamamoto_ has joined #openstack-infra00:24
*** openstackgerrit_ has joined #openstack-infra00:24
jeblairfungi: how did you open it?00:24
*** bapalm has quit IRC00:24
*** doug-fish has joined #openstack-infra00:24
fungictrl-o and then pasted the change number and hit enter00:24
fungiseeing if i can recreate, though likely it was state-dependent00:25
jeblairfungi: any chance you had full debug logging going on around then?00:25
anteayamike meyers took too long in the entrance hall00:25
fungiyeah, it loaded fine after i restarted gertty. i also have the log but not on full debug no00:25
anteayathey had to hurry him along00:25
*** doug-fis_ has quit IRC00:26
clarkb10-20 are all done. 01-09 are going to be done on the next pass00:26
anteayaclarkb: yay00:27
anteayaare 10-20 coming up again?00:27
*** doug-fis_ has joined #openstack-infra00:27
* anteaya assumes we are talking logstash_workers00:28
jeblairfungi: what change were you viewing when you opened it?00:28
*** yamamoto_ has quit IRC00:28
*** doug-fish has quit IRC00:29
*** sc68cal has quit IRC00:29
*** bpokorny_ has quit IRC00:29
clarkbanteaya: yes they are all fully operational00:29
jeblairfungi: do you have the activating the link in your terminal thing working?  (any chance you could have remotely opened the change while copying the change number?)00:29
clarkbanteaya: on logstash 2.000:29
*** doug-fish has joined #openstack-infra00:29
*** sdake has joined #openstack-infra00:29
*** bpokorny has joined #openstack-infra00:29
fungijeblair: i was loading ian's 291498 change, and no i don't have anything set up to trigger remote loading in gertty (yet at least)00:30
jeblairfungi: the error appears to be actually refreshing the change you were on, not the one you were going to, which is really weird.00:31
anteayaclarkb: yay!00:31
*** gildub has joined #openstack-infra00:31
*** doug-fis_ has quit IRC00:32
fungiclarkb: jeblair: any chance one of you want to officially weigh in on the grant option addition (291493) so i can get started reupdating jjb configs everywhere while things are quiet?00:32
*** armax has quit IRC00:33
fungiit's going to impact lots of jobs, but hopefully this is the last tweak to these macros for a long while00:33
*** doug-fish has quit IRC00:34
fungionce that lands i'll get jjb updates going again00:34
*** armax has joined #openstack-infra00:35
*** armax has quit IRC00:35
fungithis is kinda a neat/convenient tool
fungithe auto-importing it does makes it a bit less of a hassle than straight python -c too00:36
*** ashleighfarnham has quit IRC00:39
openstackgerritMerged openstack-infra/project-config: Add ability to grant to openstack_citest mysql user
fungiand now i'll get ta' updatin'00:40
*** bapalm has joined #openstack-infra00:41
fungiclarkb: how were you suggesting to do multi-threaded updates?00:42
fungii'll start looking at the jjb docs00:42
fungijust realized you had wanted me to benchmark that next00:43
*** xarses has quit IRC00:43
clarkbfungi: there is a flag to n way it00:44
*** ddieterly has joined #openstack-infra00:44
fungi--workers 000:45
jeblairincrease that very very slowly :)00:45
fungiapparently that does as many workers as you have processors00:45
fungi--workers 2 may be a good place to start ;_00:45
*** claudiub has quit IRC00:46
*** sarob has quit IRC00:48
*** openstackgerrit_ has quit IRC00:48
*** EricGonczer_ has quit IRC00:49
*** openstackgerrit_ has joined #openstack-infra00:49
*** nwkarsten has joined #openstack-infra00:49
anteaya;_ looks sinister00:49
clarkbwe be interseting to see if jenkins can handle N way updates to its configs without serializing00:50
*** pvaneck has quit IRC00:50
fungiyeah, i think i remember when jjb briefly defaulted to --workers 0 by default. if memory serves that was not a great day00:52
*** chlong has joined #openstack-infra00:52
fungianyway, we have 8 jenkins masters currently running 2-way parallel jjb updates00:52
fungiload average on zuul is hovering around 200:53
clarkbzuul is about ot merge a bunch of stuff too00:53
fungiactually its load is falling again00:53
*** takashi has joined #openstack-infra00:53
fungiyeah, it's more likely from that00:53
jeblairwe should keep an eye on jenkins itself and nodepool00:54
jeblairwith the gearman change, i think we're more likely to see trouble from api timeouts than gearman traffic00:55
fungiyeah, i'm expecting this to make jenkins01 and 02 unhappy since they already have a devil of a time avoiding api timeouts/proxy errors00:55
anteayawhy are 01 and 02 so different from the others?00:57
*** sridhar_ram1 has quit IRC00:57
*** tsg has quit IRC00:57
fungibuilt earlier, i think on older/slower instance flavors00:57
*** ajmiller has quit IRC00:57
anteayacan they be upgraded?00:57
fungiyeah, they can be rebuilt, it's just nobody's gotten around to it because there are manual steps involved in copying configuration over00:58
anteayaso something for a slow day00:59
anteayawhen we are all staring at each other and twiddling our thumbs00:59
fungiright, luckily we're thoroughly parallel wit these servers, so could take one offline for a while to rebuild it with no real negative impact00:59
anteayaor could we spin up two more and set them up and just drop these?01:00
anteayanothing needs to be migrated, yes?01:00
fungiright, we could build 08 and 09 and phase these out. a fairly similar amount of work01:01
*** ddieterly has quit IRC01:01
anteayayes, just nothing to be saved or transfered01:01
fungiwe're no longer so tight on quota that it's a problem01:01
*** sridhar_ram has joined #openstack-infra01:01
anteayathink it needs a spec?01:02
anteayaI can put up a spec if it would be useful01:02
fungino, just needs a root admin with available time to get it done01:02
anteayaa spec would be easier to come by01:02
*** Qiming has joined #openstack-infra01:07
*** angdraug has quit IRC01:12
*** esker has quit IRC01:21
*** esker has joined #openstack-infra01:22
anteayajesusaur: merge conflict on your patch:
*** SumitNaiksatam has quit IRC01:23
fungiso far the only ones i've found which weren't up to date by the time i started running on them were jenkins.o.o (27m2s to complete) and jenkins01 (which is still running)01:27
jesusauranteaya: thanks for pointing that out01:27
fungiall the rest were "0 jobs updated" and completed in around 6-8 minutes01:27
clarkbfungi: I think somethibg is wrong with caching on 0201:28
clarkbiirc the cache tells jjb whether or not its locsl jobs match the jenkins?01:28
*** zhurong_ has joined #openstack-infra01:29
anteayajesusaur: welcome01:29
fungiclarkb: i can look at the cache and see if i spot anything odd, i guess01:29
clarkbfungi: maybe it isnt readable or being written?01:30
fungiit's empty at the moment01:30
fungimaybe it isn't being written01:30
*** dmsimard|pto has quit IRC01:31
*** resker has joined #openstack-infra01:35
*** FallenPegasus has quit IRC01:35
*** sridhar_ram has quit IRC01:35
*** esker has quit IRC01:37
*** dimtruck is now known as zz_dimtruck01:39
*** tphummel has joined #openstack-infra01:40
*** sdake has quit IRC01:40
*** Sukhdev has joined #openstack-infra01:41
*** _ari_ has joined #openstack-infra01:41
jeblairjhesketh: v3 is almost caught back up to master; see the parents of
*** nikhil_k has quit IRC01:42
*** hamzy has joined #openstack-infra01:45
*** nwkarsten has quit IRC01:45
*** tzn_ has quit IRC01:45
jheskethjeblair: wouldn't we be better off doing another merge of master into zuulv3 rather than redoing tags?01:46
*** nwkarsten has joined #openstack-infra01:46
jheskethjeblair: I actually had started that yesterday, but didn't finish resolving the merge conflicts01:46
* jhesketh can do so now01:46
jeblairjhesketh: if you want... it's mostly unimplemented because it affects parts that are radically different... it's basically just adding the test debt so we know to fix it later01:47
*** Sukhdev has quit IRC01:47
jheskethjeblair: right, but if we redo it won't that make merging master into v3 (or even v3 into master if we choose to do it that way instead of just switching branches over) more difficult?01:48
jeblairjhesketh: yeah, it might make the git history nicer01:48
jheskethjeblair: do you think we should get into master first (and then merged into v3)01:49
jeblairjhesketh: probably so.  i'll try to review it soon.01:50
*** nwkarsten has quit IRC01:50
* craige cleared another Phabricator hurdle this morning. Feeling the momentum :-D01:50
jeblairjhesketh: oh, you know what that change makes sense to me right now.  +3.  :)01:51
*** xarses has joined #openstack-infra01:52
jheskethjeblair: cheers01:52
jheskethjeblair: the follow on change ( needs some discussion if you have a moment too01:52
jeblairjhesketh: so feel free to go ahead and bash that in and do whatever you need to do to catch v3 up to master; then if you have a moment to look at 268386 that would be cool.01:52
jheskethjeblair: I know you said you don't want to change the way connections do a reload, but if not, should we look at copying the cache across connections01:53
jheskethjeblair: will do01:53
jeblairjhesketh: yeah, i want to think about that more in the v3 context rather than v2.  i don't have everything i need in my head yet though.01:54
jheskethjeblair: I guess it only matters for the v2 case if it's affecting the memory leak (that I've failed to track down)01:54
jheskethit should (at least in theory, I haven't checked) decrease the load, but maybe we don't care about that until v301:55
*** nwkarsten has joined #openstack-infra01:55
jeblairjhesketh: yeah, i just desperately want v2 to be stable and stop moving at all so we can work on v301:55
jeblairwe can deal with the load01:55
*** tphummel has quit IRC01:55
openstackgerritMerged openstack-infra/zuul: Cache is held and managed by connections
*** tphummel has joined #openstack-infra01:57
*** bpokorny_ has joined #openstack-infra02:00
openstackgerritMerged openstack-infra/project-config: CentOS AFS build : use homedir
*** EricGonczer_ has joined #openstack-infra02:02
jeblairjhesketh: each time we reload, we create a new driver object and register its use with a connection, right?02:02
jheskethjeblair: correct.. we also create a new connection object02:02
*** mtanino has quit IRC02:02
jeblairjhesketh: do we?02:02
jheskethjeblair: that's what that follow up change/discussion is about02:03
*** bpokorny has quit IRC02:04
*** dingyichen has quit IRC02:04
jeblairjhesketh: ok good.  that's the safer thing to do for now.02:05
*** bpokorny_ has quit IRC02:05
*** dims has quit IRC02:08
*** dalgaaf has quit IRC02:08
*** gyee has quit IRC02:10
*** bapalm has quit IRC02:11
*** nwkarsten has quit IRC02:13
*** nwkarsten has joined #openstack-infra02:14
*** asettle has quit IRC02:15
*** esikachev has joined #openstack-infra02:15
*** asettle has joined #openstack-infra02:15
*** tzn has joined #openstack-infra02:16
*** bapalm has joined #openstack-infra02:16
*** Jeffrey4l has joined #openstack-infra02:16
*** dims has joined #openstack-infra02:17
*** gildub has quit IRC02:17
*** nwkarsten has quit IRC02:18
*** esikachev has quit IRC02:19
openstackgerritClint Adams proposed openstack-infra/project-config: Add periodic job to autogenerate grenade plugins list
*** EricGonc_ has joined #openstack-infra02:24
*** yamamoto_ has joined #openstack-infra02:25
*** EricGonczer_ has quit IRC02:27
*** moravec has joined #openstack-infra02:27
*** rguillebert has quit IRC02:27
openstackgerritJoshua Hesketh proposed openstack-infra/zuul: Merge branch 'master' into feature/zuulv3
jheskethjeblair: ^02:28
jheskethjeblair: if you'd like I can rebase your changes (268386 and above) onto that merge change (probably dropping the re-implementation of tags)02:30
openstackgerritIan Wienand proposed openstack-infra/nodepool: Fail on bad options to fake-image-create
openstackgerritClint Adams proposed openstack-infra/project-config: Add periodic job to autogenerate tempest plugins list
*** fawadkhaliq has quit IRC02:32
clarkb#success Infra is running logstash 2.0 now02:32
openstackstatusclarkb: Added success to Success page02:32
*** hichihara has joined #openstack-infra02:32
*** Guest38484 is now known as mfisch02:33
*** mfisch is now known as Guest1374902:34
*** nikhil has joined #openstack-infra02:34
*** kevinshort has quit IRC02:34
fungijenkins01 is still jjb'ing02:36
*** woodster_ has quit IRC02:37
*** kevinshort has joined #openstack-infra02:37
*** EricGonc_ has quit IRC02:38
*** kevinsho_ has joined #openstack-infra02:40
openstackgerritIan Wienand proposed openstack-infra/project-config: [wip] Revert "Add CentOS AFS build job to project-config experimental"
openstackgerritIan Wienand proposed openstack-infra/project-config: CentOS AFS build : move output to $WORKSPACE
*** nikhil has quit IRC02:43
clarkblooks like kibana4 wants to create an index and bails out early if it cna't02:44
*** tqtran-afk has quit IRC02:45
*** nwkarsten has joined #openstack-infra02:45
clarkbI amgoing to manually create that index and see if we can trick it into running without being able to write to it02:47
clarkb(this is a first step in deploying kibana4)02:47
*** yamamoto_ has quit IRC02:48
clarkbhrm that didn't do it...02:49
*** nwkarsten has quit IRC02:50
*** gildub has joined #openstack-infra02:50
*** sdake has joined #openstack-infra02:51
clarkblooks like kibana 4 lets anyone configure it via the web ui? will have to figure out how to avoid that02:52
clarkb"caution you can break things here" probably not great to expose that on the internet02:53
*** Sukhdev has joined #openstack-infra02:54
*** sdake_ has joined #openstack-infra02:55
*** ajmiller has joined #openstack-infra02:56
*** sdake has quit IRC02:56
clarkbthis is very new and different02:58
mtreinishclarkb: where's your sense of adventure :)02:58
*** dims has quit IRC02:58
*** ayoung has joined #openstack-infra02:58
*** sputnik13 has quit IRC03:00
*** ajmiller_ has joined #openstack-infra03:00
clarkbI mean it queries things03:01
clarkbit just has so many other knobs and bells and whistles and scary config that appear exposed ot everyone that I need to figure out03:01
*** kevinsho_ has quit IRC03:03
*** sdake_ has quit IRC03:03
*** ajmiller has quit IRC03:03
*** kevinsho_ has joined #openstack-infra03:06
*** kevinshort has quit IRC03:09
*** Sukhdev has quit IRC03:09
*** kevinshort has joined #openstack-infra03:09
*** kevinsho_ has quit IRC03:11
*** kevinsho_ has joined #openstack-infra03:11
fungigrr, 01 finally hit a proxy error03:11
*** kevinsho_ has quit IRC03:12
*** kevinsho_ has joined #openstack-infra03:12
*** nwkarsten has joined #openstack-infra03:13
*** dims has joined #openstack-infra03:13
*** kevinshort has quit IRC03:14
*** kevinshort has joined #openstack-infra03:14
openstackgerritIan Wienand proposed openstack-infra/nodepool: Add some tracking for image sizes
ianw^ that should make for a fun graph at some point03:15
*** nwkarsten has quit IRC03:18
*** dims has quit IRC03:18
*** thorst__ has quit IRC03:20
*** thorst has joined #openstack-infra03:20
*** zul_ has joined #openstack-infra03:25
*** yamahata_ has joined #openstack-infra03:25
*** mknawabi_ has joined #openstack-infra03:25
*** rhallisey has quit IRC03:27
*** hieulq has joined #openstack-infra03:28
*** thorst has quit IRC03:29
*** mtreinish_ has joined #openstack-infra03:29
*** notmyname_ has joined #openstack-infra03:30
*** krotscheck has quit IRC03:30
*** yamahata has quit IRC03:30
*** ildikov has quit IRC03:30
*** zul has quit IRC03:30
*** notmyname has quit IRC03:30
*** mtreinish has quit IRC03:30
*** mknawabi has quit IRC03:30
*** ildikov has joined #openstack-infra03:30
*** mtreinish_ is now known as mtreinish03:30
*** notmyname_ is now known as notmyname03:30
*** krotscheck has joined #openstack-infra03:31
*** zz_dimtruck is now known as dimtruck03:31
*** hdd has joined #openstack-infra03:33
*** aeng has quit IRC03:33
*** aeng has joined #openstack-infra03:34
*** yamamoto has joined #openstack-infra03:36
*** yamahata_ has quit IRC03:36
*** aeng has quit IRC03:36
*** apoorvad has quit IRC03:37
*** tqtran-afk has joined #openstack-infra03:42
*** sdake has joined #openstack-infra03:42
fungiinitial pass on jenkins01 was 174m17s but i've resumed it now to deal with the api errors it hit that time around03:45
*** links has joined #openstack-infra03:45
*** tqtran-afk has quit IRC03:46
*** dimtruck is now known as zz_dimtruck03:52
*** nikhil has joined #openstack-infra03:54
*** nikhil_k has joined #openstack-infra03:54
*** ildikov has quit IRC03:55
*** ildikov has joined #openstack-infra03:55
*** Nakato_ is now known as Nakato03:56
bswartzshouldn't pushing a tag cause the post pipeline to run again?03:56
bswartzor am I missing something?03:56
clarkbrelease pipeline03:57
* bswartz might be an idiot03:57
bswartzis there an example of a job that publishes tagged tarballs?04:01
clarkbto pypi?04:01
bswartzto tarballs.o.o04:01
*** jpr has quit IRC04:03
*** baoli has joined #openstack-infra04:04
bswartzactually I think I'm looking for the {name}-tarball job04:04
bswartzwhich nearly every project uses04:04
clarkb line 49304:04
*** jpr has joined #openstack-infra04:05
clarkbbut it is only run if you have the template on line 493 or explicitly run it04:05
*** dingyichen has joined #openstack-infra04:07
bswartzokay so what's the difference between "release" and "release-post"?04:07
bswartzwouldn't "release-post" be a more appropriate place to upload to tarballs.o.o?04:08
bswartzoh I see that pipe only has a single purpose04:08
clarkbI would have04:08
clarkbwould have tonread the release-post pipeline tonfigure out what it is for04:09
*** tphummel has quit IRC04:09
bswartzdescription: This pipeline runs release-process-critical jobs that operate after specific changes are merged.04:09
bswartzit's referred to in exactly 1 place04:09
bswartzthe openstack/release project04:09
*** amrith is now known as _amrith_04:10
fungiyeah, it's basically a separate "post" pipeline used for generating release notes so that we can up the priority on it, since the normal post pipeline gets lowest priority04:11
*** sdake has quit IRC04:12
openstackgerritBen Swartzlander proposed openstack-infra/project-config: Run publish job on release of manila-image-elements
*** esikachev has joined #openstack-infra04:16
*** fedexo has joined #openstack-infra04:17
*** esikachev has quit IRC04:20
docaedobswartz: do you still think you'll want to link to an image in app catalog? (which I'm highly in favor of) - just want to make sure I'm staying on top of it in case you or anyone else on the manila team needs help04:20
bswartzdocaedo: absolutely04:20
bswartzin fact that's what I'm working on atm04:20
bswartzon Wednesday I set out to push a change to app-catalog but I realized we didn't have a tagged build yet that we could use04:21
bswartzthat let me to realize we had ACL issues and other project-config issues to solve first04:21
docaedobswartz: sweet! I'm going to be stepping away for reading to the kids, but will be around either later tonight, or else early(ish) tomorrow morning, so I'm up to help however I can04:21
bswartzso I'm working through the issue so I can get a tagged build first04:22
bswartzonce I have a tagged build you'll see my change go in04:22
docaedoawesome, sounds like that build will be a tomorrow thing then :)04:22
bswartzyeah don't worry about it tonight04:22
bswartzthe wheels of zuul turn slowly04:23
*** sdake has joined #openstack-infra04:23
docaedoindeed .. also when you make your app catalog PR, be sure the first line of the commit message does not end in a period, and the body of the message should be in haiku format, though we'll accept a limerick if necessary04:23
docaedoas long as we can all agree that your poem conjures the majesty of OpenStack, we'll be able to merge quickly I'm sure.04:24
fungigrr, jenkins01 has hit another api timeout on the second jjb update pass as well04:26
*** thorst has joined #openstack-infra04:26
*** sdake_ has joined #openstack-infra04:27
*** sdake has quit IRC04:29
*** baoli has quit IRC04:29
*** [1]Thelo has joined #openstack-infra04:30
*** hieulq has quit IRC04:32
*** jaosorior has joined #openstack-infra04:32
*** ajmiller_ has quit IRC04:33
*** Thelo has quit IRC04:33
*** [1]Thelo is now known as Thelo04:33
*** thorst has quit IRC04:34
*** treaki_ has joined #openstack-infra04:35
*** treaki__ has quit IRC04:39
openstackgerritCraige McWhirter proposed openstack-infra/system-config: Added RemoteUser Provider & OpenID configs
*** sdake_ has quit IRC04:48
openstackgerritBen Swartzlander proposed openstack-infra/project-config: Refactor manila check/gate queue
*** zz_dimtruck is now known as dimtruck04:57
*** tqtran-afk has joined #openstack-infra04:58
*** jamespage has quit IRC05:00
*** gnuoy has quit IRC05:00
*** gongysh has quit IRC05:00
*** gnuoy has joined #openstack-infra05:00
*** jamespage has joined #openstack-infra05:00
*** tqtran-afk has quit IRC05:02
*** rossella_s has quit IRC05:03
*** rossella_s has joined #openstack-infra05:03
fungisecond pass on jenkins01: 84m27s05:09
fungiresuming again05:09
openstackgerritBen Swartzlander proposed openstack-infra/project-config: Refactor manila check/gate queue
*** kdas__ is now known as kushal05:17
*** kushal has quit IRC05:18
*** kushal has joined #openstack-infra05:18
*** armax has joined #openstack-infra05:20
*** sridhar_ram has joined #openstack-infra05:23
*** _nadya_ has joined #openstack-infra05:23
*** nikhil has quit IRC05:23
*** fedexo has quit IRC05:27
*** _nadya_ has quit IRC05:30
*** thorst has joined #openstack-infra05:32
*** thorst has quit IRC05:39
*** heyongli has joined #openstack-infra05:39
*** heyongli has quit IRC05:41
*** heyongli has joined #openstack-infra05:41
*** sarob has joined #openstack-infra05:45
*** heyongli has quit IRC05:46
*** heyongli has joined #openstack-infra05:46
*** Jeffrey4l has quit IRC05:47
*** cburgess has quit IRC05:49
*** kingia_ has quit IRC05:50
*** rlandy has quit IRC05:50
*** kingia has joined #openstack-infra05:50
*** sarob has quit IRC05:50
*** rlandy has joined #openstack-infra05:50
*** amotoki has joined #openstack-infra05:50
*** amotoki has quit IRC05:50
*** melwitt has quit IRC05:51
*** Slower has quit IRC05:52
*** daemontool__ has quit IRC05:52
*** cburgess has joined #openstack-infra05:52
*** Slower has joined #openstack-infra05:52
*** armax has quit IRC05:53
*** zhurong_ has quit IRC05:53
*** tzn has quit IRC05:55
*** armax has joined #openstack-infra05:55
*** tzn has joined #openstack-infra05:55
*** dimtruck is now known as zz_dimtruck05:55
*** heyongli has quit IRC05:56
*** zhurong_ has joined #openstack-infra05:56
openstackgerritAndreas Jaeger proposed openstack-infra/project-config: puppet: fix cookiecutter job template
*** jaosorior has quit IRC05:56
*** ayoung has quit IRC05:56
*** Douhet has quit IRC05:56
*** bryan_att has quit IRC05:56
*** fkautz has quit IRC05:56
*** Ng has quit IRC05:56
*** zxiiro has quit IRC05:56
*** fdegir has quit IRC05:56
*** simonmcc has quit IRC05:56
*** ashtokolov has quit IRC05:56
*** dansmith has quit IRC05:56
*** yonglihe has quit IRC05:56
*** evgenyl has quit IRC05:56
*** mwhahaha has quit IRC05:56
*** Volundr has quit IRC05:56
*** mescanef has quit IRC05:56
*** cfarquhar has quit IRC05:56
*** heyongli has joined #openstack-infra05:57
*** melwitt has joined #openstack-infra05:57
*** yamahata_ has joined #openstack-infra05:59
*** mariojv has quit IRC05:59
*** krotscheck has quit IRC06:01
*** mariojv has joined #openstack-infra06:02
*** ayoung has joined #openstack-infra06:02
*** Douhet has joined #openstack-infra06:02
*** fkautz has joined #openstack-infra06:02
*** bryan_att has joined #openstack-infra06:02
*** Ng has joined #openstack-infra06:02
*** zxiiro has joined #openstack-infra06:02
*** fdegir has joined #openstack-infra06:02
*** simonmcc has joined #openstack-infra06:02
*** ashtokolov has joined #openstack-infra06:02
*** dansmith has joined #openstack-infra06:02
*** yonglihe has joined #openstack-infra06:02
*** evgenyl has joined #openstack-infra06:02
*** mwhahaha has joined #openstack-infra06:02
*** Volundr has joined #openstack-infra06:02
*** mescanef has joined #openstack-infra06:02
*** cfarquhar has joined #openstack-infra06:02
*** vponomaryov has quit IRC06:03
*** vponomaryov has joined #openstack-infra06:04
*** jaosorior has joined #openstack-infra06:04
*** heyongli has quit IRC06:06
*** heyongli has joined #openstack-infra06:07
*** krotscheck has joined #openstack-infra06:09
*** yamahata_ has quit IRC06:10
openstackgerritOpenStack Proposal Bot proposed openstack-infra/project-config: Normalize projects.yaml
*** esikachev has joined #openstack-infra06:16
*** heyongli has quit IRC06:17
*** heyongli has joined #openstack-infra06:17
*** openstackgerrit_ has quit IRC06:17
*** cloudtrainme has joined #openstack-infra06:17
*** openstackgerrit_ has joined #openstack-infra06:18
*** cloudtrainme has quit IRC06:18
*** Guest68942 has joined #openstack-infra06:18
openstackgerritMerged openstack-infra/project-config: puppet: fix cookiecutter job template
*** esikachev has quit IRC06:21
*** hdd has quit IRC06:22
*** eezhova has quit IRC06:23
*** eezhova has joined #openstack-infra06:23
*** heyongli has quit IRC06:27
*** Sukhdev has joined #openstack-infra06:28
*** hieulq has joined #openstack-infra06:35
*** thorst has joined #openstack-infra06:37
*** baoli has joined #openstack-infra06:41
openstackgerritPurandhar Sairam Mannidi proposed openstack/diskimage-builder: Add support for building images capable of UEFI
*** thorst has quit IRC06:43
openstackgerritPurandhar Sairam Mannidi proposed openstack/diskimage-builder: Add support for building images capable of UEFI
openstackgerritOpenStack Proposal Bot proposed openstack/requirements: Updated from generate-constraints
*** baoli has quit IRC06:46
*** noorul has joined #openstack-infra06:46
*** mrmartin has joined #openstack-infra06:47
noorulI am trying to setup openstack-ci locally06:48
noorulI see mysql reference everywhere but can't find puppet manifest to setup mysql06:48
*** dalgaaf has joined #openstack-infra06:49
*** rcernin has joined #openstack-infra06:49
*** watanabe_isao has joined #openstack-infra06:50
*** Jeffrey4l has joined #openstack-infra06:55
*** sridhar_ram has quit IRC06:56
*** _nadya_ has joined #openstack-infra06:59
nibalizernoorul: most of the time we use trove instances06:59
*** gildub has quit IRC07:04
*** sbalukoff has joined #openstack-infra07:05
*** asettle has quit IRC07:07
*** camunoz has quit IRC07:07
*** asettle has joined #openstack-infra07:08
*** jaosorior has quit IRC07:10
*** mariojv has quit IRC07:10
*** ayoung has quit IRC07:10
*** Douhet has quit IRC07:10
*** bryan_att has quit IRC07:10
*** fkautz has quit IRC07:10
*** Ng has quit IRC07:10
*** zxiiro has quit IRC07:10
*** fdegir has quit IRC07:10
*** simonmcc has quit IRC07:10
*** ashtokolov has quit IRC07:10
*** dansmith has quit IRC07:10
*** yonglihe has quit IRC07:10
*** evgenyl has quit IRC07:10
*** mwhahaha has quit IRC07:10
*** Volundr has quit IRC07:10
*** mescanef has quit IRC07:10
*** cfarquhar has quit IRC07:10
*** zhurong_ has quit IRC07:11
*** zhurong_ has joined #openstack-infra07:12
*** rlandy has quit IRC07:16
noorulnibalizer: thank you!07:17
nibalizermysql::server from the mysql module works pretty well07:17
*** asettle has quit IRC07:18
*** jaosorior has joined #openstack-infra07:21
*** mariojv has joined #openstack-infra07:21
*** ayoung has joined #openstack-infra07:21
*** Douhet has joined #openstack-infra07:21
*** fkautz has joined #openstack-infra07:21
*** bryan_att has joined #openstack-infra07:21
*** Ng has joined #openstack-infra07:21
*** zxiiro has joined #openstack-infra07:21
*** fdegir has joined #openstack-infra07:21
*** simonmcc has joined #openstack-infra07:21
*** ashtokolov has joined #openstack-infra07:21
*** dansmith has joined #openstack-infra07:21
*** yonglihe has joined #openstack-infra07:21
*** evgenyl has joined #openstack-infra07:21
*** mwhahaha has joined #openstack-infra07:21
*** Volundr has joined #openstack-infra07:21
*** mescanef has joined #openstack-infra07:21
*** cfarquhar has joined #openstack-infra07:21
*** scheuran has joined #openstack-infra07:24
*** Sukhdev has quit IRC07:24
*** hieulq has quit IRC07:24
*** hieulq has joined #openstack-infra07:26
*** jtomasek has joined #openstack-infra07:27
*** amotoki has joined #openstack-infra07:29
noorulnibalizer: I installed mysql manually for the time being07:30
noorulnibalizer: I am now stuck at
*** mikelk has joined #openstack-infra07:40
*** thorst has joined #openstack-infra07:41
*** gildub has joined #openstack-infra07:42
openstackgerritAndreas Scheuring proposed openstack-infra/system-config: Correct Logstash Query Example
*** Guest68942 has quit IRC07:45
openstackgerritAndreas Scheuring proposed openstack-infra/system-config: Correct Logstash Query Example
*** asettle has joined #openstack-infra07:45
*** jaosorior has quit IRC07:47
*** mariojv has quit IRC07:47
*** ayoung has quit IRC07:47
*** Douhet has quit IRC07:47
*** bryan_att has quit IRC07:47
*** fkautz has quit IRC07:47
*** Ng has quit IRC07:47
*** zxiiro has quit IRC07:47
*** fdegir has quit IRC07:47
*** simonmcc has quit IRC07:47
*** ashtokolov has quit IRC07:47
*** dansmith has quit IRC07:47
*** yonglihe has quit IRC07:47
*** evgenyl has quit IRC07:47
*** mwhahaha has quit IRC07:47
*** Volundr has quit IRC07:47
*** mescanef has quit IRC07:47
*** cfarquhar has quit IRC07:47
*** nikhil has joined #openstack-infra07:47
*** thorst has quit IRC07:49
*** asettle has quit IRC07:50
*** mrmartin has quit IRC07:50
*** mrmartin has joined #openstack-infra07:52
*** fhubik has joined #openstack-infra07:52
*** jaosorior has joined #openstack-infra07:54
*** mariojv has joined #openstack-infra07:54
*** ayoung has joined #openstack-infra07:54
*** Douhet has joined #openstack-infra07:54
*** fkautz has joined #openstack-infra07:54
*** bryan_att has joined #openstack-infra07:54
*** Ng has joined #openstack-infra07:54
*** zxiiro has joined #openstack-infra07:54
*** fdegir has joined #openstack-infra07:54
*** simonmcc has joined #openstack-infra07:54
*** ashtokolov has joined #openstack-infra07:54
*** dansmith has joined #openstack-infra07:54
*** yonglihe has joined #openstack-infra07:54
*** evgenyl has joined #openstack-infra07:54
*** mwhahaha has joined #openstack-infra07:54
*** Volundr has joined #openstack-infra07:54
*** mescanef has joined #openstack-infra07:54
*** cfarquhar has joined #openstack-infra07:54
*** mrmartin has quit IRC07:56
AJaegerGood morning! I have one cleanup for the ubuntu-trusty jobs: Rename of tox-nodb to to: Could a second project-config core review this, please?07:58
*** kzaitsev_mb has joined #openstack-infra07:59
*** harlowja_at_home has quit IRC08:00
openstackgerritMerged openstack-infra/project-config: Delay release for charm projects
*** dizquierdo has joined #openstack-infra08:02
openstackgerritMerged openstack-infra/project-config: Normalize projects.yaml
openstackgerritOleg Gelbukh proposed openstack-infra/project-config: Add project 'fuel-tuning-box'
*** jlanoux has joined #openstack-infra08:08
*** fhubik has quit IRC08:13
*** gongysh_ has joined #openstack-infra08:16
*** claudiub has joined #openstack-infra08:18
*** e0ne has joined #openstack-infra08:21
*** claudiub|2 has joined #openstack-infra08:21
*** HeOS has quit IRC08:21
openstackgerritOleg Gelbukh proposed openstack-infra/project-config: Add project 'fuel-tuning-box'
*** vincentfrancoise has joined #openstack-infra08:23
*** jamielennox is now known as jamielennox|away08:23
*** claudiub has quit IRC08:24
*** dtardivel has joined #openstack-infra08:26
*** matrohon has joined #openstack-infra08:28
*** hashar has joined #openstack-infra08:32
*** arxcruz has joined #openstack-infra08:33
*** _nadya_ has quit IRC08:34
*** bauzas is now known as bauwser08:34
*** shardy has joined #openstack-infra08:35
*** yamahata_ has joined #openstack-infra08:35
*** sarob has joined #openstack-infra08:37
*** flepied has quit IRC08:37
*** vincentll has joined #openstack-infra08:38
*** pcaruana has joined #openstack-infra08:41
*** Jeffrey4l has quit IRC08:41
*** sarob has quit IRC08:42
*** ihrachys has joined #openstack-infra08:45
*** abregman has joined #openstack-infra08:45
*** thorst has joined #openstack-infra08:46
*** HeOS has joined #openstack-infra08:48
*** jeeloo_ has joined #openstack-infra08:49
*** chlong has quit IRC08:49
*** zeih_ has joined #openstack-infra08:51
*** jeeloo has joined #openstack-infra08:52
*** jeeloo_ has quit IRC08:52
*** thorst has quit IRC08:54
*** kzaitsev_mb has quit IRC08:54
*** ifarkas has joined #openstack-infra08:55
*** noorul has quit IRC08:57
*** watanabe_isao has quit IRC09:01
*** rossella_s has quit IRC09:03
*** dizquierdo has quit IRC09:03
*** rossella_s has joined #openstack-infra09:03
*** sfinucan has joined #openstack-infra09:04
jeelooHi, I'm trying to set up zuul in my own environment and I have managed to get most of the flow working. The issue that I have run into at the moment is that every time there is more than 2 changes in ZUUL_CHANGES the zuul-cloner fails with "IndexError: No item found with id u'origin/HEAD'"  I'm on Zuul 2.1.0, any thougts or suggestions wold be appreciated.09:06
*** lucas-afk is now known as lucasagomes09:06
openstackgerritAlexey Ovchinnikov proposed openstack-infra/project-config: [WIP] Add publishing capability to buildimage-lxd job
*** salv-orlando has joined #openstack-infra09:09
*** zhurong_ has quit IRC09:10
*** hichihara has quit IRC09:10
*** sbelous_ has joined #openstack-infra09:11
*** flepied has joined #openstack-infra09:13
*** electrofelix has joined #openstack-infra09:15
*** daemontool has joined #openstack-infra09:16
*** esikachev has joined #openstack-infra09:18
*** ramishra has quit IRC09:19
*** jordanP has joined #openstack-infra09:22
*** abregman is now known as abregman|afk09:23
*** jistr has joined #openstack-infra09:24
*** ramishra has joined #openstack-infra09:25
*** salv-orlando has quit IRC09:25
*** yamahata_ has quit IRC09:26
*** pblaho has joined #openstack-infra09:27
*** tzn has quit IRC09:28
*** tzn has joined #openstack-infra09:28
*** takashi has quit IRC09:29
*** jordanP has quit IRC09:29
*** jordanP has joined #openstack-infra09:31
*** Daisy has joined #openstack-infra09:32
*** Daisy has quit IRC09:37
*** derekh has joined #openstack-infra09:40
*** yamamoto has quit IRC09:40
*** vgridnev has joined #openstack-infra09:42
*** _nadya_ has joined #openstack-infra09:45
*** _nadya_ has quit IRC09:50
*** _nadya__ has joined #openstack-infra09:50
*** thorst has joined #openstack-infra09:51
*** esikachev has quit IRC09:53
*** placko_ has joined #openstack-infra09:57
*** abregman|afk has quit IRC09:57
*** thorst has quit IRC09:59
*** esikachev has joined #openstack-infra10:02
*** rvasilets__ has joined #openstack-infra10:02
*** daemontool_ has joined #openstack-infra10:02
*** andreykurilin__ has joined #openstack-infra10:06
*** nijaba has quit IRC10:06
*** daemontool has quit IRC10:06
*** hieulq has quit IRC10:07
*** nijaba has joined #openstack-infra10:07
*** _degorenko|afk is now known as degorenko10:08
*** tpsilva has joined #openstack-infra10:09
*** Qiming has quit IRC10:11
*** Guest20150 has quit IRC10:14
*** kushal has quit IRC10:14
*** Hal has joined #openstack-infra10:16
*** rguillebert has joined #openstack-infra10:16
*** Hal is now known as Guest6986210:16
openstackgerritmathieu bultel proposed openstack-infra/shade: Add normalize stack function for heat stack_list
openstackgerritSwapnil Kulkarni (coolsvap) proposed openstack-infra/storyboard: Replace deprecated LOG.warn with LOG.warning
*** kzaitsev_mb has joined #openstack-infra10:28
*** jcoufal has joined #openstack-infra10:30
*** sdague has joined #openstack-infra10:30
*** sambetts|afk is now known as sambetts10:31
AJaegermorning, yolanda !10:35
yolandahow are things today?10:35
yolandai'm seeing gerrit a bit slow, but i checked logs and there are no relevant errors10:36
*** samuelBartel has joined #openstack-infra10:36
AJaegeryeah, it's slowing down, had a few 502s ;(10:37
*** gongysh_ has quit IRC10:38
yolandaAJaeger, about ... do we have any policy for enabling new vhosts in infra?10:40
openstackgerritMerged openstack-infra/project-config: Publish Anchor to pypi
openstackgerritMerged openstack-infra/project-config: Modify freezer job for tempest tests
AJaegeryolanda: I don't know ;( If you look at the referenced email, fungi even suggested it but you might want to double check with him.10:42
AJaegeryolanda: I fear that adding the chnage needs some manual work for DNS.10:42
* AJaeger would love to see this move forward10:42
yolandayes, that's sure10:42
*** daemontool__ has joined #openstack-infra10:43
openstackgerritMerged openstack-infra/project-config: Added npm-install macro to swagger build job
yolandai can +2 but wait for approval until we check with more infra-root10:43
AJaegeryolanda: please include and in your review session10:44
yolandaseems that fungi agrees so this should move forward10:44
AJaegeryolanda: it needs a second +2 anyhow ;)10:44
yolandayep, system-config...10:44
*** daemontool__ is now known as daemontool10:45
* AJaeger rebases 291349 now...10:45
yolandai was just going to tell you :)10:46
*** daemontool_ has quit IRC10:46
AJaegerthe anchor change broke it ;(10:47
*** sarob has joined #openstack-infra10:48
openstackgerritAndreas Jaeger proposed openstack-infra/project-config: Rename tox-nodb-ENV to tox-ENV
AJaegerrebased ^10:48
yolandacool i'll review10:48
yolandagerrit is slowering for moment, i think is time for restart10:49
yolandai started to get 502 on every review10:49
AJaegeryeah, it's getting worse. Let's restart while the load is light if you have time10:50
yolanda#status alert Gerrit is going to be restarted due to bad performance10:50
openstackstatusyolanda: sending alert10:50
AJaegeryolanda: Send the status ok only after the #status alert has finished... you might confuse the bot otherwise. I think last time you were too quick ;)10:51
AJaegerbut it worked out just fine :)10:51
yolandaah yes, or alert too slow :)10:51
AJaegertoo many channels ;)10:52
sbelous_sdague: ping10:52
*** sarob has quit IRC10:53
-openstackstatus- NOTICE: Gerrit is going to be restarted due to bad performance10:53
*** ChanServ changes topic to "Gerrit is going to be restarted due to bad performance"10:53
*** dims has joined #openstack-infra10:54
Zarathat message always makes me feel so sorry for gerrit10:54
yolandapoor gerrit... it has been restarted already, but bot is too slow announcing...10:55
openstackstatusyolanda: finished sending alert10:56
yolandahere it comes...10:56
AJaegeryolanda: next time perhaps just a #status notice? That does not change the topic and just leaves a message...10:56
yolanda#status ok Gerrit has been restarted successfully10:56
openstackstatusyolanda: sending ok10:56
yolandayes, it may work better10:56
*** thorst has joined #openstack-infra10:56
AJaegerNot sure what's the best practice is - or if one exists. But it looks like the restart is soo quick that nobody notices with the 502s;)10:57
*** asettle has joined #openstack-infra10:57
*** asettle has quit IRC10:57
*** mrmartin has joined #openstack-infra10:57
yolandayes, it takes much more time to send the alerts than to restart gerrit10:57
openstackgerritPurandhar Sairam Mannidi proposed openstack/diskimage-builder: Add support for building images capable of UEFI
AJaegeryolanda: can you kill ? Looking at zuul status, the job runs for four hours but jenkins03 hasn't seen any action for 3+ hours ;(10:58
*** ChanServ changes topic to "Discussion of OpenStack Developer and Community Infrastructure | docs | bugs | source | channel logs"10:58
-openstackstatus- NOTICE: Gerrit has been restarted successfully10:58
openstackgerritIlya Shakhat proposed openstack-infra/project-config: Add new project "performa"
AJaegerthat's on vexxhost for the record10:59
*** matrohon has quit IRC10:59
yolandapabelanger told that vexxhost was having problems due to slow i/o yesterday10:59
*** Qiming has joined #openstack-infra11:01
openstackstatusyolanda: finished sending ok11:01
*** kzaitsev_mb has quit IRC11:02
*** yamamoto has joined #openstack-infra11:02
openstackgerritSimon McCartney proposed openstack-infra/infra-ansible: Reworked to support Debian & Darwin/OSX
openstackgerritMerged openstack-infra/system-config: Add a simple LP OpenId reverse lookup utility
*** thorst has quit IRC11:04
*** yamamoto_ has joined #openstack-infra11:06
openstackgerritMerged openstack-infra/puppet-jenkins: Use default ruby packages for Xenial
openstackgerritMerged openstack-infra/infra-manual: creators.rst: Fix broken internal hyperlinks
*** dizquierdo has joined #openstack-infra11:08
openstackgerritMerged openstack-infra/infra-manual: Update translation setup instructions
*** yamamoto has quit IRC11:09
*** ociuhandu has quit IRC11:11
*** kzaitsev_mb has joined #openstack-infra11:15
*** daemontool_ has joined #openstack-infra11:16
*** vgridnev has quit IRC11:16
*** d0ugal has quit IRC11:16
*** d0ugal has joined #openstack-infra11:17
*** mrmartin has quit IRC11:18
*** daemontool has quit IRC11:19
*** Daisy has joined #openstack-infra11:20
*** mrmartin has joined #openstack-infra11:21
*** gnuoy_ has joined #openstack-infra11:22
*** gnuoy_ has quit IRC11:22
*** e0ne has quit IRC11:23
openstackgerritMerged openstack-infra/system-config: Fix file link to groups.txt in sysadmin docs
*** vincentfrancoise has quit IRC11:23
openstackgerritMilan Kováčik proposed openstack-infra/project-config: enable stack sudo for ironic discovery gate job
*** Daisy has quit IRC11:24
*** yamamoto_ has quit IRC11:25
openstackgerritmathieu bultel proposed openstack-infra/shade: Add normalize stack function for heat stack_list
*** yamamoto has joined #openstack-infra11:26
*** e0ne has joined #openstack-infra11:26
*** salv-orlando has joined #openstack-infra11:28
*** kzaitsev_mb has quit IRC11:30
*** yamamoto has quit IRC11:30
openstackgerritMerged openstack-infra/project-config: Add experimental gate to kolla-mesos
*** mrmartin has quit IRC11:31
openstackgerritMerged openstack-infra/system-config: Add roles for CI users
*** asilenkov has quit IRC11:31
openstackgerritSteven Hardy proposed openstack-infra/tripleo-ci: Move into tripleo-ci repo
*** tesseract has joined #openstack-infra11:33
*** jtomasek has quit IRC11:34
*** tesseract has quit IRC11:34
*** Guest16009 has joined #openstack-infra11:34
*** HeOS has quit IRC11:36
*** vgridnev has joined #openstack-infra11:36
*** HeOS has joined #openstack-infra11:36
*** asilenkov has joined #openstack-infra11:37
*** Guest16009 has quit IRC11:37
*** ildikov has quit IRC11:37
openstackgerritMerged openstack-infra/storyboard-webclient: Add a tooltip when hovering on the "Task notes" button
*** salv-orlando has quit IRC11:39
*** jtomasek has joined #openstack-infra11:39
*** tesseract- has joined #openstack-infra11:40
yolandamm AJaeger, look at that failure in puppet linters..11:40
*** kushal has joined #openstack-infra11:40
yolanda2016-03-11 11:35:36.012 | + sudo rm -f /etc/sudoers.d/jenkins-sudo11:40
yolanda2016-03-11 11:35:36.013 | + sudo -n true11:40
yolanda2016-03-11 11:35:36.013 | sudo: a password is required11:40
yolandalooks related to the move?11:40
AJaegerit does not ring a bell ;(11:41
yolandagoing to hold a node and test11:41
AJaegerlet me check the job definition and change to it...11:41
*** mrmartin has joined #openstack-infra11:42
*** yamamoto has joined #openstack-infra11:42
AJaegeryolanda: EmilienM updated this job already on the 26th of February...11:43
AJaeger  looks fine11:43
*** yamamoto has quit IRC11:43
*** erlon has joined #openstack-infra11:44
*** _amrith_ is now known as amrith11:44
*** HeOS has quit IRC11:44
yolandaapart from this error, i see11:45
yolanda2016-03-11 11:35:36.455 | + rake lint11:45
yolanda2016-03-11 11:35:36.455 | /tmp/ line 23: rake: command not found11:45
*** HeOS has joined #openstack-infra11:45
AJaegeryolanda: exactly ;) The sudo is fine, the missing rake is the problem11:45
*** thorst has joined #openstack-infra11:46
yolandawe need to add exec rake macro?11:46
AJaegerI wonder why it suddenly fails ;(11:47
yolandayes, that's strange11:47
*** mrmartin has quit IRC11:48
*** ociuhandu has joined #openstack-infra11:48
openstackgerritMerged openstack-infra/project-config: CentOS AFS build : move output to $WORKSPACE
*** vgridnev has quit IRC11:49
yolandaAJaeger, it may be a particular use case of that project11:49
yolandano metadata.json or Gemfile... so it just executes rake lint11:50
yolandacheck in jenkins/jobs/macros.yaml, for the puppet-lint job11:50
AJaegerI agree. The repository has not seen merges since November and I guess the missing Gemfile or metadata.json is the culprit.11:50
yolandaok going to amend it11:51
yolandabut seems our fallback for rake lint is broken anyway11:51
AJaegeryes, it is - it should install rake11:51
AJaegermight be that bare-trusty had rake on it so this didn't hit us ;(11:52
yolandalooks like. We are hitting some interesting bugs with the move to ubuntu-trusty :)11:52
AJaegerwon't be the last one ;(11:52
yolandai will send a fix11:53
AJaegerthen get that one in - and afterwards recheck the puppet-etherpad changes...11:53
*** vgridnev has joined #openstack-infra11:54
yolandamm, git review is taking ages11:55
yolandais that the issue you had yesterday?11:55
*** thorst has quit IRC11:55
*** thorst has joined #openstack-infra11:56
AJaegerI had the issue with a review that contained a stacked change11:56
yolandait may be that my connection sucks...11:56
openstackgerrityolanda.robla proposed openstack-infra/puppet-etherpad_lite: Add missing Gemfile to the project
yolandatook a life!11:57
*** kzaitsev_mb has joined #openstack-infra11:59
AJaegerare you going to fix the job as well so that the fallback works?11:59
*** aysyd has joined #openstack-infra11:59
yolandaworking on it, but i want to test it12:00
*** jaosorior has quit IRC12:02
*** jaosorior has joined #openstack-infra12:03
*** thorst has quit IRC12:05
*** aysyd has quit IRC12:05
*** aysyd has joined #openstack-infra12:05
*** ldnunes has joined #openstack-infra12:05
*** rvasilets__ has quit IRC12:06
*** rhallisey has joined #openstack-infra12:06
*** lucasagomes is now known as lucas-hungry12:07
*** zhenguo_ has quit IRC12:10
*** rfolco has joined #openstack-infra12:10
*** |-paul-| has joined #openstack-infra12:12
*** sdake has joined #openstack-infra12:16
*** kushal has quit IRC12:22
*** sdake has quit IRC12:23
*** jtomasek has quit IRC12:25
*** jtomasek has joined #openstack-infra12:27
*** gongysh has joined #openstack-infra12:28
yolandaAJaeger, stlll looking because it seems for this case, lint will break even with rake installed12:29
*** yamamoto has joined #openstack-infra12:31
*** ildikov has joined #openstack-infra12:32
*** dims has quit IRC12:33
*** amrith is now known as _amrith_12:33
yolandait may be a problem of the project itself, because Rakefile needs another deps, such as spec_helper12:35
*** kushal has joined #openstack-infra12:36
*** sdake has joined #openstack-infra12:36
*** dims has joined #openstack-infra12:36
*** dguitarbite has quit IRC12:37
*** yamamoto has quit IRC12:40
*** thorst has joined #openstack-infra12:40
*** baoli has joined #openstack-infra12:41
openstackgerrityolanda.robla proposed openstack-infra/project-config: Install rake lint when no Gemfile is found
yolandaAJaeger, that made it work for me, although i'm not 100% sure of all cases12:44
pabelangeryolanda: AJaeger: Ya, I've seem some general IO slowness with vexxhost, we should have them check into why12:45
yolandapabelanger, i have not been much involved on that setup. Do you know the person to contact?12:47
*** _amrith_ is now known as amrith12:47
openstackgerritRicardo Carrillo Cruz proposed openstack-infra/shade: Add domain_id param to project operations
openstackgerritMerged openstack-infra/system-config: login.defs/redhat: set SYS_UID_MIN and SYS_GID_MIN
*** abregman|nb has joined #openstack-infra12:49
*** |-paul-| has quit IRC12:50
pabelangeryolanda: I cannot remember of the top of my head, need to check logs12:50
openstackgerrityolanda.robla proposed openstack-infra/puppet-etherpad_lite: Add missing Gemfile to the project
pabelangerAJaeger: yolanda: are we ready to land 291349?12:51
*** Guest69862 has quit IRC12:51
yolandai think is fine for me, we'll need to monitor jjb runs12:52
*** Hal has joined #openstack-infra12:52
pabelangerokay, I can try and help too, if you don't mind mentoring on it :)12:53
*** Hal is now known as Guest7505612:53
yolandapabelanger, i see dmsimard involved in some conversations about vexxhost. Do you remember that name?12:53
*** kaisers1 has joined #openstack-infra12:54
*** e0ne has quit IRC12:54
yolandapabelanger, what i normally do for monitoring jjb, is that i tail puppet logs on puppetmaster, to check if jenkins is successful12:54
*** yamamoto has joined #openstack-infra12:54
yolandaalso is good to see config history on all jenkins, do you have access to do it?12:54
*** e0ne has joined #openstack-infra12:54
pabelangeryolanda: Ya, dmsimard will know the contact, they are both from MTL12:54
pabelangeryolanda: okay, let me ssh into puppetmaster and tail12:55
pabelangeryolanda: where do you check config history?12:55
yolandapabelanger for example on
yolandaif you have perms you will see something, if not, that will be empty12:56
yolandayou need to be sure you have openstack-ci-admins group enabled12:56
yolandawhen login from sso12:56
pabelangeryolanda: Ah, no empty here12:56
pabelangerafter sso12:56
pabelangerso will ask fungi12:57
* yolanda needs to run to have lunch12:57
yolandawill be back in short12:57
*** amrith is now known as _amrith_13:00
openstackgerritMerged openstack-infra/project-config: Rename tox-nodb-ENV to tox-ENV
*** yamamoto has quit IRC13:01
*** rossella_s has quit IRC13:03
*** alaski is now known as lascii13:03
*** e0ne has quit IRC13:03
*** dims_ has joined #openstack-infra13:03
*** rossella_s has joined #openstack-infra13:03
*** e0ne has joined #openstack-infra13:04
*** dims has quit IRC13:04
*** EricGonczer_ has joined #openstack-infra13:05
*** sdake has quit IRC13:10
*** sdake has joined #openstack-infra13:11
*** jaosorior has quit IRC13:12
*** jaosorior has joined #openstack-infra13:12
*** mugsie has quit IRC13:13
*** mugsie has joined #openstack-infra13:13
yolandapabelanger, when you login into jenkins, you see a checkbox asking for openstack-ci-admins membership?13:17
*** akshai has joined #openstack-infra13:19
*** dprince has joined #openstack-infra13:19
*** sdake_ has joined #openstack-infra13:20
*** notnownikki has joined #openstack-infra13:23
*** vincentfrancoise has joined #openstack-infra13:23
pabelangeryolanda: no, just the SSO page from launchpad, then redirected back to jenkins13:23
*** sdake has quit IRC13:23
*** yamamoto has joined #openstack-infra13:26
*** pradk_ has joined #openstack-infra13:26
*** pradk_ has quit IRC13:26
*** dingyichen has quit IRC13:28
*** jpr has quit IRC13:28
pabelangeryolanda: okay, so I think puppet just failed on jenkins13:28
*** zeih_ has quit IRC13:28
*** lucas-hungry is now known as lucasagomes13:29
*** yaume has joined #openstack-infra13:29
*** akshai has quit IRC13:29
yolandapabelanger, then there is a playbook to run jjb manually on the failed jenkins13:30
*** akshai has joined #openstack-infra13:30
AJaegermorning, pabelanger !13:30
yolandathe run_jjb.yaml13:30
yolandayou can execute that from puppetmaster, but need to be sure that jjb is not already running on the jenkins host, to avoid cache corruption13:30
pabelangeryolanda: okay, can we do a screen share, so I can watch you do it?13:31
pabelangerAJaeger: o/13:31
*** baoli_ has joined #openstack-infra13:31
AJaegeryolanda: I'm not sure about your rake change, the if -f GEMFILE condition looks wrong, you reorder commands.13:32
AJaegerpabelanger, yolanda, please double check13:32
AJaegerpabelanger: thanks for reviewing13:33
*** Daisy has joined #openstack-infra13:34
*** baoli has quit IRC13:34
yolandaAJaeger, i will amend, it was consequence of my tests i guess13:34
*** jeeloo1 has joined #openstack-infra13:36
*** jeeloo has quit IRC13:36
*** keedya has joined #openstack-infra13:36
openstackgerritMerged openstack-infra/system-config: Remove interpret_puppet_exitcodes
*** abregman|nb is now known as abregman|afk13:37
*** links has quit IRC13:39
*** dizquierdo has quit IRC13:40
*** Daisy has quit IRC13:40
*** Daisy has joined #openstack-infra13:41
*** matrohon has joined #openstack-infra13:41
*** links has joined #openstack-infra13:42
openstackgerrityolanda.robla proposed openstack-infra/project-config: Install rake lint when no Gemfile is found
AJaegerseems that the tox job rename was small enough and it was applied to all Jenkinses - if my interpretation of puppetboard is correct13:42
*** eharney has quit IRC13:42
AJaegerEmilienM: could you check , please?13:44
EmilienMAJaeger: sure13:44
EmilienMAJaeger: nice catch, and +113:45
AJaegerEmilienM: background : We have at least one repo that does not include a Gemfile and ubuntu-trusty has no rake isntalled by default.13:45
*** Daisy has quit IRC13:45
AJaegerEmilienM: thanks for reviewing!13:45
*** kushal has quit IRC13:45
EmilienMyou will need a Gemfile though :-)13:46
*** jeeloo1 has quit IRC13:47
*** jeeloo has joined #openstack-infra13:47
pabelangerAJaeger: Ya, that is what I am trying to confirm ATM13:48
*** kushal has joined #openstack-infra13:49
AJaegerthanks, pabelanger !13:49
*** akshai has quit IRC13:51
*** hieulq has joined #openstack-infra13:53
yolandaok back.. going to check job config history to be sure13:53
*** yaume has quit IRC13:53
*** doug-fish has joined #openstack-infra13:54
yolandathx for the review EmilienM13:54
*** Qiming has quit IRC13:54
pabelangerAJaeger: yolanda: Ya, things seem good looking at puppet now13:54
*** Qiming has joined #openstack-infra13:54
AJaegerglad to hear!13:55
*** kgiusti has joined #openstack-infra13:55
*** akshai has joined #openstack-infra13:56
* AJaeger has no plans for further mass renames this week ;)13:56
pabelangerno doubt13:57
fungipabelanger: yeah, need to add you to a group on lp we're using for openid-based access control on the jenkinses. i can take care of it in a few moments13:57
AJaegeryolanda's change touches a few dozen jobs...13:57
AJaegermorning, fungi!13:57
pabelangerfungi: thanks13:57
AJaegerfungi, yolanda asked earlier whether there are policies for adding vhosts in the context of this change: - the neutron review board.13:58
* AJaeger would love to see this land to help neutron folks with their workflow13:58
yolandaAJaeger, so as a sample, that job was created on jenkins01
yolandagoing to check all jenkins13:59
AJaegeryolanda: great!13:59
*** sdake_ has quit IRC13:59
*** HeOS has quit IRC14:00
fungiyolanda: AJaeger: yeah, when i mentioned "add a vhost" i meant something generic for aggregating review information (as a replacement for or alternative to reviewday). i didn't realize they wanted to run something neutron-only, so am not really in favor of that choice14:00
yolandafungi, i misunderstood your comments on the thread then. So you prefer an independent project to be created?14:01
openstackgerritMerged openstack-infra/project-config: Install rake lint when no Gemfile is found
*** jeeloo has left #openstack-infra14:01
Shrewsfungi: do you know if there are any developer docs that describe running nodepool tests locally? none seem to be in the repo, so i'm guessing "no"14:01
*** HeOS has joined #openstack-infra14:02
yolandaAJaeger, pabelanger i see that renames applied in all jenkins properly14:02
*** gildub has quit IRC14:03
*** nwkarsten has joined #openstack-infra14:05
*** sdake has joined #openstack-infra14:05
*** nwkarsten has quit IRC14:05
*** nwkarsten has joined #openstack-infra14:06
fungiyolanda: doesn't have to be an independent project, just the job which publishes review details doesn't need to be an exclusive neutron-only thing and the site it's published under shouldn't be either14:06
*** dims_ has quit IRC14:07
fungiShrews: by running nodepool tests locally you mean other than the unit tests which are in its repo?14:07
AJaegerrossella_s: see above14:07
AJaegerfungi, so rossella_s misunderstood you...14:08
pabelangerwhat is the status of the pypi.* servers? They've been replaced with AFS right?14:08
*** kushal has quit IRC14:08
pabelangernotice they are still referenced in puppet and wonder if they need to be cleaned up14:08
fungiShrews: mmmpork wrote up some documentation on making a test deployment of nodepool to perform functional testing of it. let me see if that landed in the repo too14:08
* AJaeger should have known on Tuesday when he was with rossella_s at the bug smash - we could have worked something out then...14:09
rossella_sAJaeger, thanks for pinging it's not clear to me what I should do14:09
Shrewsfungi: just trying to run the unit tests. apparently there is some db setup that is assumed. trying to reverse engineer that part14:09
fungiShrews: oh! databases, yes. i can show you the script we use to set up databases on our job workers14:10
AJaegerfungi, did you had something in mind like for rossella_s? Where each project could run a periodic job that publishes to let's say and then there's a central index page with one entry per project?14:13
fungiShrews: the mysql-prep script at followed by the mysql-db script at
*** ajmiller has joined #openstack-infra14:13
fungiAJaeger: rossella_s: can the periodic job which generates the review details be extended to run for any team, not just neutron? then it doesn't need to run multiple times and can just create the index when it updates review info for whatever teams it's configured to support14:14
Shrewsfungi: thx14:14
*** mriedem has joined #openstack-infra14:14
openstackgerritAttila Darazs proposed openstack-infra/tripleo-ci: Use IPv6 on the ceph gate job
fungiAJaeger: rossella_s: i haven't seen the publication job yet, so not sure what the options are there14:14
*** gordc has joined #openstack-infra14:14
* AJaeger checks the job at
fungiaha, i see you just added a link to it in a review comment. that helps me gain some context on how this is expected to work14:15
*** ajmiller_ has joined #openstack-infra14:15
rossella_sfungi depends on the team, we don't have the same way of assigning priorities to reviews I think. My script was customized according to the requests of the neutron cores.14:15
fungiso tools/ is something that lives in the neutron repo?14:16
AJaegerfungi: yes14:17
*** dims has joined #openstack-infra14:17
AJaegerrossella_s: you can't hardcode mitaka-3 either in the jenkins job...14:17
*** links has quit IRC14:17
rossella_sAJaeger, I can fix that14:17
*** hdd has joined #openstack-infra14:18
fungimy goal is just that if this becomes popular and a lot of other teams decide it's cool and want to do the same thing, that we make sure we don't end up with dozens of disconnected copies of this tooling and dozens of slightly different websites under separate domain names14:18
AJaegerrossella_s: that's a second step ;) Let's figure out with fungi the first one ;)14:18
*** akshai has quit IRC14:18
*** ajmiller has quit IRC14:18
fungiso planning to make it usable for everyone (a solution for openstack) instead of only one team (a solution for neutron) is preferable14:18
AJaegerfungi: I should have challenged you, I misunderstood you as neutron specific suggestion as well ;(14:18
fungiyeah, it's more that i'd like to see someone fix/take over/replace reviewday since it seems to have fallen into disuse14:19
*** sdake_ has joined #openstack-infra14:19
*** akshai has joined #openstack-infra14:20
openstackgerritsebastian marcet proposed openstack-infra/openstackid-resources: Fix on missing Summit Entity Events
rossella_sfungi, I agree with you...the script can be extended for other teams actually, it's not so hard. I imagine the most of the requirements are in common14:20
*** _ari_ has joined #openstack-infra14:20
fungirossella_s: so to start, let's pick a generic name for the site which isn't team-specific, and we could publish this into a team-specific subdirectory while you try it out and see what needs adjusting14:21
*** xyang1 has joined #openstack-infra14:21
AJaegerand once more projects come up, we can generalize, add index page etc?14:21
rossella_sfungi, great! let's do that14:21
*** sdake has quit IRC14:21
*** vincentfrancoise has left #openstack-infra14:22
*** kaisers1 has quit IRC14:22
fungithen later, if it's something others want to do, the tooling can be extracted from the neutron repo and put somewhere more central, generalized and made more configurable, and we could start running it for more projects and publishing an index to their team dashboards or whatever14:22
AJaegerfungi, good plan.14:22
fungibut that should be the step that happens when a second team decides to use it, so we don't get a messy proliferation14:23
*** yaume has joined #openstack-infra14:23
AJaegerSo, for now: Rename the vhost in system-config14:23
rossella_sfungi, AJaeger thanks a lot, it's a good plan!14:23
*** dims has quit IRC14:23
openstackgerritMerged openstack-infra/openstackid-resources: Fix on missing Summit Entity Events
AJaegerFor project-config change: Resolve merge conflict, remove mitaka-3 as argument and publish to neutron subdirecotry14:24
AJaegerrossella_s: Add to the system-config commit message a "Needed-By: GerritID-of-project-config" line14:24
* AJaeger will be back later...14:25
AJaegerthanks, fungi!14:25
*** ikutukov has joined #openstack-infra14:26
rossella_sAJaeger, thanks a lot!14:26
*** ericksonsantos has quit IRC14:26
*** nikhil has quit IRC14:26
*** ericksonsantos has joined #openstack-infra14:27
*** nikhil has joined #openstack-infra14:27
fungipabelanger: you are now a member of
*** placko_ has quit IRC14:28
pabelangerfungi: thanks, so just so I know, what does that actually mean :)14:28
fungipabelanger: log out of the jenkins webui, then when you're logging back in pay attention to the openid details confirmation page where it prompts you to check the checkbox to include group membership details for that14:28
openstackgerritDoug Hellmann proposed openstack-infra/release-tools: add version numbers to tracking dashboard
fungipabelanger: that's pretty much the only thing we use that group for any more14:28
pabelangerfungi: ack14:28
pabelangerfungi: okay, I see the difference now14:29
fungipabelanger: it should remember you've checked that when logging into _that_ jenkins master in the future, but you'll need to do it again for any one of them you haven't logged into yet14:29
*** burgerk has joined #openstack-infra14:30
*** vgridnev has quit IRC14:31
openstackgerritRossella Sblendido proposed openstack-infra/system-config: Add vhost for Neutron reviews dashboard
*** [1]Thelo has joined #openstack-infra14:32
*** _amrith_ is now known as amrith14:32
rossella_sfungi, I fixed it, hope I got what you meant, thanks!14:32
*** dims has joined #openstack-infra14:32
*** mriedem is now known as dontrump14:32
fungirossella_s: thank you, i'll have a look14:32
rossella_sfungi thanks a lot :)14:33
openstackgerrityolanda.robla proposed openstack-infra/system-config: Add playbook to manage power state of ironic servers
*** Thelo has quit IRC14:34
*** [1]Thelo is now known as Thelo14:34
*** dansmith is now known as superdan14:36
*** zeih has joined #openstack-infra14:36
*** rlandy has joined #openstack-infra14:37
*** vgridnev has joined #openstack-infra14:37
*** hdd has quit IRC14:38
EmilienMa patch merged in project config like 8 hours ago but JJB has not been updated everywhere I think
fungithat's disheartening if so. i was up until after midnight my time getting jjb configs updated and in sync on all our jenkins masters14:40
EmilienMif you look at the job logs, the path is still not correct, something we fixed in project-config14:41
fungii should be able to take a closer look in a few minutes14:43
openstackgerritRossella Sblendido proposed openstack-infra/project-config: Add job to update the link for Neutron Review Dashboard
*** amitgandhinz has joined #openstack-infra14:44
openstackgerritMonty Taylor proposed openstack-infra/nodepool: Use shade for all OpenStack interactions
jeblairfungi, rossella_s: is that creating an actual gerrit dashboard?14:46
*** BigWillie has joined #openstack-infra14:46
*** vgridnev has quit IRC14:46
rossella_sjeblair, yes using gerrit-dash-creator14:46
*** vgridnev has joined #openstack-infra14:46
jeblairfungi, rossella_s: those can be hosted in gerrit itself, per-project14:47
*** ajmiller_ has quit IRC14:48
rossella_sjeblair, oh really? do you have any link or any pointer that explain how to do that?14:48
*** jaosorior has quit IRC14:48
pabelangerfungi: EmilienM: jenkins.o.o is correct14:49
pabelangerchecking others now14:49
jeblairfungi, rossella_s: someone just needs to update jeepyb to support installing them14:49
jeblairfungi, rossella_s:
*** zz_dimtruck is now known as dimtruck14:50
jeblairfungi, rossella_s: there's just stored under refs/meta/dashboards/...14:50
rossella_sjeblair, wow, didn't know that. Thanks for the hint, I will look into that!14:51
rossella_sfungi, jeblair, AJaeger I might bother you again on I need to go, thanks a lot again for your help!!14:51
mordredjeblair, AJaeger, pabelanger, fungi: did we get anywhere with databases and nodepool and gate tests yesterday?14:51
*** hieulq has quit IRC14:51
pabelangerfungi: jenkins02, 03, 04, 05, 06 and 07 looks out of sync14:52
zigopabelanger: Hi there! I was wondering if I could get voluptuous upgraded to its latest upstream version.14:52
pabelangermordred: I think ianw submitted a fix last night14:52
zigopabelanger: I need it for the latest version of tooz, which itself is needed by the latest Gnocchi.14:52
zigopabelanger: Did you see the Debian bug I filed?14:52
*** rbrndt__ has joined #openstack-infra14:52
*** sc68cal has joined #openstack-infra14:53
mordredpabelanger: cool. I'll go look for it14:53
pabelangermordred: 291493 I believe14:54
pabelangerzigo: upgraded where?14:54
*** krtaylor has quit IRC14:54
openstackgerritZara proposed openstack-infra/storyboard-webclient: Display task notes from board
jeblairAJaeger: ^ also see discussion above about dashboards14:55
mordredpabelanger: coverage job is not happy14:55
zigopabelanger: Upgrade the voluptuous package in Debian Sid.14:55
zigopabelanger: You are one of the package maintainers of this package.14:55
zigopabelanger: Which is why I'm asking you the permission to do it.14:56
zigopabelanger: Though I don't have access to the DPMT Git repo, so I wont be able to upload my changes there.14:56
pabelangerzigo: Ya, it's likely going to my old email address.  Let me check14:56
mordredpabelanger, AJaeger: what's the right way to get the db stuff installed for a coverage job?14:56
pabelangerzigo: I can also add you to update too14:56
mordredoh - looks like the db is installed on converage jobs ...14:57
jeblairmordred: i think there's a macro14:57
pabelangermordred: that's what I am checking14:57
mordredso perhaps that's just related to "fungi: jenkins02, 03, 04, 05, 06 and 07 looks out of sync"14:57
openstackDebian bug 817269 in src:voluptuous "Please package voluptuous >= 0.8.6" [Important,Open]14:57
* mordred thinks pabelanger is already working on this :)14:57
*** rfolco has quit IRC14:57
*** Qiming has quit IRC14:58
pabelangermordred: not the outof sync part yet, I don't know how we deal with that ATM14:58
*** Qiming has joined #openstack-infra14:58
zigopabelanger: Even if you add me as Uploaders, I wont have Git write access to /git/python-module, so the only thing I can do is ugrade the package as an NMU. Is this fine by you? You could later on commit the changes.14:58
*** Guest13749 is now known as mfisch14:58
*** mfisch is now known as Guest8807414:59
pabelangermordred: Once we solve the sync issue with JJB, I belive it will be fixed14:59
*** rossella_s has quit IRC14:59
jeblairpabelanger, mordred: yes, the job defn on 04 lacks the grant option which is in the current jjb config, so this is jenkins not updated14:59
*** salv-orlando has joined #openstack-infra15:00
pabelangerSo, I don't know how we kick off resyncs manually, yolanda send me a playbook, but figure I'll hold off a bit and just watch it happen15:00
pabelangerzigo: sure, that works for me15:00
yolandayes, the run_jjb playbook , do you want me to throw it?15:01
*** Guest88074 is now known as mfisch15:01
*** mfisch has quit IRC15:01
*** mfisch has joined #openstack-infra15:01
jeblairbefore we run it, we should see what fungi learned about the --workers option to jjb....15:01
*** jcoufal has quit IRC15:02
jeblairhe was trying it yesterday with 2 workers, but i haven't checked scrollback to learn how well that worked15:02
pabelangerzigo: FWIW: I don't think I have every upload voluptuous before.  hashar  might be a better person15:02
fungijeblair: no conclusion. most of them were already in sync via puppet/ansible in the few minutes it took me to get around to start running them15:03
zigohashar: Hi there!15:03
fungijeblair: of the ones that weren't, jenkins.o.o took just shy of 30 minutes and jenkins01 needed several tries and about 4 hours of wall clock time15:03
pabelangerzigo: I'm actually surprised I have uploads :)15:04
hasharzigo: pabelanger: the Debian package i build for voluptuous   was my very first deb package.  It should have been migrated in the alioth git repos15:04
zigohashar: Can you allow me to NMU a new upstream version of Voluptuous in Sid?15:04
hasharI might just have pilled a bunch of emails in the source package :-}15:04
pabelangerhashar: hehe15:04
*** rfolco has joined #openstack-infra15:04
hasharzigo: can you be added as a maintainer as well ?   Your .deb skills are way above my league :}15:05
jeblairfungi: it doesn't sound like there was a 2x difference.15:05
jeblairfungi: it sounds like it was a 1x difference.15:05
*** tonytan4ever has joined #openstack-infra15:05
zigohashar: Yeah, I can add myself, though I have lost access to the DPMT Git repository, so I wont be able to push the changes there.15:05
pabelangerStepping out for an hour, have to run an errand.15:05
fungijeblair: or even necessarily a discernible difference given our variability of performance15:05
hasharzigo: such a pity15:05
*** sigmavirus24_awa is now known as sigmavirus2415:06
SamYaplehello. who wants to review some shade code?!
hasharzigo: should I craft a commit to add you to maintainer list  and then add your commit and push it for you ?15:06
fungijeblair: the only consistent number i managed to get out of the exercise is that when updating 0 jobs a full jjb run on most of our jenkins masters takes somewhere in the 7 minute range15:07
*** kmartin has quit IRC15:07
zigohashar: I can send to your email some "git format-patch" series, yes.15:07
fungijeblair: most of that time is spent compiling xml and checking it against the cache, i think15:07
hasharzigo: please spam me at and I will be more than happy to push !15:07
*** dims_ has joined #openstack-infra15:07
jeblairfungi: yeah, i think so15:07
zigohashar: Thanks, doing that right away.15:08
*** dims has quit IRC15:08
jeblairso it sounds like we shouldn't bother with --workers15:08
hasharzigo: the X-Vcs still reference subversion but afaik  the team has moved to git
fungijeblair: i'm inclined to agree15:08
hasharzigo: will push later tonight or tomorrow15:08
*** xarses has quit IRC15:08
jeblairi guess pabelanger isn't going to run it; yolanda did you want to?15:09
yolandayes i can do it15:09
fungijeblair: rossella_s: oh! i didn't realize all that was doing was creating gerrit dashboards. i figured there was a lot more going on15:09
yolandainfra-root, btw, can i get feedback about ? this is to prevent the cache corruption on jjb plays15:10
fungimordred: on databases and nodepool tests, yes ianw discovered we needed with grant option so we updated all the jobs using the mysql-prep macro (thousands) to add that and then i made sure all the jenkins masters were in sync, so nodepool tests should be passing now15:11
*** woodster_ has joined #openstack-infra15:11
nibalizeryolanda: i don't see how you'd get into a place where jjb had a lockfile but didn't exist in the process table15:11
*** dims_ has quit IRC15:11
nibalizermaybe if the puppet cron hit a timeout15:11
fungimordred: if it's still not passing, i suppose it's possible that just updating /etc/project-config and then rerunning jjb update isn't updating our jobs for some reason15:12
yolandanibalizer, by your comments, you meant introducing a lockfile feature on jjb?15:12
nibalizeri think it was paul that suggested it but it makes a ton of sense to me15:13
jeblair(jjb doesn't need to know about the lockfile)15:13
fungifor everything else, there's flock(1)15:13
yolandayep, pabelanger had some valid suggestions. Also limiting the number of jobs that jjb updates per run15:13
*** hdd has joined #openstack-infra15:13
yolandaso add a lockfile extra layer on our puppet jjb runs?15:13
*** dims has joined #openstack-infra15:13
*** salv-orlando has quit IRC15:14
jeblairwe could do that.  though, since jjb writes a cache, it may make sense for it to understand a lockfile internally.15:14
nibalizerwhy would you limit the number of jobs that jjb updates per run?15:14
jeblairwrite the lockfile next to the cache15:15
nibalizerthat just sounds like you'd have all the masters in super inconsistent states15:15
SamYapledont everyone jump on reviewing that shade patch all at once now15:15
jeblairnibalizer: to be fair, that's the actual result of what we get now with puppet15:15
nibalizerjeblair: nice15:15
fungimordred: also i have a somewhat bleary-eyed recollection of grepping "with grant option" in our jenkins xml on one of the masters which claimed to have 0 jobs updated, and saw it in numerous jobs, so pretty sure at least my manual updates last night were complete15:15
nibalizerdo we know why it takes 87 minutes to apply our jjb to a master?15:15
nibalizeris it becuase the master is busy or something?15:15
funginibalizer: yeah, calls to the jenkins api can be slooooow15:16
yolandanibailzer it depends on the amount of jobs affected15:16
fungiand also sometimes break/timeout completely15:16
*** salv-orlando has joined #openstack-infra15:16
fungiwhen our masters get into their thread leak condition, it gets a lot worse too15:16
jeblairfungi: the job mordred was looking at does not have the grant option:
*** ajmiller has joined #openstack-infra15:17
yolandai think it will be a good idea to add this lockfile internally to jjb, as concurrent runs have proved to break cache15:17
*** harlowja_at_home has joined #openstack-infra15:17
jeblairyolanda: wfm15:18
*** thiagop is now known as thiagop-afk15:18
fungiindeed it is missing on that one15:18
jeblairfungi: it does have it on jenkins.o.o15:18
nibalizeryolanda: i think thats a great idea15:18
jeblairfungi: and 01 and 0215:19
openstackgerritAdam Coldrick proposed openstack-infra/storyboard-webclient: Highlight the notes button if a task has notes
fungiand on 0715:20
fungiwhat gives?!?15:20
fungii ran `sudo grep -ir 'with grant option' /var/lib/jenkins/jobs` on jenkins03 last night to double-check15:21
jeblairfungi: it's in gate-nodepool-python27 on jenkins0315:22
*** vincentfrancoise has joined #openstack-infra15:22
jeblairfungi: but not in nodepool-coverage on jenkins0315:22
fungihow does that happen? is caching wacky?15:22
fungithey should be using the same macro15:22
jeblaircaching + workers=2 ?15:23
fungiand jjb said it updated 0 jobs when i ran it, implying the cache matched15:23
openstackgerritRicardo Carrillo Cruz proposed openstack-infra/shade: Follow name_or_id pattern on domain operations
fungiohh, maybe --workers 2 bit me on the rump with this15:23
AJaegerjeblair: the challenge with rossella_s' dashboard is that neutron looks up first bugs in launchpad and prioritizes according to them.15:23
AJaegerjeblair: they want to generate a dashboard based on current bug priorities - so would need to update the gerrit dashboard daily15:24
jeblairAJaeger: yep, so we just need to figure out (a) where to run that and (b) how to push that to the special gerirt ref15:24
fungiso maybe some job which pushes updates into gerrit dashboards?15:24
fungianyway, is anyone rerunning jjb on the jenkins masters yet? if not, i'll go ahead and fire it up again15:25
*** kzaitsev_mb has quit IRC15:25
yolandafungi, i'm doing that15:26 finished15:26
openstackgerritAdam Coldrick proposed openstack-infra/storyboard-webclient: Highlight the notes button if a task has notes
fungiyolanda: okay, thanks15:26
fungiyolanda: without --workers 2 hopefully ;)15:26
*** gongysh has quit IRC15:26
yolandajust the run_jjb play15:26
*** tsg has joined #openstack-infra15:26
fungii still haven't tried that one. how does it deal with jjb hitting jenkins api timeouts? does it loop rerunning it there until it completes, or do you end up rerunning it manually after that?15:27
yolandai think it doesn't work good with timeouts15:28
yolandahave to launch again if that fails15:28
fungirunning jjb on the jenkins masters is not hard. it's keeping on top of which ones broke and restarting them that's annoying15:28
*** jcoufal has joined #openstack-infra15:28
AJaegerjeblair: so, that's a system wide option to push a dashboard? Coool... That makes it much easier!15:28
yolandafungi but is becoming a daily task lately...15:28
*** sdake_ has quit IRC15:29
yolandajenkins01 finished15:29
fungiyolanda: well, for the past week we've been iterating on modifications that hit thousands of jobs15:29
fungias we discover corner cases and need to adjust job macros15:29
*** bdemers has quit IRC15:29
yolandayes, that massive reconfig is the cause, i know15:29
fungii've been babysitting jjb runs almost every day to keep things in sync15:30
*** kzaitsev_mb has joined #openstack-infra15:30
yolandayep, same here in the mornings, along with AJaeger15:30
AJaegerAnd I thought that yolanda's fix for the puppet-lint build was not touching that many jobs that we run into this ;(15:30
fungioh, so we've been doing it twice a way? :/15:31
yolandalooks as some days it was the case15:31
fungier, twice a day15:31
AJaegerfungi, not every day  as far as I remember.15:31
*** sdake has joined #openstack-infra15:31
yolandanot daily , just on massive jobs15:31
fungii just meant we've had massive job updates almost daily for the past week, so i've resorted to running jjb manually for each of those15:32
*** esikachev has quit IRC15:33
*** regXboi has joined #openstack-infra15:34
fungiunfortunately today i've got a bunch of errands i need to disappear to take care of, which is why i was making sure things were synced up before i went to sleep last night (or so i thought anyway)15:34
openstackgerritRussell Bryant proposed openstack/requirements: Update ovs to 2.5.0.
fungisorry about that!15:34
*** HeOS has quit IRC15:34
*** esikachev has joined #openstack-infra15:34
*** tonytan4ever has quit IRC15:35
*** salv-orlando has quit IRC15:35
*** xarses has joined #openstack-infra15:35
*** tonytan4ever has joined #openstack-infra15:35
fungimy worry now is that we've got a cache out of sync with reality. so if the current run doesn't actually make any difference, we may need to blow away the cache and try again15:36
AJaegerfungi, I think on Tuesday morning it was just a jjb run on jenkins02 - the others were fine...15:36
yolandait can take hours15:36
*** vgridnev has quit IRC15:36
openstackgerritRussell Bryant proposed openstack/requirements: Add ovs with Python 3 support.
*** esikachev has quit IRC15:39
*** annegentle has joined #openstack-infra15:39
*** fedexo has joined #openstack-infra15:40
*** HeOS has joined #openstack-infra15:40
AJaegershould we schedule job approvals somehow? Or will we run into this JJB problems more often if we have them all together since more jobs get updated - and thus should not mass-approve?15:42
ikutukovHello! I have a review that stuck in 'Patch in Merge Conflict’, i’ve tried rebasing it all possible way and the only result was ’no changes’, in local merge there is no conflicts. How i could troubleshoot what gone wrong in Jenkins merge process? Thank you!15:42
openstackgerritRicardo Carrillo Cruz proposed openstack-infra/shade: Follow name_or_id pattern on domain operations
openstackgerritDan Prince proposed openstack-infra/tripleo-ci: Metrics tracking for TripleO deployment tasks
AJaegerikutukov: try the following: Create new branch from master HEAD, run git review -x 288635, send it in...15:44
*** vgridnev has joined #openstack-infra15:44
ikutukovOk, thank you15:45
*** Jeffrey4l has joined #openstack-infra15:45
*** jaypipes is now known as leakypipes15:45
AJaegerikutukov: that gives me a conflict on nailgun/nailgun/db/sqlalchemy/models/deployment_graph.py15:45
AJaegerikutukov: you always need to pull from master first and rebase on top of master to resolve merge conflicts15:45
*** claudiub|2 has quit IRC15:45
*** yamahata_ has joined #openstack-infra15:46
openstackgerritMerged openstack-infra/storyboard-webclient: Highlight the notes button if a task has notes
ikutukovThere are a chain of dependant changes under this one, maybe this have played role15:47
fungiAJaeger: i have a feeling today's issue is more due to a bug in jjb, but we'll need some more focused attempts to reproduce before we can say for sure. i don't think it's necessary to schedule approvals for normal job additions/changes, but we do still need to be mindful of any which change widely-used templates/macros15:47
ildikovhi infra team15:48
ildikovI have a basic meeting orgainzation question15:48
*** scheuran has quit IRC15:49
*** yamamoto has quit IRC15:49
ildikovdo I have to reserve a meeting slot if the meeting is only for one occasion or it is ok to just pick a channel which is available for that slot and organize/announce the meeting on the ML?15:49
*** mtanino has joined #openstack-infra15:49
*** yamamoto has joined #openstack-infra15:49
openstackgerritAna Krivokapic proposed openstack-infra/project-config: New project for TripleO UI
fungiildikov: for a one-time meeting, i wouldn't reserve it as long as there's not a significant risk of someone else picking the same channel and time15:50
*** denisra has joined #openstack-infra15:50
*** yamamoto has quit IRC15:50
fungiildikov: though if it's a cross-project meeting you may want to use #openstack-meeting-cp for it15:50
ildikovfungi: I guess if I pick a slot where multiple channels are available, then it should be ok then15:50
fungiildikov: yeah, that seems safest15:50
*** ifarkas has quit IRC15:51
fungithat way it's easy to move the meeting to another available channel quickly in the case a conflict does arise15:51
ildikovfungi: hmm, I wasn't aware of #openstack-meeting-cp :)15:51
*** yamamoto has joined #openstack-infra15:51
fungiildikov: it was created specifically to host cross-project meetings (for things like tc formed working groups, cross-project specs, et cetera)15:52
ildikovfungi: now I found it, it's just not listed explicitely on the wiki I found15:52
fungioh, we might have forgotten to update the channel list in the wiki when that was created15:52
anteayaI seem to not be around at the moment and I will continue to not be around for a bit longer as I have errands to run in town15:53
anteayayay logstash 2.015:53
fungianteaya: i'm about to be the same way. enjoy errands (as much as you can, i guess)15:53
ildikovfungi: I will try to find a slot when that's available as I'm trying to organize a cross-project one and pick another which is also available as a backup option15:53
ildikovfungi: thanks much for the help!15:53
pabelangerjeblair: Ya, just holding off for a few days and just watch how it is done :)15:53
anteayafungi: and to you15:53
openstackgerritMonty Taylor proposed openstack-infra/nodepool: Use shade for all OpenStack interactions
ildikovfungi: is it ok, if I update the wiki too with this channel?15:54
openstackgerritSergey Belous proposed openstack-infra/project-config: Initial job for Nova with os-vif
fungiildikov: i'm already doing it15:54
ildikovfungi: ok, cool, that was fast :)15:54
ildikovfungi: tnx again! :)15:54
yolandamordred, around? did you see my proposal to fix glean tests?15:56
*** yamamoto has quit IRC15:56
fungiildikov: i've added it to and now15:56
*** ldnunes has quit IRC15:57
*** tonytan_brb has joined #openstack-infra15:57
*** tonytan4ever has quit IRC15:58
ildikovfungi: I cannot see it at the latter one also there is now an item that looks odd at the top of the page:  #openstack-meeting-alt</code>15:59
ildikovfungi: I found this page:
openstackgerritBen Swartzlander proposed openstack-infra/project-config: Refactor manila check/gate queue
fungiildikov: yeah, i mistyped, just a sec15:59
ildikovfungi: meeting channels are listed there as well15:59
fungii should look closer at the preview15:59
openstackgerritRicardo Carrillo Cruz proposed openstack-infra/infra-specs: Add extension to openstackci for next phase of work
fungii'll update that too, thanks16:00
ildikovfungi: I hear ya, I mistype everything all the time16:00
*** bdemers has joined #openstack-infra16:00
ildikovfungi: np, at least my question turned out to be useful :)16:00
*** eharney has joined #openstack-infra16:01
*** hieulq has joined #openstack-infra16:02
openstackgerritAdam Coldrick proposed openstack-infra/storyboard-webclient: Stop updating the board in-place when moving lanes
fungiildikov: yep, corrected the typo and updated the other page you mentioned. thanks!16:03
*** kencjohnston has joined #openstack-infra16:03
*** nwkarsten has quit IRC16:05
*** nwkarsten has joined #openstack-infra16:05
ildikovfungi: cool, looks good now everywhere, tnx16:06
*** thorst is now known as thorst_afk16:06
*** vgridnev has quit IRC16:06
*** ramishra has quit IRC16:08
*** kmartin has joined #openstack-infra16:08
openstackgerritRicardo Carrillo Cruz proposed openstack-infra/shade: Follow name_or_id pattern on domain operations
*** esikachev has joined #openstack-infra16:08
*** ramishra has joined #openstack-infra16:09
*** nwkarsten has quit IRC16:10
*** vgridnev has joined #openstack-infra16:10
openstackgerritAndreas Jaeger proposed openstack-infra/project-config: Report number of changed jobs
AJaegerfungi, what do you think of the above debug output? ^16:11
*** Qiming has quit IRC16:11
openstackgerritAndreas Jaeger proposed openstack-infra/project-config: DO NOT MERGE - Testing
fungiAJaeger: seems like a fine addition to me16:12
*** yaume has quit IRC16:12
fungicould also use grep -c instead of piping to wc -l16:12
openstackgerritMonty Taylor proposed openstack-infra/nodepool: Use shade for all OpenStack interactions
AJaegerwill update...16:13
jeblairAJaeger: thank you :)16:13
fungithough also, looking for ^--- gets you the number of "removed or changed" jobs16:13
AJaegerfungi, that change showed me that the puppet-lint change touched 120 jobs - more than I thought16:13
*** esikachev has quit IRC16:13
*** rhedlind has quit IRC16:14
*** xwizard_ has quit IRC16:14
fungiand i wonder whether putting it after the potentially verbose diff output might make it easier to spot16:14
*** hieulq has quit IRC16:14
*** akrivoka has joined #openstack-infra16:15
yolandajenkins02 failed when removing old jobs16:15
yolandagoing to launch again16:15
AJaegerfungi, added as well. But if you have a better grep line, I can change it16:15
AJaegerI could also check for ^diff -r - see
*** skolekonov is now known as _skolekonov16:16
AJaegerthat adds a new job16:16
* AJaeger changes the test..16:16
*** nwkarsten has joined #openstack-infra16:16
openstackgerritMartin Oemke proposed openstack-infra/project-config: Enable Translation for tacker
*** _nadya__ has quit IRC16:18
openstackgerritRicardo Carrillo Cruz proposed openstack-infra/shade: Follow name_or_id pattern on domain operations
openstackgerritAndreas Jaeger proposed openstack-infra/project-config: Report number of changed jobs
openstackgerritAndreas Jaeger proposed openstack-infra/project-config: DO NOT MERGE - Testing
dolphmwhat does it take to get a channel added to eavesdrop?16:19
*** bdemers_ has joined #openstack-infra16:20
*** markvoelker has quit IRC16:21
dolphmi assumed all it would take would be adding it to accessbot
clarkbdolphm: our meetbot is actually the bot that provides channel logging16:21
clarkbthere is a hiera file to edit iirc let me get a link16:21
dolphmclarkb: ah thanks16:22
clarkbdolphm: add to that list16:22
dolphmclarkb: thank you, will do!16:22
*** bdemers has quit IRC16:23
clarkbfungi: sounds like using multple workers may not actually update things properly? fun16:23
openstackgerritMartin Oemke proposed openstack-infra/project-config: Enable Translation for tacker
*** jlanoux has quit IRC16:24
*** asettle has joined #openstack-infra16:25
*** thiagop-afk_ has joined #openstack-infra16:26
openstackgerritDolph Mathews proposed openstack-infra/system-config: Add #osic to meetbot
clarkblogstash 2.0 looks happy16:27
clarkbhappier than 1.3 so far16:27
openstackgerritAna Krivokapic proposed openstack-infra/project-config: New project for TripleO UI
bookwarhi, there is a small patch to add delay-release option to couple of projects reviews welcome :)16:28
*** mikelk has quit IRC16:28
yolandajenkins04 finished16:28
*** sbelous_ has quit IRC16:28
*** HeOS has quit IRC16:29
*** nwkarsten has quit IRC16:29
*** nwkarsten has joined #openstack-infra16:30
*** asettle has quit IRC16:30
rcarrillocruzbtw yolanda, fungi, i have this change
rcarrillocruznot sure if taht would've help on the jjb runs you are doing16:32
yolandaah i remember that16:32
*** zeih_ has joined #openstack-infra16:32
*** penguinolog has quit IRC16:32
yolandaadding  flush cache , or workers, looks useful to me16:32
*** bpokorny has joined #openstack-infra16:33
clarkbbookwar: I don't see delay-release as a jeepyb flag, any idea what that affects?16:33
bookwarclarkb: bot which changes bug status when patch is merged16:33
clarkbyolanda: we used workers yesterday and it is why we are in this situation16:33
bookwarclarkb: i'll try to find a reference..16:33
clarkbyolanda: so I don't think we should use that until it is working properly16:34
clarkbyolanda: and flush cache means we will update all jobs every time which we definitely do not want16:34
*** sdake_ has joined #openstack-infra16:34
yolandaclarkb, workers was a sample... rcarrillocruz change is a feature for enabling additional parameters16:34
yolandaso you can pass that on demand16:34
clarkbyolanda: I am just pointing out I don't think we want either of those flags16:34
openstackgerritAttila Darazs proposed openstack-infra/tripleo-ci: WIP: Use IPv6 on the gate jobs a test
*** zeih has quit IRC16:35
yolandaclarkb, --flush-cache can be useful to be able to run jjb with a clean cache, in case we detect corruption16:35
yolandaright now, if that happens, we need to run it directly on jenkins16:35
*** nwkarsten has quit IRC16:35
clarkbyolanda: jjb can and should address corruption direclt16:35
yolandaclarkb how it can? because we've hit that cache corruption quite often16:36
*** sdake has quit IRC16:36
clarkbyolanda: the cache corrupts when puppet times it outand kills it16:36
clarkbyolanda: one way to avoid that is to write it to a tmpfile until completion then mv it over16:36
clarkband if on start a corrupt cache is found jjb should just delete it16:37
yolandawell yes, but that happens... i agreed to work on some lockfile for jjb, but i think that having the ability to send some extra parameters on jjb runs cannot hurt16:37
*** nikhil has quit IRC16:37
clarkbI am not saying that16:37
*** zeih_ has quit IRC16:37
*** bdemers_ has quit IRC16:37
openstackgerritEmilien Macchi proposed openstack-infra/project-config: Add reno jobs to some Puppet OpenStack modules
clarkbI am saying those two flags shoudl be considered harmful for these reasons16:37
*** bdemers has joined #openstack-infra16:37
clarkband both should be fixed in jjb16:38
yolandai agree with you, jjb should not arrive to fail due to corrupted cache16:38
*** dims has quit IRC16:38
yolandajenkins02 finished now16:38
yolandai want to work on that next week16:38
*** matrohon has quit IRC16:38
*** apoorvad has joined #openstack-infra16:39
*** apoorvad has quit IRC16:39
*** vgridnev has quit IRC16:39
openstackgerritAndreas Jaeger proposed openstack-infra/project-config: Report number of changed jobs
openstackgerritAndreas Jaeger proposed openstack-infra/project-config: DO NOT MERGE - Testing
clarkbthe worker one is odd, it claimed ot update but apparently didn't16:40
clarkbmaybe it isn't splitting the job uploads properly between workers?16:40
clarkbso some subset of jobs gets updated but the others dont16:40
*** apoorvad has joined #openstack-infra16:40
*** pblaho has quit IRC16:42
*** vgridnev has joined #openstack-infra16:43
yolandahow is worker feature supposed to work? it divides the load of jobs to update?16:43
*** zul_ has quit IRC16:43
clarkbya, I think it uses a single thread to compile and determine which jobs need updates then it splits the update of those jobs between different threads16:43
yolandamaybe there are races updating the cache?16:44
*** ikutukov has quit IRC16:44
yolandaok all jenkins claimed to be updated now16:44
openstackgerritEric Harney proposed openstack-infra/project-config: Add NFS devstack plugin
*** zul has joined #openstack-infra16:45
*** esikachev has joined #openstack-infra16:47
pabelangerjust a heads up, will be offline a few hours this afternoon.  Should be back my 20:00 UTC16:47
openstackgerritAna Krivokapic proposed openstack-infra/project-config: New project for TripleO UI
*** krtaylor has joined #openstack-infra16:48
*** automagically_ has joined #openstack-infra16:48
*** tonytan_brb is now known as tonytan4ever16:48
automagically_FYI, seeing problems download check log:, I get and nothing more. Been seeing similar issues throughout the week16:48
*** pcaruana has quit IRC16:50
*** dims has joined #openstack-infra16:50
*** ashleighfarnham has joined #openstack-infra16:50
*** salv-orlando has joined #openstack-infra16:51
clarkbautomagically_: wgetting that file similarly fails which means it isn't the wsgi app that annotates the file failing16:51
*** abregman|afk is now known as abregman16:53
clarkbthta file doesn't exist on disk on the log server which implies it is a swift upload16:54
automagicallyappreciate you looking into it clarkb16:54
clarkbeither we aren't downloading a complete console log, we aren't uplodaing a complete console log, or swift is funny16:54
*** automagically_ has left #openstack-infra16:54
SamYapledoesnt it have to be one?16:54
SamYapleor does it rather16:54
clarkbI think the fact that the connection attempts to continue implies swift is being funny16:54
automagicallyAnd not funny haha16:55
*** tesseract- has quit IRC16:55
clarkbbaically swift is saying there is more data here and it is never showing up16:55
* clarkb looks at more logs16:55
*** thiagop-afk_ has quit IRC16:55
*** annegentle has quit IRC16:56
fungimore like "funny uhoh"16:56
*** dtantsur is now known as dtantsur|afk16:56
yolandaAJaeger, my puppet change passed. The fix for puppet-lint worked fine16:56
fungianyway, have fun with that. i need to go do errands for a while now. should be back in a few hours worst case16:56
*** dims_ has joined #openstack-infra16:58
automagicallySame issue with:
clarkbshould also find the console log on the jenkins master /me looks for it16:58
*** vincentll has quit IRC16:59
*** dims has quit IRC16:59
*** ashleighfarnham has quit IRC17:00
clarkbautomagically: is the full log on the jenkins side17:01
automagicallyThank you sir17:01
clarkbautomagically: there are no errors reported by the swift upload17:01
*** Jeffrey4l has quit IRC17:01
*** rcernin has quit IRC17:01
*** samuelBartel has quit IRC17:01
clarkband the file downloaded to be uploaded to swift reports 2423kb17:01
*** vincentfrancoise has left #openstack-infra17:02
clarkbwhich is much larger than what we are getting back out of swift again17:02
openstackgerritEmilien Macchi proposed openstack-infra/project-config: Add reno jobs to some Puppet OpenStack modules
clarkbI will check swift directly next17:03
openstackgerritRicardo Carrillo Cruz proposed openstack-infra/shade: Add domain_id param to project operations
*** e0ne has quit IRC17:05
*** BigWillie has quit IRC17:05
*** bpokorny has quit IRC17:06
automagicallyclarkb, thanks again. That Jenkins log link gave me everything I needed17:08
*** HeOS has joined #openstack-infra17:08
automagicallyBig props to the tireless efforts of the infra team17:08
*** dims_ has quit IRC17:08
*** markvoelker has joined #openstack-infra17:09
*** salv-orlando has quit IRC17:09
yolandapabelanger, more failures with vexxhost, git failures17:09
clarkbjudging by the speed at which the file brwoser thing is going we may be giving it a sad17:09
clarkbmight have to switch to the cli client17:09
pleia2good morning17:10
clarkbpleia2: good morning17:10
nibalizerohai pleia217:10
*** roxanagh_ has joined #openstack-infra17:10
*** annegentle has joined #openstack-infra17:11
*** gyee has joined #openstack-infra17:11
clarkb 14048946 files using about 1.5TB of disk17:12
jlvillalShould the gate start using pip > 8 again?  I notice it keeps using pip 7.1.2.  I had posted this patch:   to allow it to follow the pip releases again.17:12
*** abregman is now known as abregman|afk17:13
*** dims has joined #openstack-infra17:13
clarkbjlvillal: probably17:13
clarkbI will approve it now17:13
jlvillalclarkb: Thanks!17:13
clarkbautomagically: I can get the full file from swift directly17:16
clarkbmaybe it is something in our wsgi app17:16
*** sdague has quit IRC17:17
automagicallyclarkb: Hmm…probably worth opening a bug about, since I know I’ve seen it at least 5-8 times this week17:17
automagicallyAnd it prompts a recheck typically, which just consumes more resources17:17
*** MarkAtwood has joined #openstack-infra17:17
*** hashar has quit IRC17:19
pabelangerand back17:20
clarkbos-loganalyze and swiftclient are both older, so I don't think a code change has caused this17:20
pabelangeryolanda: okay, lets ask fungi or clarkb who the vexxhost contact it again17:20
clarkbpabelanger: yolanda: mnaser17:20
pabelangerclarkb: thanks17:20
*** ldnunes has joined #openstack-infra17:21
pabelangermnaser: do you have a moment to help troubleshoot some IO issue we are seeing on vexxhost?17:21
*** arxcruz has quit IRC17:21
*** BobBall_ has joined #openstack-infra17:21
mnaserfinishing up lunch17:21
mnaserafterwards sure!17:21
*** BobBall_ is now known as Guest4538317:21
openstackgerritMerged openstack-infra/system-config: Revert "Pin pip to less than version 8 in"
*** jordanP has quit IRC17:23
*** BobBall has quit IRC17:24
*** vgridnev has quit IRC17:25
openstackgerritMerged openstack-infra/puppet-etherpad_lite: fix module to work with latest mysql version
*** Guest45383 has quit IRC17:26
*** BobBall has joined #openstack-infra17:26
*** BobBall is now known as Guest6512417:26
openstackgerritZara proposed openstack-infra/storyboard-webclient: Edit task notes from board
*** vgridnev has joined #openstack-infra17:27
*** derekh has quit IRC17:27
*** leakypipes has quit IRC17:30
*** nwkarsten has joined #openstack-infra17:31
mnaserpabelanger: im a bit busy with something for most of the day today, however, if you have a list of instance id's that have been getting these problems, i could likely track them down and see if (for example) it's a bad host or something.17:34
*** Guest65124 is now known as GnomeKing17:34
*** aysyd has quit IRC17:34
*** jaypipes has joined #openstack-infra17:34
automagicallyclarkb: Yet another example:
*** krotscheck has quit IRC17:34
*** aysyd has joined #openstack-infra17:34
clarkbpabelanger: you can get that from the nodepool log by grepping for the nodepool ids of those servers, that should get you vexxhost uuids17:35
clarkbmtreinish: looking at logstash the push-subunit job claims to have 253 jobs running but only 2 workers17:35
*** pcaruana has joined #openstack-infra17:35
clarkbmtreinish: not sure if that is expected but could be source of the OOMs if it is actually running more than one job per worker17:35
*** nwkarsten has quit IRC17:36
*** krotscheck has joined #openstack-infra17:37
clarkbautomagically: my first suggestion would be to switch away from swift logging for that job17:37
*** ericksonsantos has quit IRC17:37
clarkbautomagically: just to work around the problem for now while we sort out what is causing it17:37
clarkbI cna push that chnage up if you are ok with it17:38
automagicallyclarkb: I’m good with that17:38
*** vgridnev has quit IRC17:38
*** arxcruz has joined #openstack-infra17:39
automagicallyAgain, appreciate your help17:40
automagicallyclarkb: Will they just be accessible via Jenkins?17:40
*** admin0 has joined #openstack-infra17:41
*** stevelle has joined #openstack-infra17:41
*** bpokorny has joined #openstack-infra17:41
openstackgerritClark Boylan proposed openstack-infra/project-config: Ansible role jobs copy console to disk not swift
*** vgridnev has joined #openstack-infra17:42
clarkbautomagically: no, they will still be accessible via, just changing the backend that we store the data in17:42
openstackgerritZara proposed openstack-infra/storyboard-webclient: Edit task notes from board
automagicallyAh okay, clarkb17:42
clarkbautomagically: right now we have the ability to save to a 12TB disk array or to a swift container17:42
*** sfinucan has quit IRC17:42
automagicallyI’m +1 on that17:43
automagicallyAnd let the community in #openstack-ansible know17:43
openstackgerritJames E. Blair proposed openstack-infra/zuul: Make all configuration in-repo configuration
openstackgerritJames E. Blair proposed openstack-infra/zuul: Fix race in tests
*** dizquierdo has joined #openstack-infra17:46
openstackgerritMerged openstack-infra/zuul: Merge branch 'master' into feature/zuulv3
*** kzaitsev_mb has quit IRC17:48
clarkbpleia2: fungi: for the sprint we are sharing with ironic and qa again or is that copy pasta from last time?17:48
*** ildikov has quit IRC17:48
*** bpokorny has quit IRC17:49
*** bdemers has quit IRC17:49
jeblairthere's something strange going on with nodepool allocations17:50
*** bpokorny has joined #openstack-infra17:50
jeblairwe've been consistently under capacity for a while, but we still have post jobs waiting for hours17:50
pleia2clarkb: copy pasta, fixing :)17:50
clarkbjeblair: its because we have a large extra ready node17:50
*** baoli_ has quit IRC17:50
clarkbwhich may be due to bare-trusty? not sure if we have cleaned that out yet17:51
*** lucasagomes is now known as lucas-afk17:51
*** stevelle has left #openstack-infra17:51
jeblairclarkb: yeah, there are a bunch of bare-trusty nodes than are 4h ready17:51
*** EricGonc_ has joined #openstack-infra17:52
jeblairclarkb: ah, we need a zuul geard restart now don't we?17:52
clarkbjeblair: yes I think that is still pending17:52
*** _nadya_ has joined #openstack-infra17:52
jeblairclarkb: which dovetails nicely into why i was checking that page anyway17:52
jeblairclarkb: i want to restart it to pick up jhesketh's cache change (and there's an outside chance that may have an effect on the leak too, though i don't think so)17:53
jeblairso i will restart zuul now17:53
*** nwkarsten has joined #openstack-infra17:53
clarkbthen we can tell nodepool to delete any leftover bare-trusty17:53
clarkbcrinkle: re summit planning, what do you think the hcances are of infra-cloud hosts being in their new home by summit time? we could possibly use the sprint to bring it back up again and put it into production17:54
clarkbcrinkle: I know west is sort of in a weird layer 8 state but maybe we can do east?17:55
*** EricGonczer_ has quit IRC17:55
jeblairzuul restarted; nodepool deleting17:55
* clarkb adds it to the list, worst case we pick other things off the list17:55
jeblairclarkb: i like the idea17:55
*** e0ne has joined #openstack-infra17:56
*** vgridnev has quit IRC17:56
jeblairall bare-trusty nodes are in delete state17:56
*** samuelBartel has joined #openstack-infra17:56
openstackgerritAndreas Jaeger proposed openstack-infra/project-config: Report number of changed jobs
openstackgerritAndreas Jaeger proposed openstack-infra/project-config: DO NOT MERGE - Testing
pabelangerclarkb: sure, let me check now17:57
*** auggy is now known as frauggy17:57
*** nwkarsten has quit IRC17:57
clarkbpabelanger: note the nodepool log is on an 8 hour rotation so you may have to look at the previous several log files17:57
AJaegeryolanda: great to hear that the puppet-lint works!17:58
pabelangerclarkb: okay17:58
crinkleclarkb: if everything goes as expected then at least east will be rehomed by the summit (but i worry about counting on things going as expected)17:58
*** esikache1 has joined #openstack-infra17:58
*** esikachev has quit IRC17:58
clarkbcrinkle: thats fair, I put it on the list and we can tackle it if we are able17:59
crinkleclarkb: timeline was about three weeks from last monday which gives a couple extra weeks of buffer17:59
*** sarob has joined #openstack-infra18:00
*** asettle has joined #openstack-infra18:00
*** tongli has joined #openstack-infra18:01
*** yamahata_ has quit IRC18:01
*** vgridnev has joined #openstack-infra18:01
*** samuelBartel has quit IRC18:02
openstackgerritMerged openstack-infra/zuul: Fix race in tests
*** dims_ has joined #openstack-infra18:04
*** dims has quit IRC18:04
*** asettle has quit IRC18:05
*** tphummel has joined #openstack-infra18:05
*** dizquierdo has quit IRC18:06
openstackgerritMonty Taylor proposed openstack-infra/nodepool: Use shade for all OpenStack interactions
*** ihrachys has quit IRC18:06
*** admin0 has left #openstack-infra18:07
*** vgridnev has quit IRC18:08
*** annegentle has quit IRC18:09
*** sbelous_ has joined #openstack-infra18:09
*** annegentle has joined #openstack-infra18:09
*** annegentle has quit IRC18:10
*** rossella_s has joined #openstack-infra18:10
*** icey has quit IRC18:11
*** flepied has quit IRC18:11
*** dizquierdo has joined #openstack-infra18:11
*** mriedem has joined #openstack-infra18:12
* AJaeger sees a NOT_REGISTERED for gate-bindep-fallback-devstack-centos7 in zuul18:12
openstackgerritDan Prince proposed openstack-infra/tripleo-ci: Metrics tracking for TripleO deployment tasks
AJaegerWe didn't change that job AFAIK18:13
clarkbAJaeger: zuul was just restarted, we may not have had a devstack-centos7 ready to register18:13
AJaegerclarkb: ah...18:13
*** icey has joined #openstack-infra18:13
*** binbincong has quit IRC18:13
*** dontrump has quit IRC18:13
*** shardy has quit IRC18:13
*** baoli has joined #openstack-infra18:14
*** nwkarsten has joined #openstack-infra18:14
*** fernnest_ has joined #openstack-infra18:14
*** fernnest has quit IRC18:15
*** vgridnev has joined #openstack-infra18:15
*** fedexo has quit IRC18:16
*** annegent_ has joined #openstack-infra18:16
*** hdd has quit IRC18:17
*** noorul has joined #openstack-infra18:17
*** annegent_ has quit IRC18:17
*** nwkarsten has quit IRC18:18
*** annegentle has joined #openstack-infra18:18
noorulI am trying to setup openstack ci locally18:20
noorulI think I am almost there18:20
*** binbincong has joined #openstack-infra18:21
*** hdd has joined #openstack-infra18:21
*** esikache1 has quit IRC18:21
noorulOnce I add new project information to project-config repo, I suppose I should run puppet apply on gerrit machine18:21
clarkbAJaeger: ya no ready or used centos7 right now18:21
clarkbone building in internap18:21
noorulSo that it adds this new project to gerrit18:22
*** sbelous_ has quit IRC18:22
*** jistr has quit IRC18:22
noorulpuppet apply fails at /usr/local/bin/manage-projects -d -l /var/log/manage_projects.log18:22
*** austin81 has joined #openstack-infra18:22
*** degorenko is now known as _degorenko|afk18:22
clarkbthat is what adds the project, check the log file there18:23
clarkbcould be authentication or permissions issue18:23
noorul/usr/local/lib/python2.7/dist-packages/paramiko/ UserWarning: Unknown ssh-rsa host key for18:23
clarkbthat means that you need to accept the ssh-rsa key for gerrit18:23
AJaegerclarkb: thanks...18:23
*** annegent_ has joined #openstack-infra18:24
*** resker has quit IRC18:24
*** abregman|afk has quit IRC18:24
*** esikache1 has joined #openstack-infra18:24
*** annegent_ has quit IRC18:24
noorulclarkb: Not sure what needs to be done18:24
*** annegent_ has joined #openstack-infra18:25
clarkbnoorul: you have to accept the ssh host key for the gerrit service18:25
clarkbas the user running manage-projects (probably root) ssh to gerrit's ssh server (so port 29418 not 22) and accept the key18:25
*** austin81 has left #openstack-infra18:26
*** rbrndt__ has quit IRC18:26
AJaegerso, my change for reporting the number of changed jenkins jobs is ready for review if anybody wants some more information:
*** annegentle has quit IRC18:28
*** sdake_ has quit IRC18:28
noorulclarkb: I am not sure which key and user it is using to ssh18:28
clarkbnoorul: if using our puppet modules it will be root18:29
nooruland the key /home/gerrit2/review_site/etc/ssh_host_rsa_key ?18:29
clarkbyes thats the key, but you need the ssh client side to store it in known hosts18:29
clarkbso just run the ssh command against it18:29
*** asselin has left #openstack-infra18:30
noorulBut I get access denied18:30
clarkbthats fine, did you accept the host key? if so that problem is solved18:30
clarkbas for access denied you probably have to set the username for the remote side18:30
clarkbsince root isn't going to be a user in gerrit18:30
*** tsg has quit IRC18:30
*** vgridnev has quit IRC18:30
pabelangerclarkb: mnaser: I've added 3 nodes for veexhost that are failing.
pabelangertrying to find a few with slow apt-get install issues18:31
*** binbincong has quit IRC18:31
noorulbut when using puppet it will anyhow use root, right?18:32
clarkbnoorul: it will run the command as root but you can always spcify the remote username18:32
noorulclarkb: Where should I specify that?18:32
*** binbincong has joined #openstack-infra18:33
clarkbnoorul: it is part of the jeepyb config iirc18:33
noorulclarkb: I am not sure whether I missed any step here18:34
openstackgerritAndreas Jaeger proposed openstack/requirements: Update upper-constraints for fairy-slipper 0.2.0
*** claudiub|2 has joined #openstack-infra18:34
*** nwkarsten has joined #openstack-infra18:34
noorulI am following
*** arxcruz has quit IRC18:35
noorulclarkb: I am trying to find out where is jeepyb config located18:36
*** nwkarste_ has joined #openstack-infra18:36
noorulI am running this from gerrit node18:37
clarkbnoorul: looks like you set the user in the projects.yaml18:37
clarkbgerrit-user: foo18:37
*** dtardivel has quit IRC18:38
clarkbnoorul: so hceck your projects.yaml and project.ini18:38
clarkbthats the config jeepyb/manage-projects uses18:38
*** sdake has joined #openstack-infra18:39
AJaegerwow, still no dsvm-centos7 nodes ;(18:39
clarkbnoorul: that is in /home/gerrit218:39
*** nwkarsten has quit IRC18:39
pabelangerAJaeger: did we loose it?18:39
*** thiagop-afk is now known as thiagop18:39
clarkbAJaeger: the one in internap has been building for an hour :/18:39
clarkband I tried deleting the one in ovh that is in a deleting state manualy and that failed18:40
openstackgerritSimon McCartney proposed openstack-infra/infra-ansible: Reworked to support Debian & Darwin/OSX
*** claudiub has joined #openstack-infra18:40
AJaegerpabelanger: NOT_REGISTERED for any job that uses dsvm-centos7 ;(18:40
clarkbI can try deleting the one that is building18:40
*** sdake_ has joined #openstack-infra18:41
AJaegermin-ready is 1 for that one18:41
*** nwkarste_ has quit IRC18:41
clarkbAJaeger: we may want to rebalance that to a higher number18:41
clarkbok we have one ready now18:41
*** yamahata_ has joined #openstack-infra18:41
AJaegergreat, thanks18:42
*** claudiub|2 has quit IRC18:42
AJaegerwe also do not run dsvm-centos7 in all clouds. Is that intentional?18:42
openstackgerritJames E. Blair proposed openstack-infra/zuul: Move gerrit logic from source to connection
openstackgerritJames E. Blair proposed openstack-infra/zuul: Sever the forward reference from connections to their users
AJaegerrax is missing18:42
clarkbAJaeger: may be due to dib and glean18:42
clarkbwe probably could add it to rax now if we test that networking is configured properly on boot18:42
*** sputnik13 has joined #openstack-infra18:43
pabelangerclarkb: Once ianw gets a new release of diskimage-builder we can do 29097418:44
pabelangerlast I checked centos-minimal was working on rax, according to ianw18:44
*** sdake has quit IRC18:44
clarkbpabelanger: that has the opposite problem of only adding it to rax :)18:44
*** thorst_afk is now known as thorst18:44
clarkbbut ya with that in place we can rax and everywhere else because glean18:45
pabelangerclarkb: Ya, I think ianw wanted to get it working on rax first, then other coulds.  Since he has always battled rax in the past18:45
AJaeger817 jobs run on devstack-centos7 - complete distribution (but one day old tree):
mnaserpabelanger does run on ipv6 too?18:45
AJaegerSo, more devstack-centos7 (817) jobs than devstack-trusty (539)?18:46
jeblairmnaser: yes18:46
pabelangermnaser: ya, it resolves to 2001:4800:7819:103:be76:4eff:fe06:63c18:47
slaglehi, we're seeing NOT_REGISTERED for all of the gate-tripleo-* jobs18:47
mnaserpabelanger jeblair ok, we've been working on bringing ipv6 up and we noticed a few networking issues that we're trying to iron out.  i suspect that is related.18:47
AJaegerclarkb: what value do you suggest for devstack-centos7 min-ready instead of 1?18:47
SamYapleshameless begging for an ack review on my shade service w/ keystone v3 support patch
slagleis there any maintenance going on, or something we can check in the tripleo cloud?18:47
pabelangerdprince: ^18:48
clarkbslagle: we restarted the gearman server which means that all the jenkins slaves had to reregister themselves, if there are no tripleo slaves available then they won't register18:48
jeblairslagle: i restarted nodepool and zuul.  all tripleo nodes are in use except one which has been in delete state for 14 days18:48
*** flepied has joined #openstack-infra18:48
jeblairslagle: i can delete all those nodes, and some should be rebuild and will register jobs18:48
jeblair(this would happen on its own once the jobs finish, but they take a while)18:49
slagleok, thanks18:49
pabelangermnaser: clarkb: Ya, I noticed some network issue with the AFS mirror in vexxhost too.  So we are seeing some network issues too18:49
pabelanger404 timeout for example18:49
pabelanger404 not found18:50
mnaserwe're working on it, i don't think it'll be much longer to clear things out now18:50
jeblairslagle: all deleted, should correct itself soon18:50
dprincejeblair: thanks18:50
slaglejeblair: thanks18:50
*** ajmiller_ has joined #openstack-infra18:50
openstackgerritAndreas Jaeger proposed openstack-infra/project-config: Increase min-ready for devstack-centos7
AJaegerclarkb: very conservative change to 2 only ^ - I'm fine to increase further if there's some guidance18:52
*** jcoufal has quit IRC18:52
prometheanfirewhile we were down, I proposed this patch increaded the libvirt-python upper constraint to 1.3.218:52
*** ajmiller__ has joined #openstack-infra18:53
*** ajmiller_ has quit IRC18:53
*** ajmiller has quit IRC18:53
noorulclarkb: I remember creating Project Bootstrappers group18:54
noorulclarkb: Adding account project-creator18:54
noorulclarkb: That is in projects.ini18:54
clarkbnoorul: then you need to make sure the ssh key for project-creater is the one that manage_projects uses and that the ssh host key is accepted by ssh18:55
openstackgerritAndreas Jaeger proposed openstack-infra/project-config: Increase min-ready for devstack-centos7
fungionly a couple hundred lines of scrollback while i was erranding. surprisingly non-chatty, but i guess that's a friday for you18:57
*** nwkarsten has joined #openstack-infra18:57
AJaegerfungi, have a look at the numbers of nodes and jobs in 291861 or - that was interesting to me18:58
*** sridhar_ram has joined #openstack-infra18:58
fungiclarkb: that's what we _had_ last time. what i told ttx and thingee was "shared sprint space" was fine for us again as it worked out well before18:59
noorulclarkb: I am not sure what is wrong, somehow auth is denied18:59
clarkbnoorul: are you still getting the ssh host key error?18:59
clarkbnoorul: if so that is the problem18:59
openstackgerritAndreas Jaeger proposed openstack-infra/project-config: Add missing node to puppet-cookiecutter check
noorulI have project-creator account on gerrit19:00
noorulI have the public ssh key added to the account19:00
AJaegersomehow we lost the node declaration on two jobs, would be nice to get the change above in today ^19:00
AJaegerEmilienM: FYI ^19:00
fungijeblair: i wanted to restart zuul last night but was trying to get the last of the jjb updates in sync everywhere first to make sure we didn't reregister bare-trusty on any. then lenkins01 took so long/many retries to finish i was too sleepy to be confident i wouldn't make a mess and figured we could restart zuul today or tomorrow when it gets quiet again19:00
openstackgerritMerged openstack-infra/project-config: Setup neutron-lbaas-dashboards for release
clarkbnoorul: yup, the ssh host key is what the ssh server identifies itself with as an assurance you are not getting man in the middled19:01
*** sridhar_ram1 has joined #openstack-infra19:01
*** nwkarsten has quit IRC19:01
fungias for the cloud formerly known as infra-cloud west, i suspect nobody knows an actual timeline for that hardware rematerializing in a connected location. at least that's how the last update sounded19:02
noorulclarkb: Is there a way I can go and see the user information in gerrit for those accounts created using gerrit create-account command?19:02
fungiburied under a data center full of red tape19:02
clarkbnoorul: if you are getting the host key errors still that isn't an issue with the gerrit user, it is a problem with the ssh client side configuration19:02
clarkbnoorul: if you run the ssh command and accept the key that should go away19:02
pabelangerfungi: where did you see the update by chance?19:03
EmilienMAjaeger: ack19:03
mtreinishclarkb: so there are only 2 workers, but each should only be handling 1 thing at a time on the gearman19:03
*** sridhar_ram has quit IRC19:03
clarkbmtreinish: ok, that isn't what gearman is claiming. May possibly be related to OOMing?19:03
clarkbnoorul: once that is done, rerun manage-projects and see if the error changes and go from there19:04
noorulclarkb: This is what I am getting
mtreinishclarkb: they pull the subunit files from the log server and as soon as it downloads it puts it on a queue and returns job complete19:04
*** sdake_ has quit IRC19:05
mtreinishand if something goes wrong with the pulling it returns a job exception (or whatever it is)19:05
clarkbnoorul: that says your ssh key is not a valid ec key19:05
clarkbnoorul: I would use rsa not sure how well gerrit does ecdsa19:05
mtreinishclarkb: the ooming I assumed was that it was putting things on the queue faster than it could pop them off19:05
clarkbmtreinish: I see19:05
mtreinishwhich is why we moved to 2 workers19:05
mtreinishI don't think it's OOmed since nibalizer and I got subunit-worker02 up19:05
*** doug-fis_ has joined #openstack-infra19:05
*** tqtran-afk has joined #openstack-infra19:06
noorulclarkb: I used ssh-key-gen command from that section19:07
*** doug-fish has quit IRC19:08
nibalizermtreinish: would it say it oomed in syslog?19:08
clarkbnoorul: ok you need to figure out why it is thinking that file is an ecdsa key19:08
clarkbnoorul: examine it and see if the first bytes are not ssh-rsa19:08
mtreinishnibalizer: syslog, dmesg, or kern.log. I forget where things end up in ubuntu land19:09
clarkbfungi: ya but it sounds like maybe we will have east19:09
mtreinishbut you'd see it in 1 of those places (if not all of them, assuming they all exist)19:09
fungiyolanda: did jjb updates complete successfully?19:09
clarkband we can go ahead an work on making east a thing19:09
*** eharney has quit IRC19:09
nibalizermtreinish: i see no evidence19:10
*** sdake has joined #openstack-infra19:10
nibalizeralso our subunit worker has been running since Mar07 and puppet isn't configured to restart it anyways19:10
fungiyolanda: i think it must have worked because has "WITH GRANT OPTION" now19:11
fungiso that's a good sign at least19:11
yolandafungi, everything worked19:11
fungimordred: should be safe to recheck i think, if you haven't already ^19:11
pabelangerfungi: Ah, ya, that was a little cryptic19:11
*** tqtran-afk is now known as tqtran19:11
*** alex_xu has quit IRC19:12
openstackgerritMerged openstack-infra/puppet-etherpad_lite: Fix races in httpd install and config
mtreinishnibalizer: yeah that's probably because I'm pretty sure we'd drop whatever's been queued when we restart it :)19:12
fungipabelanger: if you followed the discussion at the sprint, that's basically that they didn't get around to unracking and removing our hardware before the deadline for that data center to go on "lockdown" for a full reinventory19:12
*** e0ne_ has joined #openstack-infra19:13
mordredfungi: woot!19:14
*** e0ne has quit IRC19:14
pabelangerfungi: wow19:14
*** binbincong has quit IRC19:14
*** ferhat has quit IRC19:16
*** alex_xu has joined #openstack-infra19:16
*** nwkarsten has joined #openstack-infra19:17
jeblairpabelanger: (even though we shut it down early to make sure they could do so)19:17
automagicallyclarkb: Any idea why didn’t verify. I don’t see any Jenkins checks with a failure19:17
*** ildikov has joined #openstack-infra19:18
fungiyeah... them: we're worried this hardware may get stuck in the datacenter for a reinventory if we don't remove it soon... us: okay, we'll shut it down today so you can get started right away!19:18
clarkbautomagically: its the NOT_REGISTERED job if you recheck that should be fixed now19:18
automagicallyaha, its been submitted for recheck, so I’ll wait and see19:19
openstackgerritMerged openstack-infra/project-config: Add missing node to puppet-cookiecutter check
crinklefungi: latest comment on the jira ticket is "shipment postponed while Switch and HP finalize the move out"19:19
clarkbcrinkle: for west or east or both?19:20
crinkleclarkb: for west19:20
crinkleeast is on its way19:20
jeblairclarkb: okay... so after the restart..... there's something weird going on with nodepool allocations; post jobs are waiting 1h for jobs even though we're well under capacity.  :)19:20
fungii guess "Switch" is a department? or a vendor?19:20
openstackgerritMerged openstack-infra/devstack-gate: Use dd to create swapfile, ensure swap amount == 8GB
clarkbjeblair: hrm and the ready nodes are all gone19:20
crinklefungi: i am guessing a vendor19:21
noorulclarkb: Looks like an issue with the key19:21
fungijeblair: last night i saw a bunch of ubuntu-trusty nodes in a building state in rax-iad for up to an hour, so nodepool was counting those toward demand fulfillment19:21
noorulclarkb: I used admin account to create the project19:21
jeblairfungi: ah, that could do it19:21
pabelangerjeblair: Exactly19:21
fungiif iad is still having trouble...19:21
jeblairfungi: most building nodes are <5m; maybe ~10 at 30+m19:22
openstackgerritAleksandra Fedorova proposed openstack-infra/project-config: Add missing delay-release option in Fuel projects
jeblairfungi: all 30m+ are iad19:22
jeblairi wonder if we should lower our build timeout19:22
jrollcrinkle: fungi: last I saw "Switch" was building a big colo - not sure if they only do colo or what19:23
openstackgerritDan Prince proposed openstack-infra/tripleo-ci: Metrics tracking for TripleO deployment tasks
pabelanger1hr boot is pretty gracious19:24
jeblairthere is a switch dc in vegas19:24
pabelangerI'd be raging after 5mins :)19:24
jeblairso maybe that's where west is moving out of19:24
mordredthat is where west was19:25
mordredit's an AMAZING datacenter19:25
Clintwith slot machines in every rack19:25
jeblairmordred: they have another one called "tahoe-reno" even though it would be *slightly* more accurate to call it "black rock city"19:26
mordredjeblair: something tells me that calling it black rock city would get way to many people showing up at its doors19:27
*** annegent_ has quit IRC19:28
SamYaplemordred: trying to figure out the ending of LOST still huh? black rock was just a ship move on19:29
*** ajmiller__ is now known as ajmiller19:29
mordredSamYaple: heh. sorry man - wrong cultural reference19:29
openstackgerritMerged openstack-infra/shade: Add update_service()
*** annegentle has joined #openstack-infra19:30
jeblairokay i'm wrong, it is slightly closer to lake tahoe than brc19:30
noorulNow the project is created. I submitted a patch for review19:31
noorulZuul log shows
jeblairpabelanger: yes, though we also have to account for the fact that nodepool is slow when under load, so it probably wouldn't be good to make it shorter than 30m at the moment19:31
*** Guest75056 has quit IRC19:32
*** sambetts is now known as sambetts|afk19:33
pabelangerjeblair: Was just to see if we logged the boot times in statsd, I don't think so19:34
noorulCan someone help me to figure what is wrong?19:34
*** ociuhandu has quit IRC19:36
openstackgerritMerged openstack-infra/project-config: Ansible role jobs copy console to disk not swift
*** akrivoka has quit IRC19:37
*** amrith is now known as _amrith_19:38
*** daemontool_ has quit IRC19:39
*** e0ne_ has quit IRC19:40
jeblairmordred: i don't know about actual hosting, but looking at the gallery, i will give them *high* marks for theater.19:40
jeblairmordred:   <-- actual internet tubes :)19:40
mordredjeblair: yup :)19:40
jeblairpabelanger: best we have is 'time to ready'19:41
mordredjeblair: so - one of the neat things they have there is that their air handling system mixes outside air depending on air quality and humidity19:41
mordredjeblair: cause it turns out the air in the desert at night is actually very much the type of air you want19:41
jeblairpabelanger: which is what nodepool is paying attention to.  the trick is that because of the nodepool thread starvation under heavy load, it can actually be several minutes before nodepool gets around to processing a node that has become ready19:41
jeblairpabelanger: we will fix that in v319:41
jeblairmordred: i love the air in the desert at night19:42
pabelangerjeblair: Ah, interesting19:42
jeblairalmost as much as i love it at dawn19:42
mordredthe other thing is that they have ex-marines outside who patrol a few-miles-wide perimeter and who are empowered to commandeer fuel in case of crisis for the generators19:42
*** HeOS has quit IRC19:43
jeblairso, like, don't drive past in a diesel truck in times of crisis19:43
*** bdemers has joined #openstack-infra19:43
jeblairmordred: or they will take your truck and fuel and put you in the holding cell:
*** HeOS has joined #openstack-infra19:44
mordredjeblair: it's not a bad holding cell to be in in times of crisis that are crisis enough that the hoover dam is not producing electricity19:44
* jeblair wonders if the empowerment comes from the constitution or the guns their holding, and whether that's an academic question19:44
mordredI mean, let's be clear about what all has to fail before those fellows are out taking your truck19:44
*** _nadya_ has quit IRC19:44
jeblairthey're even19:45
mordredjeblair: apparenlty, currently, it's an AND19:45
jeblairthe constitution is weird in nevada19:45
mordredjeblair: the empowerment comes from the DOD hosting strategic servers in the facility19:45
mordredthe power grid shuts off power to hospitals before that data center for the same reason19:46
mordredwhich is, in itself, creepy19:46
jeblairnoorul: is that your zuul or ours?19:46
jeblairnoorul: sorry, meant is that your gerrit or ours?19:46
nooruljeblair: mine19:47
jeblairnoorul: i'd try running "ssh -p 29418 $hostname gerrit stream-events" with the ssh key zuul is using and see what it's outputting19:47
noorulI did that19:48
nooruland I get19:48
noorulCapability streamEvents is required to access this resource19:48
noorulzuul user I created is under Continuous Integration Tools group19:48
*** esikache1 has quit IRC19:48
noorulI am figuring out how to give access to stream-events for users under that group19:48
noorulany idea?19:49
*** sdake_ has joined #openstack-infra19:49
jeblairnoorul: you'll want to modify acls for the "All-Projects" project19:49
*** sridhar_ram1 is now known as sridhar_ram19:49
openstackgerritMerged openstack-infra/project-config: Enable Translation for tacker
*** sdake has quit IRC19:49
*** tonytan4ever has quit IRC19:50
jeblairnoorul: under 'global capabilities' you should be able to add the 'stream events' capability and add the group to that capability19:50
*** nwkarsten has quit IRC19:50
jeblairnoorul: here's ours:,access19:50
*** electrofelix has quit IRC19:50
*** tonytan4ever has joined #openstack-infra19:51
jeblairnoorul: (we give it to all users, but giving it to any group your zuul user is in, such as ci tools, will be fine)19:51
*** thiagop has quit IRC19:51
fungimordred: my old employer's boston datacenter did similar outside air mixing to save on cooling in the winter19:52
mordredfungi: ++19:52
*** Slower has quit IRC19:53
nooruljeblair: Instead I added zuul user to Non Interactive Users group19:54
fungiit was actually a very smart system, did inline (de)humidification and filtration on the intake19:54
nooruljeblair: Now I get
openstackgerritMerged openstack-infra/nodepool: Be more specific in logging timeout exceptions
*** esikache1 has joined #openstack-infra19:54
*** esikache1 has quit IRC19:54
*** _nadya_ has joined #openstack-infra19:56
openstackgerritSimon McCartney proposed openstack-infra/infra-ansible: Reworked to support Debian & Darwin/OSX
*** sdake_ has quit IRC19:59
*** tonytan4ever has quit IRC20:02
*** nwkarsten has joined #openstack-infra20:03
*** nwkarsten has quit IRC20:04
*** nwkarsten has joined #openstack-infra20:04
*** pots has quit IRC20:09
*** gnuoy_ has joined #openstack-infra20:11
*** madorn_ is now known as madorn20:11
*** pots has joined #openstack-infra20:12
*** nikhil_k has quit IRC20:12
*** flepied has quit IRC20:13
AJaegerEmilienM: the cookiecutter job still fails ;( I have no idea why ;( Hope somebody else can help and tell what's wrong with paths in
jeblairnoorul: that'll be a config file syntax error; you need a "projects:" section in the layout.yaml file20:16
*** sdake has joined #openstack-infra20:17
EmilienMAjaeger, OK I'm currently on my phone afk but I'll look tonight. It's not urgent at all and thanks a lot for your help. I'll debug20:17
*** ociuhandu has joined #openstack-infra20:19
*** eil397 has joined #openstack-infra20:19
*** x00350071 has quit IRC20:19
*** x00350071 has joined #openstack-infra20:20
*** ihrachys has joined #openstack-infra20:21
nooruljeblair: You are right.20:22
nooruljeblair: I fixed that!20:22
noorulNow Zuul is up20:22
jeblairnoorul: yay!20:22
*** pots has quit IRC20:23
noorulBut it is not responding to events20:23
noorulI posted a review20:23
jeblairnoorul: can you paste your layout.yaml?20:23
*** pots has joined #openstack-infra20:24
*** zul has quit IRC20:25
*** pots has quit IRC20:25
*** eharney has joined #openstack-infra20:26
*** hashar has joined #openstack-infra20:27
jeblairnoorul: two things: 1) you may want to start with a much simpler layout.yaml that only has the jobs you're working on, and doesn't use our paratemer functions, etc.20:27
jeblairnoorul: 2) the debug log will usually help you figure out why zuul ignored an event or didn't run a job, etc.20:27
noorulAnother issue I have is20:28
noorulwhen I run zuul using service command it is not starting properly20:29
noorulbut when I run "sudo -u zuul zuul-server -d", it is running20:29
*** pots has joined #openstack-infra20:29
*** ajmiller_ has joined #openstack-infra20:30
*** ajmiller has quit IRC20:30
*** pots has quit IRC20:30
noorulLooks like it is getting the event20:31
*** apoorvad has quit IRC20:32
fungiAJaeger: pwd on the job is likely the git worktree at /home/jenkins/workspace/gate-puppet-openstack-cookiecutter-puppet-3-cookiecutter-check-nv/ and so the relative path to the script would be ./contrib/ not ./contrib/openstack/puppet-openstack-cookiecutter/contrib/bootstrap.sh20:33
*** |-paul-| has joined #openstack-infra20:33
*** kencjohnston has quit IRC20:34
fungiAJaeger: it's probably adding an extra $ZUUL_PROJECT/contrib/ in the middle of that or something20:34
*** pots has joined #openstack-infra20:34
openstackgerritK Jonathan Harker proposed openstack-infra/puppet-elasticsearch: Make total_shards_per_node configurable
*** twm2016_ has joined #openstack-infra20:35
twm2016_Has anyone here encountered problems related to installing deps while running a tox test?20:35
*** asselin has joined #openstack-infra20:35
pleia2fungi: doh, thanks20:35
twm2016_I've tried the solution here: but haven't had any luck.20:36
*** kencjohnston has joined #openstack-infra20:36
*** thorst has quit IRC20:38
fungitwm2016_: "problems related to installing deps while running a tox test" is... well the word vague would be an understatement here. can you link to an example of the particular "problems" you have?20:39
*** thorst has joined #openstack-infra20:39
AJaegerthanks for checking fungi, let me look at it again and udpate...20:40
fungitwm2016_: are you seeing the same "Not uninstalling py at /usr/lib/python2.7/dist-packages, owned by OS" message mentioned in that github issue?20:40
twm2016_fungi: sorry for the vague description. Here is the output I get after the test runs
AJaegerfungi, I should have never approved that last change ;( It made it worse...20:40
AJaegerthanks, fungi20:40
pleia2AJaeger: heh, I missed it too20:41
fungitwm2016_: for some reason it's calling /opt/stack/swift/.tox/py27/bin/easy_install instead of pip. do you happen to know why?20:41
openstackgerritAndreas Jaeger proposed openstack-infra/project-config: Fix path for puppet-openstack-cookiecutter test
AJaegerpleia2, fungi, EmilienM : Let's try again ^20:42
fungitwm2016_: is it possible you're using a very old release of tox? or maybe a heavily modified configuration?20:42
*** asettle has joined #openstack-infra20:42
AJaegerthanks for reviewing!20:44
twm2016_fungi: after running tox --version it says it is "2.3.1 imported from /usr/local/lib/python2.7/dist-packages/tox/__init__.pyc"20:45
*** krtaylor has quit IRC20:45
AJaegertwm2016_: that version works fine for me ;)20:45
fungitwm2016_: okay, that's the latest version. have you modified the tox.ini for swift?20:45
twm2016_fungi: I have not made any modifications to tox.ini.20:46
openstackgerritMerged openstack-infra/project-config: enable stack sudo for ironic discovery gate job
*** asettle has quit IRC20:47
fungitwm2016_: and workinf with a recent checkout of swift source code?20:48
twm2016_Ajaeger: I believe has something to do with my proxy. I'm looking at this patch to help: and when I run the pip commands I get "ConnectTimeoutError". I found this link to confirm my problem:
fungitwm2016_: yeah, the tox.ini for swift definitely says "install_command = pip install --allow-external netifaces --allow-insecure netifaces -U {opts} {packages}" but instead of pip it's calling easy_install20:49
twm2016_fungi: yes cloned, thirty minutes ago.20:49
fungiin your example output20:50
fungiand then easy_install thinks you're asking it to install a package called "install" because it doesn't take subcommands like that20:50
*** apoorvad has joined #openstack-infra20:50
*** annegentle has quit IRC20:51
twm2016_fungi: I've tried both, sorry I must have given you different output.20:51
fungiis it possible there's a very, very old version of virtualenv being used? what does `/opt/stack/swift/.tox/py27/bin/pip --version` say?20:51
fungitwm2016_: tried both what?20:51
openstackgerritSam Yaple proposed openstack-infra/project-config: Add python3 non-voting job to kolla-mesos
*** rbrndt__ has joined #openstack-infra20:52
twm2016_fungi: both pip install and easy_install20:52
twm2016_fungi: The version output is : pip 8.1.0 from /opt/stack/swift/.tox/py27/local/lib/python2.7/site-packages (python 2.7)20:52
fungioh, can you provide another paste where you haven't tried to modify it to use easy_install? the error in the one you provided is definitely along the lines of "easy_install has no idea what you're asking it to do"20:53
fungiso it's not going to have anything at all to do with whatever error you're getting from proper pip usage20:54
twm2016_fungi: Here it is,
fungitwm2016_: the big bold errors there telling you to install the header packages on your system for liberasurecode are your problem20:55
AJaegerRegarding JJB: I think it would be nice to see the number of changed jobs inthe compare output merged sooner - reviews for are welcome20:55
openstackgerritMerged openstack-infra/project-config: Add missing delay-release option in Fuel projects
fungitwm2016_: looks like you're on debian, so `sudo apt-get install liberasurecode-dev`20:55
* AJaeger waves good-bye and wishes everybody a great weekend!20:56
fungithanks AJaeger and have a great weekend!20:56
pleia2AJaeger: take care20:56
twm2016_fungi: You know I tried doing that before, I must have got the package name wrong because it couldn't find it. But that command definetly worked.20:56
twm2016_Thank you for your help!20:56
fungitwm2016_: no problem, have a great weekend20:57
AJaegerfungi, pleia2 enjoy the weekend as well!20:57
*** sdake has quit IRC20:58
jeblairif that doesn't work, just try this:
bswartzXKCD always has something appropriate21:04
*** sigmavirus24 is now known as sigmavirus24_awa21:05
openstackgerritMerged openstack-infra/project-config: Fix path for puppet-openstack-cookiecutter test
*** gordc has quit IRC21:06
openstackgerritMerged openstack-infra/project-config: Add python3 non-voting job to kolla-mesos
*** gildub has joined #openstack-infra21:08
openstackgerritDan Prince proposed openstack-infra/tripleo-ci: Metrics tracking for TripleO deployment tasks
*** dprince has quit IRC21:13
*** alex_xu has quit IRC21:14
ianwpabelanger: afs rpms are at least building now :
*** ldnunes has quit IRC21:15
*** kzaitsev_mb has joined #openstack-infra21:15
*** alex_xu has joined #openstack-infra21:15
mordredianw: woot!21:16
*** yolanda has quit IRC21:18
*** yolanda has joined #openstack-infra21:19
ianwpabelanger: ahh, so close!  it's one directory deeper nested than i thought it would be.21:19
clarkbis plan to have puppet grab from tarballs and install that rpm?21:20
*** kgiusti has quit IRC21:20
clarkbwhen building the wheel builder slave?21:20
*** frauggy is now known as auggy21:20
ianwclarkb: is the plan21:21
*** sigmavirus24_awa is now known as sigmavirus2421:22
*** roxanagh_ has quit IRC21:23
*** roxanagh_ has joined #openstack-infra21:24
*** baoli has quit IRC21:24
*** baoli has joined #openstack-infra21:25
*** nikhil_k has joined #openstack-infra21:28
*** nwkarste_ has joined #openstack-infra21:28
*** baoli has quit IRC21:31
*** nwkarsten has quit IRC21:32
openstackgerritIan Wienand proposed openstack-infra/project-config: [wip] Revert "Add CentOS AFS build job to project-config experimental"
openstackgerritIan Wienand proposed openstack-infra/project-config: Centos AFS rpm : copy to workspace
*** devananda has quit IRC21:35
*** nwkarste_ has quit IRC21:35
*** thorst has quit IRC21:35
*** nwkarsten has joined #openstack-infra21:35
*** mriedem has quit IRC21:36
*** rlandy has quit IRC21:37
*** nwkarste_ has joined #openstack-infra21:37
*** sarob has quit IRC21:37
*** annegentle has joined #openstack-infra21:38
openstackgerritClark Boylan proposed openstack-infra/system-config: Don't install ES on logstash workers
*** annegentle has quit IRC21:39
*** nwkarsten has quit IRC21:40
clarkbanteaya: ^ last step in logstash 2.0 upgrade21:41
*** annegentle has joined #openstack-infra21:41
anteayaclarkb: any other post logstash 2.0 install patches to review?21:41
clarkbanteaya: that should be it21:41
anteayayup, caught it, thanks21:41
anteayawonderful, you have my +121:41
*** noorul has quit IRC21:41
anteayaso far so good?21:41
*** nwkarste_ has quit IRC21:42
jesusaurclarkb: do we want to also ensure the service is stopped and the package in purged?21:42
jesusauror will that be done manually?21:43
*** _nadya_ has quit IRC21:43
clarkbjesusaur: the current puppet-elasticsearch module can't express that, I was just going to do it manually21:44
jesusaurclarkb: right, it can't be done through the module, but we could replace the class resource with a service and package resource, then remove those in a follow-up patch21:45
*** sarob has joined #openstack-infra21:45
*** annegentle has quit IRC21:45
clarkbnah I will just clear it out manually, but can make the puppet richer if people prefer that21:45
*** annegentle has joined #openstack-infra21:46
jesusaurnah, just wanna make sure that step doesn't get missed21:46
anteayaI have no preference21:46
anteayaas long as we soon get to be able to discuss steps to impliment kibana 421:46
anteayaand as long as anyone else should they have to touch a logstashworker, knows what to do21:47
*** devananda has joined #openstack-infra21:48
mtreinishanteaya: why the big push for kibana 4? what's so much better about it?21:48
* mtreinish hasn't even looked at it or the differences21:48
*** kencjohnston has quit IRC21:48
fungii've recently heard it referred to as "munroe's law" (akin to godwin's law) that once the discussion has been summarized by an xkcd link there's no point in continuing it21:48
*** nwkarsten has joined #openstack-infra21:49
*** aysyd has quit IRC21:50
anteayafungi: ha ha ha21:50
anteayamtreinish: it is our end state for front ends21:50
*** gnuoy_ has quit IRC21:50
SamYapleits crazy how many laws there are that are similiar fungi. we should come up with one law that is a standard21:50
anteayamtreinish: I took an an action item at the last summit, to bring on more folks able to debug the gate21:51
*** tongli has quit IRC21:51
anteayamtreinish: then we had to address logstash/kibana upgrades21:51
anteayamtreinish: the kibana interface for 3 was much different than 221:51
anteayaand I'm told, the kibana 4 interface is much different again21:51
SamYapleanteaya: I would be happy help but the learning curve is super high...21:51
mtreinishanteaya: well it's the end state until they release kibana 5 :)21:51
mtreinishanteaya: yeah, they seem to like to change things a lot between releases21:52
mtreinish(and IMO not always for the better)21:52
anteayaso I didn't see the point to try to teach folks kibana 321:52
anteayaSamYaple: yes the learning curve is high, even when the interfaces are not changing out from under you21:52
anteayamtreinish: points on both21:52
anteayaso I made a decision21:52
anteayaand didn't communicate it widely21:52
anteayaprefering to focus my energy on doing what I could to support achieving the end state21:53
anteayahence my actions21:53
SamYaplewho would have thought deploying openstack would be simplier than debugging the gate?21:53
*** nwkarsten has quit IRC21:53
anteayaso anytime there was an opportunity to support the move to logstash 2.0 to get to kibana 4, I tried to help21:54
anteayaand i hope my actions are construed as helping21:54
*** gnuoy_ has joined #openstack-infra21:54
anteayaSamYaple: anyone who hasn't tried to debug the gate21:54
fungiSamYaple: to debug gate jobs you basically become an openstack operator at a very small it shop where you're the only one around to support the entire cloud and it's suddenly suffered a catastrophic cascade failure21:55
*** sc68cal has quit IRC21:56
*** thorst has joined #openstack-infra21:56
*** tsg has joined #openstack-infra21:56
anteayamtreinish: so that's my reason21:56
*** tpsilva has quit IRC21:57
anteayanow I have to go back to the cross project session this summit and hang my head in shame as I did not do what I said I would do this release21:57
*** rguillebert has quit IRC21:57
anteayabut I do hold the hope we can get to kibana 4 by austin so at least I can offer a silver lining21:57
SamYaplefungi: i was an openstack operator and i run a cloud with public ips at my house.... but no one dpends on me....21:58
anteayabut I didn't tell the whole story before because clark has had enough on his plate this release21:58
anteayaso decided to just offer to help21:58
fungianteaya: well, the up side is that most of the answers to "kibana 3 is terrible" were "wait for kibana 4, we've totally solved that"21:58
mtreinishanteaya: ok, so there isn't a killer feature or any known improvement. It's just to make sure we have the latest21:58
anteayafungi: exactly21:58
mtreinishfungi: well I'm sure they said the same things about 2 vs 321:58
anteayaI'm hoping for that21:58
anteayaclarkb says it is better than 321:58
anteayaI am trusting clarkb21:59
fungiand indeed, it was probably true. what they never tell you is what other features you depended on that they decided to completely break21:59
mtreinishfungi: like permalink :)21:59
*** e0ne has joined #openstack-infra21:59
*** scottda is now known as scottda_on_vacat21:59
fungithat's the main one i had in mind when i said that21:59
fungiso painful clarkb basically reimplemented it in javascript22:00
*** scottda_on_vacat is now known as scottda_vacation22:00
*** _ari_ has quit IRC22:00
fungior somebody did (that was clarkb right?)22:00
clarkbwell its "better"22:00
*** rfolco has quit IRC22:01
clarkbI ran it locally last night and it needs some work22:01
clarkbthey expose all the config knobs to everyone22:01
clarkband have big scary things like "lol this can break things"22:01
clarkbthanks kibana for letting everyone break things22:01
*** ajmiller has joined #openstack-infra22:02
*** ajmiller_ has quit IRC22:02
*** jaypipes has quit IRC22:02
fungii bet it's a really cool feature, as long as you don't ever let anyone use it22:02
clarkbits also a lot more visually noisy22:03
*** dmsimard has joined #openstack-infra22:03
*** e0ne has quit IRC22:04
clarkbon the list of things then for someone (maybe me) to figure out is how to configure or otherwise hobble kibana 4 to prevent that access22:06
clarkbit does still use elasticsearch itself to save the data for links and stuff in a .kibana index but we would give the appliation access to that not the user or their browser so that is fine22:07
anteayado we have a not noisy option?22:08
anteaya2 was friendly in that it was very quiet22:08
clarkbanteaya: no thats part of the new and improved kibana22:08
clarkbyou can save visualizations and dashboards22:08
clarkband share those22:08
*** lascii is now known as alaski22:08
anteayaall I want to be able to do is compose a query22:09
anteayashare a query22:09
clarkbyup thats still there too22:09
anteayaand teach others to do those two actions22:09
*** GnomeKin1 has joined #openstack-infra22:09
anteayaokay thanks22:09
clarkb is what needs to be hobbled22:10
anteayaoh my22:10
anteayayes, can you just remove the html that makes that available?22:10
anteayahtml/javascript/language of choice22:10
*** GnomeKing has quit IRC22:12
clarkbor set up a proxy that null routes that22:14
clarkbor maybe there is an on disk config to update22:14
*** GnomeKin1 has quit IRC22:14
*** GnomeKing has joined #openstack-infra22:14
anteayawe can hope?22:14
*** GnomeKing is now known as Guest8280722:14
*** amitgandhinz has quit IRC22:15
*** dmsimard is now known as dmsimard|pto22:15
*** melwitt has quit IRC22:18
*** melwitt has joined #openstack-infra22:18
anteayaI am on day three of working on this expense report22:21
clarkbya, i am sure there is a way even if it is making the proxy smarter than it should be22:21
clarkbbut the logstsah upgrade itself looks good22:21
anteayaI bet I spend 30% of my time logging back in after it kicks me out22:21
*** regXboi has quit IRC22:21
clarkbit is keeping up with demand better than old version likely due to the extra available memory22:21
anteayaclarkb: yay logstash 2.022:21
anteayaclarkb: ohhh, good to hear22:21
*** krtaylor has joined #openstack-infra22:22
sc`i'm attempting to get a centos7 gate working, but the build fails because ifconfig isn't there, despite net-tools being installed22:25
clarkbsc`: probably a path issue22:26
clarkbifconfig is in /sbin/ I htink? and that isn't always in default paths22:26
sc`looks like for c7 it lives in /usr/sbin22:26
clarkbyup just checked on a devstack-centos7 machine22:27
clarkbI would make sure that is in your path22:28
sc`yeah. trying with that in my path22:28
*** twm2016_ has quit IRC22:29
*** gnuoy_ has quit IRC22:30
*** Sukhdev has joined #openstack-infra22:30
*** Slower has joined #openstack-infra22:31
*** esker has joined #openstack-infra22:32
*** asettle has joined #openstack-infra22:33
*** krtaylor has quit IRC22:35
openstackgerritMerged openstack-infra/project-config: Report number of changed jobs
*** melwitt has quit IRC22:36
*** melwitt has joined #openstack-infra22:36
*** krtaylor has joined #openstack-infra22:37
clarkbinfra-root care to ack then I can clear out the ES installs on the logstash workers?22:37
*** andymaier has joined #openstack-infra22:37
*** david-lyle has quit IRC22:37
clarkblast step in logstash 2.0 upgrade22:37
pleia2having a look22:37
nibalizerclarkb: lgtm22:37
*** andymaier has quit IRC22:37
*** jtomasek has quit IRC22:38
*** asettle has quit IRC22:38
pleia2clarkb: there you go22:39
clarkbthank you!22:39
*** krtaylor has quit IRC22:42
*** krtaylor has joined #openstack-infra22:43
*** jlk has quit IRC22:43
clarkbmordred: so I have discovered that seems to need a complete rewrite now?22:43
clarkbthis diff is not very lcean22:43
mordredclarkb: oh. piddle22:44
mordredclarkb: I thought that had landed22:44
mordredclarkb: I'll port your wor22:44
clarkbmordred: ok thanks, I think the actually tricky stuff is in the shell scripts and those don't conflict22:44
clarkbits just in the logic/workflow that needs merging22:44
mordredthe patch in launch-node itself all makes immediate sense to me22:44
openstackgerritMerged openstack-infra/system-config: Don't install ES on logstash workers
*** dimtruck is now known as zz_dimtruck22:46
*** rcernin has joined #openstack-infra22:47
sc`looks like i'm unable to override PATH?22:47
sc`never mind. fat finger22:47
jeblairianw: see comments in 29191922:48
*** jlk has joined #openstack-infra22:48
*** jlk has quit IRC22:48
*** jlk has joined #openstack-infra22:48
*** Slower has quit IRC22:48
clarkbmordred: oh is build server not ansibling? I could probably do this if you want me to22:48
clarkbmordred: its just the file copying that is ansibling now?22:49
*** Sukhdev has quit IRC22:49
*** alex_xu has quit IRC22:49
clarkbhrm does git appy have a -e flag? this will be awesome if so22:49
mordredclarkb: build server is not yet ansibling - ricky actually has some work I want to build on - but I don't think we need to ansible it yet22:49
*** tsg has quit IRC22:49
mordredclarkb: right now it's just ansibling the initial puppet run22:49
clarkbno git apply -e :(22:50
*** alex_xu has joined #openstack-infra22:51
sc`PATH appears to be getting set, but not making it to the point when 'env' is run22:51
clarkbsc`: how are you setting it? is it on the other side of a sudo?22:52
*** dmsimard|pto has quit IRC22:52
*** claudiub has quit IRC22:52
sc`before sudo. might have to put it in one of the sudo commands i'm thinking22:53
clarkbyes sudo wipesout env22:53
*** ihrachys has quit IRC22:53
*** IlyaG has joined #openstack-infra22:53
SamYapleunless using -E22:54
clarkbthe first easy check would be to use a full path to ifconfig22:54
clarkbif that works then figure out PATH22:54
sc`i'm not directly calling ifconfig22:55
sc`"something" is22:56
*** EricGonczer_ has joined #openstack-infra22:56
sc`ah. never mind. reverse engineering fun times. i think i see it22:57
*** sarob has quit IRC22:57
*** Slower has joined #openstack-infra22:57
*** sarob has joined #openstack-infra22:58
*** EricGonc_ has quit IRC22:59
*** bauwser is now known as bauzas23:00
*** xyang1 has quit IRC23:01
*** jlk has quit IRC23:01
*** asettle has joined #openstack-infra23:01
*** jlk has joined #openstack-infra23:01
*** jlk has joined #openstack-infra23:01
*** bpokorny_ has joined #openstack-infra23:02
openstackgerritJames E. Blair proposed openstack-infra/project-config: Pass tags through to jobs
openstackgerritJames E. Blair proposed openstack-infra/project-config: Add neutron tags to some jobs
jeblairmtreinish: our running zuul supports that now ^23:03
jeblairmtreinish: (and tags can be a list; i just added that neutron tag as an example more or less)23:04
*** bpokorny_ has quit IRC23:04
*** amotoki has quit IRC23:04
*** bpokorny_ has joined #openstack-infra23:04
anteayaI'd like to see them in action23:05
anteayathanks jeblair23:05
clarkblooking at puppet runs and puppetboard we could stand to do some cleanup23:05
jeblairanteaya: yeah, nothing uses them yet, but we can merge that and see the output on the jenkins params page for those jobs23:05
*** bpokorny has quit IRC23:05
anteayaclarkb: anything a non-root can help with?23:05
clarkbthe old pypi mirrors all fail and appear to still be in puppetboard23:05
clarkbanteaya: mostl yI just need to understand how to clear it up23:06
anteayajeblair: yeah, I figured it wouldn't hurt anything to merge them and see where they take us23:06
clarkbI think we need a puppet cert delete and maybe a cache removal for ansible23:06
jeblairclarkb: if i had a veto on deleting those, i hereby lift it if i hadn't already.  :)23:06
anteayaclarkb: ah, do say if I can help at all23:06
mnaserclarkb pabelanger ipv6 issues should be resolved by now.23:06
mnaserso the git checkout timeouts should be improved and favor ipv6 too23:06
anteayamnaser: yay!23:06
*** rcernin has quit IRC23:06
jeblairmnaser: ipv6!23:06
mnaserdual stack by default for all machines23:07
clarkbwhen I am not cleaning up after ES I will try to look into that more23:07
anteayamnaser: nice work23:07
clarkb(I am watching puppet run before cleaning up es)23:07
anteayaclarkb: yup23:07
jeblairi'll look into the pypi stuff23:07
mtreinishjeblair: very cool, I'll push a patch up shortly to expand on that23:07
anteayahappy to help if I can figure out what to do (and have permissions)23:07
mnaserclarkb: looking at the addressing, can now resolve to 2604:e100:1:0:f816:3eff:feb1:40bd and it'll #justwork23:08
*** rcernin has joined #openstack-infra23:08
clarkbmnaser: neat we will have to add AAAA records23:08
mnaseri dont have ipv6 at home but according to online pings it work23:08
clarkbmnaser: are you slaacing?23:09
mnaserwe're doing a few more tweaks because one of our upstreams is involved in a little peering drama23:09
mnaseryes clarkb23:09
clarkbWondering if we need to do anyhting on our side to pick that up23:09
mnasernothing at all23:09
mnaserall your instances are all pingable by ipv6 now automatically23:09
mnaserand also they have ipv6 listed in nova show / api23:09
*** asettle has quit IRC23:09
openstackgerritRicardo Carrillo Cruz proposed openstack-infra/shade: Make delete_project to call get_project
mnaserif any network issues happen again, please let me know, however, there shouldn't be23:12
jeblairclarkb: we deleted the pypi servers....23:12
jeblairwhere is the ansible cache?23:13
clarkblaunch node has a path to it iirc23:13
jeblair/var/cache/ansible-inventory/ansible-inventory.cache ?23:13
anteayamnaser: thanks for being so available23:13
mnaseranteaya: no problem, it's my pleasure!23:13
anteayamnaser: makes it so lovely and easy23:13
jeblairclarkb: no pypis in there23:14
mnaseri make an effort!23:14
clarkbhuh why do they show up in puppet_run_all.log23:14
mnaseranyways, time for a little break.  take care folks23:14
anteayamnaser: you do so23:15
anteayamnaser: have a good weekend23:15
anteayathanks again23:15
jeblairthey are in /etc/ansible/hostfile and hosts/generated-groups23:15
*** daemontool has joined #openstack-infra23:16
*** akshai has quit IRC23:16
jeblairand they show up in ansible 'pypi*' --list-hosts23:16
jeblairwhich is how those get generated23:16
clarkband list-hosts is generating those via the inventory script?23:17
jeblairclarkb: the generated groups are via a script that runs ansible --list-hosts; my understanding is that should be consulting the cache, but clearly my understanding is incomplete23:18
jeblairmordred: and similar hosts appear in "ansible 'pypi*' --list-hosts" but do not appear in /var/cache/ansible-inventory/ansible-inventory.cache.  what am i missing?23:18
openstackgerritMatthew Treinish proposed openstack-infra/project-config: Add periodic-qa tag to periodic qa jobs
mtreinishjeblair: ^^^23:20
*** gyee has quit IRC23:21
jeblairclarkb, mordred: i think i figured it out23:21
jeblairclarkb, mordred: it's slightly circular...23:21
clarkbelasticsearch is removed on all logstash workers23:21
pleia2clarkb: hooray23:21
jeblairclarkb, mordred: /etc/ansible/hosts/generated-groups appears to _also_ be input to ansible --list-hosts23:21
clarkbjeblair: nice!23:21
jeblairclarkb, mordred: so once a host exists and gets put in that file, it will be in the inventory forever23:22
*** sigmavirus24 is now known as sigmavirus24_awa23:22
clarkbwe should probably make the only input the inventory cache?23:23
clarkbor rathe rthe only "these are the hosts that exist" input23:23
openstackgerritJames E. Blair proposed openstack-infra/system-config: Remove pypi ansible group
jeblairclarkb, mordred: in this specific case, of removing all matching hosts, this will correct the problem, but we should still solve for the case of deleting a single host that's part of a group ^23:24
anteayaclarkb: yay23:24
anteayamtreinish: any way we could regex that up a bit?23:25
mtreinishjeblair: once we have some examples in the subunit2sql db with tags I think it's probably worth a ML post about tags and periodic jobs23:25
anteayamtreinish: 29194323:25
mtreinishanteaya: I didn't see an easy way23:25
openstackgerritJames E. Blair proposed openstack-infra/system-config: Remove pypi hosts from cacti
*** rguillebert has joined #openstack-infra23:25
mtreinishbecause there are other jobs that fit the basic pattern23:25
mtreinishanteaya: like: periodic-tempest-dsvm-oslo-latest-full-master23:26
mtreinishwhich isn't in the set I wanted to tag23:26
jeblairmtreinish: ++ on the change and the ml post23:26
jeblairanteaya: to be fair, my neutron tags may be too loose as well.23:27
anteayaso we'll make them work and hopefully tighten them up a bit when we can?23:27
jeblairthey'll look a lot nicer in zuulv3 too.23:28
anteayaoh good23:28
anteayathat makes me feel better, thanks23:28
SamYapleeta on zuulv3?23:28
SamYaplewaiting for that sweet sweet reconnect feature23:28
anteayaI'm +2 on all of them, but won't +A mtreinish's since the others haven't been approved yet23:29
anteayaSamYaple: that is entirely based on patches and reviews done on patches23:29
anteayaso reviewing and writing patches will speed things along23:29
anteayaif you leave it all up to jeblair it is as fast as he is able to create space to write code23:30
SamYaplelast time this rolled around i did ask if tehre was a spec/blueprint/wrok items where i could pick up some slack on23:30
*** mgkwill has quit IRC23:30
*** claudiub|2 has joined #openstack-infra23:30
SamYapleso if someone links me to that I am sure i can help there23:30
*** eil397 has quit IRC23:30
*** rcernin has quit IRC23:31
*** eil397 has joined #openstack-infra23:31
*** annegentle has quit IRC23:32
anteayaSamYaple: this would be a good place to begin:
*** annegentle has joined #openstack-infra23:32
SamYaplethanks anteaya will review23:32
anteayaand if you would like to review:
anteayathank you23:33
*** flepied has joined #openstack-infra23:34
*** annegentle has quit IRC23:37
*** bryan_att has quit IRC23:37
* rcarrillocruz waves as his latest change has voting tests in green23:37
rcarrillocruzhave a nice weekend all23:37
anteayarcarrillocruz: thanks, you too23:39
anteayacongratulations for all green on your change23:39
anteayahave a nice weekend23:39
sc`i keep running into permission denied bits. sudo wrapping the command gets me past it, but i shouldn't be using sudo for everything23:41
*** notnownikki has quit IRC23:41
*** daemontool_ has joined #openstack-infra23:41
jeblairSamYaple: nodepool-shade is in the critical path for zuulv3, so thanks for reviewing that23:42
*** thorst is now known as thorst_afk23:42
jeblairSamYaple: we're going to branch nodepool after it lands23:42
*** asettle has joined #openstack-infra23:42
*** daemontool has quit IRC23:43
*** mgkwill has joined #openstack-infra23:45
*** annegentle has joined #openstack-infra23:46
SamYaplejeblair: okiedokie. ill try to keep up with the reviews since it sounds like that may be where im of most help23:47
*** erlon has quit IRC23:48
*** hdd has quit IRC23:49
*** asettle has quit IRC23:49
*** annegentle has quit IRC23:54
*** roxanagh_ has quit IRC23:54
*** annegentle has joined #openstack-infra23:54
docaedoanyone feel like taking a few minutes to help debug an app-catalog issue? Would require either ssh'ing to the box, or critiquing a fun pipe-chain shell script :)23:54
*** nwkarsten has joined #openstack-infra23:55
docaedo(but I figure most everyone is weekending by now, so can pick this up Monday)23:55
*** roxanagh_ has joined #openstack-infra23:56
openstackgerritSimon McCartney proposed openstack-infra/infra-ansible: Reworked to support Debian & Darwin/OSX
sc`it seems as though /home/jenkins/workspace/stuff isn't writeable by the jenkins user...23:57
*** rbrndt__ has quit IRC23:57
mordredSamYaple, jeblair: why nodepool-shade fail???23:57
sc`seems that the git clone is happening as root. hm... have to see where/why/how23:58
*** annegentle has quit IRC23:59
anteayaas a follow in to last night's conversation, some may enjoy this:
sc`may as well just use sudo in this case. no sense in debugging permissions and other sundry things23:59
anteaya*follow on23:59

Generated by 2.14.0 by Marius Gedminas - find it at!