Thursday, 2016-03-10

*** openstack has joined #openstack-infra00:09
*** asettle has quit IRC00:09
clarkbI Think it cleans up after breaking so logs for fail will be before cleanup00:10
cody-somervillejeblair: figured it out. Good catch. Fixed uploaded ^^00:11
fungiyeah, i end up scrolling up a few dozen lines to look for errors about missing files or whatnot00:11
craigeOMG, we have comments mordred!00:11
cody-somervillejeblair: FYI, looks like --open argument in Python3 is broken00:11
*** hdd_ has joined #openstack-infra00:11
*** hdd has quit IRC00:12
*** asettle has joined #openstack-infra00:12
pleia2bot rescued successfully00:12
craigeI'll need to updated the etherpad but as far the actual data migration to Phabricator is concerned, all blocking issues are resolved fungi.00:13
anteayapleia2: nice work00:13
craigeso I'd love more people to hit it and find things wrong (preferrably right) as I stand up
anteayapleia2: I told meeting-4 the bot is back00:14
craigeCurrent dev instance is
*** davideag_ has quit IRC00:14
anteayapleia2: but the meeting in there was already over00:14
SamYaplethis is the closest ERROR i have
*** davideagnello has joined #openstack-infra00:14
SamYaplehave you seen it before and can you save me troubleshooting time?00:14
SamYaple(ubuntu1 is this hose)00:14
pleia2anteaya: ah yeah, I didn't bother with -3 or -4 since there wasn't anything going on right now00:14
anteayapleia2: yup00:15
*** placko_ has quit IRC00:16
*** Qiming has quit IRC00:16
*** placko_ has joined #openstack-infra00:17
*** openstackgerrit has quit IRC00:17
*** dimtruck is now known as zz_dimtruck00:17
fungicraige: awesome! now we need people to poke the demo more aggressively and report back on whether there are missing features we need before we can consider a go-live cutover00:17
fungittx: ^ (when you're not busy travelling)00:18
mordredcraige: WOOT!!!!00:18
*** openstackgerrit has joined #openstack-infra00:18
*** mescanef has joined #openstack-infra00:18
*** tonytan4ever has joined #openstack-infra00:18
SamYaplemordred: you see this dib failure?
*** flepied has quit IRC00:22
craigeMuch w00t mordred much w00tness.00:23
craigeYes, that's what i'm hoping fungi.00:23
craigeI'm also trying to now stand up to have a more accurate instance for poking. However people should absolutely hammer my dev instance in the interim.00:24
*** nwkarsten has quit IRC00:24
craigemtreinish, yolanda ^^00:24
*** nwkarsten has joined #openstack-infra00:25
dimsfungi : clarkb : anyone seen weirdness in just simple pep8 jobs? apt-get times out after 30 mins -
*** bpokorny has quit IRC00:28
fungidims: pabelanger also noted some very poor i/o performance in vexxhost on another job00:28
ianwfungi: i *really* learnt C thanks to an internship at SGI where i had not too much to do, and plenty of time to play with irix's often hilariously lax security with the benefit of source code.  IndyCam's were freaking awesome00:29
*** bpokorny has joined #openstack-infra00:29
fungiianw: far better than my experience learning korn shell by trying to fix sco openserver!00:29
craigeoh, ew.00:29
*** yamahata has joined #openstack-infra00:30
dimsfungi : ack thanks00:30
*** nwkarsten has quit IRC00:30
* ianw now knows fungi is good for a korn shell port of devstack00:30
fungidims: apt/dpkg are pretty hard on disk i/o, so while i think this is likely a deeper performance issue there are probably also some simple things we can do to make that a lot bit faster00:30
*** xarses has quit IRC00:31
*** asettle has quit IRC00:31
fungiianw: i'd sooner port it to almquist00:32
*** asettle has joined #openstack-infra00:32
fungi(which granted would not be hard at all)00:32
fungiooh, i shoulda said "plan 9 rc"00:34
*** flepied has joined #openstack-infra00:34
fungithat would be sorta fun00:34
*** dims has quit IRC00:35
fungialso, on the topic of operating systems which went missing in action, whatever happened to beos?00:36
openstackgerritCraige McWhirter proposed openstack-infra/system-config: Added RemoteUser Provider & OpenID configs
fungioh, right, haiku still exists as a descendant00:37
pleia2yeah, haiku is still active00:37
craigebeos and OS/2 are having a little tea party together.00:38
* fungi warps to the sofa00:38
craigeWarps 4 the sofa?00:38
pleia2about 10 years ago I would attend a user group that started with os/2 and morphed into a general systems meetup00:38
pleia2but their membership didn't, I was always the youngest, by far (and the only woman)00:39
craigesounds like many LUGs.00:39
pleia2it was fun though :)00:39
fungiin the places i lived, the bugs were always more fun than the lugs00:39
craigeI loved OS/2, until Linux came out.00:39
fungicraige: which granted was 1992 if you were spending any time downloading source from usenet00:40
SamYaplemmmm i still have a PS/2 sitting i a closet somewhere00:40
craigeNo, I linux didn't come out for me until 1994, when I bought an infomagic double CD in Chinatwon, the only place in Sydney to get it.00:41
fungiooh, infomagic. that brings back memories00:41
craigeI'd never even used UNIX before, went home, replaced Os/2 with Linux and start there at the blinking slackware darkstar # prompt with no idea how to do ANYTHING.00:42
* craige then went and bought a book...00:42
SamYaplei few years back i had this customer how was using a ps/2 running os/2 for all of thier vulcanization equipment. been running there for like 30 years then it broke00:42
pleia2SamYaple: sounds like a very bad day00:42
SamYapleyou would think so but no00:42
SamYaplebattery had died. went down to radio shack and bought a replacement. cause you know, they had it on the shelf00:43
SamYaplethats what was most shocking00:43
SamYaplethe battery, not a ps/200:43
pleia2gah, so they didn't stop using it?00:43
*** denisra_ has joined #openstack-infra00:43
pleia2bad day postponed :)00:43
* fungi remembers when you could buy basic circuit components at radio shack. also parts for radios00:43
SamYaplethey would have had to spend millions to replace that00:43
*** BobBall_ has joined #openstack-infra00:43
*** asselin_ has joined #openstack-infra00:44
*** bpokorny has quit IRC00:44
*** BobBall_ is now known as Guest310200:44
SamYaplei did convince them it was cheaper to buy all the ps/2 stuff on ebay they could find00:44
Clintand forrest mims books00:44
SamYaplethat included a 51/2" floppy drive00:44
SamYaplethey spent a few thouhsand and have all the spare parts you could want00:44
*** bpokorny has joined #openstack-infra00:44
SamYaplei think it had 4MB of ram00:44
*** Volundr has joined #openstack-infra00:45
jokke_fungi: yeah, now ye get them from ebay ;)00:46
fungii went into a radio shack recently, it was like an outpost for best buy. cell phones and flat screens. want a 20-pack of 10kohm .25w 5% resistors? sorry, what decade are you from?00:46
*** isq_ has joined #openstack-infra00:46
*** mwhahaha_ has joined #openstack-infra00:46
*** cbader_ has joined #openstack-infra00:46
fungii'm a digikey junkie these days00:46
*** kuromagi^ has joined #openstack-infra00:46
SamYapleive got a store near me with all that stuff if i dont want to wait for china00:46
*** evgenyl_ has joined #openstack-infra00:47
bkerofungi: Oh, you're one of *rolls eyes* THOSE customers. The parts bin is in the back. Don't ask us about it.00:47
*** aysyd has quit IRC00:47
*** prometheanfire has quit IRC00:47
*** kukacz_ has joined #openstack-infra00:47
*** Nakato_ has joined #openstack-infra00:48
bkeroI still have/use the original IBM Model M from my first computer, an IBM 386.00:48
fungii just had such fond memories of the "old" radio shack. it's sad to be reminded of what they've become00:48
bkeroI think it was a 386DX. Fancy fancy.00:48
*** sheeprine_ has joined #openstack-infra00:48
bkerofungi: there were some efforts to revitalize it with 'hardware for a newer generation'. Arduinos and shields.00:49
jokke_fungi: icbm controller accurate enough for hobbyist use is so much cheaper to build nowadays :P00:49
*** ruhe has quit IRC00:49
*** mwhahaha has quit IRC00:49
*** ptoohill has quit IRC00:49
*** eglute has quit IRC00:49
*** evgenyl has quit IRC00:49
*** mkoderer__ has quit IRC00:49
*** denisra has quit IRC00:49
*** aeng has quit IRC00:49
*** harlowja_at_home has quit IRC00:49
*** zzzeek has quit IRC00:49
*** kushal has quit IRC00:49
*** cbader has quit IRC00:49
*** gmann has quit IRC00:49
*** Kiall has quit IRC00:49
*** BobBall has quit IRC00:49
*** whoops has quit IRC00:49
*** jbernard_ has quit IRC00:49
*** sheeprine has quit IRC00:49
*** sbadia has quit IRC00:49
*** asselin__ has quit IRC00:49
*** pcrews has quit IRC00:49
*** kuromagi has quit IRC00:49
*** isq has quit IRC00:49
*** kukacz has quit IRC00:49
*** madorn has quit IRC00:49
*** verdurin has quit IRC00:49
*** Nakato has quit IRC00:49
*** ruhe2 has joined #openstack-infra00:49
*** tqtran is now known as tqtran-afk00:49
*** Kiall has joined #openstack-infra00:49
*** ruhe2 is now known as ruhe00:49
*** madorn_ has joined #openstack-infra00:49
*** eglute_s has joined #openstack-infra00:49
*** evgenyl_ is now known as evgenyl00:49
*** mwhahaha_ is now known as mwhahaha00:49
*** gmann has joined #openstack-infra00:49
*** ptoohill has joined #openstack-infra00:49
*** mkoderer___ has joined #openstack-infra00:49
*** aeng has joined #openstack-infra00:49
*** sbadia has joined #openstack-infra00:50
*** harlowja_at_home has joined #openstack-infra00:50
jokke_bkero: those are great, but try to find store selling them (or any components) without ridiculous prices00:50
*** kushal has joined #openstack-infra00:50
bkerojokke_: Radio Shcak00:50
bkeroErr, Radio Shack*00:50
*** jbernard_ has joined #openstack-infra00:50
*** zzzeek has joined #openstack-infra00:50
*** verdurin has joined #openstack-infra00:50
*** prometheanfire has joined #openstack-infra00:50
*** pcrews has joined #openstack-infra00:50
fungipatronize as much as you can. they're awesome00:51
*** cody-somerville has quit IRC00:51
jokke_bkero: so they actually stock components still as well, got bit different impression from fungi above ;)00:51
jokke_fungi: them or sparkfun.com00:51
jokke_sparkfun stocks bit weirder stuff as well00:52
bkeroI usually sparkfun, adafruit, or seeed00:52
fungijokke_: they do. there's usually a corner in the back of the store littered with empty soda cups and old napkins with some inaccurately-marked bins of parts they rarely get around to restocking00:52
fungiyeah, the sparkfun crowd are great too. frencil is a hoot00:52
fungii've not checked out seeed but doing now00:53
Clintisn't that a german reggae band00:54
bkeroThere's a really cool PCB marketplace startup in Portland. I'm trying to remember the name of it.00:54
fungiClint: you're thinking of ß"eed00:54
* Clint twitches.00:55
SamYaplebkero: dvdshack?00:55
bkeroheh no00:55
jokke_we have store chain called maplin here. Like I tried to get some basic components from them few months back. Can't remember exactly what, but it was something you would expect any hobbyist needing. Their individual store saldo was 2 ... an not like "We've sold everything out, get more next week" But their restock was actually 2 ... and component out sale price was like .30EUR00:55
bkero(not Tinder)00:55
bkero just scroll down00:55
fungitheir motto totally should have been "just scroll down"00:56
SamYaplefungi: i think "swipe up" would work better for "tindie"00:58
clarkbdmsimard: that job failed and I wonder if it is because we ran the setup twice for grenade00:59
*** Jeffrey4l has joined #openstack-infra00:59
SamYaplealright i can do this no more. i have no idea why dib is breaking00:59
clarkbSamYaple: sorry just getting back to computer and can look at some pastes00:59
clarkbbuying a house is literally the worst thing ever01:00
clarkbhighly recommend doing it before you have twins01:00
*** asettle has quit IRC01:00
*** rhallisey has quit IRC01:00
jokke_I grew up about 15miles out of small city (<90k people) and we had 3 electronics stores in that city ... you got project idea at Fri, made BOM and picked up Saturday morning (normally had to go max to the second store to get all you wanted) you were done Sunday. Now you get project idea at fri evening, you order the parts, sit on your thumb for the weekend and maybe get the parts by Tue or Wed01:00
SamYapleclarkb: that one just finished no idea what to make of it01:00
SamYapleclarkb: i just bought mine (and sold one out of state)01:00
SamYaplesome places its fine, others it is not01:01
SamYaplein TX it was all online, here in SC it was all on paper :/01:01
clarkbin portland its just the crazyness of the market making everything crazy01:02
SamYaplejokke_: get your ideas on mon or tue. problem solved!01:02
clarkbFound house we liked then they drop the bomb on you that it burned down 3 years ago01:02
clarkbya nope01:03
pleia2clarkb: wow :(01:03
SamYapleyea and there are some new 2016 laws on the books making actually buying it even harder01:03
*** tonytan4ever has quit IRC01:03
SamYaplethe lawyers have more to do now which slows things down01:03
clarkbyes apparently closing times have increased ~33%01:03
*** markus_z has quit IRC01:04
*** Jeffrey4l has quit IRC01:04
fungiSamYaple: where in sc? i'm practically next door (nc)01:04
SamYaplewell the federal delay only adds 3 days, but the new laws mean people havent streamlined the process01:04
clarkbSamYaple: that log looks like puppet is breaking for some reason01:04
SamYaplefungi: greenville01:04
SamYapleclarkb: youve gotten as far as me!01:04
SamYaplefungi: i go to ashville and charlotte a bit too01:05
clarkbbut good news is we found something else that we ust put an offer on so we shall see how that goes01:05
jokke_fungi: where in nc are you?01:05
fungioof. you're practically next door to my parents in hendersonville nc, but i'm at the opposite end of the state any more01:05
fungijokke_: kill devil hills, out on the barrier islands01:05
greghaynescan confirm, PDX house buying is the worst01:06
jokke_fungi: uuuh ... looks nice01:06
jokke_fungi: although not exactly coffee meet distance for next week :P I'm landing RDU Sun night and leaving Fri morning01:07
*** tonytan4ever has joined #openstack-infra01:07
SamYapleclarkb: any ideas on the puppet breakage?01:07
fungijokke_: it is nice. also slow to get to/from. closest "major" airport (and i use the term loosely) is 1.5 hours by car. rdu is 3 hours01:07
*** rguillebert has quit IRC01:07
*** markusz has joined #openstack-infra01:08
fungii lived in the raleigh area for 15 years. got tired of it01:08
jokke_fungi: bit like Galway, Ireland (where I'm at) closest international is about an hour away and Dublin Airport just over 2hrs ... Over here needs effort though to manage to spend 3 hours driving from point to point :P01:09
*** angdraug has quit IRC01:09
fungiwe desperately need to have a summit in the dublin area01:10
jokke_fungi: so anything special I definitely should visit there? (You know any good breweries or coffee roasteries around Raleigh?)01:10
jokke_fungi: that would be handy01:10
openstackgerritMerged openstack-infra/puppet-askbot: Add captcha to the askbot venv
*** sridhar_ram has quit IRC01:11
*** Daisy has joined #openstack-infra01:11
fungijokke_: oh, sure, there are plenty. "cup-a-joe" is the local coffee chain, they do their own roasting. also lots of brewery-pub establishments now. are you staying downtown or nearer the airport?01:11
fungiif you're in downtown raleigh there are a few easy walking distance (i'll omit the "irish" pubs)01:12
jokke_downtown ... sheraton01:13
fungiyeah, i should be able to give you some recommendations... just a sec01:13
fungimy favorite is aviator, but they're not downtown. you're still likely to find a lot of their stuff on tap at various restaurants there01:15
*** yonglihe has joined #openstack-infra01:15
*** thorst_ has quit IRC01:15
*** Daisy has quit IRC01:16
jokke_yeah, time is kind of a problem. I'm expecting to sit majority in the office (not exactly a holiday trip)01:17
fungii see there's a "boylan bridge" brewpub, which is likely an easy walk, though that's a new one on me01:17
fungiclarkb should go check that out if he ever gets to raleigh ;)01:17
*** ddieterly has joined #openstack-infra01:17
*** thorst_ has joined #openstack-infra01:17
fungicrank arm brewery isn't too bad01:17
clarkbwait what01:18
*** dalgaaf has quit IRC01:18
jokke_fungi: cheers, appreciated.01:19
fungiraleigh brewing on neil st is right off hillsborough (the main drag through downtown next to the main ncsu campus)01:20
*** Sukhdev has joined #openstack-infra01:20
fungitrophy brewing is also right around there01:20
fungisub noir and big boss breweries aren't right downtown but probably still walkable in a reasonable timeframe01:21
*** Sukhdev has quit IRC01:22
fungithere are a bunch more up on the north side of town where i used to live, but there's not much point in me recommending them01:22
*** cody-somerville has joined #openstack-infra01:22
*** nikhil_k has quit IRC01:22
jokke_that sounds already like a list that I will need to use taxis from half way of the beer night and hope that they have wheelbarrow at the hotel to deliver me into my room :P01:23
*** Odd_Bloke has quit IRC01:23
*** Odd_Bloke has joined #openstack-infra01:23
fungiheh, they might01:23
*** lezbar__ has quit IRC01:24
*** thorst_ has quit IRC01:24
*** Sukhdev has joined #openstack-infra01:24
fungithere's also lots of great restaurants and bars right around where the downtown sheraton is. you shouldn't have any trouble finding something to do with your evenings01:24
*** mriedem has joined #openstack-infra01:25
*** pvaneck has quit IRC01:25
*** gyee has quit IRC01:25
jokke_sounds good. As long as I don't need to touch Miller and such but get something local for my taste buds I'll be happy01:25
fungiyeah, raleigh has an excellent local brewing scene. i always drank local beers when i lived there01:27
*** Sukhdev has quit IRC01:27
fungipretty much any place with more that 2 taps had at least one local keg on01:28
*** Sukhdev has joined #openstack-infra01:28
*** aorourke is now known as aorourke-afk01:28
openstackgerritMatt Riedemann proposed openstack/requirements: Exclude python-troveclient 2.1.0
*** jpr has joined #openstack-infra01:29
jeblairdrat, southend closed01:29
jeblairi used to walk there from my house01:30
*** dims has joined #openstack-infra01:31
jokke_fungi: I'm bit spoiled. My local is
fungijeblair: 10 years ago, apparently. there's still a southend brewery in charlotte, fwiw01:32
fungijokke_: that looks amazing. now i have another reason to visit galway01:33
bkerojokke_: Busy Bee Cafe was pretty nice when I was there01:34
clarkbjokke_: is it ok for me to complain that the sign says "america craft beer" then has a blue moon logo?01:35
fungijeblair: apparently the locations in charlotte nc and charleston sc were affiliated with the one in raleigh01:35
*** persia has quit IRC01:35
jeblairfungi: yeah.  good bbq _and_ good beer.01:36
jokke_clarkb: absolutely01:36
jeblairgood ribs even.01:36
clarkbok good, blue moon isn't exactly craft beer01:36
*** Qiming has joined #openstack-infra01:37
jeblairthey probably make less of it than guiness so it counts as craft?  i dunno :)01:37
*** persia has joined #openstack-infra01:38
jokke_clarkb: I think you should read it as American, craft beer & Blue Moon :P01:38
*** jpr has quit IRC01:38
clarkbI am just proud to live in the state that drinks the most beer brewed locally per capita01:38
jokke_clarkb: sound like a place I wanna visit :D01:39
*** jpr has joined #openstack-infra01:39
*** Sukhdev has quit IRC01:39
*** Sukhdev has joined #openstack-infra01:39
bkeroalso most microbreweries per capita01:41
bkeroin the world AIUI01:41
*** placko_ has quit IRC01:41
jokke_Vancouver was nice place beer wise01:41
*** Sukhdev has quit IRC01:41
*** Sukhdev has joined #openstack-infra01:42
jeblairfungi: last time i was in raleigh, my friends took me to all the nice new hip bars and restaurants... but when they weren't looking i went to places they take for granted like the seafood market, clyde coopers, big ed's, etc.01:42
SamYapleasheville, nc pumps out alot of beer01:43
SamYaplethe water is just right there so they run it through some charcoal filters and use it01:43
fungijeblair: yeah, there are still some great old staples there01:43
SamYaplemost other places need alot more01:43
*** kzaitsev_mb has quit IRC01:43
*** mtanino has quit IRC01:43
clarkbSamYaple: thats a big reason for it here too. Our water has almost nothing in it01:43
*** aeng has quit IRC01:44
clarkbthe water department apparently calls all the breweries when they switch to the well feed01:44
fungijeblair: seafood market as in the one at the entrance to the farmers market?01:44
clarkb"hey you might need to treat your water now"01:44
jeblairfungi: no, the one on the right in the middle of the stands that basically just serves fried-anything-from-the-ocean01:44
*** edmondsw has quit IRC01:44
SamYaplei just had a bunch of west coast beer01:44
fungijeblair: aha, yep. good stuff01:44
bkeroclarkb: AIUI our water is pretty soft01:45
SamYaplefirestone walker stuff01:45
SamYapleit was good01:45
clarkbbkero: yup, its like 3 ppm total additives01:45
clarkbbkero: many breweries add stuff to our water01:45
bkero"Portland’s water is very soft. The hardness of Bull Run water is typically 3-8 parts per million (ppm) – approximately ¼ to ½ a grain of hardness per gallon. Portland’s groundwater hardness is approximately 80 ppm (about 5 grains per gallon), which is considered moderately hard."01:45
bkeroclarkb: Eugene breweries just take it right out of the Willamette01:45
*** armax has quit IRC01:46
*** Sukhdev has quit IRC01:46
*** Sukhdev_ has joined #openstack-infra01:46
fungithe asheville area is starting to get overrun with larger chain breweries opening though. in the past few years sierra nevada, fat tire and oskar blues have muscled in01:46
clarkbbkero: I don't recommend that up here01:47
fungimostly because of water supply, again01:47
jokke_I've lived my whole life max 15miles from sea ... I'd have a trust issue towards sea food market that is located xxx miles inland :P01:47
SamYaplejeez. in TX we had 3ppm of clhorine....01:47
bkeroclarkb: the Willamette is different down there. Not nearly as dirty.01:47
clarkbSamYaple: and when somone pees in the resevoir they drain it and refill01:47
clarkbnevermind all the animals in it01:48
*** cody-somerville has quit IRC01:48
fungijokke_: raleigh is actually not very far up river. most of the fresh seafood there was caught and hauled ashore just a few miles from my house around sunrise, then spent a few hours in a refrigerated truck01:48
SamYaplei always like to point that out to drinkers of "spring bottled water"01:48
*** Sukhdev has joined #openstack-infra01:49
fungijokke_: but i agree, much better here right on the ocean ;)01:49
*** Sukhdev_ has quit IRC01:49
*** sarob has quit IRC01:49
SamYapleim from myrtle beach originally. they have some good seafood too01:49
*** fawadk has quit IRC01:49
* Clint salivates.01:50
clarkbin portland the seafood swims upriver to make it easier to eat it01:50
jeblairi think it was easier to get fresh seafood in raleigh than sf01:50
jeblairthere are some places here, but it's taken me a while to find them, and they're not where you'd expect.01:50
*** takashi has joined #openstack-infra01:50
Clintbaltimore had that problem for a couple years01:50
clarkbjeblair: is it mostly shellfish and salmon?01:50
clarkbjeblair: thats our problem01:50
SamYapleclarkb: i couldnt figure out the puppet thing so i just had it not report error codes01:51
*** jpr has quit IRC01:51
clarkbSamYaple: probably good enough for what you are doing01:51
*** sputnik13 has quit IRC01:51
*** Sukhdev has quit IRC01:51
SamYaplei "fixed" it!01:51
clarkbSamYaple: I will have to dig in toorrow and see if our builds are failing too01:51
jokke_I have to admit that while living in Dublin, I made sure that the seafood I got was either from West or North, not from the Irish sea ;)01:51
fungiyeah, nice thing about here is we can go right down to the warf and buy right from catches as they're coming off the boat. great if you're, say, having a party and want 15 lbs of fresh shrimp or something01:51
*** Sukhdev has joined #openstack-infra01:52
SamYaplewhen i was a kid i used to go down and take lobsters out of the lobster traps. i was told they were there for everyone01:52
*** thorst_ has joined #openstack-infra01:52
SamYaplei did not have the best parent in that situation01:52
SamYaplebut it was delicious lobster01:52
jeblairclarkb: most of our salmon is _your_ salmon is the big problem :|01:52
fungiSamYaple: the error code reporting in puppet is insane, so this does not surprise me. they have several non-zero "success" exit codes01:53
jeblairca runs are in very bad shape01:53
*** Sukhdev has quit IRC01:53
SamYaplefungi: in this case its _probably_ validly reporting non-zero, but i dont know anything about puppet01:53
SamYapleand frankly id like to keep it that way01:53
clarkbya its a valid error, it can't write to /etc/dhcp because the dir doesn't exist01:54
*** Sukhdev has joined #openstack-infra01:54
SamYaplethat should all be in the chroot environment, right?01:54
clarkbSamYaple: yup, its not your system's etc/dhcp01:54
jokke_fungi: that's the lovely part living here by Atlantic. 6 days a week day's catch available from local wholesale at the harbour01:54
SamYapleyea then i dont know01:54
SamYapleeither way ill have the fstrim results soo01:54
clarkbSamYaple: I am going to guess its something like simple-init element missing01:55
clarkbSamYaple: maybe add the simple-init next time around if you have patience for it01:55
SamYapleit takes about 10m to run the script so ill do it again01:55
*** gongysh has joined #openstack-infra01:55
*** Sukhdev has quit IRC01:56
*** Sukhdev has joined #openstack-infra01:56
*** Jeffrey4l has joined #openstack-infra01:57
SamYapleclarkb: cool. the results are in and are what i expected. 4.6GB01:58
*** gildub has quit IRC01:58
clarkbSamYaple: nice!01:58
SamYaplepatch incoming01:58
*** gildub has joined #openstack-infra01:58
*** Sukhdev has quit IRC01:58
clarkband this will work with xfs if we choose to make centos7 images that way01:58
*** Sukhdev has joined #openstack-infra01:59
*** ddieterly has quit IRC01:59
prometheanfireSamYaple: mind letting me know what you think?
SamYapleclarkb: correct02:00
SamYapleor btrfs (one can hope)02:00
clarkbSamYaple: I was reading an lkml article from 4 years ago where they said btrfs will be default everywhere soon02:00
*** Sukhdev has quit IRC02:01
*** Sukhdev has joined #openstack-infra02:01
fungion oracle linux02:01
prometheanfireI like my zfs :P02:02
openstackgerritSam Yaple proposed openstack/diskimage-builder: Use fstrim to prep the block device
SamYaplewell considering the devs didnt announce it "stable for the core features" until about 18 months ago that article sounds hopeful02:02
SamYapleprometheanfire: yea i jump back and forth02:02
SamYapleclarkb: with the above patch we really dont need zerofree, but i left it in there for now. it just doesnt run if it detects fstrim02:03
fungiwas 18 months ago about the point where it got a usable fsck?02:03
SamYaplewho needs to fsck a CoW filesystem?02:03
prometheanfirezpool scrub02:04
fungiyeah, repairing errors? it's not like those ever happen02:04
prometheanfireand a friend had to use the btrfs recovery stuff once02:04
SamYaplei mean it has checksum errors02:04
SamYapleif thats what you mean by fsck its had that for a long time02:04
SamYapleyou can read all the data and check for errors02:04
fungioh yeah, i basically lost a very large btrfs some years back. it was nigh unrecoverable and there was no filesystem checker for it02:04
prometheanfirethere's one now, but it's discouraged02:05
SamYaplei mean some years back it wasnt annouced as stable :P02:05
*** fawadkhaliq has joined #openstack-infra02:05
SamYapleprometheanfire: its not your standard fsck02:05
SamYapleprometheanfire: there is a btrfs scrub02:05
clarkbwhat version of the kernel is it considered stable in?02:05
SamYapleits not teh same thing02:05
SamYapleclarkb: 3.14 i believe02:05
SamYaplemaybe a bit newer, but in the 3.x branch02:06
prometheanfireya, scrubs aren't the same02:06
fungiright, there was no basic file recovery for btrfs the last time i tried to use it. luckily i was making frequent backups02:06
SamYapleyou just said zfs had that and that was the same thing02:06
SamYapleit has the same checks zfs has to keep it 'stable'02:06
prometheanfiredidn't say it was the same thing02:06
SamYaplefungi: not likely to happen either i think. cow is nigh impossible to recover02:07
SamYapleprometheanfire: i guess you made it saound that way02:07
SamYaple02:04:28 < SamYaple> who needs to fsck a CoW filesystem?02:07
SamYaple02:04:55 < prometheanfire> zpool scrub02:07
fungigood point02:07
prometheanfireI know of some recoveries of zfs that have been done (mainly due to zil stuff dieing)02:08
SamYaplebtrfs has scrubs and checksums, it can be used to validate integrity02:08
fungiessentially the block device started having some sporadic errors. btrfs more or less gave up being mountable. if it were ext3 or reiser or something, some of the files would have just become unreachable but i could still have at least copied most of the content off with it mounted read-only02:08
prometheanfirealso, it helps that I've never lost data on zfs02:09
SamYapleprometheanfire: were you using it before it was announced as stable :P02:09
SamYaplenot really a direct comparision then02:09
prometheanfirewere you using it before it was announced as stable02:09
fungii was definitely using btrfs before it was officially stable. this was probably 2009? hard to remember now02:09
prometheanfireI was not using it before it was stable02:09
SamYapleprometheanfire: btrfs? yes i was02:09
prometheanfireonly after02:09
SamYaplelol yea that was a while back fungi02:10
prometheanfireso, comparison applicable :P02:10
fungiseems recent to me, but yeah. i guess it's been a while02:10
*** yamamoto_ has joined #openstack-infra02:10
SamYaplei dont think there has been a "all your data are gone now" since its been announced stable02:10
SamYaplebut you know if you lost the wrong block on zfs you have to rekick the whole thing, btrfs not so much02:11
prometheanfirethere's been a few bad ones though02:11
fungii do keep meaning to try it again02:11
SamYaplebefore it was called stable, yes. same with zfs02:11
prometheanfireand after02:11
prometheanfirenot the same with zfs02:11
SamYapleshow me one?02:11
* fungi hopes he02:12
prometheanfirehave to reclone02:12
fungi's not going to need to make a plea for civility02:12
SamYaplelol fungi me and prometheanfire are good02:12
fungiit's almost like someone called a vi vs emacs throwdown in here02:12
prometheanfireso might be a few :P02:13
SamYaplewe both like ZFS, i lean toward btrfs though02:13
SamYaplefungi: so thats what were going to do today? fight?02:13
prometheanfireand I lean toward btrfs02:13
SamYaplei have it in writing!02:13
prometheanfirethere are some things I wish zfs had that btrfs does02:13
prometheanfireoh, meh02:13
SamYaplesame with zfs02:13
SamYaplei love creaintg zvols02:13
prometheanfireReceiving objects:   1% (79164/4583176), 35.07 MiB | 408.00 KiB/s02:13
SamYaplehate that btrfs will never have it02:14
prometheanfireIf oracle doesn't finish their zfs cinder driver I'm going to do it myself02:14
prometheanfirethey've been 'working on it'02:15
SamYaplei wrote a zfs driver for opennebula02:16
SamYapletell me prometheanfire , what is "${VAR##*:}" doing?02:16
SamYaple##*: is in question02:16
openstackgerritJoshua Hesketh proposed openstack-infra/project-config: Add experimental job to test xstatic packaging
prometheanfire## remove longest match from the front of the string02:17
prometheanfire*: remove anything up til the last : from the front of the string02:17
Clintand including the :02:17
prometheanfirebash substring manipulation is awesome02:17
prometheanfireClint: yes02:17
SamYaplegot it02:18
SamYaplegrep some of my scripts turns out ive done this before02:18
SamYaplei cannot remember these things02:18
prometheanfireelements_path was /foo/bar/baz:/x/y/z:/What/I/want02:18
*** thorst_ has quit IRC02:18
prometheanfireya, I have to google every time02:19
prometheanfireReceiving objects:   9% (448897/4583176), 181.92 MiB | 446.00 KiB/s02:19
SamYapleyea i would have gone |awk -F: '{print $(NF)}'02:19
prometheanfirestill going lol02:19
*** thorst_ has joined #openstack-infra02:19
SamYaplebut yours is bashier02:19
prometheanfireya, didn't want to call out02:19
Clinti remember # vs. % because the 3 is left of the 502:20
fungiexcept on a 10-key pad02:20
prometheanfirethat removing from the end?02:20
Clinti haven't seen one of those in a while02:21
fungiso true02:21
Clintprometheanfire: # from the left, % from the right02:21
Clint(# is on the left of $, % is on the right of $)02:21
Clinton us layout anyway02:22
*** Roamer` has quit IRC02:22
SamYapleman bash02:23
fungiever wanted to waste several hours of your life you'll never get back? man bash!02:24
*** Sukhdev_ has joined #openstack-infra02:24
*** Sukhdev has quit IRC02:24
SamYapleim very happy the new lts for ubuntu is about to land02:24
SamYapleumount -R is your friend02:25
fungii've slowly grown a finger memory for forward-searching to specific sections of the bash manpage02:25
prometheanfireSamYaple: with zfs support :P02:25
SamYaple(its like a requirement with systemd!)02:25
fungiSamYaple: what's -R get you?02:25
SamYaplerecursive unmount!02:25
fungiyeah, that would be handy02:25
prometheanfireoh, that is nice02:25
fungiespecially with chroots02:25
*** hdd_ has quit IRC02:26
SamYaplei actually pull in the new umount binaries on my 14.04 system because i need that thing02:26
SamYaplei neeeeeed it02:26
fungii always forget to umount sys or proc or something when cleaning up a chroot02:26
*** roxanagh_ has quit IRC02:26
fungii still don't get why /dev/pts has to be a separate virtual fs from /dev02:27
SamYapleumount -R would get right of this crap
*** IlyaG has joined #openstack-infra02:27
prometheanfirethat's my 'fun'02:27
*** baoli has joined #openstack-infra02:27
fungithat one makes a little more sense as it's just a ramdisk02:27
prometheanfireSamYaple: oh god02:27
*** kevinsho_ has quit IRC02:28
SamYapleprometheanfire: right!02:28
prometheanfired34dh0r53: wtf you doing here?02:28
*** thorst_ has quit IRC02:28
SamYapleclearly he wanted to join the cool kids02:28
*** Sukhdev_ has quit IRC02:30
*** Sukhdev has joined #openstack-infra02:31
*** IlyaG has quit IRC02:33
*** Sukhdev has quit IRC02:35
*** Sukhdev has joined #openstack-infra02:35
*** Roamer` has joined #openstack-infra02:36
*** EricGonczer_ has joined #openstack-infra02:37
*** _ody has quit IRC02:38
*** _ody has joined #openstack-infra02:38
*** ociuhandu has quit IRC02:38
*** Sukhdev has quit IRC02:38
*** ociuhandu has joined #openstack-infra02:39
*** Sukhdev has joined #openstack-infra02:39
*** baoli has quit IRC02:40
*** eharney has quit IRC02:40
*** Jeffrey4l has quit IRC02:41
*** Sukhdev has quit IRC02:42
*** Sukhdev has joined #openstack-infra02:42
*** EricGonc_ has joined #openstack-infra02:43
*** Jeffrey4l has joined #openstack-infra02:44
*** kzaitsev_mb has joined #openstack-infra02:45
*** Sukhdev has quit IRC02:45
*** Sukhdev has joined #openstack-infra02:46
*** EricGonczer_ has quit IRC02:46
*** Sukhdev has quit IRC02:48
*** Sukhdev has joined #openstack-infra02:49
*** yamamoto_ has quit IRC02:49
*** Sukhdev has quit IRC02:52
*** binbincong has quit IRC02:52
*** Sukhdev has joined #openstack-infra02:52
*** fawadkhaliq has quit IRC02:53
*** EricGonc_ has quit IRC02:53
*** tonytan4ever has quit IRC02:55
*** Sukhdev has quit IRC02:55
*** Sukhdev_ has joined #openstack-infra02:55
*** tqtran-afk has quit IRC02:56
*** SergeyLukjanov has quit IRC02:56
*** NikitaKonovalov has quit IRC02:56
*** kzaitsev_mb has quit IRC02:57
*** Sukhdev_ has quit IRC02:58
*** Sukhdev has joined #openstack-infra02:59
openstackgerritIan Wienand proposed openstack/diskimage-builder: centos-minimal does not provide base
*** HeOS has quit IRC03:00
*** jlk has quit IRC03:01
*** dansmith has quit IRC03:01
*** darvon has quit IRC03:01
*** dansmith has joined #openstack-infra03:01
*** jlk has joined #openstack-infra03:01
*** jlk has quit IRC03:01
*** jlk has joined #openstack-infra03:01
*** Sukhdev has quit IRC03:01
openstackgerritTristan Cacqueray proposed openstack-infra/shade: Add get_stack_outputs method
*** Sukhdev has joined #openstack-infra03:02
*** darvon has joined #openstack-infra03:02
*** HeOS has joined #openstack-infra03:02
*** SergeyLukjanov has joined #openstack-infra03:02
*** gildub has quit IRC03:03
*** NikitaKonovalov has joined #openstack-infra03:03
*** Sukhdev has quit IRC03:05
*** Sukhdev_ has joined #openstack-infra03:05
*** Sukhdev_ has quit IRC03:08
*** Sukhdev has joined #openstack-infra03:08
*** ajmiller has quit IRC03:10
*** thorst_ has joined #openstack-infra03:10
*** hdd has joined #openstack-infra03:11
*** Sukhdev_ has joined #openstack-infra03:11
*** Sukhdev has quit IRC03:11
*** tphummel has quit IRC03:14
*** Sukhdev_ has quit IRC03:14
*** thorst_ has quit IRC03:14
*** dims has quit IRC03:14
*** Sukhdev has joined #openstack-infra03:14
*** baoli has joined #openstack-infra03:14
*** thorst_ has joined #openstack-infra03:15
*** grassass has joined #openstack-infra03:15
*** Sukhdev has quit IRC03:15
*** zz_dimtruck is now known as dimtruck03:17
openstackgerritMatt Riedemann proposed openstack/requirements: Exclude python-troveclient 2.1.0
*** mriedem has quit IRC03:17
*** apoorvad has quit IRC03:18
openstackgerritMerged openstack/requirements: kilo: pin pysaml2<4.0.4
*** SergeyLukjanov has quit IRC03:19
*** thorst_ has quit IRC03:23
*** bapalm has quit IRC03:24
*** baoli has quit IRC03:24
*** fawadkhaliq has joined #openstack-infra03:24
*** SergeyLukjanov has joined #openstack-infra03:24
*** asettle has joined #openstack-infra03:26
*** bpokorny has quit IRC03:27
*** bpokorny has joined #openstack-infra03:27
*** markvoelker has joined #openstack-infra03:29
*** hdd has quit IRC03:32
*** bpokorny has quit IRC03:32
*** asettle has quit IRC03:33
*** fawadkhaliq has quit IRC03:34
*** yamamoto_ has joined #openstack-infra03:36
*** gildub has joined #openstack-infra03:37
*** bapalm has joined #openstack-infra03:43
*** markvoelker has quit IRC03:43
*** rlandy has quit IRC03:45
*** yamahata has quit IRC03:49
*** links has joined #openstack-infra03:50
*** kzaitsev_mb has joined #openstack-infra03:53
*** kencjohnston has joined #openstack-infra03:55
*** woodster_ has quit IRC03:57
*** kencjohnston has quit IRC03:57
*** kzaitsev_mb has quit IRC03:58
openstackgerritIan Wienand proposed openstack/diskimage-builder: Clear up "already provided" message
*** dims has joined #openstack-infra04:01
*** jaosorior has quit IRC04:02
*** jaosorior has joined #openstack-infra04:03
openstackgerritTristan Cacqueray proposed openstack-infra/shade: Add get_stack_outputs method
*** gongysh has quit IRC04:11
*** zeih has joined #openstack-infra04:12
*** dims has quit IRC04:13
*** nikhil has joined #openstack-infra04:15
*** nikhil_k has joined #openstack-infra04:15
*** zeih has quit IRC04:17
*** tphummel has joined #openstack-infra04:20
*** thorst_ has joined #openstack-infra04:21
*** takashi has quit IRC04:27
*** takashi has joined #openstack-infra04:27
*** thorst_ has quit IRC04:28
*** hichihara has joined #openstack-infra04:28
*** kevinsho_ has joined #openstack-infra04:29
openstackgerritIan Wienand proposed openstack-infra/project-config: Add centos-minimal build, experimental job
openstackgerritIan Wienand proposed openstack-infra/project-config: Remove Fedora23 install-glean-from-git flag
*** ajmiller has joined #openstack-infra04:33
*** kevinsho_ has quit IRC04:33
*** IlyaG has joined #openstack-infra04:37
*** treaki__ has joined #openstack-infra04:37
*** amotoki has joined #openstack-infra04:38
*** harlowja_at_home has quit IRC04:39
*** harlowja_at_home has joined #openstack-infra04:39
*** zeih has joined #openstack-infra04:39
*** treaki_ has quit IRC04:40
openstackgerritIan Wienand proposed openstack-infra/project-config: Add centos-minimal build, experimental job
*** jamielennox is now known as jamielennox|away04:41
*** zeih has quit IRC04:44
openstackgerritReinhard Tartler proposed openstack-infra/jenkins-job-builder: Add support for disable-strict-forbidden-file-verification
*** HeOS has quit IRC04:45
*** HeOS has joined #openstack-infra04:45
ianwwhile c++ still wins for "one character wrong ridiculous length of backtrace", jjb is getting up there ->
*** DinaBelova has quit IRC04:51
*** kragniz has quit IRC04:52
*** DinaBelova has joined #openstack-infra04:53
*** kragniz has joined #openstack-infra04:54
*** heyongli has joined #openstack-infra04:57
*** ajmiller has quit IRC05:00
*** heyongli has quit IRC05:00
*** heyongli has joined #openstack-infra05:00
*** HeOS has quit IRC05:01
*** IlyaG has quit IRC05:02
*** HeOS has joined #openstack-infra05:03
*** kragniz has quit IRC05:05
*** kragniz has joined #openstack-infra05:05
*** heyongli has quit IRC05:10
*** heyongli has joined #openstack-infra05:10
*** nwkarsten has joined #openstack-infra05:13
*** Jeffrey4l has quit IRC05:15
*** nwkarsten has quit IRC05:17
*** fedexo has joined #openstack-infra05:19
*** beisner has quit IRC05:19
*** sheeprine_ has quit IRC05:19
*** sheeprine has joined #openstack-infra05:20
*** heyongli has quit IRC05:20
*** heyongli has joined #openstack-infra05:20
*** beisner has joined #openstack-infra05:21
*** dimtruck is now known as zz_dimtruck05:25
*** zz_dimtruck is now known as dimtruck05:25
*** thorst_ has joined #openstack-infra05:26
*** hdd has joined #openstack-infra05:29
*** heyongli has quit IRC05:30
*** hieulq has joined #openstack-infra05:31
*** flepied has quit IRC05:31
*** hieulq has quit IRC05:31
*** thorst_ has quit IRC05:33
*** baoli has joined #openstack-infra05:36
*** harlowja_at_home has quit IRC05:36
*** infra-red has joined #openstack-infra05:38
*** sheeprine has quit IRC05:40
*** jaosorior has quit IRC05:40
*** dansmith has quit IRC05:40
*** yonglihe has quit IRC05:40
*** evgenyl has quit IRC05:40
*** mwhahaha has quit IRC05:40
*** Volundr has quit IRC05:40
*** mescanef has quit IRC05:40
*** dingyichen has quit IRC05:40
*** cfarquhar has quit IRC05:40
*** baoli has quit IRC05:40
*** flepied has joined #openstack-infra05:43
*** asettle has joined #openstack-infra05:45
*** binbincong has joined #openstack-infra05:45
*** sheeprine has joined #openstack-infra05:46
*** dansmith has joined #openstack-infra05:46
*** yonglihe has joined #openstack-infra05:46
*** evgenyl has joined #openstack-infra05:46
*** mwhahaha has joined #openstack-infra05:46
*** Volundr has joined #openstack-infra05:46
*** mescanef has joined #openstack-infra05:46
*** dingyichen has joined #openstack-infra05:46
*** cfarquhar has joined #openstack-infra05:46
*** jaosorior has joined #openstack-infra05:46
*** jaosorior has quit IRC05:46
*** kushal has quit IRC05:47
*** nwkarsten has joined #openstack-infra05:49
*** asettle has quit IRC05:49
*** armax has joined #openstack-infra05:50
*** jtomasek has joined #openstack-infra05:53
*** ayoung has quit IRC05:53
*** nwkarsten has quit IRC05:54
*** melwitt has quit IRC05:54
*** kukacz_ has quit IRC05:55
*** kzaitsev_mb has joined #openstack-infra05:55
*** med_ has quit IRC05:56
*** mfisch has quit IRC05:56
openstackgerritAndreas Jaeger proposed openstack-infra/project-config: WIP: Remove bare-trusty from projects.yaml
*** kukacz has joined #openstack-infra05:56
*** med_ has joined #openstack-infra05:56
*** med_ is now known as Guest9366405:56
*** melwitt has joined #openstack-infra05:59
*** kzaitsev_mb has quit IRC06:00
*** sheeprine has quit IRC06:02
*** dansmith has quit IRC06:02
*** yonglihe has quit IRC06:02
*** evgenyl has quit IRC06:02
*** mwhahaha has quit IRC06:02
*** Volundr has quit IRC06:02
*** mescanef has quit IRC06:02
*** dingyichen has quit IRC06:02
*** cfarquhar has quit IRC06:02
*** dimtruck is now known as zz_dimtruck06:03
*** ayoung has joined #openstack-infra06:04
*** infra-red has quit IRC06:06
*** fawadkhaliq has joined #openstack-infra06:07
*** sheeprine has joined #openstack-infra06:08
*** dansmith has joined #openstack-infra06:08
*** yonglihe has joined #openstack-infra06:08
*** evgenyl has joined #openstack-infra06:08
*** mwhahaha has joined #openstack-infra06:08
*** Volundr has joined #openstack-infra06:08
*** mescanef has joined #openstack-infra06:08
*** dingyichen has joined #openstack-infra06:08
*** cfarquhar has joined #openstack-infra06:08
*** binbincong has quit IRC06:10
*** abregman has joined #openstack-infra06:14
*** hdd has quit IRC06:15
openstackgerritOpenStack Proposal Bot proposed openstack-infra/project-config: Normalize projects.yaml
*** bapalm has quit IRC06:23
*** Guest38484 has joined #openstack-infra06:27
*** binbincong has joined #openstack-infra06:29
*** fawadkhaliq has quit IRC06:30
*** thorst_ has joined #openstack-infra06:31
*** kushal has joined #openstack-infra06:31
openstackgerritAndreas Jaeger proposed openstack-infra/project-config: gerrit-plugin-jobs: Use ubuntu-trusty
openstackgerritOpenStack Proposal Bot proposed openstack/requirements: Updated from generate-constraints
openstackgerritAndreas Jaeger proposed openstack-infra/project-config: Change jobs to use ubuntu-trusty
*** thorst_ has quit IRC06:38
openstackgerritPurandhar Sairam Mannidi proposed openstack/diskimage-builder: Add support for building images capable of UEFI
*** bapalm has joined #openstack-infra06:42
*** melwitt has quit IRC06:45
AJaegerfungi, pabelanger, clarkb: Thanks for reviews! We're down to 6 bare-trusty jobs06:45
AJaeger5 of them are changed with  and
AJaegerfungi, clarkb, pabelanger: The final one needs discussion:  gate-infra-puppet-apply-bare-trusty06:46
AJaegerDo we have servers running bare-trusty that we need this for? Do we need to setup a job for ubuntu-trusty?06:47
AJaegermore later today - just wanted to dump it here while I investigated06:47
*** sheeprine has quit IRC06:49
*** sheeprine has joined #openstack-infra06:49
*** melwitt has joined #openstack-infra06:50
*** melwitt is now known as Guest9853406:50
*** gongysh has joined #openstack-infra06:51
*** jtomasek has quit IRC06:53
openstackgerritPurandhar Sairam Mannidi proposed openstack/diskimage-builder: Add support for building images capable of UEFI
*** Guest98534 is now known as melwitt06:55
*** fedexo has quit IRC06:58
*** melwitt has quit IRC06:58
fungiAJaeger: no, the server puppet manifests just need to be tested on a representative of the distro/release, so the testing we do on ubuntu-trusty now validates the puppetry for our servers which run on trusty06:59
*** melwitt has joined #openstack-infra07:00
openstackgerritMerged openstack-infra/project-config: Normalize projects.yaml
*** kushal has quit IRC07:03
*** melwitt has quit IRC07:03
*** melwitt has joined #openstack-infra07:04
*** melwitt is now known as Guest3702007:05
*** skraynev has quit IRC07:05
*** skraynev has joined #openstack-infra07:06
*** yamahata has joined #openstack-infra07:07
lennybanteaya, mriedem: Hi, I will check the issue with the email. I've changed it to my personal email until the issue is solved.07:07
*** sarob has joined #openstack-infra07:11
*** esikachev has joined #openstack-infra07:14
openstackgerrityolanda.robla proposed openstack/diskimage-builder: Set default locale to image in ubuntu-minimal
*** sarob has quit IRC07:16
*** Kennan has quit IRC07:18
*** zeih has joined #openstack-infra07:22
*** sc68cal has quit IRC07:22
*** ildikov has quit IRC07:22
*** Kennan has joined #openstack-infra07:23
*** Jeffrey4l has joined #openstack-infra07:26
*** zeih has quit IRC07:27
*** roxanagh_ has joined #openstack-infra07:27
*** roxanagh_ has quit IRC07:28
*** andreykurilin__ has joined #openstack-infra07:29
*** melwitt_ has joined #openstack-infra07:30
*** melwitt_ has quit IRC07:32
*** scheuran has joined #openstack-infra07:33
*** rcernin has joined #openstack-infra07:33
*** ildikov has joined #openstack-infra07:34
*** thorst_ has joined #openstack-infra07:36
*** esikachev has quit IRC07:38
*** melwitt_ has joined #openstack-infra07:39
*** melwitt_ is now known as melwitt07:40
*** melwitt has quit IRC07:40
*** melwitt has joined #openstack-infra07:41
*** fhubik has joined #openstack-infra07:42
*** thorst_ has quit IRC07:43
*** esikachev has joined #openstack-infra07:43
openstackgerritMerged openstack-infra/puppet-storyboard: Update SSLProtocol and SSLCipherSuite
*** nikhil has quit IRC07:45
*** fhubik has quit IRC07:46
*** nikhil has joined #openstack-infra07:47
yolandauh AJaeger, i'm having that error on a diskimage-builder test07:47
yolanda2016-03-10 07:45:27.466 |   File "/opt/stack/new/neutron-fwaas/neutron_fwaas/tests/tempest_plugin/services/", line 18, in <module>07:47
yolanda2016-03-10 07:45:27.466 |     from import base07:47
yolanda2016-03-10 07:45:27.467 | ImportError: cannot import name base07:47
yolandado you know about it?07:47
yolandai'll as in qa as well07:48
*** baoli has joined #openstack-infra07:48
dmelladommm yolanda, could this be that tox it's not using the system packages?07:50
*** baoli has quit IRC07:53
*** dizquierdo has joined #openstack-infra07:53
*** e0ne has joined #openstack-infra07:53
*** mrmartin has joined #openstack-infra07:54
*** jlanoux has joined #openstack-infra07:55
AJaegeryolanda: no idea on that one. Did we switch jobs on that test to ubuntu-trusty? IT shouldn't change tempest python package installation...07:56
*** nmagnezi has joined #openstack-infra07:57
*** kzaitsev_mb has joined #openstack-infra07:57
*** _nadya_ has joined #openstack-infra08:00
*** cyraxjoe has joined #openstack-infra08:00
*** ifarkas has joined #openstack-infra08:02
*** kzaitsev_mb has quit IRC08:02
esikachevguys, who can help me with ci?08:05
esikachevtrigger by time not triggered08:05
yolandaAJaeger dmellado, so question answered in qa, there are patches in review for this08:06
yolanda and
*** ihrachys has joined #openstack-infra08:07
*** ihrachys has quit IRC08:08
openstackgerritAlexey Ovchinnikov proposed openstack-infra/project-config: [WIP] Add publishing capability to buildimage-lxd job
yolandadmellado , you seem to be more aware on these tempest tests than me08:09
*** daemontool_ has joined #openstack-infra08:11
*** kzaitsev_mb has joined #openstack-infra08:11
*** ihrachys has joined #openstack-infra08:13
*** daemontool has quit IRC08:14
*** fhubik has joined #openstack-infra08:16
*** armax has quit IRC08:17
*** flepied has quit IRC08:17
*** zeih_ has joined #openstack-infra08:20
*** [HeOS] has joined #openstack-infra08:20
*** kushal has joined #openstack-infra08:21
*** HeOS has quit IRC08:21
*** e0ne has quit IRC08:21
*** kushal has joined #openstack-infra08:25
*** e0ne has joined #openstack-infra08:25
*** matrohon has joined #openstack-infra08:26
*** zeih_ has quit IRC08:27
*** lezbar has joined #openstack-infra08:27
*** arxcruz has joined #openstack-infra08:28
*** [1]Thelo has joined #openstack-infra08:30
*** kevinsho_ has joined #openstack-infra08:31
dmelladoyolanda: ?08:31
*** zeih_ has joined #openstack-infra08:31
yolandadmellado, i'm talking about the tempest failures, i was seeing your comments in openstack_qa08:32
*** mikelk has joined #openstack-infra08:32
*** dguitarbite has quit IRC08:32
*** davideagnello has quit IRC08:32
*** Thelo has quit IRC08:33
*** [1]Thelo is now known as Thelo08:33
dmelladoyolanda: what usually works for the tempest plugin is that they add an entry point for testr08:34
*** pcaruana has joined #openstack-infra08:34
*** davideagnello has joined #openstack-infra08:35
dmelladoso depending on the repo and implementation there's usually a tox env that exposes the global packages so it could get to tempest too08:35
dmelladoanyway this seemed related to the patch ;)08:35
*** kevinsho_ has quit IRC08:36
*** andreykurilin__ has quit IRC08:36
*** betherly has quit IRC08:36
*** fkautz has quit IRC08:36
*** fdegir has quit IRC08:36
*** simonmcc has quit IRC08:36
*** etoews has quit IRC08:36
*** sylwesterB has quit IRC08:36
*** hrybacki has quit IRC08:36
*** ashtokolov has quit IRC08:36
*** zhiyan has quit IRC08:36
*** edwarnicke has quit IRC08:36
*** Ng has quit IRC08:36
*** GheRivero has quit IRC08:36
*** zhenguo_ has quit IRC08:36
*** fmccrthy has quit IRC08:36
*** teran has quit IRC08:36
*** skath_ has quit IRC08:36
*** kgalanov has quit IRC08:36
*** ams_ has quit IRC08:36
*** igorbelikov has quit IRC08:36
*** zzzeek has quit IRC08:36
*** bryan_att has quit IRC08:36
*** lindsayk_ has quit IRC08:36
*** dham1 has quit IRC08:36
*** samyewell has quit IRC08:36
*** ameade has quit IRC08:36
*** scottda has quit IRC08:36
*** jpmaxman has quit IRC08:36
*** zxiiro has quit IRC08:36
*** esikachev has quit IRC08:37
*** zzzeek has joined #openstack-infra08:38
yolandadmellado ah, i understand. For my particular failure i think that this patch will solve my issue08:39
*** ashtokolov has joined #openstack-infra08:39
*** skath_ has joined #openstack-infra08:39
*** zhiyan has joined #openstack-infra08:39
*** scottda has joined #openstack-infra08:39
*** ameade has joined #openstack-infra08:39
*** simonmcc has joined #openstack-infra08:40
*** etoews has joined #openstack-infra08:40
*** betherly has joined #openstack-infra08:40
*** fdegir has joined #openstack-infra08:40
*** lindsayk_ has joined #openstack-infra08:40
*** thorst_ has joined #openstack-infra08:41
dmelladoyolanda: if it doesn't work for you let me know and we'll have a look08:41
*** kgalanov has joined #openstack-infra08:42
*** teran has joined #openstack-infra08:42
dmelladonp ;)08:42
*** GheRivero has joined #openstack-infra08:42
*** fmccrthy has joined #openstack-infra08:42
*** igorbelikov has joined #openstack-infra08:42
*** dham1 has joined #openstack-infra08:42
*** zxiiro has joined #openstack-infra08:42
*** e0ne has quit IRC08:42
*** jpmaxman has joined #openstack-infra08:43
openstackgerritMerged openstack-infra/system-config: Add a playbook and script for doing adhoc puppet
*** zhenguo_ has joined #openstack-infra08:44
*** Ng has joined #openstack-infra08:44
*** sylwesterB has joined #openstack-infra08:44
*** hrybacki has joined #openstack-infra08:44
*** andreykurilin__ has joined #openstack-infra08:44
*** bryan_att has joined #openstack-infra08:44
*** fkautz has joined #openstack-infra08:44
*** samyewell has joined #openstack-infra08:44
*** edwarnicke has joined #openstack-infra08:44
*** kzaitsev_mb has quit IRC08:45
*** ams_ has joined #openstack-infra08:45
*** dingyichen has quit IRC08:45
*** vincentll has joined #openstack-infra08:46
*** thorst_ has quit IRC08:48
*** zeih_ has quit IRC08:48
*** gildub has quit IRC08:48
openstackgerritRicardo Carrillo Cruz proposed openstack-infra/system-config: Use set-hostnames playbook in launch-node
openstackgerritMerged openstack-infra/system-config: Make the shade cache and groups writable by admin
*** Jeffrey4l has quit IRC08:50
openstackgerritMerged openstack-infra/system-config: Update for ansible and puppet apply
*** Jeffrey4l has joined #openstack-infra08:51
*** jaosorior has joined #openstack-infra08:52
*** salv-orlando has joined #openstack-infra08:55
*** fhubik is now known as fhubik_brb08:55
*** flepied has joined #openstack-infra08:56
openstackgerritMerged openstack-infra/project-config: Added IoTronic project.
*** jordanP has joined #openstack-infra08:56
*** esikachev has joined #openstack-infra08:56
openstackgerritMerged openstack-infra/project-config: use the correct branch when proposing a new config file
stevemaryolanda: *poke*08:57
stevemaryolanda: can you also please review :)08:57
*** fhubik_brb is now known as fhubik08:58
yolandahi stevemar taking a look, i'm going through the review queue now08:58
stevemaryolanda: thank you! :]08:58
rcarrillocruzlooks like the set-hostnames change ran in a jenkins without the job fixed09:00
rcarrillocruzthe locale thing09:00
yolandaolder node? ok let's recheck09:00
rcarrillocruzi'm rechecking, but the fact the change merged yesterday and some jenkins not having it is worrying09:00
yolandamm, and that fix was done on the job directly right?09:00
yolandamaybe jjb problems?=09:00
*** hashar has joined #openstack-infra09:01
yolandagoing to check jjb runs09:01
*** zz_dimtruck is now known as dimtruck09:03
*** shardy has joined #openstack-infra09:05
*** Jeffrey4l has quit IRC09:05
openstackgerritMerged openstack-infra/project-config: Add a non-voting iPXE job to ironic-inspector
yolandarcarrillocruz, i don't see failures in jjb in logs. But going to check job config history in them09:06
*** _nadya_ has quit IRC09:06
openstackgerritMerged openstack-infra/project-config: don't fail if commands overlap
*** jaosorior has quit IRC09:09
*** jaosorior has joined #openstack-infra09:09
*** jistr has joined #openstack-infra09:09
yolandarcarrillocruz, just started looking and jenkins02 has not picked the job updates09:10
rcarrillocruz:/, recheck didn't seem to go thru, i'll retry09:10
AJaegerSo, manual JJB run on jenkins02 again? ;( Any other jenkins needed?09:11
*** dtantsur|afk is now known as dtantsur09:11
yolandaAJaeger, going to check all09:12
yolandamanual updates yes09:12
*** dimtruck is now known as zz_dimtruck09:12
*** jcoufal has joined #openstack-infra09:12
yolandajenkins02 is failing with jjb quite often...09:13
*** rossella_s has joined #openstack-infra09:13
AJaegerthanks, yolanda !09:14
*** lucas-dinner is now known as lucasagomes09:14
rcarrillocruz\o/ now green...09:14
*** _skolekonov is now known as skolekonov09:14
*** amotoki has quit IRC09:15
yolandaall the other jenkins are fine09:15
*** salv-orlando has quit IRC09:20
yolandajjb doing its job on jenkins02, looks as it hadn't run for a long time, because it's reconfiguring a bunch of gate jobs09:26
*** dtardivel has joined #openstack-infra09:29
*** openstackgerrit has quit IRC09:30
*** openstackgerrit_ has joined #openstack-infra09:30
*** openstackgerrit_ is now known as openstackgerrit09:31
*** openstackgerrit has quit IRC09:31
*** openstackgerrit_ has joined #openstack-infra09:31
*** pblaho has joined #openstack-infra09:31
*** openstackgerrit_ is now known as openstackgerrit09:32
*** openstackgerrit has quit IRC09:32
*** openstackgerrit_ has joined #openstack-infra09:32
*** flepied has quit IRC09:33
*** openstackgerrit_ is now known as openstackgerrit09:33
*** openstackgerrit has quit IRC09:33
*** e0ne has joined #openstack-infra09:33
*** openstackgerrit_ has joined #openstack-infra09:34
*** sfinucan has joined #openstack-infra09:34
*** openstackgerrit_ is now known as openstackgerrit09:34
*** abregman has quit IRC09:35
*** sarob has joined #openstack-infra09:38
openstackgerritMerged openstack-infra/system-config: Add pabelanger to infra root
yolandafinally! this test has failed a lot of times ^09:40
openstackgerritMerged openstack-infra/system-config: Remove pabelanger from asterisk and stackalytics
*** christian_ has joined #openstack-infra09:41
*** electrofelix has joined #openstack-infra09:41
*** _degorenko|afk is now known as degorenko09:42
*** sarob has quit IRC09:42
*** dguitarbite has joined #openstack-infra09:44
*** dguitarbite has quit IRC09:44
*** pkarikh has left #openstack-infra09:44
*** thorst_ has joined #openstack-infra09:45
*** daemontool__ has joined #openstack-infra09:46
*** abregman has joined #openstack-infra09:47
*** daemontool_ has quit IRC09:49
*** amotoki has joined #openstack-infra09:50
*** acabot has joined #openstack-infra09:51
*** flepied has joined #openstack-infra09:51
*** kushal has quit IRC09:52
*** thorst_ has quit IRC09:53
*** placko_ has joined #openstack-infra09:56
*** esikachev has quit IRC10:00
*** markusz is now known as markus_z10:00
*** tpsilva has joined #openstack-infra10:00
*** [HeOS] has quit IRC10:02
*** HeOS has joined #openstack-infra10:03
*** Guest3102 is now known as BobBall10:04
*** esikachev has joined #openstack-infra10:04
*** hichihara has quit IRC10:10
hasharmaar roots !10:10
*** sambetts|afk is now known as sambetts10:11
*** kzaitsev_mb has joined #openstack-infra10:11
*** derekh has joined #openstack-infra10:13
gmannyolanda: AJaeger:  regarding tempest plugin failure10:13
gmanni have opened a bug  -
openstackLaunchpad bug 1555524 in tempest "Any tempest plugin import failure leads other plugin tests fails" [Undecided,New] - Assigned to Ghanshyam Mann (ghanshyammann)10:13
gmannlet's see we can find goos way to avoid such failure in future10:13
yolandagmann that makes sense yes10:15
yolandawe shall only care about the plugins we are testing10:15
gmannyolanda: basically, other project error should not leads to any project tests failure10:15
gmannyolanda: exactly.10:15
gmannyolanda: we take care while loading, running or registering config etc10:16
*** jtomasek has joined #openstack-infra10:16
gmannbut this is special case where testr needs to list all tests and run requested one10:16
gmannand it is import error so comes while listing the tests10:17
yolandai understand, do you see an easy way to avoid it? enclosing imports in some try block?10:17
gmannyolanda: no, i think we should make it on tempest side, so while discovery if there is any issue in any plugin just log the error and move on10:20
*** esikachev has quit IRC10:21
gmannbut catching exception from loader.discovery is not straight forward10:21
gmannanyways this will fix issue as of now -
gmanni will ping fwaas cores to merge it soon10:22
Kennanhi gmann:10:23
gmannKennan: hi10:23
Kennanwhen will tempest failure be fixed ? I notice you are working on that10:23
Kennangmann: seems this ImportError: cannot import name base10:24
openstackLaunchpad bug 1555524 in tempest "Any tempest plugin import failure leads other plugin tests fails" [Undecided,New] - Assigned to Ghanshyam Mann (ghanshyammann)10:24
gmannKennan: this patch fix this issue -
gmannKennan: m checking if somehow we can avoid such issues in future10:25
Kennangmann: so tempest have changes which cause such failure ?10:25
openstackgerritPurandhar Sairam Mannidi proposed openstack/diskimage-builder: Add support for building images capable of UEFI
gmannKennan: theoretically yes but that is not exactly the way other plugin should import10:26
gmannbasically tempest interfaces are not stable interfaces except tempest/lib10:27
gmannso if someone importing and using those, we cannot guarantee that those will not be changed10:27
gmannall interfaces under tempest/lib are stable one and there we make sure about any backward incompatibility10:28
Kennanok gmann: i notice our tempest also failed for that, hope that your change can be merged soon10:28
gmannKennan: i am trying to reach fwaas core to merge that soon :)10:29
*** flepied has quit IRC10:29
*** dizquierdo_ has joined #openstack-infra10:35
openstackgerritRicardo Carrillo Cruz proposed openstack-infra/shade: Follow name_or_id pattern on domain operations
*** dizquierdo has quit IRC10:37
*** flepied has joined #openstack-infra10:42
*** dizquierdo_ has quit IRC10:43
*** dizquierdo has joined #openstack-infra10:45
*** rguillebert has joined #openstack-infra10:46
*** dims has joined #openstack-infra10:47
*** tzn has joined #openstack-infra10:48
*** kushal has joined #openstack-infra10:48
*** thorst_ has joined #openstack-infra10:50
*** bapalm has quit IRC10:51
*** dims has quit IRC10:52
*** yamamoto_ has quit IRC10:52
*** placko_ has quit IRC10:53
*** dizquierdo is now known as dizquierdo_afk10:55
*** dims has joined #openstack-infra10:56
*** ociuhandu has quit IRC10:56
*** fhubik is now known as fhubik_brb10:57
*** salv-orlando has joined #openstack-infra10:57
*** thorst_ has quit IRC10:58
*** sdague has joined #openstack-infra11:00
openstackgerritAttila Darazs proposed openstack-infra/tripleo-ci: Use IPv6 on the ceph gate job
openstackgerritThomas Morin proposed openstack-infra/project-config: net-bgpvpn: control gate settings from rc scripts
*** asettle has joined #openstack-infra11:07
*** larainema has quit IRC11:08
*** bapalm has joined #openstack-infra11:08
*** gongysh has quit IRC11:09
*** pblaho has quit IRC11:11
*** _nadya_ has joined #openstack-infra11:11
*** fhubik_brb is now known as fhubik11:11
dimssdague : is ready, i print the count of lines the warning message is repeated (example -
*** dguitarbite has joined #openstack-infra11:14
sdaguedims: awesome, thanks11:16
openstackgerritRicardo Carrillo Cruz proposed openstack-infra/shade: Follow name_or_id pattern on domain operations
*** dims has quit IRC11:19
*** dims has joined #openstack-infra11:21
*** fhubik is now known as fhubik_brb11:22
*** fhubik_brb is now known as fhubik11:24
openstackgerritRicardo Carrillo Cruz proposed openstack-infra/infra-specs: Add extension to openstackci for next phase of work
*** bapalm has quit IRC11:24
*** asettle has quit IRC11:27
*** kmartin has quit IRC11:29
*** Douhet has joined #openstack-infra11:29
*** salv-orlando has quit IRC11:30
openstackgerritAdam Coldrick proposed openstack-infra/storyboard: Fix removing due dates from boards
*** ldnunes has joined #openstack-infra11:43
*** bapalm has joined #openstack-infra11:44
*** yamamoto has joined #openstack-infra11:44
*** yamamoto has quit IRC11:44
*** yamamoto has joined #openstack-infra11:44
*** Guest88500 has quit IRC11:45
*** yamamoto_ has joined #openstack-infra11:45
*** yamamoto has quit IRC11:50
*** Qiming has quit IRC11:50
*** rfolco has joined #openstack-infra11:50
openstackgerrityolanda.robla proposed openstack-infra/system-config: Add playbook to manage power state of ironic servers
*** thorst_ has joined #openstack-infra11:56
*** dims has quit IRC11:56
*** sfinucan has quit IRC11:56
*** yamamoto_ has quit IRC11:57
*** aysyd has joined #openstack-infra11:59
*** kzaitsev_mb has quit IRC12:00
*** jaosorior has quit IRC12:02
*** thorst_ has quit IRC12:03
*** jaosorior has joined #openstack-infra12:03
*** dims has joined #openstack-infra12:04
*** erlon has joined #openstack-infra12:05
*** fhubik is now known as fhubik_brb12:07
*** fhubik_brb is now known as fhubik12:08
*** _amrith_ is now known as amrith12:16
*** xiangxinyong has quit IRC12:19
lucasagomeshi all, ironic seems to be broken due some tempest import error:
openstackgerritMerged openstack-infra/storyboard: Fix removing due dates from boards
*** placko_ has joined #openstack-infra12:21
*** kzaitsev_mb has joined #openstack-infra12:22
*** |-paul-| has joined #openstack-infra12:23
*** x00350071 has joined #openstack-infra12:25
lucasagomesthis patch in tempest broken ironic
AJaegerlucasagomes: Read backsroll, this sounds like
openstackLaunchpad bug 1555524 in tempest "Any tempest plugin import failure leads other plugin tests fails" [Undecided,New] - Assigned to Ghanshyam Mann (ghanshyammann)12:26
yolandalucasagomes, gmann was working on it12:26
gmannlucasagomes: this will fix it -
*** Daisy has joined #openstack-infra12:28
lucasagomesyolanda, gmann AJaeger thanks12:28
*** yamamoto has joined #openstack-infra12:28
gmannand in parallel m checking if we can have permanent solution to such case12:28
lucasagomesyeah it breaks us pretty badly12:28
lucasagomesI will add a patch in ironic depending on that to see if that works12:28
lucasagomesgmann, thanks a lot12:28
*** placko_ has quit IRC12:29
*** placko_ has joined #openstack-infra12:29
*** gordc has joined #openstack-infra12:31
*** weshay has joined #openstack-infra12:31
*** kevinsho_ has joined #openstack-infra12:32
*** Daisy has quit IRC12:33
*** Daisy has joined #openstack-infra12:34
*** lucasagomes is now known as lucas-hungry12:34
*** hamzy has quit IRC12:35
*** thorst_ has joined #openstack-infra12:35
*** kevinsho_ has quit IRC12:37
*** Daisy has quit IRC12:38
*** doug-fish has quit IRC12:38
*** yamamoto has quit IRC12:39
*** doug-fish has joined #openstack-infra12:39
*** Qiming has joined #openstack-infra12:40
*** christian_ has quit IRC12:40
*** fesp has joined #openstack-infra12:43
*** yamamoto has joined #openstack-infra12:43
*** doug-fish has quit IRC12:44
*** pcaruana has quit IRC12:45
*** gongysh has joined #openstack-infra12:46
*** ildikov has quit IRC12:48
openstackgerritMerged openstack-infra/project-config: Move Manila experimental job to check pipeline
*** fesp has quit IRC12:51
*** |-paul-| has quit IRC12:52
*** e0ne_ has joined #openstack-infra12:53
*** asettle has joined #openstack-infra12:53
*** e0ne has quit IRC12:54
*** pblaho has joined #openstack-infra12:55
*** baoli has joined #openstack-infra12:55
*** fhubik_real has joined #openstack-infra12:56
*** Guest8646 is now known as zeus12:56
*** zeus has quit IRC12:56
*** zeus has joined #openstack-infra12:56
*** fhubik has quit IRC12:57
*** hdd has joined #openstack-infra12:57
*** rhallisey has joined #openstack-infra12:57
*** dizquierdo_afk has quit IRC12:58
*** asettle has quit IRC12:58
*** jpr has joined #openstack-infra12:59
*** yamahata has quit IRC13:00
*** pcaruana has joined #openstack-infra13:00
*** dprince has joined #openstack-infra13:01
*** rossella_s has quit IRC13:02
*** cj has quit IRC13:03
*** sarob has joined #openstack-infra13:03
*** rossella_s has joined #openstack-infra13:03
*** cj has joined #openstack-infra13:05
*** EricGonczer_ has joined #openstack-infra13:05
*** esikachev has joined #openstack-infra13:07
*** sarob has quit IRC13:08
*** Daisy has joined #openstack-infra13:09
*** Daisy has quit IRC13:11
*** Daisy has joined #openstack-infra13:11
*** Hal has joined #openstack-infra13:12
*** kgiusti has joined #openstack-infra13:12
*** Hal is now known as Guest9906513:12
*** pradk has joined #openstack-infra13:13
*** Guest99065 has quit IRC13:14
openstackgerritDavanum Srinivas (dims) proposed openstack-infra/project-config: Grab upper-constaints file for oslo master jobs
*** Daisy has quit IRC13:15
*** lucas-hungry is now known as lucasagomes13:17
openstackgerritMerged openstack-infra/tripleo-ci: Kill CI job if it doesn't get a testenv quickly
*** larainema has joined #openstack-infra13:18
openstackgerritgordon chung proposed openstack/requirements: bump upper constraint for ceilometerclient
*** edmondsw has joined #openstack-infra13:21
*** hamzy has joined #openstack-infra13:21
*** zeih has joined #openstack-infra13:22
*** cj has quit IRC13:24
*** zeih has quit IRC13:25
*** zeih has joined #openstack-infra13:25
*** doug-fish has joined #openstack-infra13:26
*** mtanino has joined #openstack-infra13:27
*** doug-fish has quit IRC13:27
*** doug-fish has joined #openstack-infra13:27
openstackgerritAlexis Lee proposed openstack/requirements: WIP oslo.config>=3.9.0
*** moravec has quit IRC13:31
*** cj has joined #openstack-infra13:33
openstackgerritDerek Higgins proposed openstack-infra/project-config: Reduce max-servers for tripleo rh1
derekhSmall nodepool config tweak if anybody has a minute to take a look ^^13:36
*** acabot has quit IRC13:38
*** dims has quit IRC13:38
*** ogelbukh_ has joined #openstack-infra13:39
*** dims has joined #openstack-infra13:39
*** fhubik_real has quit IRC13:39
*** vgridnev has joined #openstack-infra13:40
anteayalennyb: thank you13:40
*** cj has quit IRC13:41
*** EricGonczer_ has quit IRC13:43
*** gongysh has quit IRC13:43
*** burgerk has joined #openstack-infra13:43
*** fhubik_real has joined #openstack-infra13:44
*** Daisy has joined #openstack-infra13:45
*** sarob has joined #openstack-infra13:46
lennybanteaya: np, I just wasn't aware of the issue and was unreachable yesterday.13:48
*** links has quit IRC13:48
AJaegerdims: do you have some mintues to discuss your upper-constraints change?13:49
*** degorenko is now known as _degorenko|afk13:49
dimsAJaeger : sure, i was looking for why the file does not get downloaded in my job and was trying to see if this was an option13:50
*** acabot has joined #openstack-infra13:50
*** markvoelker has joined #openstack-infra13:50
*** vgridnev has quit IRC13:50
AJaegerIn general we download the file as part of setup, see
AJaegerzuul-git-prep-upper-constraints sets up the constraints file in the location for tox to use13:51
*** sarob has quit IRC13:51
*** vgridnev has joined #openstack-infra13:51
dimsAJaeger : ah so i should stick in zuul-git-prep-upper-constraints somewhere?13:51
AJaegercan we do something similar for your job? We might need to do this in a generic way - since other integration tests need it as well13:51
dimsAJaeger : yes of course13:51
AJaegerdims: I'm currently looking at options, not yet on the drawing board ;)13:52
*** rfolco_ has joined #openstack-infra13:52
dimsthanks AJaeger : i've W-1'ed my review13:52
AJaegerdims: you can change gerrit-git-prep to the zuul variant13:53
*** rfolco has quit IRC13:53
AJaegerI mean replace branch-git-prep with a zuul variant13:53
AJaegerdims: might need some more discussion with sdague, clarkb or fungi.13:54
dimsAJaeger : here right?
dimsAJaeger : i'll still need to be able to specify branch?13:55
AJaegerand then use zuul-git-branch-prep-upper-constraints with master13:55
anteayalennyb: I've forwarded you the email mriedem tried to send you yesterday, did you recieve it?13:55
*** placko_ has quit IRC13:56
AJaegerdims: you still might need to change the shell script to link run-tox together with this...13:56
dimsAJaeger : ok. will wait for a bit for clarkb and fungi13:57
AJaegerperhaps copy the file around? But that's my suggestion...13:57
AJaegerthanks, dims !13:57
*** fhubik_real is now known as fhubik|rdotest13:57
pabelangeryolanda: So the failures are a result of IO slowness in vexxhost13:57
AJaegerwe could also pass UPPER_CONSTRAINTS_FILE in and only set to $(pwd)/upper-constraints.txt if its not set13:57
pabelangerI'm about to check logstash and see how many are passing and failing in vexxhost13:57
*** ddieterly has joined #openstack-infra13:58
AJaegeryolanda, pabelanger, anteaya: and are 5 out of the last 6 bare-trusty node changes (the final one I'll send once those two are in). If you have time to review this sometime this week, it would be great13:59
*** mriedem has joined #openstack-infra13:59
pabelangerAJaeger: checking13:59
*** salv-orlando has joined #openstack-infra13:59
yolandaAJaeger ok13:59
*** jaosorior has quit IRC13:59
*** rlandy has joined #openstack-infra14:01
*** EricGonczer_ has joined #openstack-infra14:02
anteayaAJaeger: looks like you have what you need14:02
openstackgerritSergey Belous proposed openstack-infra/project-config: Initial job for Nova with os-vif
AJaegerthanks, yolanda, anteaya and pabelanger!14:03
*** keedya has joined #openstack-infra14:03
*** burgerk has quit IRC14:03
*** openstackgerrit has quit IRC14:03
*** openstackgerrit_ has joined #openstack-infra14:03
*** akshai has joined #openstack-infra14:04
*** amrith is now known as _amrith_14:04
*** denisra_ has quit IRC14:04
*** ifarkas has quit IRC14:04
*** openstackgerrit_ is now known as openstackgerrit14:04
*** openstackgerrit has quit IRC14:04
*** openstackgerrit_ has joined #openstack-infra14:05
*** openstackgerrit_ is now known as openstackgerrit14:05
*** openstackgerrit has quit IRC14:05
*** openstackgerrit_ has joined #openstack-infra14:06
*** hdd has quit IRC14:06
*** Hal has joined #openstack-infra14:06
*** openstackgerrit_ is now known as openstackgerrit14:06
*** openstackgerrit has quit IRC14:07
*** Hal is now known as Guest4067214:07
*** openstackgerrit_ has joined #openstack-infra14:07
AJaegerpoor openstackgerrit ;(14:07
*** gongysh has joined #openstack-infra14:07
*** openstackgerrit_ is now known as openstackgerrit14:08
*** openstackgerrit has quit IRC14:08
*** openstackgerrit_ has joined #openstack-infra14:08
*** openstackgerrit_ is now known as openstackgerrit14:09
*** openstackgerrit has quit IRC14:09
*** openstackgerrit_ has joined #openstack-infra14:09
*** amotoki has quit IRC14:10
*** openstackgerrit_ is now known as openstackgerrit14:10
*** Guest41345 has joined #openstack-infra14:10
*** hashar has quit IRC14:10
*** hashar has joined #openstack-infra14:11
*** burgerk has joined #openstack-infra14:12
silehtAJaeger, hi, I got a issue with the new ubuntu-trusty image, the locale seems not the same as base-trusty, making some telemetry tests that loads yaml files to fail14:12
*** amitgandhinz has joined #openstack-infra14:13
silehtAJaeger, it works on python2.7 but we have trouble with python3.4 jobs14:13
AJaegersileht: we discussed this on the channel here yesterday but I don't remember the final conclusion.14:14
* sileht will dig into channel logs14:14
AJaegerMy personal statement: IF your test is dependent on a specific sort/locale, your test should set it.14:14
AJaegersileht: otherwise, it might fail as well for any developer...14:14
silehtAJaeger, agreed14:15
AJaegersileht: So, sorry for the breakage - but can you make your test robust on this, please?14:15
*** nwkarsten has joined #openstack-infra14:15
*** nwkarsten has quit IRC14:15
silehtAJaeger, the main issue is under underlying dep, I will continue to investigate14:16
*** kushal has quit IRC14:16
*** nwkarsten has joined #openstack-infra14:16
*** claudiub has joined #openstack-infra14:16
AJaegerfungi, anteaya: Was there any solution that we want to do on the infra side for this?14:16
pabelangersileht: AJaeger do you have a URL to the failing test?14:16
lennybanteaya: yes. I will catch him later14:16
anteayalennyb: did you recieve the email?14:17
anteayaAJaeger: there was a long discussion on the topic yesterday, as you point out14:17
anteayaand as sileht mentioned there seems to be a locale default difference between the two images14:17
lennybanteaya: yes, I received the email. thank you.14:18
openstackgerritAttila Darazs proposed openstack-infra/tripleo-ci: Use IPv6 on the ceph gate job
pabelangersileht: you likely need to set en_US.utf8 for your locale:
anteayaif there was a final conclusion I lost it in the noise, so yes sileht's decision to look at the logs is the one I would support14:18
anteayalennyb: great thanks14:18
silehtpabelanger, as AJaeger said, tests should work in any locale14:18
*** salv-orlando has quit IRC14:19
AJaegersileht: I said: Work in any developers/image locale. So, if your test is locale dependent, set it to well defined value as pabelanger suggests14:19
anteayaI was reading backscroll at the time of the conversation and still couldn't follow the conclusion if any14:19
*** jpr has quit IRC14:19
pabelangersileht: right, but we don't set any default now in ubuntu-trusty14:19
*** ajmiller has joined #openstack-infra14:19
silehtAJaeger, yeah if the system doesn't have the defined locale it will fallback to C and I will got the same issue14:20
AJaegersileht: are you setting a locale?14:20
anteayathat was a fast nova meeting14:20
anteaya<-- breakfast14:20
*** sheeprine has quit IRC14:20
AJaegeranteaya: enjoy!14:20
silehtAJaeger, no and I don't want to14:20
clarkbsileht: then your tests need to work against a C locale14:21
silehtAgreed, I will fix the lib that cause issue14:22
clarkbthe rough conclusion yesterday was we will set a utf8 locale by default, but as you say tests that actually care about utf8 or sort order shpuld explicitly say so14:22
clarkbAJaeger: ^14:22
*** mtanino has quit IRC14:22
yolandapabelanger, clarkb, i fix some fixes to ubuntu-minimal to set a locale
*** nmagnezi has quit IRC14:23
AJaegerclarkb: thanks14:23
*** EricGonc_ has joined #openstack-infra14:23
AJaegerclarkb: good morning, you're up early -everything ok?14:23
yolandahowever the change is not passing tests, due to a bug in tempest14:23
clarkbAJaeger: ya and mostly not here yet :)14:24
mordredclarkb: aw - I was going to start poking you for reviews :)14:25
*** Daisy has quit IRC14:25
mordredpabelanger: how about you - you wanna +A this? ? :)14:25
*** ildikov has joined #openstack-infra14:25
AJaegermordred: that has a +A already, hasn't it...14:26
*** Daisy has joined #openstack-infra14:26
AJaegermordred: you first need to merge as dependent change14:26
*** salv-orlando has joined #openstack-infra14:26
pabelangermordred: sure, checking14:26
*** nmagnezi has joined #openstack-infra14:26
AJaegerWow, Zuul: "Queue lengths: 15292 events, 295 results. "14:27
mordredpabelanger: it's actually that needs help14:27
*** EricGonczer_ has quit IRC14:27
mordredAJaeger: good point14:27
*** nwkarste_ has joined #openstack-infra14:28
clarkbmordred: shade-launch-node is a thing now too14:28
*** dalgaaf has joined #openstack-infra14:28
clarkbwe might want to delete old launch node or rename the shade one14:28
*** dims has quit IRC14:29
* yolanda is currently using shade-launch-node successfully14:29
pabelangerYay for playbooks14:29
*** moravec has joined #openstack-infra14:29
*** salv-orlando has quit IRC14:29
mordredclarkb: yup. that's 25106314:29
mordredclarkb: I kinda went on a tear yesterday because there was still parts of the old code trying to import novaclient v1_1 which doesn't exist anymore14:30
mordredclarkb: seemed like time to just finish the job14:30
fungiAJaeger: sileht: i believe yolanda has a patch up to address it in diskimage-builder. the biggest issue is that the default locale provided by ubuntu's glibc in trusty was not a utf-8 locale, but it does have an available utf-8 locale so we're going to see about setting the environment default to that on new images14:30
*** Daisy has quit IRC14:30
*** ociuhandu has joined #openstack-infra14:30
yolandai set that to C.UTF-8 ... is waiting for a fix in tempest to pass tests, and review from cores14:30
*** nwkarsten has quit IRC14:31
*** gongysh has quit IRC14:32
*** akrivoka has joined #openstack-infra14:32
anteayaclarkb: thanks that helps me14:32
AJaegerfungi, yolanda: Thanks!14:32
odyssey4meIt seems like the zuul queue lengths are just climbing and climbing14:33
AJaegerodyssey4me: impressive, isn't it?14:33
*** xyang1 has joined #openstack-infra14:34
openstackgerritMerged openstack-infra/project-config: Change jobs to use ubuntu-trusty
*** nwkarste_ has quit IRC14:34
anteayafungi: inbound zuul traffic is spiking, a runaway ci commenting?
*** nwkarsten has joined #openstack-infra14:34
AJaegeryolanda: is manual JJB run on jenkins02 finished?14:34
anteayaAJaeger: thanks for the alert on the zuul queue lengths14:34
yolandayes, it finished14:34
openstackgerritMerged openstack-infra/project-config: gerrit-plugin-jobs: Use ubuntu-trusty
AJaegerthanks, yolanda14:35
AJaegeranteaya: thanks for digging into it14:35
*** dims has joined #openstack-infra14:35
*** christian_ has joined #openstack-infra14:36
*** tonytan4ever has joined #openstack-infra14:36
anteayawell hopefully just finding what fungi can dig into14:36
*** zz_dimtruck is now known as dimtruck14:36
anteayathat was the only anomoly I see on the zuul cacti graphs14:36
yolandaAJaeger, glad to help. job config history in jenkins02 looks much better now14:36
*** asettle has joined #openstack-infra14:37
*** _ari_ has joined #openstack-infra14:37
fungiesikachev: you said you have some problem with "trigger by time" earlier that i don't see addressed by anyone yet. care to elaborate?14:37
anteayafungi: do we have a runaway ci commenting causing long zuul queue lengths?14:38
fungiyolanda: rcarrillocruz: jjb had been manually run very recently on jenkins02 (all of the jenkins masters in fact). it's been less than 48 hours14:38
yolandafungi so jenkins02 this morning was buggy again, it didn't pick the utf8 changes fro yesterday14:39
yolandaso i had to do again today14:39
anteayaqueue lengths have decreased AJaeger14:39
AJaegerindeed, back to normal, anteaya14:39
AJaegerdrastic - from 20k to 10 in seconds...14:39
anteaya20K is very large though14:40
anteayaI'm used to 3k or so14:40
fungidims: AJaeger: you mention wanting to discuss some upper-constraints change with me, clarkb or sdague but don't mention what the change is... details?14:40
* AJaeger never saw it over 10k before. 2k is normal...14:40
anteayaAJaeger: yeah me neither14:41
AJaegerfungi: is the change14:41
*** pcaruana has quit IRC14:41
dimsfungi : this one
AJaegerbackground: nova (and glance, neutron*) use constraints in their tox.ini file14:41
AJaegerBut integration tests do not setup the constraints file at all ;(14:41
dimsfungi : i am seeing failures in oslo master periodic jobs14:41
*** sheeprine has joined #openstack-infra14:41
AJaegerdims run into this with the olso-master periodic job on nova14:41
AJaegerSo, question is how to setup constraints for integration tests in the best way?14:42
AJaegeroslo master is the first one to run into it, others will follow for sure...14:42
dimsAJaeger : these are py27/py34 jobs (periodic-nova-py27-with-oslo-master)14:42
*** asettle has quit IRC14:43
AJaegerfungi, it's periodic-{name}-{python}-with-oslo-master job -defined in jenkins/jobs/python-jobs.yaml14:44
*** masayukig has quit IRC14:45
*** masayukig has joined #openstack-infra14:47
fungidims: AJaeger: yep, looks like we're already doing that in other periodic jobs14:47
fungiAJaeger: dims: oh, you're worried that the constraints file will cause pip not to install libs from source?14:48
*** tonytan4ever has quit IRC14:48
fungii believe constraints are ignored if you include a requirement from local file path, url, et cetera14:49
*** kevinsho_ has joined #openstack-infra14:49
AJaegerfungi: I haven't thought about that one yet ;)14:49
*** tonytan4ever has joined #openstack-infra14:49
openstackgerritMerged openstack-infra/system-config: Retire old
dimsfungi : i think if that change looks good, we should merge and see what happens :)14:49
yolandamordred, are you ok in approving ? the glean hostname fix14:49
*** ajmiller has quit IRC14:49
AJaegerfungi: I was just looking at dims change and didn't consider it correct and wanted to discuss the bigger picture here - but have no energey right now to help with the solution ;/14:49
*** salv-orlando has joined #openstack-infra14:50
openstackgerritMerged openstack-infra/system-config: Zero out the inventory cache instead of removing it
dimsAJaeger : i'll work through it if that review does not help with the situation, i'll dig deeper14:50
*** _amrith_ is now known as amrith14:50
fungiAJaeger: what do you think is incorrect about it?14:50
AJaegerI was thinking of using a zuul setup macro instead of running a wget...14:51
openstackgerritMerged openstack-infra/system-config: Use set-hostnames playbook in launch-node
mordredyolanda: done14:51
fungiAJaeger: oh, in the script?14:51
*** eharney has joined #openstack-infra14:52
AJaegerI meant use zuul-git-branch-prep-upper-constraints and then copy the upper-constraints file around14:52
fungiAJaeger: copy it around to where?14:52
AJaegerto the place that run-tox expects it14:52
*** esikachev has quit IRC14:52
*** esikachev has joined #openstack-infra14:52
fungiAJaeger: doesn't zuul-git-branch-prep-upper-constraints do that already?14:52
anteayamordred yolanda is today the day we get ip addresses on the new 60GB vm? we do need to announce as soon as possible so we can give folks at least 4 weeks to adjust14:52
*** Zhongjun_ has joined #openstack-infra14:52
anteayaand we have set the date for migration as April 11th14:53
yolandaanteaya, shade-launch-node finally worked so i have a vm now with an ip14:53
AJaegerfungi, I suggest to double check that it copies it to the right place14:53
yolandai was just testing it14:53
anteayayolanda: oh how lovely14:53
fungiAJaeger: or are you saying projects that use this particular job have tox.ini files not set up the way other projects do?14:53
anteayayolanda: congratulations!14:53
*** pcaruana has joined #openstack-infra14:53
anteayayolanda: yup, awesome, thank you14:53
yolandakudos to mordred14:53
* anteaya offers mordred kudos14:54
yolandai'm testing the puppet runs, it seems to be giving some problems, but the vm is ready14:54
*** _nadya_ has quit IRC14:54
eharneyianw: did you see my reply on ?  curious what you think14:54
yolandaanteaya, shall it be enough to send an announce to openstack-infra mailing list?14:54
fungiAJaeger: that macro does this: cp $REQS_DIR/upper-constraints.txt ./14:54
*** pradk_ has joined #openstack-infra14:54
anteayayolanda: yes, the ips are all you need to announce14:55
yolandano need to enter something in the wiki, or fix any other documentation ?14:55
fungiAJaeger: so it's copying the file into the top level directory of the project's clone workdir14:55
anteayayolanda: then you/we have 4 weeks to prepare it14:55
AJaegerfungi, I just double checked - I was udner the impression the job changed directories but does not. So, just using the zuul macro should indeed fix this.14:55
anteayayolanda: I haven't seen that we have done that before, what were you thinking?14:55
anteayayolanda: whatever you think we need to tell folks the ips14:55
yolandafirst time that i will do that, so i wanted to be sure14:55
yolandamostly to be sure that the announce in mailing list is not ignored14:56
fungiyolanda: ip address and our tentative maintenance date/time14:56
*** _nadya_ has joined #openstack-infra14:56
*** hashar has quit IRC14:56
*** jaosorior has joined #openstack-infra14:56
anteayayolanda: here is the announcement from last time:
*** _degorenko|afk is now known as degorenko14:56
yolandafungi anteaya, it will be a good idea to announce again every week until we approach to the maintenance window14:56
fungiyolanda: make sure to include both ipv4 and ipv6 addresses of the new server and, for reference, the ones on the current production server14:56
AJaegerfungi: I've been looking to much at integration jobs like jenkins/jobs/bandit.yaml - where there is a cd involved. I expect those jobs to fail the same way like the olso one soon - and those do change directories14:56
anteayayolanda: I don't know as we did that before14:56
*** thiagop has joined #openstack-infra14:57
anteayathe thread before was one post long:
anteayaI'll be sharing the post at the third party meetings14:57
fungiAJaeger: yes, i think some more complex jobs need more complex solutions if they're going to use constraints. for example, devstack jobs take a completely different (and more complicated) approach to using it14:57
yolandaok i'll send something like that14:57
anteayayolanda: awesome, if you want to create a draft in an etherpad we can take a peek before you post, if you like14:58
yolandasounds good14:58
anteayathanks for taking care of this14:58
yolandai did a similar work for Gozer, to happy to do it in upstream this time14:58
openstackgerritMerged openstack-infra/project-config: Reduce max-servers for tripleo rh1
mordredfungi, yolanda: actually - before we send out the email ...14:59
mordredopenstack supports fixed ips that are not tied to a specific server but that are not floating ips15:00
mordredrackspace does not allow us to create them15:00
mordredmaybe we should ask if they'd let us make one15:00
mordredso that we could put it on the new gerrit, but have this be the last time we have to tell people about a new IP?15:00
*** _nadya__ has joined #openstack-infra15:00
*** _nadya_ has quit IRC15:01
anteayamordred: whom could we ask?15:01
mordredI have no idea how hard that would be permissions-wise. maybe I should shut up and we should just do that next time15:01
anteayawell I don't think it would hurt to ask15:01
yolandamordred, and how does it work on terms of a server replacement? you update the ip, moving to the new server you create?15:01
*** mtanino has joined #openstack-infra15:01
anteayaas long as we don't delay announcing waiting on the reply15:01
mordredyolanda: same as with floating ips15:01
yolandaah you associate/disassociate on demand15:02
mordredyolanda: you make the new server, then you detach the port from the old server and attach it to the new server15:02
yolandawho can we talk in rax for that?15:02
mordredjroll: should I ping pvo about that ? ^^15:02
jrollmordred: sup15:02
* AJaeger waits for ages for a "git review" to send changes to our gerrit ;(15:03
jrollmordred: so you want a fixed ip you can move around?15:03
jrollmordred: or just a regular port?15:03
*** dimtruck is now known as zz_dimtruck15:04
*** Jeffrey4l has joined #openstack-infra15:04
openstackgerritMerged openstack-infra/puppet-lodgeit: fix puppet-lodgeit to support latest mysql
*** mrmartin has quit IRC15:06
*** burgerk has quit IRC15:06
mordredjroll: uh. either? I mean, those two things sound the same to my dumb lizard brain15:06
mordredjroll: nova fixed-ip == neutron port best I can tell?15:06
mordredbut yes - would love to allocate an IP that we can add to a server, and then remove from it, and then add to a diffrent server15:07
openstackgerritMerged openstack-infra/project-config: Remove cloudv-ostf-adapter from Infrastructure Systems
*** sigmavirus24_awa is now known as sigmavirus2415:08
openstackgerritMerged openstack-infra/project-config: Remove redundant Manila Tempest job
dmsimardclarkb: so you think the failures in are legit ?15:08
* dmsimard looks15:08
mtreinishinfra-root: when someone gets a chance can they check the logs on health.o.o it looks like something is up with the api server15:09
mordredmtreinish: I blame jaypipes15:09
openstackgerritMerged openstack-infra/project-config: Remove Fedora23 install-glean-from-git flag
mtreinishmordred: heh, works for me :)15:09
openstackgerritAndreas Jaeger proposed openstack-infra/project-config: Remove bare-trusty from projects.yaml
*** zeih has quit IRC15:11
AJaegerthe change above just passed as recheck without any JJB changes in the diff output, so ready to merge!15:11
jrollmordred: I have learned things15:11
*** rbrndt__ has joined #openstack-infra15:12
AJaegerfungi, do you want to review the removal of bare-trusty from projects.yaml? ^15:12
mtreinishmordred: oh, yay it's returning 500 on that status page, that's no good15:12
*** vgridnev has quit IRC15:12
jrollmordred: tl;dr: it's possible to do so, but you have to land in the same cell, among other gotchas, I wouldn't do it15:12
*** tonytan_brb has joined #openstack-infra15:13
*** vgridnev has joined #openstack-infra15:13
openstackgerritAna Krivokapic proposed openstack-infra/project-config: New project for TripleO UI
* jaypipes blames mordred 15:15
*** salv-orlando has quit IRC15:16
*** vgridnev has quit IRC15:16
*** mrmartin has joined #openstack-infra15:16
yolandajroll, mordred, so fixed ip is not a good idea?15:17
*** Guest40672 has quit IRC15:17
mordredjroll: gotcha15:17
*** tonytan4ever has quit IRC15:17
jrollmordred: yolanda: correct, I'd recommend waiting for floating ips15:17
mordredjroll: when rax gets floating ips - it's not going to take away our current lovely and wonderful real public ips is it?15:18
anteayajroll: thanks for the fast turn around on the query15:18
jrollmordred: I don't believe so :)15:18
jrollanteaya: of course :)15:18
mordredjroll: \o/15:18
*** ddieterly is now known as ddieterly[away]15:18
*** Hal has joined #openstack-infra15:18
mordredthe fact that rax does not require floating ips for things is one of my favorite things about it15:18
jrollheh, I know15:19
*** Hal is now known as Guest2015015:19
* mordred just wants to make sure everyone is aware his mind has not magically changed recently15:19
*** zeih has joined #openstack-infra15:19
yolandamordred, jroll ++ for not floating ips :)15:19
anteayamordred: so noted15:19
mordredbut ... you're right, the use case I have with wanting a sticky ip for review.o.o is the one actual useful usecase for floating ips15:19
*** vgridnev has joined #openstack-infra15:19
*** zz_dimtruck is now known as dimtruck15:19
mordredit's rare - less than 1% of our servers need it - but it _is_ a useful thing for that 1%15:19
jrollmordred: I even believe that floating IPs will not be implemented via NAT, but am not 100% sure on that15:19
mordredjroll: WOOT15:20
mordredjroll: that makes your floating ips useful15:20
mordredwhoever thought NAT for servers was a feature should be very ashamed of themselves15:20
fungiAJaeger: i do want to review that, yes. i'll add it to my priority list for this morning15:20
*** esikachev has quit IRC15:20
fungistill trying to catch up on e-mail15:20
AJaegerfungi: no urgency on that - just happy to see that progress ;)15:21
AJaegeronce that's in, we need to review whether infra-manual needs an update as well...15:21
*** ddieterly[away] is now known as ddieterly15:21
*** vgridnev has quit IRC15:22
AJaegerI can ssh to but I cannot push a change using "git review -d". Any ideas?15:22
ClintAJaeger: push with -d?15:22
fungijroll: yes! for ages i've wanted nova to support binding additional arbitrary addresses to instance network interfaces directly (aliases in the traditional term) rather than only supporting solving that use case through network address translation15:22
*** jaosorior has quit IRC15:23
jrollfungi: :)15:23
AJaegerClint: "git review -y"15:23
*** vgridnev has joined #openstack-infra15:23
*** ajmiller has joined #openstack-infra15:23
AJaeger(those patches on top of each other)15:23
fungii used to attach multiple ip addresses to network interfaces in virtual machines all the time, but that was in the dark ages where i managed my virtual machines by hand15:23
anteayaI don't know the -y flag15:23
openstackgerritAndreas Jaeger proposed openstack-infra/project-config: Remove final bare-trusty job
AJaegeranteaya: don't ask for "yes" if you have stacked changes15:24
AJaegerAh, that one works on its own15:24
*** tsg has joined #openstack-infra15:24
jeblair30532 opensta+  20   0 3790004 3.184g   7964 S 100.4 82.6 613:11.88 apache215:25
*** dizquierdo has joined #openstack-infra15:25
openstackgerritAndreas Jaeger proposed openstack-infra/project-config: Remove bare-trusty from nodepool
AJaegerBye, bye bare-trusty! ^15:25
openstackgerritAttila Darazs proposed openstack-infra/tripleo-ci: Use IPv6 on the ceph gate job
AJaegerso, I manged to push the two patches through as single ones...15:25
EmilienMhello infra, would it be possible to land this patch about login.defs on redhat systems? thanks
fungiAJaeger: wow, that change deletes 758 lines! i have a feeling we need to update the examples in the infra manual too15:26
jeblair[Thu Mar 10 15:22:07.549262 2016] [:error] [pid 30532:tid 139826121930496] [remote] OperationalError: (pymysql.err.OperationalError) (2013, 'Lost connection to MySQL server during query') [SQL: u'SELECT runs.run_at AS runs_run_at, runs.passes AS runs_passes, runs.fails AS runs_fails, runs.skips AS runs_skips, run_metadata.value AS run_metadata_value \\nFROM runs INNER JOIN run_metadata ON = ...15:26
jeblair... run_metadata.run_id \\nWHERE run_metadata.`key` = %(key_1)s AND runs.run_at >= %(run_at_1)s AND runs.run_at <= %(run_at_2)s'] [parameters: {u'key_1': u'project', u'run_at_1': datetime.datetime(2016, 2, 10, 14, 52, 3, 165000, tzinfo=tzlocal()), u'run_at_2': datetime.datetime(2016, 3, 10, 14, 52, 3, 165000, tzinfo=tzlocal())}]15:26
mtreinishjeblair: hmm, I would have expected it to recover from a loss in connectivity to the db15:27
jeblairmtreinish: there are several of those, that's the most recent15:27
jeblairmtreinish: and did you see the line from top above ^15:27
jeblair[Thu Mar 10 14:58:33.988192 2016] [cache:info] [pid 31777:tid 139825881478912] [client] AH00780: Cache lock file for '/runs/key/build_queue/gate/recent' too old, removing: /tmp/mod_cache-lock/b/Q/bQW2ktpSMvVsBFA0rNFeJw, referer:
pabelangerAJaeger: \o/15:28
mtreinishjeblair: about the memory consumption?15:28
jeblairmtreinish: and cpu consumption15:28
openstackgerritMerged openstack-infra/system-config: Decouple openstack_project::server from puppetdb class
*** Daisy has joined #openstack-infra15:29
openstackgerritMerged openstack-infra/system-config: Add periodic pipeline test result to infra docs
openstackgerritAndreas Jaeger proposed openstack-infra/infra-manual: Remove bare-trusty lines
AJaegerfungi, just two lines for infra-manual15:30
mtreinishjeblair: hmm, I kinda wish there was a big obvious stack trace in the logs. That's easier to debug :)15:30
*** aysyd has quit IRC15:30
mtreinishI'm not quite sure why it's going off the rails15:30
*** kingia_ has joined #openstack-infra15:30
*** aysyd has joined #openstack-infra15:30
jeblairmtreinish: there are several of those, but i haven't yet correlated them with the timeouts, which are more recent.15:31
fungiAJaeger: yep, figured it would be simple, but it reduces the number which will get reintroduced through copying (not by much though, since people mostly seem to copy project creation changes they or others submitted in the past rather than cribbing from the instructions each time)15:31
mtreinishjeblair: well, at the very least I guess that's evidence that mod_cache is doing something. I wasn't 100% that was actually configued correctly :)15:32
yolandamordred then we leave the external ip for another time?15:32
jeblair[Thu Mar 10 07:55:41.802144 2016] [:error] [pid 30532:tid 139826138715904] [remote] mod_wsgi (pid=30532): Exception occurred processing WSGI script '/opt/openstack-health/.venv/bin/openstack-health'.15:32
*** eharney has quit IRC15:32
jeblairmtreinish: first timeout ^15:32
*** kingia has quit IRC15:33
mtreinishjeblair: ok, yeah that makes more sense15:33
mtreinishdoes it log the exception anywhere?15:33
*** aorourke-afk is now known as aorourke15:34
mordredyolanda: yes. let's continue with the current plan15:34
openstackgerrityolanda.robla proposed openstack/diskimage-builder: Generate fedora-atomic images using dib
*** burgerk has joined #openstack-infra15:35
*** eharney has joined #openstack-infra15:35
openstackgerritAna Krivokapic proposed openstack-infra/project-config: New project for TripleO UI
openstackgerritMerged openstack-infra/project-config: Fix oslo master jobs failures - missing upper constraints
mtreinishjeblair: hmm, so it lost the connection to the db and that query failed15:36
mtreinishI'd expect that 1 request to get a 500 and then for everything to move on15:37
mtreinishI'm kinda surprised that's enough to bring everything down15:38
openstackgerritMonty Taylor proposed openstack-infra/system-config: Remove interpret_puppet_exitcodes
*** IlyaG has joined #openstack-infra15:38
mtreinishjeblair: like it's returning 500 on and those don't even make db queries15:39
jeblairmtreinish: i expect all of its threads are busy15:39
*** Qiming has quit IRC15:39
mtreinishah, that could do it15:40
mtreinishjeblair: is there still a connectivity issue to the db from health.o.o?15:40
jeblairthere's basically no network traffic right now...15:40
*** sigmavirus24 is now known as sigmavirus24_awa15:40
*** cody-somerville has joined #openstack-infra15:41
jeblairmtreinish: i can connect to the port via telnet.  there is no mysql client installed on that host.15:41
*** Daisy has quit IRC15:42
*** darvon has quit IRC15:42
mtreinishhah, I guess that's a disadvantage of using pymysql, mysql never gets installed on the host because you don't need the c bindings anymore15:42
*** Daisy has joined #openstack-infra15:42
*** pfallenop has quit IRC15:43
jeblairi connected from a different host, mysql is idle15:43
openstackgerritAna Krivokapic proposed openstack-infra/project-config: New project for TripleO UI
*** vgridnev has quit IRC15:44
mtreinishjeblair: right, I've been able to connect locally since I woke up ~1hr ago. I was thinking it might have been a connectivity issue more local to the api server15:44
jeblairmtreinish: did you install a thread-dump handler by any chance?15:44
mtreinishjeblair: I'm tempted to just say lets just restart apache and cross our fingers this doesn't happen again15:44
mtreinishjeblair: I don't think so15:44
*** sdake has joined #openstack-infra15:44
jeblairthat would tell us what the threads are doing... right now strace just says they are all waiting on futexes15:45
mtreinishjeblair: if you have a package name in mind I can add it to the puppet policy for the api server15:45
*** darvon has joined #openstack-infra15:45
*** pfallenop has joined #openstack-infra15:45
jeblairthere are several connections to the mysql server that are in close_wait15:45
jeblairmtreinish: for thread dumping?  i mean in the program.  we have them in nodepool and zuul.15:46
*** sigmavirus24_awa is now known as sigmavirus2415:46
*** sdake_ has joined #openstack-infra15:46
jeblaircould probably copy it pretty easily, it's fairly standalone15:47
mtreinishjeblair: oh, flask might have something builtin, but I didn't put anything explicitly in the api server15:47
*** esikachev has joined #openstack-infra15:47
yolandainfra-root (or any interested) that is a draft of the email to send for gerrit ip: . Any feedback?15:48
dmsimardclarkb: found the issue for the swap patch .. fixing15:49
jeblairmtreinish: i'm out of ideas without further access to the internal state.  i can restart now if you want.15:49
*** dimtruck is now known as zz_dimtruck15:49
*** sdake has quit IRC15:49
*** baoli has quit IRC15:50
mtreinishjeblair: yeah go for it. I'll work on adding the thread dump to the api server should this happen again15:50
*** ildikov has quit IRC15:50
*** baoli has joined #openstack-infra15:51
sdagueoh... hey, I just discovered this in the gerrit docs -
jeblairenv: htcacheclean: No such file or directory15:51
jeblairmtreinish: ^ that is emitted when it starts15:51
*** angdraug has joined #openstack-infra15:52
sdagueseems like if we specify that correctly bug:1523668 would be a valid search bit15:52
*** yamahata has joined #openstack-infra15:52
*** zz_dimtruck is now known as dimtruck15:53
* yolanda needs to step out for a while15:53
mtreinishjeblair: hmm, I haven't seen that before. Shouldn't the apache package install that?15:53
jeblairmtreinish: apparently it's in the apache2-utils package15:54
jeblairwhich is not installed15:54
*** xarses has joined #openstack-infra15:54
pabelangermordred: thanks for the 1.5.1 release of shade, it fixed some packaging issues I was having for fedora :)15:56
pabelangerI should finally be able to push this up to rawhide today or tomorrow15:56
openstackgerritMatthew Treinish proposed openstack-infra/puppet-openstack_health: Add apache2-utils package to api server
mtreinishjeblair: ^^^15:58
*** mrmartin has quit IRC15:59
openstackgerritBen Swartzlander proposed openstack-infra/project-config: Give manila-image-elements its own acls
*** jistr has quit IRC15:59
*** Jeffrey4l has quit IRC16:00
*** armax has joined #openstack-infra16:00
openstackgerritZara proposed openstack-infra/storyboard-webclient: WIP task notes
*** jtomasek has quit IRC16:00
openstackgerritMatt Riedemann proposed openstack/requirements: Exclude python-troveclient 2.1.0
*** mrmartin has joined #openstack-infra16:02
mordredpabelanger: woot!16:03
*** esikachev has quit IRC16:03
anteayayolanda: I have added my thoughts to the etherpad, thank you16:03
*** pradk has quit IRC16:06
*** pradk_ is now known as pradk16:06
*** oomichi_ has joined #openstack-infra16:06
*** esikachev has joined #openstack-infra16:07
*** oanson has quit IRC16:09
*** sdake_ is now known as sdake16:10
openstackgerritMerged openstack-infra/system-config: Upgrade grafana to 2.6.0
openstackgerritZara proposed openstack-infra/storyboard-webclient: Update Fontawesome to 4.5.0
*** kmartin has joined #openstack-infra16:16
*** zeih has quit IRC16:17
*** hashar has joined #openstack-infra16:17
mtreinishjeblair: did you just manually install apache2-utils to get it going?16:17
openstackgerritJames E. Blair proposed openstack-infra/system-config: Add storyboard exim router/transports
jeblairmtreinish: no, i ignored it16:17
jeblairmtreinish: it did not seem to be a fatal error16:17
*** oomichi_ has quit IRC16:18
*** oomichi_ has joined #openstack-infra16:19
mtreinishah, ok16:19
fungiyolanda: lgtm other than including the current ip addresses, which i just added (this makes it easier for people who have to submit requests to their corporate firewall overlords to tell them what ip addresses are replacing what other ip addresses)16:19
*** arxcruz has quit IRC16:20
*** esikachev has quit IRC16:21
jeblairanyone know why we still have two entries for the health dashboard on the status site?16:22
*** baoli has quit IRC16:23
*** tonytan_brb has quit IRC16:23
jeblairah here it is
*** pblaho has quit IRC16:24
*** tonytan4ever has joined #openstack-infra16:24
*** bpokorny has joined #openstack-infra16:24
openstackgerritAttila Darazs proposed openstack-infra/tripleo-ci: Use IPv6 on the ceph gate job
*** christian_ has quit IRC16:26
*** ZZelle has quit IRC16:27
*** EricGonc_ has quit IRC16:28
*** tonytan4ever has quit IRC16:28
*** tonytan4ever has joined #openstack-infra16:28
*** ildikov has joined #openstack-infra16:29
anteayaclarkb: when next you are about, logstash 2.016:29
*** baoli has joined #openstack-infra16:29
anteayaclarkb: I'm wondering if there is any way we can get to kibana 4 by summit16:29
*** sc68cal has joined #openstack-infra16:30
anteayaI don't dare to dream yet, just wondering what you think is possible16:30
*** nwkarsten has quit IRC16:30
*** nwkarsten has joined #openstack-infra16:30
*** tzn has quit IRC16:31
*** sc68cal_ has joined #openstack-infra16:31
openstackgerritMerged openstack-infra/system-config: Remove QA health link from status page
*** _nadya__ has quit IRC16:32
*** amitgandhinz has quit IRC16:32
*** nwkarste_ has joined #openstack-infra16:32
*** amitgandhinz has joined #openstack-infra16:32
*** harlowja_at_home has joined #openstack-infra16:33
*** mrmartin has quit IRC16:34
*** annegentle has joined #openstack-infra16:34
*** nwkarsten has quit IRC16:35
openstackgerritMerged openstack-infra/project-config: Build AFS RPMs for Centos7 periodically
*** jlanoux has quit IRC16:36
SamYapleclarkb: for this fstrim patch, based on what the comments were in teh review it seems like _only_ presenting fstrim and not zerofree would be prefered, in which case i think the cleanest thing would be a revert of the zerofree patch16:37
SamYapleyour thoughts?16:37
*** mikelk has quit IRC16:37
openstackgerritMerged openstack/gertty: Use diff long options and uncap GitPython
openstackgerritMerged openstack/gertty: Add config option for git clone URL
clarkbSamYaple: I dont think there is anything in the zerofree patch that needs to survive without zerofree so that should be fine16:38
SamYapleyea i checked it over, i agree with you there16:38
SamYapleok will do16:38
*** vincentll has quit IRC16:39
openstackgerritJames E. Blair proposed openstack/gertty: Add configuration information to docs
openstackgerritMerged openstack/gertty: Make topic selection a select list
openstackgerritMerged openstack/gertty: Indent projects under topics
*** jpr has joined #openstack-infra16:39
clarkbanteaya: kibana4 work can happen independent of logstash 2.0 I just dont have the time to do both at the same time. I do however intend on getting logstash 2.0 in real soon now. so we shall see16:39
openstackgerritMerged openstack/gertty: Support batch abandon/restore
*** Daisy has quit IRC16:39
openstackgerritMerged openstack/gertty: Handle more than one change result when searching
nibalizeryolanda: that blurb looks good to me16:39
*** Daisy has joined #openstack-infra16:40
*** flepied has quit IRC16:40
*** rfolco has joined #openstack-infra16:40
jeblairpabelanger: would you please reconsider ?16:40
anteayaclarkb: is there something like an etherpad or storyboard in place such that I can understand the current status and figure out how to help, or can I compose such a thing?16:41
*** sigmavirus24 is now known as sigmavirus24_awa16:42
*** sigmavirus24_awa is now known as sigmavirus2416:42
pabelangerjeblair: WFM16:43
*** rfolco_ has quit IRC16:43
clarkbanteaya: just the changes in review16:43
anteayaclarkb: do they have topics?16:43
anteayaif yes, what are they?16:43
pabelangerjeblair: left comment about git.o.o, that what my only concern16:44
jeblairpabelanger: i agree, it would be nice.16:44
*** Daisy has quit IRC16:44
anteayaclarkb: I only see two patches with logstash-2.0 as the topic:
clarkbanteaya: looks like 'logstash' and 'logstash-2.0'16:45
anteayathank you16:45
anteayaclarkb: any objection to me creating a storyboard for these?16:45
anteayaif only for my own benefit?16:45
clarkbthere are only 3 changes total, just need to get them applied16:45
*** sridhar_ram has joined #openstack-infra16:45
anteayafor kibana 4 as well?16:45
pabelangerwon't mind a review of Specifically the manually installation of wiki extensions, I left a comment explaining what needs to be done moving forward16:45
clarkbanteaya: sure, though there isnt much to say other than apply changes16:46
clarkbkibana 4 is a completely unknown entity at this point16:46
openstackgerritMerged openstack-infra/devstack-gate: Generate a log with deprecation warnings
anteayaoh okay, thanks16:46
*** e0ne_ has quit IRC16:46
*** pcaruana has quit IRC16:47
markvanAJaeger: ping16:48
*** ihrachys has quit IRC16:48
*** scheuran has quit IRC16:49
anteayaclarkb: looks like yolanda is leaving for you to approve and babysit16:49
anteayamarkvan: AJaeger prefers if you just ask your question16:49
openstackgerritJames E. Blair proposed openstack-infra/system-config: Add storyboard exim router/transports
markvananteaya: k, thx.   Looking for help on a job-template question.   Is it possible to define different job timeouts for templated jobs?  Example anywhere?16:50
fungiAJaeger: did you have a change proposed to replace gate-infra-puppet-apply-bare-trusty with gate-infra-puppet-apply-ubuntu-trusty? i'm also noticing we're singling out the centos7 version of that job with a different job-template which i'm looking into now to see if it can be simplified16:52
anteayacan you share which job template you are looking to change?16:52
markvananteaya: sure, it's this one in Heat:   Looks like LBaaSv2 increases devstaack deploy by 10-15min and then testing are almost 5min16:53
*** regXboi has joined #openstack-infra16:53
markvanso would like to have only lbaasv2 side of that template have a longer timeout 80 -> 12016:54
*** asettle has joined #openstack-infra16:54
clarkbanteaya oh good, though ideally all the changes go in together I can at least get the refactor in first16:54
jeblairwhy does lbaasv2 take so long to deploy?16:55
*** cody-somerville has quit IRC16:55
*** fhubik|rdotest has quit IRC16:55
anteayaclarkb: ah okay16:55
anteayaclarkb: reviewing them now16:55
markvanjeblair: looks like it the octavia_amphora packages and images, but have not dug into that yet16:55
*** flepied has joined #openstack-infra16:56
yolandathanks for the feedback, i'll send now16:56
rm_workwe use DIB to build our amp16:56
*** ashleighfarnham has joined #openstack-infra16:56
rm_workwhich takes... a bit16:56
rm_work^^ octavia devstack build times16:56
rm_workalso we pull in barbican as a dep too16:56
rm_workand neutron and nova and glance etc16:57
anteayayolanda: did you remove the last sentence?16:57
rm_workso large dep list16:57
anteayayolanda: or do you disagree?16:57
rm_workjeblair: ^^16:57
yolandai'll remove16:57
anteayayolanda: thanks16:57
yolandai was used to send emails with that bit at the end :)16:57
jeblairrm_work: do the amphora images need to be rebuilt for all of these jobs, or just certain ones?16:57
markvananteaya: I was thinking I could add a timeout in the project job descriptions and then ref that in the template as {timeout}, would that work?16:58
anteayamarkvan: as for the orginal question, I do not believe we have a mechinism to change timeouts for various variables in a job16:58
jeblairrm_work: the trove folks (and some other projects) have a system where they build images and put them on tarballs.o.o for most jobs to use, then you only need to rebuild them in live tests when you're changing something on them.16:58
*** fhubik|rdotest has joined #openstack-infra16:58
*** asettle has quit IRC16:58
anteayamarkvan: as jeblair is noting, asking to do so might be indicative of needing to solve a problem within on part of them16:58
anteayayolanda: I understand16:59
amrithjeblair, we are moving away from that system16:59
*** fhubik|rdotest has quit IRC16:59
fungiamrith: what did the challenges end up being?16:59
jeblairmarkvan: when we're talking about increasing timeouts because expected run times of jobs are > 1 hour, we should really pause and reflect.16:59
amrithit sounds like a good idea but we've found a lot of challenges with it.16:59
fungiamrith: also do you know if ironic is experiencing similar issues as they also do that?16:59
anteayamarkvan: not that I'm aware and as jeblair is pointing out, this is cause for reflection16:59
amrithfungi: let me write up a longer response and then paste-bomb it.16:59
fungiamrith: ml thread might make more sense in that case16:59
*** moravec has quit IRC17:00
amrithfungi, no. it would be ironic if they were facing the same problem ;)17:00
amrithkidding, could not resist.17:00
amrithI don't know.17:00
yolandaok sent17:00
rm_workjeblair: well17:00
anteayayolanda: thank you17:00
yolandathx for the feedback17:00
rm_workjeblair: so it is something we have tried17:00
amriththe issues have mostly to do with frequency of update17:00
*** ihrachys has joined #openstack-infra17:00
amrithwe used tarballs in two cases; guest images and trove client.17:00
rm_workjeblair: so the images we generate bake in code from our current codebase which any patch could be changing17:01
amrithI'll summarize in a short email17:01
rm_workso we kinda need to build it?17:01
openstackgerritDavanum Srinivas (dims) proposed openstack-infra/devstack-gate: Log deprecations in horizon and keystone
amrithrm_work, we work around that in a different way.17:01
rm_workhmm ok17:01
*** HeOS has quit IRC17:01
openstackgerritMerged openstack/requirements: Cap cliff-tablib <2.0 in stable/kilo
amrithtrove guest images that we use for testing/development that is17:01
*** tzn has joined #openstack-infra17:01
amriththe way we do that is that the guest images have a stub of upstart code that rsync's the guest agent code from the host.17:01
rm_worklike i said, we have tried it, we did download an image from a CDN for a while instead of building17:01
rm_workbut it caused us problems17:01
markvananteaya: jeblair was looking for short term solution to get the basic lbaas v2 heat test up and going since the resources were added.17:02
amriththat way, if I'm making changes to the code base, server and guest, then launching a new instance automagically gets you the latest guest agent.17:02
rm_workamrith: i am sure if you have experience you are willing to share, we would be interested in hearing it17:02
amrithwould love to17:02
rm_workamrith: if you want to talk to us in #openstack-lbaas that would be great17:02
rm_workamrith: specifically johnsom was working on it17:02
amrithsure thing, let me add that group17:02
*** rossella_s has quit IRC17:03
mgagnepabelanger: is the grafana upgrade still in progress?17:03
*** rossella_s has joined #openstack-infra17:03
amrithrm_work, hailing you in #openstack-lbaas17:03
*** lucasagomes is now known as lucas-afk17:04
*** flepied has quit IRC17:04
openstackgerritMerged openstack-infra/system-config: Customize cgit max-repo-count and set it to 1500
*** dims has quit IRC17:04
*** derekh has quit IRC17:05
anteayamarkvan: yes, we are aware, and I believe the point jeblair is making is that in cases such as this the most effcient way of dealing with the issue is to understand the problem rather than attempting to cover it over and creating technical debt17:05
anteayaand if indeed that is his point, I agree with him17:05
openstackgerritMerged openstack-infra/system-config: Cacti: add hosts to alternate graph trees
openstackgerritMerged openstack-infra/storyboard-webclient: Update Fontawesome to 4.5.0
*** dims has joined #openstack-infra17:06
jeblairamrith, rm_work: okay, so we have a pretty flexible system with a lot of options for working through this, and this is the first time i'm hearing about either of the issues.  there are a lot of folks around here who may have creative ideas for how to deal with this, so please do start a mailing list thread and invite infra folks to join.17:06
anteayamarkvan: also given there is no mechinism to do what you want, I don't see there is a short term solution available17:06
*** moravec has joined #openstack-infra17:06
rm_workjeblair: we actually dealt with the timeout and long build issues about 3-4 months ago17:06
markvananteaya: sure I can see that point.  Thinking down the road when devstack starts to enable lbaasv2 as default, could impact many jobs17:06
rm_workdealt with, as in, it was a huge issue for us and neutron and we ended up just cutting out testing down17:07
rm_workand splitting our testing into like 5 different jobs17:07
rm_workif you remember that whole mess17:07
anteayamarkvan: why would devstack enable lbaasv2 by default?17:07
anteayaI don't know as we even have neutron enabled by default in devstack yet17:07
markvananteaya: ok, so my only choice is to create a separate job just for this (it's experiemental right now) with a longer timeout17:07
anteayasomeone correct me if I'm wrong17:07
jeblairmarkvan: what's the expected runtime of the job?17:08
anteayamarkvan: well I have been trying to say, work with the folks who have created the longer runtime to solve that issue17:08
markvananteaya: was thinking as folks move from v1 which is there by default to v2, they will need that enabled.  Not sure on roadmap17:08
anteayamarkvan: that would be something you would have to discuss with the devstack folks17:08
markvanjeblair: right now it's at 80, and looks to have about 10-20 minutes left17:08
anteayaand if lbaasv1 is enabled by default in devstack, that is news to me17:09
jeblairmarkvan: we try to keep jobs under 1 hour17:09
openstackgerritZara proposed openstack-infra/storyboard-webclient: WIP task notes
markvanjeblair: sure, that's reasonable goal17:09
*** vincentfrancoise has joined #openstack-infra17:10
anteayaclarkb: rechecked a few of the patches to get fresh logs17:10
*** gyee has joined #openstack-infra17:10
*** HeOS has joined #openstack-infra17:11
anteayaasselin mmedvede lennyb can you help ensure folks see this email post?
markvananteaya: the current lbaasv2 job is at 130, that's why I was going down this road in the first place for this new heat job.17:11
anteayawhat jeblair said17:12
mmedvedeanteaya: ok, I'll bring it up during meetings17:13
markvanyeah, puts this heat job in a bad place.  I think this heat job is the first one trying to use lbaasv2 outside of the base lbass jobs17:13
anteayammedvede: thank you17:13
jeblairmarkvan: so the heat job should not be changing any amphora code, so it's ripe for being able to use a pre-built image17:13
* sc68cal drops in17:13
anteayasc68cal: welcome17:14
*** abregman is now known as abregman|mtg17:14
markvanjeblair: yup, that sounds like what we need here17:14
*** sc68cal_ has quit IRC17:14
jeblair(even if that's more difficult for lbaasv2 itself, though i suspect there are untried solutions there as well)17:14
*** keedya has quit IRC17:14
markvanjeblair: anteaya thx for the chat, I'll take it over to the lbaas channel to discuss17:15
jeblairmarkvan: let us know if you need any help17:15
lennybanteaya: ok17:16
sc68calrm_work: still figuring out gating on an amphora image eh?17:16
anteayalennyb: thank you17:16
jeblairrm_work: do all of the jobs each build an amphora image?17:16
anteayamarkvan: thank you17:16
*** kushal has joined #openstack-infra17:17
openstackgerritZara proposed openstack-infra/storyboard-webclient: Let Users Add Notes to Tasks
*** vgridnev has joined #openstack-infra17:19
anteayaZara: yay notes17:19
openstackgerritMerged openstack-infra/system-config: Add sysstat and iotop to all hosts
anteayaZara: wooooo17:19
*** hashar has quit IRC17:20
Zaraanteaya: :) it should replace the old 'task links' patch17:20
Zarasince you can put an url in a note (even if you have to hilight and paste it)17:20
ZaraI'm hoping we can refine it in the future to make it a bit nicer, though17:20
anteayaI didn't even know about the task links patch17:20
anteayayou invoking a writhing mass of snakes17:20
anteayathat makes me less confident than I would like to be17:20
anteayais taht still how you would characterize the state of the code?17:21
Zaraah, no it has been improved since I wrote that, though there still might be odd things17:21
anteayacan you update the comment?17:21
Zarabefore I was switching between 3 variable names for the same sort of thing17:21
Zarait was ugly17:21
*** Sukhdev has joined #openstack-infra17:21
*** fedexo has joined #openstack-infra17:22
*** vgridnev has quit IRC17:22
*** dizquierdo has quit IRC17:23
jeblairinfra-root: did someone try to stop gerrit?17:25
fungijeblair: i did not17:25
mordredjeblair: I did not - but I did see the email17:25
fungiwhy do you ask?17:25
clarkbnot me17:25
yolandajeblair, the sudoers? it may come from the replacement gerrit17:25
yolandai'm working on that and tried to sudo with gerrit217:25
fungioh! yes, that makes sense17:25
* clarkb is working on paging logstash 2.0 back in17:25
fungithey are both named as far as they know17:25
yolandai may stop mailings on that for the moment17:25
jeblairit's not configured to talk to the database is it?17:25
yolandayes, they are both named the same17:25
*** ddieterly is now known as ddieterly[away]17:26
fungiyolanda: don't worry about it, we're aware and know to look at received headers17:26
pabelangerjeblair: not me17:26
*** matrohon has quit IRC17:26
fungibut yeah, let's make sure we don't start gerrit on that server17:26
jeblairbecause if it accesses the production database, it could corrupt it17:27
fungihaving it connect to the production database and alter anything there would be a "bad thing"[tm]17:27
fungiright, that17:27
fungidon't cross the streams17:27
yolandait has its own17:27
fungiyolanda: its own trove instance?17:27
yolandano, local database17:28
fungioh, interesting17:28
yolandabut anyway, ned to be careful with replication and other thigns17:29
yolandabut anyway, ned to be careful with replication and other things17:29
*** oomichi_ has quit IRC17:29
jeblairhow does it have its own trove database?17:29
jeblairer local database?17:29
*** ddieterly[away] is now known as ddieterly17:30
yolandayep local at the moment because i was doing some tests17:30
jeblairyolanda: what are you doing?17:30
yolandai was checking manage_projects, wasn't running17:31
* anteaya checked the backup server on cacti and device read/write appear to be operating on devices xvda,b,c,e17:31
*** esker has joined #openstack-infra17:32
yolandajeblair, how do you normally test these replacements ? what i did for gozer is to have its own local db, disable replication17:32
jeblairyolanda: what are we testing?17:33
*** moravec has quit IRC17:33
openstackgerritMerged openstack-infra/git-review: Correct metadata URLs
yolandajeblair, if the server comes up cleanly, and everything is up17:33
jlkfungi: jeblair: mordred: have you tried turning bluebox back on in nodepool to see if it's working now?17:33
anteayajlk: it looks good to me17:34
*** apoorvad has joined #openstack-infra17:34
anteayajlk: I looked this morning17:34
fungithough i think our grafana is busted at the moment17:34
fungipabelanger: is that an issue known to you?17:34
jeblairyolanda: we don't.  and we definitely don't make alternate manual configuration changes to servers.  puppet runs and it either works or we fix it.17:34
jlkanteaya: ah okay.17:34
fungipabelanger: at the moment i just get "{{alert.title}}" when i browse to it17:34
anteayaum, I get an odd output when I try:
anteayafungi: me too17:35
yolandajeblair so you spin up the instance, and don't do any operation until the deployment day?17:35
mordredjlk: I agree with anteaya17:35
jeblairyolanda: yep.17:35
jeblairpabelanger and i are upgrading grafana to 2.6.0.  it looks like we have succeeded. ;)17:35
*** esker has quit IRC17:35
anteayajlk: I did look this morning, no error node launch attempts17:35
*** esker has joined #openstack-infra17:35
*** nwkarste_ has quit IRC17:35
anteayajlk: thanks for asking17:36
anteayajeblair pabelanger success17:36
fungijeblair: heh!17:36
jeblairyolanda: mostly we make sure puppet and the gerrit init script are disabled so we don't accidentally start it running17:36
*** nwkarsten has joined #openstack-infra17:36
yolandamm, gerrit is stopped there, but how can i disable that on puppet? if i put on emergency hosts, the naming is common to both gerrits17:37
jeblairyolanda: there's a thing you can do with uuids i think17:37
fungiyeah, it's documented in the system-config manual17:37
*** vincentfrancoise has quit IRC17:37
jeblairyolanda: (and make sure the gerrit init script is not set to start on boot; that's a manual change that's okay to make :)17:37
jeblairyolanda: (that way if rackspace reboots the instance for some reason it doesn't start up)17:38
yolandaah that makes sense17:38
fungiyolanda: (second example)17:38
nibalizerand 20 minutes later I get the gerrit sudo email17:38
*** Zhongjun_ has quit IRC17:38
* nibalizer hi5s fastmail17:38
*** tphummel has quit IRC17:39
funginibalizer: greylisting ftw?17:39
nibalizerya thats probably what happened17:39
fungiit's probably the first e-mail they've ever seen from that ip address, so not terribly surprising17:39
nibalizeralso pabelanger now that you're root can you make stackalytics not email me so much17:39
anteayaha ha ha17:40
*** fedexo has quit IRC17:40
*** nwkarsten has quit IRC17:41
openstackgerritMerged openstack-infra/project-config: Remove bare-trusty from projects.yaml
openstackgerritZara proposed openstack-infra/storyboard-webclient: Let Users Add Notes to Tasks
*** esikachev has joined #openstack-infra17:45
pabelangerAJaeger: nice work again with bare-trusty! We can likely start looking into system-config to remove some puppet code too17:45
*** fawadkhaliq has joined #openstack-infra17:47
pleia2ooh, was that the last bare-trusty?17:47
openstackgerritAles Komarek proposed openstack-infra/project-config: Added required package python-yaml for salt-formula-test job
pabelangerpleia2: Ya, I think so17:48
AJaegerfungi, is the apply job17:48
*** SumitNaiksatam has joined #openstack-infra17:49
AJaegerpabelanger: thanks. Cleaning up system-config is a task I will *not* sign up for17:49
anteayayay AJaeger has limits!17:49
fungiAJaeger: oh, awesome--thanks. i figured you probably had since it was the only glaring omission from git grep17:49
*** yamamoto has quit IRC17:49
pabelangerAJaeger: :) I can volunteer for that17:50
*** amitgandhinz has quit IRC17:51
*** jtomasek has joined #openstack-infra17:51
pabelangerAJaeger: fungi: see my comments on ^ 291321.  I guess that will be the next step, removing custom apt-get install logic into JJB templates17:51
*** nikhil has quit IRC17:51
*** jpr has quit IRC17:51
pc_mHi folks. I had a question... was +2/+A at 8:30am, it passed check queue at 9:30am, but I don't see it in gate queue and it now almost 1pm. Any thoughts?17:52
*** Sukhdev has quit IRC17:52
*** e0ne has joined #openstack-infra17:53
*** dims has quit IRC17:54
pabelangerpc_m: is it possible it requires 289558 first?17:54
pabelangergerrit web is terrible now for me to tell17:54
anteayapabelanger: the recheck17:54
pabelangerfires up gertty17:54
anteayasorry pabelanger17:54
anteayapc_m: the recheck17:54
anteayapc_m: that sent it into check and told it to return17:55
pc_mpabelanger: Ah, that could be. I think it was based on top of that one.17:55
anteayaoh pabelanger is probably correct here17:55
anteayaI can't read the new gerrit ui dependency tree anymore17:56
pc_mpabelanger: Where is that shown in gerrit.17:56
pabelangerpc_m: anteaya: Ya, that is the issue. confirmed by gertty17:56
pabelangerpc_m: related changes17:56
pabelangerwhich is a terrible name for it17:56
pc_mpabelanger: Ah. I didn't know where to look. Used to be right in the center and easy to see. Thanks!17:56
anteayapc_m: top right corner17:56
anteayapc_m: yeah17:56
pc_manteaya: pabelanger: Thanks!17:57
anteayapc_m: gertty shows the dependenices easier I'm told17:57
pabelangeranteaya: indeed!17:57
* pc_m googling for gertty17:57
openstackgerritZara proposed openstack-infra/storyboard-webclient: Let Users Add Notes to Tasks
pc_manteaya: Thanks. Just found it. Will have to look at this17:58
*** moravec has joined #openstack-infra17:59
openstackgerritClark Boylan proposed openstack-infra/system-config: Logstash 2.0 compat ES output rule
*** baoli has quit IRC18:00
openstackgerritClark Boylan proposed openstack-infra/puppet-logstash: Install logstash 2.0
*** tphummel has joined #openstack-infra18:00
clarkbanteaya: ^ a few updates that apparently got lost when logstash got deprioritized18:00
anteayaclarkb: thanks, will re-review18:00
*** sputnik13 has joined #openstack-infra18:01
*** baoli has joined #openstack-infra18:01
asselinanteaya, mmedvede I added the link to the wiki.
*** mriedem has quit IRC18:01
anteayaI need to call the hardware store about my dryer, it seems to not be heating up reliably18:01
anteayaasselin: good idea, thank you18:01
*** dims has joined #openstack-infra18:03
AJaegerfungi, once once the apply job change is in (291234), we can remove bare-trusty also from nodepool (
*** notnownikki has quit IRC18:04
*** sambetts is now known as sambetts|afk18:04
fungiAJaeger: yep, i was about to review it, but my workstation (which still has a broken fan i can't seem to get a replacement for) shut itself off again so i'm letting it cool back down18:04
fungiwill look at that one shortly once i let it recover18:05
Zaraanteaya: thanks for reviewing! :)18:05
AJaegerfungi, no worries - just wanted to point it out. I put the nodepool one on WIP, wasn't sure whether that one can merge directly or needs to wait a bit...18:06
*** abregman|mtg is now known as abregman18:06
*** Swami has joined #openstack-infra18:07
fungiAJaeger: i'm inclined to hold off merging removal for nodepool and system-config yet, since it may be harder to temporarily pivot some jobs back to bare-trusty if we discover a particularly nasty issue which will take more time to work around18:07
fungiat least for a few more days18:07
AJaegerfungi, feel free to -2 my patch then ;)18:07
fungionce my workstation is back up, i'll gladly do that ;)18:08
fungiin the meantime, catching up on more e-mail18:08
*** ddieterly is now known as ddieterly[away]18:08
*** ybathia has joined #openstack-infra18:09
* AJaeger renames now tox-nodb to tox again...18:09
*** annegentle has quit IRC18:09
pabelangerAJaeger: so, what sort of timelines do you think about removing apt-get syntax from JJB and migrating projects to other-requirements.txt?18:11
*** nmagnezi has quit IRC18:12
*** jordanP has quit IRC18:12
fungipabelanger: we should hold off doing that until we have the database requirements worked out18:13
*** mriedem has joined #openstack-infra18:13
fungipabelanger: otherwise we're going to wind up with lots of projects adding other-requirements.txt files and unnecessarily including mysql/postgres because of implementation details of some current job-templates18:13
*** kzaitsev_mb has quit IRC18:13
AJaegerpabelanger: I need a few days break first ;) - and wait for fungi to look at the database side first18:14
pabelangerAJaeger: fungi: Yup, fair enough18:14
*** kevinsho_ has quit IRC18:15
openstackgerritAndreas Jaeger proposed openstack-infra/project-config: Rename tox-nodb-ENV to tox-ENV
openstackgerritSam Yaple proposed openstack/diskimage-builder: Revert "Zerofree the image if possible"
openstackgerritSam Yaple proposed openstack/diskimage-builder: Use fstrim to prep the block device
AJaegerpabelanger, fungi: that's the rename back to tox-ENV ^18:15
*** sputnik13 has quit IRC18:16
fungiAJaeger: awesome! you are a change-writing machine18:16
*** ybathia has quit IRC18:16
*** electrofelix has quit IRC18:17
*** abregman has quit IRC18:17
AJaegerfungi: I know how to use emacs ;)18:17
*** moravec has quit IRC18:17
AJaegerpabelanger: do you want to approve your first infra-manual change, please? Then the examples are fine as well:
*** moravec has joined #openstack-infra18:18
*** esker has quit IRC18:18
pabelangerAJaeger: done18:18
*** esker has joined #openstack-infra18:18
SamYaplehello, can I get some reviews on this shade patch? more keystone v3 things for shade18:19
*** tmorin has joined #openstack-infra18:19
AJaegerthanks, pabelanger !18:19
tmorinhi AJaeger yolanda18:19
tmorinit would help a lot if one of you (or perhaps someone else) could have a look at
AJaegertmorin: please explain what it is and what's important with it18:20
tmorinwe need this to go through to enrich DSVM jobs without having to push a change to project-config each and every time18:21
tmorinthis is a pretty straightforward job change where we move from builtin devstack parameters to parameters defined in an rcfile in the repo18:21
*** dims has quit IRC18:21
*** ybathia has joined #openstack-infra18:21
openstackgerritAndreas Jaeger proposed openstack-infra/project-config: Rename tox-nodb-ENV to tox-ENV
* AJaeger takes te last emacs reference - I do not know how to use emacs ;(18:21
SamYapleAJaeger: would you mind reviewing again, i responded to your comment18:22
* AJaeger just noticed that I forgot to safe a file for my change18:22
*** annegentle has joined #openstack-infra18:22
*** dims has joined #openstack-infra18:22
*** ihrachys has quit IRC18:23
AJaegertmorin: needs as usual a neutron infra liason to +118:23
*** esker has quit IRC18:23
openstackgerritMerged openstack-infra/infra-manual: Remove bare-trusty lines
tmorinAJaeger: yes, sure, need to find out who can do this, I believe mestery, but maybe more than one person has this role18:24
mesterytmorin: Yes, me or armax or dougwig18:25
tmorinhi kyle18:25
openstackgerritMerged openstack-infra/storyboard-webclient: Let Users Add Notes to Tasks
tmorinI've added the three of you to the change18:25
tmorinkyle, maybe you can have a look right now ?18:25
tmorinthat would be awesome :)18:25
tmorinthe change is pretty simple18:25
AJaegertmorin: I commented, the change looks wrong18:25
tmorinas I say above: we just change from builtin devstack parameters to parameters defined in an rcfile in the repo18:26
AJaegertmorin: at least you do something that is not part of commit message18:26
*** asettle has joined #openstack-infra18:26
tmorinI've replied why I think this is correct18:27
tmorinWe really are just moving things in a different place18:27
openstackgerritAndreas Jaeger proposed openstack-infra/project-config: Rename tox-nodb-ENV to tox-ENV
tmorinwe have the same lines that we remove in the rcfiles recently commited to the repo18:28
openstackgerritMerged openstack-infra/puppet-openstack_health: Add apache2-utils package to api server
*** jaosorior has joined #openstack-infra18:28
tmorinAJaeger: see
AJaegertmorin: let's ask sdague to review, please18:29
*** openstackgerrit_ has joined #openstack-infra18:30
tmorinAJaeger: sure18:30
AJaegertmorin: it would have helped if you have said in the commit message: REmove bagpipe settings, it's now part of rcfile.18:30
*** asettle has quit IRC18:30
*** [1]Thelo has joined #openstack-infra18:31
tmorinAJaeger: ok, I will update18:31
AJaegertmorin: right now you only say "enable rcfile so that parameters can be controlled"18:31
tmorinyes, true18:31
robcresswellAJaeger: I finally got to bumping the cisco-ui translation patch. Thankyou for following that up.18:31
*** tqtran-afk has joined #openstack-infra18:31
*** degorenko is now known as _degorenko|afk18:31
openstackgerritThomas Morin proposed openstack-infra/project-config: net-bgpvpn: control gate settings from rc scripts
tmorinAJaeger: I can't find sdague email address to add him as a reviewer :-(18:32
tmorinjust found it :)18:32
*** Thelo has quit IRC18:33
*** [1]Thelo is now known as Thelo18:33
*** tonytan4ever has quit IRC18:33
*** kdas__ has joined #openstack-infra18:33
*** tonytan4ever has joined #openstack-infra18:34
openstackgerritThomas Morin proposed openstack-infra/project-config: net-bgpvpn: control gate settings from rc scripts
*** kushal has quit IRC18:34
*** takashi has quit IRC18:35
*** tonytan_brb has joined #openstack-infra18:35
dougwigtmorin: just looked at it.  you got the full range of +1, 0, -1.  :)18:35
*** nwkarsten has joined #openstack-infra18:36
tmorindougwig: I will remove the if lines that are not useful anymore18:36
dougwigtmorin: ok. and is your team too small for the standard two +2's ?18:37
tmorindougwig: on the single +2 on the dependent change  -> the reason is that this dependent change is in a project were we are only two core and the change is proposed by one of them18:37
tmorindougwig: yes, exactly18:37
dougwigtmorin: figured. ok.18:37
tmorindougwig: we always apply the "two +2" rule on changes by non-core contributors18:38
tmorindougwig: :)18:38
openstackgerritThomas Morin proposed openstack-infra/project-config: net-bgpvpn: control gate settings from rc scripts
AJaegerEmilienM, pabelanger: I don't understand why is needed. What am I missing?18:39
tmorindougwig, AJaeger: I think I've now addressed your comments on
*** tonytan4ever has quit IRC18:39
anteayafungi: I wish dyson would get around to creating fans for computers:
pabelangerAJaeger: I believe the scripts EmilienM has for the job are expecting the project to be in openstack/puppet-openstack-cookiecutter and not .18:40
fungianteaya: i know, they have some awesome airfoil-based designs18:41
pabelangerAJaeger: I also think there is the added depends-on support18:41
tmorinI have to go18:41
tmorinbye folks!...18:41
*** tmorin has left #openstack-infra18:41
anteayafungi: have you considered mounting your workstaation on a block of ice?18:41
*** nwkarsten has quit IRC18:42
SamYaplethanks AJaeger18:42
mordredanteaya: he could also move the workstation to canada18:42
mordredanteaya: I hear you have ice up there18:42
*** angdraug has quit IRC18:43
SamYapleice is a natural sound absorber. thats why it seems so quiet when it snows. it is so quiet!18:43
anteayamordred: it is melting18:43
anteayamordred: we have snowbanks still18:43
AJaegerpabelanger: ok, will check again...18:44
anteayamordred: was considering tapping this season18:44
anteayabut the weather got warm and stayed warm18:44
fungianteaya: i should submerge it in a mineral oil path circulated through a chiller18:44
anteayait has to ossilate around zero for a few weeks for good tasting syrup18:44
anteayafungi: that sounds like it would work18:44
anteayayou may have to wash your hands after every time you type though18:45
anteayabeen foggy out my window since monday18:45
anteayanot good syrup weather at all18:45
*** esikachev has quit IRC18:45
fungianteaya: instead, i've found another barebones system of the same model on surplus somewhere online, so i'm ordering that and then can swap it with the current hardware and send that back to the manufacturer for fan replacement18:45
anteayafungi: ha ha ha18:46
openstackgerritSam Yaple proposed openstack/diskimage-builder: Use fstrim to prep the block device
fungisince they were disinclined to just tell me the make and model of the fan replacement18:46
anteayasounds simple and straighforward18:46
anteayafungi: :(18:46
anteayaand I bet you asked nice18:46
*** SumitNaiksatam has left #openstack-infra18:46
fungii'm quite capable of replacing the fan cartridge myself, but it has no distinguishing marks on it to indicate who made it, nor even a cryptic sequence of digits i might take to be a model identifier18:47
fungiand because of the tight tolerances in there, it's either get the exact same fan to replace this one, or take power tools to the case and saw a very large hole so i can bolt on something generic18:48
*** eharney has quit IRC18:48
fungia solution which i'm saving as a last resort18:49
SamYapleanteaya: you ever get maple syrup from the tree and poor it onto ice and then roll that onto a stick?18:49
*** amitgandhinz has joined #openstack-infra18:49
SamYapleits delicious18:49
anteayamaple tafee18:49
fungiSamYaple is now known as SayMaple18:49
anteayalost a tooth to that when I was a kid18:50
*** keedya has joined #openstack-infra18:50
SamYaplethats the one!18:50
rm_workjeblair: yes, and yes they do, but we're going to look at options again -- just needs to be revisited, and sounds like amrith can give us some pointers here18:50
anteayathe snow has to be the consistentcy of small ice pellets for that to be really good18:50
*** yamamoto has joined #openstack-infra18:50
rm_work(yes we're still figuring out amp images, and yes we build them on every job)18:50
amrithhappy to18:50
pleia2I got some of that when I was in montreal :d18:50
SamYaplethats why i said ice, not snow. there is a sublte difference18:50
rm_workamrith: johnsom is here now18:50
anteayaused to make it every year from last year's syrup while boiling down18:50
anteayajust disappears if you pour it on snow18:51
anteayafungi: I'm sure you have a spectrum of solutions, hope the manufacturer will insall a fan when you send them the workstation18:52
jeblairpabelanger: looking at the grafana server, i see the puppet run from when it upgraded, but i don't see any errors related to it18:52
anteayapleia2: yeah that serving was very large in my opinion18:52
pabelangerjeblair: I think we need to restart apache, grafana.o.o is funky18:52
anteayapleia2: but glad you got a taste18:52
jeblairpabelanger: restarted, no joy18:53
jeblairpabelanger: i see several errors in my dev console18:53
jeblaira missing css and notably, a missing js file18:53
anteayarm_work: is there any way you can get amphorae to not build an image on every code change?18:53
jeblairpabelanger: oh there's a grafana server process18:54
pleia2anteaya: that's right, I was with you then!18:54
pabelangerjeblair: that's right. Forgot about that18:54
jeblairpabelanger: ah.  that was easy.  :)18:54
anteayapleia2: :) it was fun18:54
jeblairpabelanger: probably just need to subscribe a restart event to the install in puppet and this would be automatic18:54
anteayarm_work: from my understanding, building an image on every code change seems to be the root of the issue18:55
pabelangerjeblair: Ya, think so18:55
jeblairjlk:  is up again18:55
pabelangerneed to setup grafana again, and tests elasticsearch backend18:56
*** rhallisey has quit IRC18:58
openstackgerritMerged openstack-infra/project-config: Give manila-image-elements its own acls
* AJaeger better calls it a day and waves good-bye....18:58
*** ybathia has quit IRC18:58
*** nwkarsten has joined #openstack-infra18:59
pabelangerfungi: AJaeger: clarkb: Do we want to do 282878 now? I think we needed shade 1.5.1 for that?18:59
*** yamamoto has quit IRC18:59
AJaegerpabelanger: I want it - but let fungi confirm ;)18:59
fungiokay, i've probably let the processor cool down long enough. i'll fire stuff back up and have a look at that19:00
*** cyraxjoe has left #openstack-infra19:01
*** vgridnev has joined #openstack-infra19:01
anteayaAJaeger: thank you, enjoy your evening19:03
*** eharney has joined #openstack-infra19:03
*** nwkarsten has quit IRC19:03
*** pvaneck has joined #openstack-infra19:04
anteayaclarkb: I've reviewed* and they all have fresh jenkins results19:04
anteayaany other infra-root care to review some logstash patches? yolanda I believe has reviewed many19:04
*** tonytan_brb is now known as tonytan4ever19:07
*** sarob has joined #openstack-infra19:08
*** ddieterly[away] has quit IRC19:09
pabelangeranteaya: I can look, but my logstash foo is weak19:10
anteayapabelanger: thanks it is a lot of puppet19:11
anteayapabelanger: so your puppet reviewing is greatly appreciated19:11
anteayaand you don't have to review or vote anything you don't feel comfortable voting on19:12
anteayayou can always use +1 as well if that is what you are comfortable with19:12
anteayaglad to have more eyes, thank you19:12
anteayasometimes the best review is a good clarifying question19:13
*** dimtruck is now known as zz_dimtruck19:14
pabelangermordred: so, is a thing now :) Plan on testing it out with the help of russellb19:14
pabelanger1 step close to infracloud :p19:14
russellbpabelanger: let me know how i can help19:15
openstackgerritK Jonathan Harker proposed openstack-infra/zuul: Let zuul-cloner work in post and release pipelines
*** jistr has joined #openstack-infra19:15
pabelangerrussellb: Ya, we'll need ubuntu support for sure. Any issues doing fedora 23?19:16
russellbpabelanger: f23 should be fine19:16
russellbpabelanger: puppet may need some tweaks for ubuntu since it hasn't been thested there yet19:16
*** dmsimard has quit IRC19:17
russellbpabelanger: we're also not building the networking-ovn (neutron plugin) as an ubuntu package anywhere yet19:17
pabelangerrussellb: Ya, I'm hoping EmilienM will have some CI in place to help gate on the testing19:17
EmilienMrussellb: I'll disable beaker tests for ubuntu, since no packaging. No need to waste resources.19:18
EmilienMwe are in the middle of a release, I'll put some efforts probably in 2 weeks.19:18
EmilienMto have integration testing with OVN, asap19:18
*** akrivoka has quit IRC19:19
russellbthe packaging for OVN itself is done for debian/ubuntu19:19
*** nwkarsten has joined #openstack-infra19:19
russellbjust not the neutron plugin19:19
openstackgerritDavid Shrewsbury proposed openstack-infra/project-config: Enable heat service in shade jobs
Shrewsmordred: before you get engrossed in Duke v. ND, ^^^^19:21
*** dmsimard has joined #openstack-infra19:21
*** nwkarste_ has joined #openstack-infra19:21
openstackgerritMerged openstack/requirements: Exclude python-troveclient 2.1.0
EmilienMrussellb: I'll start by using packaging in RDO and iterate with Ubuntu if needed.19:22
*** ddieterly has joined #openstack-infra19:22
EmilienMbut I'm working on Mitaka release this week and next week so it won't happen before 2 weeks I think19:22
russellbEmilienM: yep no worries19:22
russellbEmilienM: pabelanger fwiw, i have notes about where Fedora/CentOS packages can be found in the packaging section of
EmilienMit's already bookmarked :P19:24
*** nwkarsten has quit IRC19:24
*** pcrews_ has joined #openstack-infra19:24
*** zul has quit IRC19:25
*** pcrews has quit IRC19:25
*** zul has joined #openstack-infra19:26
anteayarbradfor: nice post on feedback on the bug smash event19:26
*** nwkarste_ has quit IRC19:26
anteayarbradfor: thanks for attending19:26
*** esikachev has joined #openstack-infra19:26
rbradforanteaya, thanks for the comments.19:27
rbradforI actually got a lot from the event as I got to meet 3 openstack regulars for the first time.19:27
*** gnuoy_ has joined #openstack-infra19:27
anteayawell done19:28
*** vgridnev has quit IRC19:28
anteayarbradfor: how many people would you say attended?19:28
*** dims_ has joined #openstack-infra19:28
sc68calrbradfor: 15?19:28
openstackgerritJames E. Blair proposed openstack-infra/system-config: Add storyboard exim router/transports
*** ddieterly is now known as ddieterly[away]19:28
* sc68cal crashed the party on monday19:28
anteayasc68cal: ah you were there too, very nice19:28
*** sobersabre has joined #openstack-infra19:28
rbradforand it reminded me of when I started I did all this myself, so there is merit in having beginner guides to help individuals get over the intro hurdle.19:28
anteayawell I don't see how one can crash a bug smash event19:28
anteayarbradfor: yes, I agree19:29
*** jcoufal has quit IRC19:29
anteayayou made good points there in your post19:29
*** dims has quit IRC19:29
sc68calooh, I haven't read it yet. link?19:29
rbradforsc68cal, anteaya on monday I'd say 15 in the room, and I think 5 of us were full-time contributors.  I'd say 8 of the 10 probably had not any experience with OpenStack.  It would be good to get these numbers at other sites.19:29
anteayasc68cal: dev ml19:29
*** Sukhdev has joined #openstack-infra19:29
rbradforsc68cal, See ML.19:29
sc68calack - thx19:30
*** rhallisey has joined #openstack-infra19:30
robcresswellAJaeger: Does the new translation extraction setup require additional config? Seems to be generating empty .pot files on horizon-cisco-ui.19:30
anteayaI wonder how the event was communicated such that folks with no experience were in attendance19:31
fungipabelanger: AJaeger: on 282878 it looks like nodepool-builder was last restarted at 04:06 utc on tuesday, and shade 1.5.1 was not tagged until 12:57:43 utc on tuesday, so we need a nodepool-builder (and also probably nodepoold) restart before we can rely on that fix19:31
anteayarbradfor: what other sites were doing an in person event?19:31
*** jaosorior has quit IRC19:31
ianwSamYaple: hopefully that fstrim just needs a sudo, and the ioctl failure isn't actually saying "not supported"?19:31
*** jpr has joined #openstack-infra19:32
SamYapleianw: dunno havent looked at the gate. will look into it19:32
rbradforanteaya, well the free tee shirt had a list of locations :)19:33
*** andreykurilin__ has quit IRC19:33
anteayarbradfor: nice19:33
anteayafound them, they are listed on an etherpad:
rbradforanteaya, australia, china, india, russia, united kingdom, united statse (at least)19:33
sobersabrewaynr: can you take a quick look at ?19:34
SotKanteaya: the task notes patch has made it through to now, if you're interested :)19:34
sobersabrefdegir: have you had a chance to take a look at: ?19:34
anteayaSotK: woooo, I"m interested19:34
* anteaya heads to storyboard19:34
sobersabrefdegir: you said you would, but I guess it's rarely things happen too close to the plan :-|19:35
openstackgerritJoshua Harlow proposed openstack/requirements: Bump voluptuous to 0.8.9
openstackgerritJames E. Blair proposed openstack-infra/zuul: Fix race in tests
pabelangerfungi: thanks19:35
*** dmsimard has quit IRC19:36
pabelangerianw: testing centos-minimal again locally, so far, so good19:36
*** annegentle has quit IRC19:36
anteayaSotK: so it is that black square that creates a drop down box?19:36
fungipabelanger: note that nodepool-builder restarts are disruptive to daily image updates, so probably should happen sometime later on once they're done for the day19:36
fungii'll try to plan to do one tonight19:36
anteayaSotK: with the prompt, enter task details here?19:36
*** annegentle has joined #openstack-infra19:36
pabelangerfungi: Ack. Ya, I haven't even started looking into the production systems yet.  Won't be for a while until I start restarting services :)19:37
SotKanteaya: yeah, I think the square is supposed to be a sticky-note type thing19:37
*** dmsimard has joined #openstack-infra19:37
ianwpabelanger: yeah, you might see some occasional failures, but i believe covers it19:37
anteayaSotK: okay my mind thinks, black square, not sticky note19:37
anteayaSotK: do we have the ability to add a tooltip on hover?19:37
pabelangerianw: good to know19:37
SamYapleianw: i dont have alot of time right now so ill just push a patch with sudo for the gate to test it for me19:37
*** nwkarsten has joined #openstack-infra19:38
ianwpabelanger: i spent a while trying to diff rpm-db's to get a true root cause on that, but basically, gave up.  too many layers there19:38
SamYapleianw: worst case is the loop stuff doesnt support discard and we have to use qemu-nbd, but well cross that bridge if we get there19:38
ianwand when it's probably due to version skew, it's too obscure to waste too much time on19:38
openstackgerritSam Yaple proposed openstack/diskimage-builder: Use fstrim to prep the block device
SotKanteaya: sure19:39
* SotK prepares a patch19:39
anteayaSotK: thank you19:39
ianwSamYaple: yeah, that was my concern; it's a good point that we should probably actually capture at least kernel logs for that job, we might have got something useful there19:39
ianwSamYaple: did it work locally?19:40
openstackgerritSridhar Ramaswamy proposed openstack-infra/project-config: Remove groups for python-tackerclient
SamYapleianw: of course19:40
SamYaplei posted results for you19:40
openstackgerritMerged openstack-infra/puppet-infracloud: Add examples directory
SamYapleianw: i dont push untested patches. thats crazy talk19:40
ianwSamYaple: hmm, what platform19:40
*** rcernin has quit IRC19:40
SamYapleubuntu (though it was 16.04 preview, i was testing dib xenial stuff)19:41
ianwhmm, that's going to be a much later kernel than trusty...19:41
SamYapleyea 4.4 vs 3.1319:41
SamYaplethe kernel certainly supports it for qemu-nbd19:41
SamYaplemaybe no losetup19:41
SamYapleif yo urecall i commented i was suprised it worked in the comments19:42
ianwthat doesn't fill me with hope that it's not a kernel issue :)19:42
SamYaplemy point is even if it is kernel related, there is still a path forward19:42
*** e0ne has quit IRC19:43
*** Sukhdev has quit IRC19:43
ianwi'm just happy the tests found something for once, rather than just being a PITA :)19:43
jeblairmordred, nibalizer: has just the right amount of whitespace, quotation marks, and backslashes in just the right places now.19:44
openstackgerritAleksandra Fedorova proposed openstack-infra/project-config: Add missing delay-release option in Fuel projects
ianwwhat's going to be the best way to find logs for periodic jobs?19:45
*** armax has quit IRC19:46
fungiianw: i think there's an intention to get the health dashboard to start providing a more convenient index to those at some point19:46
anteayais there?19:47
*** _nadya_ has joined #openstack-infra19:47
*** IlyaG has quit IRC19:47
*** nelsnelson has quit IRC19:47
jeblairtrue.  i actually answered the question "what is currently the best way to find logs for the afs build job?"19:47
jeblairthere's a whole multiverse of possible answers to the question as written.  :)19:48
fungiindeed. it's all about relative spacetime, after all19:49
fungii wasn't clear on ianw's current relativistic frame of reference19:49
*** baoli has quit IRC19:50
openstackgerritMerged openstack-infra/project-config: Remove final bare-trusty job
fungiAJaeger: ^ boom!19:50
SamYapleits finished19:50
SamYaplering has been dropped into the lava19:50
*** SumitNaiksatam has joined #openstack-infra19:50
openstackgerritAdam Coldrick proposed openstack-infra/storyboard-webclient: Add a tooltip when hovering on the "Task notes" button
fungiSamYaple: if only. we still have jobs running on bare-precise :/19:50
jeblairthis does not exist yet19:51
pabelangerfungi: nice!19:51
jeblairare we stuck on jjb updates?19:51
SamYaplefungi: give it a year that ubuntu will stop releaseing seucirty patcheso n that one19:51
fungijeblair: i hope not. jenkins.o.o usually manages to update even if nothing else does19:51
* mordred is excited about the death of bare-trusty19:52
pabelangerlooking forward to centos-7 next19:52
pabelangerwell, not the death of19:52
*** ayoung has quit IRC19:52
fungimordred: we still need a nodepool-builder restart for shade 1.5.119:53
fungiso we're not building these in rax at the moment19:53
anteayaSotK: thank you19:53
pabelangerianw: centos7-minimal dib successful, nice work19:53
fungiianw: jeblair: i don't see that job in project-config19:53
mordredfungi: I restarted it the other day after having applied the patch by hand - did it get restarted again with stock 1.5.0?19:53
jeblairfungi: hrm19:54
fungimordred: oh! i forgot about that. i was comparing the creation time on the pidfile to your tag time19:54
jeblairfungi: ah typo19:54
anteayaSotK: next question, after I have added a note, where should I be able to view the contents of the note?19:54
jeblairfungi: there it is
mordredfungi: I mean, we SHOULD restart to pick up 1.5.1 so that we're not running on a hand-patched file19:54
anteayaSotK: thus far I have not figured out where I should be looking19:54
openstackgerritDan Prince proposed openstack-infra/tripleo-ci: Add common bash functions to help track metrics.
openstackgerritDan Prince proposed openstack-infra/tripleo-ci: Metrics tracking for TripleO deployment tasks
*** esker has joined #openstack-infra19:55
mordredfungi: builder is ready for a restart any time19:55
fungimordred: so anyway, it's your assessment that we should be safe to merge 282878? want to +3 if i un-wip?19:55
*** gyee has quit IRC19:56
mordredfungi: yes. shall I restart the builders on actual 1.5.1 ?19:56
*** moravec has quit IRC19:56
fungimordred: it's safe to do that even if image updates are underway?19:57
mordredfungi: oh - are we updating images right now? no, I do not think it is safe to restart19:57
SotKanteaya: so far, its just in the same place as you added it19:57
* fungi doesn't know how long our image updates go these days19:57
*** rguillebert has quit IRC19:57
SotK(at the moment its just a single big text field for putting notes in)19:57
*** ildikov has quit IRC19:58
mordredfungi: there are images in building state19:58
anteayaSotK: so I have to click on the button to see what I added?19:58
* anteaya just heard a very loud noise of unknown origin19:58
*** sshnaidm_ has joined #openstack-infra19:58
*** sshnaidm has quit IRC19:59
*** hashar has joined #openstack-infra19:59
anteayaSotK: so I do, okay19:59
fungimordred: anyway, that change is ready for approval19:59
anteayahmmmmm, is there a larger vision for this feature?19:59
fungii added relevant comments19:59
ianwjeblair: yeah, i was wondering about the latest run ... i'd like to add it to my nodepool build monitoring script.  i feel like scraping dates is wrong20:00
mordredfungi: +A20:00
SotKI'd like an arbitrary number of notes to be able to be added eventually20:00
ianwfungi: it's 7am, so relativistic time is best described as "still a little blurry"20:00
SotKidk what other people's views on the feature are though20:00
*** ildikov has joined #openstack-infra20:00
anteayawell I'd like to be able to see from storyview if any task have notes20:01
fungiianw: i like to refer to that as "time dilation"20:01
anteayaso I know if clicking the button is worth my time20:01
*** rhallisey has quit IRC20:01
anteayaso if the button changed shape or colour if it had content in the form of a note20:02
*** rhallisey has joined #openstack-infra20:02
anteayaso some other way of saying clicking this button is worth it20:02
openstackgerritSam Yaple proposed openstack-infra/shade: Add update_service()
anteayaSotK: and adding multiple notes makes sense to me20:03
anteayaor the ability to do so20:03
* SotK makes a patch to change colour if there are notes20:04
anteayaSotK: thanks :)20:04
anteayaI appreciate you listening to me20:04
SotKyou're welcome :)20:05
*** sdake has quit IRC20:05
openstackgerrityolanda.robla proposed openstack-infra/glean: Place glean.lock in working dir of script
yolandamordred, what do you think about that? ^20:06
openstackgerritMatt Riedemann proposed openstack-infra/reviewstats: Support passing just a single project name
mriedemrussellb: ^ do i get ATC cred for that?20:06
mriedemthis is all new to me20:07
openstackgerrityolanda.robla proposed openstack-infra/glean: Place glean.lock in working dir of script
mriedemwas trying to find a good way to get reviewstats on stable branches20:07
openstackgerrityolanda.robla proposed openstack-infra/glean: Place glean.lock in working dir of script
russellbmriedem: i feel like it did that at one point ...20:08
mriedemrussellb: it does, but only for stable-maint-core20:08
russellbi see20:08
mriedemrussellb: so if i want to see just stable reviews on nova, it needs some work20:09
openstackgerritIan Wienand proposed openstack-infra/project-config: Add CentOS AFS build job to project-config experimental
openstackgerritIan Wienand proposed openstack-infra/project-config: [WIP] Remove CentOS AFS build experimental job
russellbmriedem: got it20:09
mriedemplus it doesn't handle gerrit group hierarchy20:09
russellbmriedem: sounds good to me, left a comment though20:09
russellbmriedem: it doesn't?  lame on my part20:09
openstackgerritAdam Coldrick proposed openstack-infra/storyboard-webclient: Highlight the notes button if a task has notes
* anteaya reviews20:09
mriedemrussellb: it probably never needed to20:09
mriedemand arguably doesn't need to...20:10
mriedemi.e. stable-maint-core is included in nova-stable-maint, but i probably care more about nova-stable-maint for just nova reviews20:10
openstackgerritSam Yaple proposed openstack-infra/shade: Test v3 params on v2.0 endpoint; Add v3 unit
openstackgerritSam Yaple proposed openstack-infra/shade: Update create_endpoint()
russellbmriedem: ok, i could go either way ... i'd probably expect it to show everything included20:11
*** e0ne has joined #openstack-infra20:12
SotKanteaya: thanks :)20:12
SamYapleianw: sudo took care of it. gate is happy20:12
*** mrmartin has joined #openstack-infra20:13
fungiso with the last bare-trusty using job out of our config, i should plan to make sure our jenkins masters are completely up to date later this evening and then restart zuul when it's quiet, so we no longer have any remaining bare-trusty registrations in gearman20:14
clarkband update nodepool config20:14
fungiyeah, it's already dialed down to min-ready: 020:15
fungias of that last change20:15
*** Swami has quit IRC20:15
ianwSamYaple: nice.  i also have up so that we can pull some longer-term graphs of image size over time20:15
SamYaplehey clarkb if you want to run with this patchseries and see if it generates the images you want, i would be interested in seeing how it compares20:15
SamYapleianw: yay i like data20:15
fungiclarkb: there's a wip change to remove the images, but i wanted to wait at least a couple more days in case we encounter a "zomg must switch back" issue we haven't spotted yet20:15
*** |-paul-| has joined #openstack-infra20:16
fungiso that we don't have to wait for new images to build in that case20:16
clarkbfungi: ya if we set max-servers yo 0 we will still build images20:16
*** aeng has joined #openstack-infra20:16
openstackgerritMerged openstack-infra/project-config: Revert "Temporarily stop booting ubuntu-trusty in rax-.*"
*** rguillebert has joined #openstack-infra20:18
anteayaclarkb: oh the -1 keeps the images from being built20:18
openstackgerritK Jonathan Harker proposed openstack-infra/zuul: Let zuul-cloner work in post and release pipelines
ianwSamYaple: yeah, the gate failures were due to the image switch and mysql users.  should be fixed now20:19
*** austin81 has joined #openstack-infra20:19
*** matrohon has joined #openstack-infra20:19
*** armax has joined #openstack-infra20:20
SamYaplei am always teetering on the edge of staying away from infra and falling into the abyss and keeping up with the goings-ons around here20:21
fungiSamYaple: there is no shallow end in our pool20:21
SamYapleif i knew more about this I would be able to tell if that was a pun or not20:22
mriedemrussellb: is it just me or does the default reviewstats run for nova take a looon time20:22
russellbyes it takes a very long time20:22
russellbespecially the first time you run it20:22
mriedemheh, i'll have to remember not to run that20:22
russellbit keeps a cache that should speed it up20:22
russellbafter it finishes the first time20:22
mriedemok, i've been running these in venvs20:22
fungiSamYaple: i just like making cryptic comments. i can be safely ignored ;)20:22
russellbmriedem: the whole thing is a hack20:22
russellbmriedem: it downloads the entire review history of a project...20:22
mriedemi have aspirations to contribute20:22
SamYaplefungi: you are a cipher wrapped in an enigma20:23
russellbwhich you can imagine hasn't scaled amazingly20:23
mriedemi thought it was just going back 14 days20:23
*** kzaitsev_mb has joined #openstack-infra20:23
*** zz_dimtruck is now known as dimtruck20:23
*** e0ne has quit IRC20:23
russellbmriedem: i think it downloads everything anyway, to populate its cache to work on20:23
*** _nadya_ has quit IRC20:24
russellbwhich is just a pickled dict written to disk heh20:24
mriedemyeah, when i did just --stable=liberty it was faster20:24
mriedemmmm pickled dict20:24
*** ociuhandu has quit IRC20:24
russellbpickles are gross though20:24
fungirussellb: mriedem: if the struggle to get into production ever ends, the hope is that we can try to get whatever statistical features and queries we're missing reimplemented there20:24
russellbfungi: yep20:24
mriedemrussellb: i have a jar in my fridge that begs to differe20:24
russellbi haven't worked on reviewstats other than approving people's info updates for the past very long time20:24
russellbpretty low cost to keep it running in the meantime20:25
clarkbI have a tasty pickle with kunch20:25
russellbi'm sorry20:25
fungii have a pickled plum for breakfast every morning. they're awesome20:25
*** annegentle has quit IRC20:26
fungihad pickled red onions on my pork belly tacos at dinner last night too20:26
fungidon't think i could get by without pickles20:27
russellbit's like i don't even know you people20:27
*** annegentle has joined #openstack-infra20:27
russellb(i don't actually feel this strongly about pickles)20:27
openstackgerritAndreas Jaeger proposed openstack-infra/project-config: Remove one bare-trusty line from projects.yaml
dmsimardclarkb: so I know what the issue is for the swap review20:28
anteayamy grandmother made mustard pickles all the time20:28
dmsimardBut I'm having a hard time finding a non-hacky way to fix it20:28
anteayathey were my favourite20:28
*** esker has quit IRC20:28
dmsimardBasically the swap amount, even if it's 8192MB can be reported as low as ~7700MB but it's not a set number20:28
*** ddieterly[away] is now known as ddieterly20:29
fungiAJaeger: i have a feeling we'll need to do a few cleanup patches like 291412 for the next... while20:29
clarkbdmsimard: is the issue we check amiunt of swap?20:29
dmsimardclarkb: yeah20:29
openstackgerritMatt Riedemann proposed openstack-infra/reviewstats: Support passing just a single project name
clarkbdmsimard: I would write a file out that acts as a guard20:29
dmsimardso, I mean, I can add a "fuzz" factor (say 500MB) and then compare "$swapcurrent -lt $(( 8192 - fuzz))" but that feels hacky20:29
clarkbdmsimard: maybe use the swapfile for that20:30
dmsimardoh, hey, that's not a bad idea. Sort of a lock file ?20:30
*** Sukhdev has joined #openstack-infra20:30
clarkbif not exists swapfile then setup swap20:30
clarkbdmsimard: ya20:30
*** esikachev has quit IRC20:30
dmsimardbut sometimes we use the ephemeral disk too.20:30
*** esker has joined #openstack-infra20:30
clarkbya write a 0 byte file in that case20:30
dmsimardyeah, I can create a dummy file.20:30
dmsimardgood idea !20:31
clarkband dony swapon20:31
austin81mtreinish, pabelanger: Did y'all manage to come to a consensus on puppet vs dib for stackviz deployment?20:31
anteayaclarkb: can this go in yet?
anteayaclarkb: I think everyone is waiting for you to say what goes in in what order20:32
anteayathe decouple log processing from logstash patch20:33
jesusaurwhen did we start enforcing subunit sizes <50MB? and what's the reasoning behind that?20:33
clarkbanteaya: that one goes in first, then when it is applied everywhere we merge the two other changes (those have to go in together)20:33
clarkbanteaya: my preference would be to not merge anything until it all has the necessary +2s20:33
anteayaclarkb: the other two changes you offered?20:33
clarkbanteaya: yes20:34
anteayainstall logstash and compat ES output rule20:34
mtreinishaustin81: consensus, I don't think so, or at least I don't rememeber ever having that. :)20:34
anteayaokay will see if I can round up some +2's20:34
clarkbanteaya: that way we can blaze through and get it all done at once20:34
clarkbanteaya: and avoid getting it half done today and finished in a couple weeks20:34
anteayaclarkb: yup, I'm all for helping you blaze through20:34
anteayaclarkb: yup20:35
anteayapabelanger: you rechecked and the jenkins jobs passed now, care to review20:35
anteayaLogstash 2.0 compat ES output rule20:35
*** esker has quit IRC20:35
*** esker has joined #openstack-infra20:35
*** esker has quit IRC20:36
anteayapabelanger: and thanks for reviewing the puppet code on install logstash 2.0:
austin81mtreinish: Gotcha, just checking :) I won't be around next week for infra meeting so I was trying to get it pushed along now if possible20:36
*** erlon has quit IRC20:36
austin81mtreinish: or at least understand the debate better20:37
*** IlyaG has joined #openstack-infra20:37
anteayaany infra-root in addition to pabelanger able to review 247756 and 247714 ?20:37
anteayapretty please?20:37
clarkbjesusaur: I think the reasoning is to keep subunit2sql sane? jesusaur large subunit files take a lot of time to process and use up more room in the database?20:37
mtreinishaustin81: personally I would move forward with dib, I still don't understand how writing a puppet module to run 1 build/install command is going to solve getting the output on the nodepool images20:38
*** angdraug has joined #openstack-infra20:38
mtreinishaustin81: you're going to have to use dib to get the content on there. Adding a puppet module doesn't make any sense to me20:38
fungiwell, we apply some puppet to our images from a dib element, but we'd like to continue to scale that back20:39
openstackgerritDavid Moreau Simard proposed openstack-infra/devstack-gate: Use dd to create swapfile, ensure swap amount == 8GB
fungithe challenge with doing it only in dib is that we're not yet to the point where devstack-based jobs are running on dib-built images in every provider (rackspace is a holdout because of our need to switch to simple-init/glean for network configuration)20:40
mtreinishfungi: well, it was dib and the equiv build script20:40
waynrsobresabre: okay, I have reviewed and given workflow +1, should be merged soonish20:40
mtreinishfungi: I believe austin81 has a patch up already20:40
SamYapleianw: alot is slowly becoming an accepted word. im making sure of it just to prove my 2nd grade teacher wrong20:41
fungibut i have hopes that we're at most weeks away from moving all devstack-trusty jobs to ubuntu-trusty and then we will be using dib in rackspace for the platform running devstack jobs20:41
SamYapledmsimard: if thats a picutre of a fuzzy animal thing, thats my spirit animal20:42
austin81fungi: spec at
*** esikachev has joined #openstack-infra20:42
SamYapleit is. thats the one. thanks dmsimard20:42
dmsimardSamYaple: :p it comes from I think20:42
jesusaurclarkb: hm, my zuul-cloner change just barely went over that limit20:43
clarkbjesusaur: oh it shouldn't affect zuul20:43
SamYapleyes dmsimard im ok with this situation20:43
fungiSamYaple: the alot is your spirit animal? i love this alot20:43
clarkbjesusaur: or is the check fialing the job?20:43
AJaeger#success All jobs moved from bare-trusty to ubuntu-trusty20:43
openstackstatusAJaeger: Added success to Success page20:43
SamYaplefungi: indeed20:43
anteayaAJaeger: congratulations20:44
nibalizerclarkb: its safe to land the logstash changes?20:44
fungia success among successes to be sure20:44
*** nwkarsten has quit IRC20:44
clarkbnibalizer: it is safe to land the first one (would prefer we land that only when the other two have the +2s)20:44
clarkbnibalizer: the other two aren't super safe because there is no forward/backward compatibile path so we have to coordinate it tightly20:45
*** IlyaG has quit IRC20:45
clarkblet me put a paste up20:45
anteayaclarkb: which is the first one? Decouple log processing from logstash?
*** sbalukoff has quit IRC20:45
openstackgerritMerged openstack-infra/project-config: Remove one bare-trusty line from projects.yaml
anteayaI think that is the first one20:46
clarkbanteaya: yes decouple is first20:46
jesusaurclarkb: yeah, the subunit filesize check failed the job, and on recheck the filesize was under but an alarm clock failed the job :(20:46
anteayais it safe to land yet?20:46
clarkbnibalizer: anteaya
clarkbyes I am fine if the decoupling changes lands now and the other two have +2s20:47
jeblairclarkb, mordred: are we ready to think about nodepool shade?20:47
openstackgerritMerged openstack-infra/jenkins-job-builder: upd google-cloud-storage plugin support
openstackgerritBilly Olsen proposed openstack-infra/project-config: Delay release for charm projects
clarkbjeblair: does it pass tests? I havne't checked in on it recently20:47
mordredjeblair: YES20:48
openstackgerritMichael Krotscheck proposed openstack-infra/project-config: Added npm-install macro to swagger build job
mordredjeblair: I need to rebase the patch after having pushed the recent logging changes20:48
anteayanibalizer: care to land 247857?20:48
mordredjeblair: lemme do that real quick20:48
jeblairmordred: okay.  cool, i think this might be a good time to try it.20:49
jeblairwe can handle some nodepool restarts at our load.20:49
AJaegerthanks everybody for reviewing these ubuntu-trusty changes in quick time!20:50
* AJaeger disappears again...20:50
nibalizeranteaya: sure, if clarkb is ready20:50
anteayaAJaeger: thanks for doing the hard work of shepherding them20:50
anteayanibalizer: well as long as the other two patches are reviewed and ready to be merged soon after20:50
clarkbnibalizer: anteaya has two +2's so I think we are ready ot merge its parent20:51
clarkbnibalizer: so go ahead :020:51
clarkber :)20:51
nibalizeri think i've reviewd the things?20:51
*** denisra has joined #openstack-infra20:51
clarkbnibalizer: you did, thats what I was saying, I am ready because you and pabelanger have reviewed20:51
clarkbnibalizer: and you saw the paste above right? outlines the rough process20:51
nibalizerclarkb: ya i saw20:51
nibalizeryou're on those steps?20:51
anteayawe still need two +2's on
clarkbnibalizer: yup I will work through it20:51
nibalizerim around and can hop on if things go sidewase20:52
sobersabrewaynr: thanks for the merge!!20:52
clarkbplan is after first change merges and applies to disable puppet in ~5 at a time and update them in chunks20:52
clarkbthat way we should never stop processing events20:52
sobersabrefinally I can remove all these cherry-picks I'm doing and use master :)20:52
*** yamahata has quit IRC20:52
anteayaand another system-config core in addition to nibalizer, if you could review please20:52
clarkbanteaya: that one I am almost comfortable self approving since its just a small config change and not a large refactor20:53
anteayaclarkb: oh okay fine, I'll stop now20:53
anteayayou have what you need20:53
anteayathank you all20:53
jeblairit also has some previous +2s20:53
anteayaeven better20:54
*** moravec has joined #openstack-infra20:55
*** matrohon has quit IRC20:55
pabelangeranteaya: clarkb reviewed20:55
pabelangerBut will yield approval to somebody else :)20:55
openstackgerritMerged openstack-infra/reviewstats: Support passing just a single project name
clarkbpabelanger: I can take care of approval, thank you20:56
*** hdd has joined #openstack-infra20:56
*** esker has joined #openstack-infra20:56
clarkbnibalizer: do we have a working "run puppet when emergency file says don't run puppet" playbook thing yet20:57
clarkbthat was being worked on during the sprint and I don't recall if we finished that20:57
clarkb(I will likely need that for this)20:57
anteayapabelanger: thanks20:57
nibalizerclarkb: monty has some code up20:57
mordredwhat did I do?20:58
mordredyah. it' smerged20:58
clarkbI don't need it, I can put all the hosts in the emergency file, wait for the module to update on puppetmaster, then remove a handful of them at a time from the emergency file and they will auto update20:58
fungiweather's too nice to spend all afternoon in front of a computer. i'm going out for a quick walk but should be back in roughly an hour20:58
openstackgerritMerged openstack-infra/puppet-subunit2sql: Switch use of cStringIO.StringIO to io.BytesIO
anteayafungi: enjoy20:59
openstackgerritMerged openstack-infra/puppet-subunit2sql: Remove inaccurate comment
mordredwell, if you nede the other thing - remote_puppet_adhoc.yaml with a --limit argument is your friend20:59
mordredclarkb: ^^20:59
clarkbmordred: remote_puppet_adhoc.yaml ?20:59
openstackgerritMerged openstack-infra/system-config: Move statusbot_auth_nicks to hiera
clarkbmordred: thanks20:59
anteayaweather is overcast and foggy here but was thinking of doing the same20:59
pabelangerany core like to review 291402? It will help ianw with testing afs builds on centos20:59
mordredclarkb: yup!20:59
openstackgerritMerged openstack-infra/system-config: Add storyboard exim router/transports
openstackgerritMerged openstack-infra/puppet-log_processor: Decouple log processing from logstash
*** denisra has quit IRC21:00
*** rfolco has quit IRC21:00
lifelessdhellmann: we fixed the pbr jobs; you may need to recheck21:00
*** sigmavirus24 is now known as sigmavirus24_awa21:00
clarkbpabelanger: done21:01
*** weshay has quit IRC21:01
pabelangerclarkb: danke21:01
*** esikachev has quit IRC21:01
*** amrith is now known as _amrith_21:01
*** esikachev has joined #openstack-infra21:01
*** rossella_s has quit IRC21:03
*** rossella_s has joined #openstack-infra21:03
*** sputnik13 has joined #openstack-infra21:04
clarkbnibalizer: hrm any idea why a good chunk of the logstash machines are reporting unreprted?21:04
clarkbnibalizer: in puppetbaord21:05
tristanCttx: is it time to tag governance repository with march-2016-elections ?21:05
nibalizerthat likely means they fail21:05
nibalizeriirc they are timing out?21:05
clarkbnibalizer: huh21:05
* clarkb pulls up ansible logs21:05
anteayatristanC: I think he would be offline by now, would he not?21:06
nibalizer2016-03-10 06:53:45,822 p=17767 u=root |  fatal: []: FAILED! => {"changed": false, "failed": true, "msg": "/usr/bin/timeout -s 9 30m /usr/bin/puppet apply --detailed-exitcodes --logdest syslog --no-noop /opt/system-config/production/manifests/site.pp failed with return code: 4", "rc": 4, "stderr": "Faraday: you may want to install system_timer for reliable timeouts\n",21:06
nibalizer"stdout": "", "stdout_lines": []}21:06
clarkbfailed with return code: 4"21:06
tristanCanteaya: you're right, he is probably afk, do you know who can tag the repository ?21:06
clarkbya will check syslog21:06
anteayatristanC: I think he is the one who does it21:06
nibalizermaybe we need the system_timer gem21:06
clarkbit is entirely possible that this set of changes fixes it if logstash stopped publishing the old stuff or similar21:06
clarkbnibalizer: I think the timer thing is mostly noise21:06
anteayatristanC: even if someone else did have permissions, I can't see them wanting to tag governance without ttx's say so21:07
*** sdake has joined #openstack-infra21:07
*** sigmavirus24_awa is now known as sigmavirus2421:07
clarkbdpkg was interrupted, you must manually run 'dpkg --configure -a' to correct the problem21:07
clarkbdpkg is unhappy, I will work on fixing21:07
*** dtardivel has quit IRC21:08
*** kencjohnston has joined #openstack-infra21:08
openstackgerritMerged openstack-infra/project-config: Add CentOS AFS build job to project-config experimental
openstackgerritMerged openstack-infra/project-config: Remove groups for python-tackerclient
*** abregman has joined #openstack-infra21:09
openstackgerritRyan Brandt proposed openstack-infra/project-config: Use TEMPEST_SERVICES variable to trigger monasca tempest tests
*** tzn has quit IRC21:10
*** tzn has joined #openstack-infra21:11
openstackgerritMerged openstack-infra/project-config: Add non-voting job for Cinder Functional tests
tristanCanteaya: oh well it's mostly for the mail announcement link, i'll send ttx a reminder mail to be sure it's tagged soon21:12
*** krtaylor has quit IRC21:13
*** jpr has quit IRC21:13
HeOSHello! Could you please review the following request about updates of jeepyb?
anteayawell that and to ensure the state of the repo is static at a certain point in time21:14
openstackgerritDeklan Dieterly proposed openstack-infra/project-config: Add pre_test_hook to freezer api gate job
anteayabut I'm sure he will tag as soon as he knows to do so21:14
ttxtristanC: I'm tagging as we speak, processing the las tchanges first21:14
anteayattx: ah there you are21:14
anteayahow fortuitous21:15
ttxcrazy heh21:15
tristanCttx: neat! thank :)21:15
anteayaso glad you are staring at a screen at this time of night21:15
anteayalucky us21:15
ttxtristanC: should I tag march-2016-elections, april-2016-elections ?21:15
openstackgerritMerged openstack-infra/jeepyb: Remove argparse from requirements
*** ldnunes has quit IRC21:16
ianwfungi: nodepool tests still failing?  do you already know about this?21:16
tristanCttx: as you want, both are fine21:16
ttxlet's say march, start of period21:17
tristanCttx: that's what we did for sept-2014-elections, it was the start date21:17
openstackgerritStanislaw Pitucha proposed openstack-infra/project-config: Publish Anchor to pypi
ttxtristanC, fungi: alright, tag is up21:18
*** esikachev has quit IRC21:19
openstackgerritMerged openstack-infra/infra-manual: Fix path to jenkins/jobs/projects.yaml
rm_workanteaya: sorry, i'm in and out today -- but yeah, that's part of it (a big part). We don't have a good solution yet that is actually still workable to validate everything works (because any change could include changes to our image agent code), so even on other projects, we could break them if they don't have the absolute latest version of our agent and they are gating against our master branch, since the agent and the controller21:20
rm_work code are tightly coupled21:20
rm_workanteaya: but i agree we should do something21:20
rm_workanteaya: we are going to see what we can learn from _amrith_21:20
*** weshay has joined #openstack-infra21:20
rm_workmaybe he has good ideas that can help us :)21:20
openstackgerritStanislaw Pitucha proposed openstack-infra/project-config: Publish Anchor to pypi
openstackgerritMerged openstack-infra/jeepyb: Use oauth token to create GitHub project if available
anteayarm_work: well infra uses disk image builder to build images daily21:21
anteayadoes that use case parallel your use case at all?21:22
anteayais _amrith_ building images?21:22
*** kencjohnston has quit IRC21:22
clarkbok I think the dpkg/apt surgery is mostly done21:23
clarkbshould make puppet happier21:24
*** gnuoy_ has quit IRC21:24
clarkbI had to run dpkg --configure -a and in one case apt-get -f install21:24
*** Sukhdev has quit IRC21:25
*** hashar has quit IRC21:26
*** kgiusti has quit IRC21:29
*** boris-42 has joined #openstack-infra21:30
Shrewsclarkb: did you see my question for you on
*** hdd has quit IRC21:32
clarkbShrews: no21:32
clarkbShrews: but, it needs a new patchset now that they have working dns and ssl21:32
*** dprince has quit IRC21:32
clarkbShrews: and yes we don't specify any versions in our clouds.yaml currently21:32
Shrewsclarkb: ack. i'll wait for the new patch then21:33
*** mrmartin has quit IRC21:33
clarkbnibalizer: puppet has been running for almost 20 minutes on, any idea if that is normal?21:34
*** flepied has joined #openstack-infra21:36
clarkbShrews: new patchset pushed with up to date auth_url21:37
clarkbShrews: and my local testing has shown that no version overrides are required21:37
anteayagoing to go for a walk myself21:38
nibalizerclarkb: that doesn't seem normal to me21:38
nibalizerwhat is it doing?21:38
clarkbnibalizer: not sure, it ends up finishing around 20 minutes iwth no errors21:38
clarkbnibalizer: would probably have to check the syslog on the server21:38
clarkbmaybe it is puppeting both servers serially?21:39
nibalizeroh huh we have a new review.o.o don't we21:39
*** Sukhdev has joined #openstack-infra21:40
*** Sukhdev has quit IRC21:40
*** jistr has quit IRC21:40
*** hamzy has quit IRC21:40
*** Sukhdev has joined #openstack-infra21:41
*** openstackstatus has quit IRC21:42
*** nikhil_k has quit IRC21:42
*** openstackstatus has joined #openstack-infra21:44
*** ChanServ sets mode: +v openstackstatus21:44
*** annegentle has quit IRC21:44
*** annegentle has joined #openstack-infra21:44
*** nwkarsten has joined #openstack-infra21:45
*** keedya has quit IRC21:46
*** tpsilva has quit IRC21:47
*** amitgandhinz has quit IRC21:47
*** annegentle has quit IRC21:47
*** annegentle has joined #openstack-infra21:48
*** ihrachys has joined #openstack-infra21:49
*** nwkarsten has quit IRC21:49
*** |-paul-| has quit IRC21:49
clarkbnibalizer: looks like the logprocessor user switch over has happened without any major fallout21:50
*** sobersabre has quit IRC21:51
clarkbwaiting for puppet to finish up and figure out why logstash-worker03 and 06 claim to have had ssh errors21:51
clarkbI can manually ssh21:51
nibalizerfrom laptop or root@puppetmaster?21:51
*** annegentle has quit IRC21:51
clarkbwell both but ya21:51
*** sdague has quit IRC21:52
*** dims has joined #openstack-infra21:52
*** annegentle has joined #openstack-infra21:52
*** jamielennox|away is now known as jamielennox21:52
*** dims_ has quit IRC21:52
* nibalizer tries a -m ping21:54
nibalizerso yea the entire logstash-worker pool replies to pings21:54
*** ociuhandu has joined #openstack-infra21:55
*** egonzalez has joined #openstack-infra21:55
*** egonzalez has quit IRC21:56
*** thorst_ has quit IRC21:56
clarkbonly 03 and 06 seem to have been affected, everything else is happy21:56
*** ociuhandu has quit IRC21:56
clarkbI may wait for it to back around again and maybe it was a fluke network blip?21:57
*** dgonzalez has quit IRC21:57
*** ociuhandu has joined #openstack-infra21:57
dmsimardclarkb: is /root/ persisted between grenade runs ?21:57
clarkbdmsimard: yes it should be21:58
openstackgerritMonty Taylor proposed openstack-infra/jenkins-job-builder: Add PLAIN default for branch|project-compare-type
clarkbdmsimard: we don't really delete anything between grenade runs, we just start services out of the other location21:58
dmsimardMy logic could be bad but it looks like it gets created twice21:58
*** doug-fish has quit IRC21:59
*** dgonzalez has joined #openstack-infra21:59
clarkbya definitely looks like it is touching it both times22:00
* clarkb rereads the logic22:00
dmsimard[[ -e /root/fixed_disk_layout ]] && return 0 || sudo touch /root/fixed_disk_layout22:00
dmsimardfor me that reads if the file exists, return 0 or else touch the file22:00
*** annegentle has quit IRC22:00
*** melwitt has quit IRC22:00
dmsimardI tested it locally22:00
clarkbdmsimard: it could be a permissions issue22:01
clarkbI actually think that is the problem22:01
*** annegentle has joined #openstack-infra22:01
dmsimardhm, yeah22:01
dmsimardI can create the file elsewhere..22:01
clarkbdmsimard: you could write it to /etc/22:01
*** HeOS has quit IRC22:01
dmsimardeither way, looks like the job failed but for other reasons this time (I haven't yet looked into)22:02
*** thorst_ has joined #openstack-infra22:02
clarkbya it seems to have been happy with the swap that time around22:02
*** HeOS has joined #openstack-infra22:02
dmsimardand it did exactly what I wanted it to, bumped the swap on a vexxhost vm22:03
openstackgerritDavid Moreau Simard proposed openstack-infra/devstack-gate: Use dd to create swapfile, ensure swap amount == 8GB
nibalizerclarkb: im going to split for dinner in a bit22:03
nibalizerneed anything else?22:03
clarkbnibalizer: not right now, its mostly working. Once I get 03 and 06 puppeted I will disable puppet on them and approve the other changes then start working through the logstash 2.0 transition22:04
*** FallenPegasus has joined #openstack-infra22:04
*** thorst__ has joined #openstack-infra22:04
*** thorst_ has quit IRC22:06
*** shardy has quit IRC22:07
*** thorst__ has quit IRC22:08
clarkbnibalizer: looks like the run that just happened worked22:08
clarkbso transient network blip is what I will attribute that to22:08
*** jpr has joined #openstack-infra22:09
clarkbonce ansible is done with this pass I will be putting all 20 logstash workers in the emergency file then approve the logstash 2.0 upgrade changes22:10
*** rcernin has joined #openstack-infra22:11
*** esker has quit IRC22:13
*** aysyd has quit IRC22:13
*** _amrith_ is now known as amrith22:14
*** dgonzalez has quit IRC22:14
*** dgonzalez has joined #openstack-infra22:15
*** dgonzalez has quit IRC22:16
*** melwitt has joined #openstack-infra22:16
*** melwitt has quit IRC22:16
*** dgonzalez has joined #openstack-infra22:16
*** dgonzalez has quit IRC22:17
*** abregman has quit IRC22:17
*** dgonzalez has joined #openstack-infra22:18
*** jtomasek has quit IRC22:18
openstackgerritMerged openstack-infra/jenkins-job-builder: Add PLAIN default for branch|project-compare-type
openstackgerritIan Wienand proposed openstack-infra/project-config: [wip] Revert "Add CentOS AFS build job to project-config experimental"
openstackgerritIan Wienand proposed openstack-infra/project-config: CentOS AFS build : don't revoke sudo
openstackgerritMerged openstack-infra/system-config: Logstash 2.0 compat ES output rule
*** thorst_ has joined #openstack-infra22:22
*** rcernin has quit IRC22:22
openstackgerritMonty Taylor proposed openstack-infra/nodepool: Use shade for all OpenStack interactions
openstackgerritMerged openstack-infra/puppet-logstash: Install logstash 2.0
clarkbok worker19 is staged to be the first updated22:24
clarkb(well 20 was updated way back when so I guess 19 is technically second)22:25
*** sdake_ has joined #openstack-infra22:25
*** thorst__ has joined #openstack-infra22:26
*** woodster_ has joined #openstack-infra22:27
*** sdake has quit IRC22:28
*** thorst_ has quit IRC22:29
fungithanks ttx!22:30
jesusaurfungi: i have the zuul-cloner change passing tests locally, but the check queue seems to either hit an alarm or create too large of a subunit file22:31
clarkbjesusaur: the alarm is the test timeout22:31
clarkbjesusaur: and for too alrge of a subunit file that is because zuul is logging all the things22:32
fungiianw: i thought that failure mode was theorized to be due to the anonymous account still existing, but i guess that wasn't the case because you can see the "DELETE FROM mysql.user WHERE User='\'''\''; FLUSH PRIVILEGES;" there in the job log22:32
jesusaurclarkb: so what is the correct thing to do to mitigate the failures? increase the limits or reduce the logging?22:33
*** esker has joined #openstack-infra22:33
clarkbjesusaur: I think reduce the logging22:34
fungiclarkb: nibalizer: puppet shouldn't be trying to apply on both review.o.o servers, yolanda disabled the new one by uuid in the emergency file today right after it was built22:34
ianwfungi: yeah ... so i presume no other jobs are seeing this22:34
clarkbjesusaur: this is a simple sanity check saying "you are logging too much22:34
clarkbfungi: ok, it has been quicker since the 20 minute run22:34
*** mriedem has quit IRC22:34
fungiclarkb: it might have had a change to stuff manage-projects cares about and triggered the exec for that? it's crazy slow22:35
clarkbthat is possible22:36
fungiianw: nodepool is the only project with a job exhibiting this regression as far as i'm aware22:36
clarkbfungi: nodepool is also one of the only projcets that does db per test22:36
ianwfungi: i haven't had luck seeing it with anything in logstash, but i also see chatter about that22:36
fungiclarkb: maybe the grant we do doesn't provide sufficient permission to create databases? unfortunately it doesn't echo the failing query in the log22:37
*** nwkarsten has joined #openstack-infra22:37
ianwfungi: might it be worth holding a node so we can manually inspect mysql?22:37
fungipymysql.err.OperationalError: (1044, u"Access denied for user 'openstack_citest'@'%' to database 'WzwysOab_6923'")22:38
clarkbwell thats disappointing. erb array.to_s is not the same as irb array.to_s22:38
jeblairclarkb: what's an irb?22:38
*** esker has quit IRC22:38
fungiianw: i can do that quite easily, yes. want to recheck and let me know the node id? i'm happy to put your ssh key on it22:38
clarkbjeblair: interactive ruby22:38
ianwfungi: just a tick.  i think that's going to be quicker than trying to replicate it22:38
*** chlong has quit IRC22:39
*** gyee has joined #openstack-infra22:40
fungino argument from me. i could probably also hold us an unused bare-trusty node before we delete the remaining one if you want a side-by-side comparison22:40
openstackgerritMathieu Gagné proposed openstack-infra/jeepyb: Add support for GitHub Enterprise
fungier, before we delete the remaining oneS22:40
clarkbit appears to be making an array just fine but instaed of to_s resulting in [] and ""s its appending all the entries together22:40
ianwfungi: ubuntu-trusty-ovh-gra1-868749922:41
ianwget it while it's hot :)22:41
clarkbcrinkle: have a chance to look at vs and help me figure that out?22:42
*** nwkarsten has quit IRC22:42
*** _ari_ has quit IRC22:42
fungiianw: held. i'll aldo grab a bare-trusty out of the fresh pile. what ssh key do you want added?22:43
*** burgerk has quit IRC22:44
*** EricGonczer_ has joined #openstack-infra22:44
anteayamorning jhesketh22:44
*** hdd has joined #openstack-infra22:45
openstackgerritEmilien Macchi proposed openstack-infra/project-config: puppet: fix cookiecutter job template
EmilienMhey infra, can someone approve this patch please ?
fungiianw: jenkins@ for the held ubuntu-trusty node and jenkins@ for an unused bare-trusty i held22:48
*** EricGonczer_ has quit IRC22:49
openstackgerritMerged openstack-infra/project-config: CentOS AFS build : don't revoke sudo
*** hdd has quit IRC22:50
fungii too would like to see EmilienM's 288244 change merged since it updates that puppeted file for consistency with centos 7's (and recent fedora's) new defaults, and corrects a definite bug on our workers for those platforms22:50
crinkleclarkb: that is weird22:51
clarkbthe to_s is apparently redundant but doesn't change the output22:53
*** arxcruz has joined #openstack-infra22:54
*** EricGonczer_ has joined #openstack-infra22:55
*** ddieterly is now known as ddieterly[away]22:56
*** doug-fish has joined #openstack-infra22:59
clarkbI think I have an alternative I just need to figure out how to rmeove the trailing ,23:00
*** dingyichen has joined #openstack-infra23:00
*** sigmavirus24 is now known as sigmavirus24_awa23:01
crinkleclarkb: i'm trying to reproduce, not seeing it23:02
clarkbthis is puppet 3.8.623:03
*** doug-fish has quit IRC23:03
crinkleruby 1.9?23:03
*** doug-fish has joined #openstack-infra23:03
clarkblooks like 1.8.7?23:03
clarkbruby1.9.1 is not installed23:04
crinkleaha with 1.8.7 i can reproduce23:04
*** krtaylor has joined #openstack-infra23:05
* clarkb does a bad hack23:06
*** arxcruz has quit IRC23:07
*** doug-fish has quit IRC23:08
*** EricGonczer_ has quit IRC23:08
ianwfungi: can you please sudoer jenkins so i can query mysql as root23:08
fungiianw: sure thing, just a sec23:09
*** derekh has joined #openstack-infra23:09
fungiianw: done23:10
*** dmsimard is now known as dmsimard|pto23:10
ianwERROR 1141 (42000): There is no such grant defined for user 'openstack_ci' on host '%'23:11
ianwwell that seems odd23:11
*** krtaylor has quit IRC23:11
*** moravec has quit IRC23:12
*** annegentle has quit IRC23:12
*** annegentle has joined #openstack-infra23:12
crinkleclarkb: I think .inspect gives you what you want in ruby 1.8.723:13
openstackgerritClark Boylan proposed openstack-infra/system-config: Use ruby 1.8 compat erb for logstash config
clarkbcrinkle: instead of to_s?23:13
openstackgerritMerged openstack-infra/devstack-gate: Update IRC channel to use #openstack-qa
clarkbI can try that23:13
crinkleclarkb: yes23:13
*** annegentle has quit IRC23:14
clarkbcrinkle: is inspect something in ruby 1.9 and 2.0?23:14
*** doug-fish has joined #openstack-infra23:14
crinkleclarkb: yes and it seems to have the same behavior23:15
clarkblooks like inspect worked, I will amke that update instead23:15
*** _tzn has joined #openstack-infra23:15
*** annegentle has joined #openstack-infra23:15
*** tzn has quit IRC23:16
openstackgerritClark Boylan proposed openstack-infra/system-config: Use ruby 1.8 compat erb for logstash config
*** tzn_ has joined #openstack-infra23:16
*** ihrachys has quit IRC23:16
*** _tzn has quit IRC23:16
clarkbI self approved that23:17
clarkbmordred: nibalizer the next puppet-ansible thing I want to tackle is fixing how we copy our modules onto the hosts23:17
nibalizerfix how23:18
clarkbmordred: nibalizer we aren't clearing out old files if they have been deleted, I think we just need a slightly better rsync command23:18
nibalizergit clean?23:18
clarkbor that23:18
nibalizeri think yea ibstall_modules should clean23:18
clarkbnibalizer: I don't think the issue is puppetmaster side23:18
clarkbnibalizer: but it might be both23:18
*** ddieterly[away] is now known as ddieterly23:18
clarkbin any case I will try to look into that soonish23:18
nibalizeryea lets do both23:19
*** doug-fish has quit IRC23:19
*** asettle has joined #openstack-infra23:19
*** tonytan4ever has quit IRC23:20
*** EricGonczer_ has joined #openstack-infra23:20
openstackgerritJames E. Blair proposed openstack-infra/puppet-exim: Always specify driver first
jeblairnibalizer: can you check my erbing?23:21
nibalizerjeblair: sorry23:22
nibalizergerrit on phobe is super bad23:22
*** Qiming has joined #openstack-infra23:22
docaedoI'm trying to debug an issue with the app catalog, wondering if there are more verbose logs from puppet (i.e. something more useful than what I find at
jeblairis anyone here not on a phone?23:25
Clinti'm not on a phone23:25
jeblairClint: do you understand ruby erb?23:25
*** krtaylor has joined #openstack-infra23:25
* Clint peers.23:25
clarkbdocaedo: unfortunately the ansible stuff has us stashing those only in syslog on the host in question23:25
mordredcrinkle: ^^ ?23:25
jeblairClint: can you let me know if looks good?23:25
mordredjeblair: it looks good to me - but I'm not sure my erb is so stellar23:26
clarkbI know my erb is bad based on recent events23:26
*** dmsimard|pto has quit IRC23:26
*** xyang1 has quit IRC23:26
jeblairclarkb: i would have asked you but i did not want to prolong your pain23:26
docaedoclarkb: ok thanks .. I think one of the scripts that builds the JSON of assets is misbehaving, just trying to figure out how best to tell23:26
mgagnewe have a couple of example in puppet-horizon with ERB:
mgagnegood luck23:27
clarkbdocaedo: I only see recent successes in syslog23:27
*** esker has joined #openstack-infra23:27
clarkbdocaedo: nothing indicating one of the execs broke23:27
*** esker has quit IRC23:28
fungiianw: are you seeing signs that the grant executed by the job didn't "stick"?23:28
*** esker has joined #openstack-infra23:28
fungior maybe never happened at all?23:28
*** nikhil_k has joined #openstack-infra23:28
jeblair<%- @neutron_options_real.sort.each do |opt_name,opt_val| -%>23:28
jeblairmgagne: wait23:28
jeblairwhat does that do?23:28
mgagnetakes a "dict", sort it and loop23:28
docaedoclarkb: yeah don't think it's anything fully broken - it's just supposed to add a last_modified field to each asset (based on what git blame tells us), and locally works perfectly, but on the server that field is blank23:28
jeblairmgagne: is that a list of hash key, value pairs sorted by key?23:28
mgagnewe sort it due to ruby not preserving order in 1.823:29
docaedoclarkb: I can recreate that behavior locally but there are a few different things that could cause it, so I don't know what to try to fix :)23:29
jeblairmgagne: yeah, that's one of the things i'm fixing in but i didn't know you could do that, i thought you had to just sort the keys and access by key23:29
*** melwitt has joined #openstack-infra23:29
clarkbdocaedo: what should I be checking to narrow it down for you?23:29
mgagneboth could work I guess23:29
jeblairyeah.  at this point, i'm inclined to leave what i now know is not perfectly elegant ruby in place because i have tested it :)23:30
*** bpokorny_ has joined #openstack-infra23:30
jeblairbut i wuld have written it that way had i known23:30
docaedoclarkb: I think I need to merge a change to a script in the app catalog first - right now tools/ creates a file that gets merged with assets.yaml, and then it deletes that file,23:30
docaedoclarkb: so that file not being there when you look would be normal behavior (and it's only there for a moment, if assets.yaml is changed)23:31
jeblairi'm self-approving because it's a bugfix and it looks okay to Clint23:31
docaedoclarkb: I can have that file live in /tmp/ though, and not remove it until the start of the *next* run .. so until I do that, I don't think there's anything you can do to help23:32
clarkbdocaedo: ok23:32
docaedoclarkb: appreciate the offer though - thank you!23:32
crinklemordred: o/23:32
*** ddieterly is now known as ddieterly[away]23:33
*** bpokorny has quit IRC23:34
*** regXboi has quit IRC23:34
*** ddieterly[away] has quit IRC23:36
*** derekh has quit IRC23:39
*** Qiming has quit IRC23:40
openstackgerritJames E. Blair proposed openstack-infra/zuul: Add job tags
openstackgerritMerged openstack-infra/system-config: Use ruby 1.8 compat erb for logstash config
*** Qiming has joined #openstack-infra23:41
*** dmsimard|pto has joined #openstack-infra23:41
*** annegentle has quit IRC23:42
*** sdake_ has quit IRC23:42
clarkbnext puppet run should puppet logstash-worker18 with the above fix23:43
openstackgerritJames E. Blair proposed openstack-infra/zuul: Make all configuration in-repo configuration
clarkbif that works I will do half and half of the remainder23:43
openstackgerritFrank Kloeker proposed openstack-infra/system-config: Manifest for configuration translation checksite
*** Sukhdev has quit IRC23:44
openstackgerritMerged openstack-infra/puppet-exim: Always specify driver first
*** Sukhdev has joined #openstack-infra23:45
*** yamahata has joined #openstack-infra23:46
*** gongysh has joined #openstack-infra23:47
ianwfungi: not sure ... think it's something in nodepool that's creating the "random string" db23:47
ianwahh, i think i see it ...23:48
openstackgerritJames E. Blair proposed openstack-infra/puppet-exim: Add some whitespace between routers/transports
* anteaya watches the prime minister arrive for the state dinner23:52
mordredanteaya: arrive at your house?23:53
jeblairSotK, Zara: the storyboard exim patches have landed23:53
*** gordc has quit IRC23:54
anteayaI don't have state dinners23:54
anteayajeblair: yay the exim patches have landed23:55
anteayamordred: now mind you if the states wanted to have the prime minister to dinner at my house, I would do my best23:55
anteayaI'd even bring out gramma's china23:55
anteayathe white house23:55
anteayaapparently this is to make the us aware canada is a thing?23:56
*** Swami has joined #openstack-infra23:56
anteayaor to help23:56
ianwfungi: ok, so it's this bit ->
anteayayou too can watch:
anteayaso far guests are being announced23:57
ianwfungi: the nodepool testing creates a new user and tries to grant them their database23:57
*** Sukhdev_ has joined #openstack-infra23:57
*** Sukhdev has quit IRC23:57
fungiianw: aha, rather than using the openstack_citest user?23:57
ianwfungi: we have to setup "openstack_citest" WITH GRANT OPTION; if we want them to be able to do that23:57
ianw(grant to other users)23:57
mordredanteaya: ah. I undersatnd now23:58
fungiianw: sounds like a reasonable addition. if you want to write up that patch we'll get it merged before i go do manual jjb runs this evening23:58
fungiclarkb: ^ sound fine?23:58
ianwfungi: the other approach is to, i guess, refactor nodepool.  i'd be willing to be with some "blame" digging it was copied from somewhere else and nobody's ever really thought that much about it23:58

Generated by 2.14.0 by Marius Gedminas - find it at!