NobodyCam | party at mordred 's house during summit??? | 00:00 |
---|---|---|
mordred | boeing field is awesome | 00:00 |
anteaya | I'm in | 00:00 |
mordred | NobodyCam: s/during/instead of/ | 00:00 |
NobodyCam | lol | 00:00 |
clarkb | ya the museum there flies many of their planes | 00:00 |
NobodyCam | ++++ | 00:00 |
anteaya | even better | 00:00 |
fungi | first flight airstrip^H^H^H^H^Hport is also fun for watching (small) aircraft | 00:00 |
mordred | fungi: ++ | 00:00 |
fungi | and is a pleasantly short walk from my house | 00:00 |
fungi | the wright bros. monument plaza at the top of kill devil hill actually has a great view of the airstrip too | 00:01 |
jeblair | i saw a hacker project that's an ads-b receiver that looks up the origin/dest airport of planes overhead and displays it; so you can be like "oh, that's the 10:30 swa from san" | 00:01 |
*** HeOS has joined #openstack-infra | 00:01 | |
pleia2 | cool | 00:01 |
clarkb | jeblair: that is really neat | 00:01 |
fungi | NEAT! | 00:01 |
anteaya | ha ha ha | 00:02 |
anteaya | purring cat in my lap | 00:02 |
clarkb | when I was at garmin all our ground test equipment could be plugged into building antennas to pick up that tpye of info as if we were stationary craft on the ground | 00:02 |
anteaya | nice | 00:03 |
anteaya | so hey | 00:04 |
anteaya | if you watch 60 minutes you soon will have a segment on our prime minister | 00:04 |
*** annegentle_ has quit IRC | 00:05 | |
anteaya | lucky 60 minutes watchers | 00:05 |
mordred | jeblair: that seems like a hacker project I should maybe install on my patio | 00:05 |
*** piet_ has joined #openstack-infra | 00:05 | |
mordred | jeblair: maybe figure out a way to project tracking positoins of flights on the ceiling | 00:05 |
anteaya | to troll the cat? | 00:06 |
mordred | hahaha | 00:06 |
mordred | hey cat - chase the 9:45 to SLC! | 00:06 |
anteaya | ha ha ha | 00:06 |
pleia2 | projected google sky map (it does stars and planets), but for airplanes | 00:06 |
mordred | yah | 00:06 |
jeblair | anteaya: Jean Poutine was the best pm. | 00:06 |
*** HeOS has quit IRC | 00:06 | |
anteaya | I like stars and planes better | 00:06 |
mordred | pleia2: in the location I'm in it would be a really cool projection | 00:06 |
anteaya | jeblair: absolutely | 00:06 |
fungi | no doubt dhs/tsa would have a fit | 00:07 |
mordred | hahaha | 00:07 |
anteaya | jeblair: can't do better than Poutine | 00:07 |
jeblair | mordred: found the blog: http://jeremybmerrill.com/blog/2016/01/flyover.html | 00:07 |
mordred | fungi: listen - if THAT's what they decide to be unhappy at me about | 00:07 |
anteaya | ha ha ha | 00:07 |
fungi | i'm pretty sure that's far down the list of things they're unhappy with _you_ about | 00:07 |
fungi | actually, with the amount you fly, they probably all know you by sight (at every major airport) | 00:08 |
mordred | jeblair: awesome. that seems totally like a thing I could build differently than the fellow there | 00:09 |
jeblair | mordred: yeah, it could use some polish :) | 00:09 |
jeblair | but a good idea and resources | 00:09 |
mordred | I | 00:09 |
mordred | I'm thinking collect all the data I can hear | 00:09 |
mordred | then define a fixed map grid (since my patio does not move) | 00:10 |
jeblair | mordred: lucky you | 00:10 |
jeblair | ours move in ca | 00:10 |
fungi | i was just thinking that ;? | 00:10 |
mordred | and display the flight info as the plane - perhaps with bigger/smaller depending on altitude | 00:10 |
anteaya | ha ha ha | 00:10 |
clarkb | SergeyLukjanov: how many items to index? | 00:10 |
anteaya | moving patio as a feature | 00:11 |
fungi | also, no idea what emoji ;? is. i think it expresses consternation at hitting shift at the wrong times | 00:11 |
pleia2 | secretly SergeyLukjanov took off for the weekend like 20 minutes ago | 00:11 |
SergeyLukjanov | clarkb 6k :( | 00:11 |
pleia2 | oh good o/ | 00:11 |
anteaya | fungi: noted | 00:11 |
jeblair | it's too late to beat the rush to tahoe | 00:11 |
anteaya | ha ha ha | 00:11 |
SergeyLukjanov | no, actually, it's bad, because it's processing nearly 1 per second... | 00:11 |
mordred | I've been imagining that SergeyLukjanov has been skiing this whole time | 00:11 |
pleia2 | mordred: haha | 00:12 |
SergeyLukjanov | :) | 00:12 |
mordred | and doing the clarkb irc-on-phone | 00:12 |
*** rbrndt has quit IRC | 00:12 | |
jeblair | i think we're learning something really interesting about this version of gerrit | 00:12 |
anteaya | yes | 00:12 |
pleia2 | "spilled snow on my laptop, sec" | 00:12 |
mordred | jeblair: yes. 'interesting' | 00:12 |
fungi | were i more industrious, i'd go back to our channel log from the upgrade and find out how long the reindex took | 00:12 |
fungi | but instead i'm catching up on e-mail | 00:12 |
*** yamamoto_ has joined #openstack-infra | 00:12 | |
clarkb | its late on friday | 00:13 |
fungi | my beer is getting lonely in the fridge | 00:13 |
clarkb | I need beer | 00:13 |
mordred | good point. it's definitely caipirinha-o'clock | 00:14 |
fungi | i still have a new batch of contributor discount codes for summit registration to generate and send once gerrit comes back up | 00:14 |
clarkb | mordred: oh! I have limes I can do that instead | 00:14 |
mordred | \o/ | 00:14 |
jeblair | clarkb: i thought you said you had LINES you could do instead | 00:14 |
clarkb | mordred: what cachaca do you use? novo fogo has treated me well but apparently its a for foreign markets only cachaca so seems less authentic | 00:14 |
jeblair | and, well, i mean, er, ... | 00:14 |
clarkb | jeblair: I am not that exciting :) | 00:14 |
SergeyLukjanov | 98% yay! | 00:14 |
clarkb | mordred: so I figure you drink something else | 00:14 |
mordred | clarkb: I usually just do leblon because it's a solid performer that I can always fine | 00:15 |
fungi | though i did analyze the >100 bounces i received from the initial batch, map it back to the contributor contact data, and come up with a list of 33 people who did not receive their invites because all their e-mail addresses bounced | 00:15 |
SergeyLukjanov | have 3 hours before cinema, hopefully, it'll end in 2 hours ;) | 00:15 |
mordred | clarkb: my favorite is jacuba, but that's harder to get my hands on | 00:15 |
*** dims has quit IRC | 00:15 | |
mordred | clarkb: avua is also a really good one you can get in the US | 00:15 |
*** hashar has quit IRC | 00:15 | |
clarkb | I think I have had leblon and prefer novo fogo to it | 00:16 |
nibalizer | fungi: we wouldn't want your beer to feel neglected | 00:16 |
SergeyLukjanov | should we update the topic? | 00:16 |
clarkb | the other two are new to me | 00:16 |
fungi | quite a few of the unlucky 33 i know, at least in passing, so need to reach out to them and see if they want a resend (and would like to update their e-mail addresses in gerrit) | 00:16 |
SergeyLukjanov | with +1 another hor? | 00:16 |
anteaya | clarkb: yay for being boring | 00:16 |
anteaya | SergeyLukjanov: I think those following along have gotten the point | 00:17 |
fungi | SergeyLukjanov: sure, update to 01:00 utc | 00:17 |
fungi | can't hurt | 00:17 |
SergeyLukjanov | #status alert Gerrit is offline for maintenance, ETA updated to 01:00 utc | 00:18 |
openstackstatus | SergeyLukjanov: sending alert | 00:18 |
mordred | clarkb: there's also one that samueldmq brought me - but I drank it all and can't remember the name | 00:18 |
fungi | samueldmq no doubt remembers the name | 00:18 |
anteaya | mordred: was it good? | 00:19 |
* Sam-I-Am realizes he missed airplane talk | 00:19 | |
*** ccrouch1 has joined #openstack-infra | 00:19 | |
clarkb | mordred: I find this is a common problem with cachaca | 00:19 |
Sam-I-Am | and beer talk | 00:19 |
anteaya | Sam-I-Am: we are now onto beer talk | 00:20 |
clarkb | caipirinha is basically a little lime then fill glass with cachcaca | 00:20 |
clarkb | you run out quickly | 00:20 |
anteaya | oh I think beer talk is still active | 00:20 |
Sam-I-Am | anteaya: i might try flying myself to the austin summit | 00:20 |
-openstackstatus- NOTICE: Gerrit is offline for maintenance, ETA updated to 01:00 utc | 00:20 | |
*** ChanServ changes topic to "Gerrit is offline for maintenance, ETA updated to 01:00 utc" | 00:20 | |
anteaya | Sam-I-Am: wooooo | 00:20 |
anteaya | Sam-I-Am: I want to go up with you sometime | 00:20 |
mordred | clarkb: novo fogo is good ... if you can get your hands on sagatiba it's excellent | 00:20 |
Sam-I-Am | end of april is potentially decent weather for that. i'll have a backup ticket on southwest if it doesnt. | 00:20 |
anteaya | Sam-I-Am: hey can we fly from austin to mordred's house? | 00:20 |
anteaya | Sam-I-Am: he has a backyard! | 00:21 |
Sam-I-Am | probably. where does mordred live? | 00:21 |
anteaya | Sam-I-Am: dallas | 00:21 |
jeblair | i don't know his backyard is _that_ big | 00:21 |
*** ybathia has joined #openstack-infra | 00:21 | |
anteaya | ha ha ha | 00:21 |
Sam-I-Am | plenty of airports there. probably an hour or so flight from austin. | 00:21 |
Sam-I-Am | jeblair: i'm installing a STOL kit | 00:21 |
Sam-I-Am | land in 50 feet. | 00:22 |
mordred | anteaya: you can. but you can also drive | 00:22 |
Sam-I-Am | flying > driving | 00:22 |
*** ccrouch has quit IRC | 00:22 | |
Sam-I-Am | well, when you're not flying commercial, that is | 00:22 |
anteaya | ++ | 00:22 |
mordred | clarkb: and http://www.mapadacachaca.com.br/wp-content/uploads/2012/08/cachaca_nega_fulo_carvalho.jpeg | 00:22 |
anteaya | mordred: is your backyard 50 feet? | 00:22 |
Sam-I-Am | its texas. its probably 500 feet. | 00:22 |
Sam-I-Am | everything is bigger there | 00:23 |
pleia2 | hehe | 00:23 |
Sam-I-Am | now mordred just needs a ford F-550 lifted to 3 feet off the ground | 00:23 |
openstackstatus | SergeyLukjanov: finished sending alert | 00:23 |
Sam-I-Am | thats how you texas | 00:23 |
* anteaya wonders if reindexing is happening on a server in texas | 00:23 | |
anteaya | ha ha ha | 00:23 |
fungi | anteaya: in fact, it IS! | 00:23 |
Sam-I-Am | texans have pride for their pride | 00:23 |
anteaya | fungi: that explains it | 00:23 |
clarkb | mordred: I will have to see what I can get in oregon | 00:24 |
fungi | our servers, including gerrit and our trove database for it, live on servers in the rackspace dallas/ft. worth datacenter | 00:24 |
clarkb | mordred: I have also just finished off my novo fogo with this caipirinha | 00:24 |
anteaya | oh perfect | 00:24 |
anteaya | they are doing friday beer | 00:24 |
mordred | clarkb: probably not nega fulo. possibly sagatiba, I think there is someone who imports it to the US | 00:24 |
Sam-I-Am | nothing beats a beer on the porch | 00:26 |
*** redixin has quit IRC | 00:26 | |
clarkb | mordred: I hear you can buy entire shopping malls in texas and use them for other things | 00:26 |
Sam-I-Am | assuming its not -20 out | 00:26 |
clarkb | Sam-I-Am: drinking beer your brewed yourself on the porch :) | 00:26 |
Sam-I-Am | clarkb: also good | 00:26 |
Sam-I-Am | i havent brewed in a while. its not easy living in a downtown condo. | 00:27 |
Sam-I-Am | space and temperature regulation | 00:27 |
clarkb | I have some friends that would use the park area with the grills | 00:27 |
clarkb | because of space issues | 00:27 |
*** rguillebert has quit IRC | 00:27 | |
Sam-I-Am | fermenting temps, at least | 00:27 |
mordred | clarkb: yup. they're great places to install very big fans | 00:28 |
Sam-I-Am | i could exchange the area my bed consumes for a chest freezer with external thermostat | 00:28 |
anteaya | mordred: you buy a shopping mall and install a very big fan? | 00:28 |
Sam-I-Am | just sleep in the chest | 00:28 |
clarkb | I actually looked at a house that had an awesome cellar space that would have been perfect for brewing, except it had no central heat which is :( | 00:28 |
anteaya | mordred: is there more to this vision? | 00:28 |
clarkb | I need my central heat | 00:28 |
mordred | anteaya: visit the castle sometime and you'll see what I mean | 00:29 |
*** piet_ has quit IRC | 00:29 | |
anteaya | clarkb: good call | 00:29 |
jeblair | clarkb: you have computers, right? | 00:29 |
clarkb | jeblair: hahahaha | 00:29 |
Sam-I-Am | clarkb: i had a place with a basement a few years ago. i could easily lager in the winter... summer was stable enough for ales | 00:29 |
anteaya | mordred: the castle? | 00:29 |
clarkb | jeblair: I have actually downsized quite a bit, I no longer have a ps3 space heater | 00:29 |
anteaya | mordred: dallas has a castle? | 00:29 |
mordred | anteaya: where our rax friends in san antonio call home | 00:29 |
anteaya | ah | 00:29 |
clarkb | jeblair: I like small fanless low heat computers | 00:29 |
anteaya | been there | 00:29 |
jeblair | i have shut off the heat register in the room i use as an office -- haven't had it on all winter | 00:29 |
mordred | anteaya: then you should have see the very large fans | 00:29 |
*** angdraug has quit IRC | 00:29 | |
pleia2 | jeblair: admittedly, "winter" | 00:30 |
jeblair | pleia2: indeed :) | 00:30 |
anteaya | oh no, sorry not in san antonio, been to rax in austin | 00:30 |
*** mtanino has quit IRC | 00:30 | |
jeblair | pleia2: there were those 2 weeks when it almost got down to freezing | 00:30 |
jeblair | must have been like, what, 45 or something | 00:30 |
anteaya | goodness me | 00:30 |
*** Sukhdev has quit IRC | 00:30 | |
anteaya | how did you survive? | 00:30 |
pleia2 | hehe | 00:30 |
anteaya | oh btw it was so lovely last night, sky was cloudless, saw all the stars | 00:31 |
jeblair | anteaya: well, like half the stars, right? | 00:31 |
fungi | not only does san antonio have a castle, it also have a very memorable catholic mission compound | 00:31 |
anteaya | -20C and my fingers were cold by the time I finished shoveling | 00:31 |
fungi | er, has | 00:31 |
anteaya | jeblair: yes half :) | 00:31 |
clarkb | jeblair: we routed the home with a cellar right after it snowed so it was chilly and I basically decided I couldn't live like that | 00:31 |
clarkb | routed == toured | 00:32 |
anteaya | and the snow made the lovely scrunching noise it does at -20 | 00:32 |
clarkb | after half a caipirinha | 00:32 |
anteaya | loved being outside last nigh | 00:32 |
anteaya | t | 00:32 |
smcginnis | anteaya: Sounds like Minnesota. :) | 00:32 |
anteaya | too warm tonight | 00:32 |
* bswartz notices the channel topic and frowns | 00:32 | |
anteaya | smcginnis: indeed mn would know the pleasure I describe | 00:32 |
anteaya | bswartz: frowning doesn't change it | 00:32 |
jeblair | bswartz: hey, be happy, i think you're the first person ever to notice it! | 00:32 |
*** |-paul-| has quit IRC | 00:33 | |
* bswartz smiles | 00:33 | |
mordred | clarkb: wow. half a caipirinha has a different effect on you than on me | 00:33 |
anteaya | much nicer | 00:33 |
fungi | if nothing else, gerrit maintenances provide an excuse to take up drinking | 00:33 |
anteaya | bswartz: we are discussing beer | 00:33 |
anteaya | and fans | 00:33 |
smcginnis | jeblair: Hah! | 00:33 |
Sam-I-Am | clarkb: its been springish enough here lately to break out the apricot blonde ale | 00:34 |
Sam-I-Am | fungi: s/gerrit maintenances/opensource | 00:34 |
clarkb | Sam-I-Am: http://www.beeradvocate.com/beer/profile/2391/42203/ that is one of my favorite beers | 00:35 |
*** gangil has quit IRC | 00:35 | |
*** sdake_ has joined #openstack-infra | 00:35 | |
*** tmcpeak has joined #openstack-infra | 00:35 | |
*** tmcpeak has left #openstack-infra | 00:35 | |
*** sdake has quit IRC | 00:35 | |
*** EricGonczer_ has quit IRC | 00:35 | |
Sam-I-Am | clarkb: mmmm | 00:36 |
Sam-I-Am | clarkb: this one is from dry dock brewing | 00:36 |
fungi | looks tasty, adding to my to-do list | 00:36 |
clarkb | fungi: next time you are in portland we should hit cascade | 00:36 |
fungi | didn't we try last time? | 00:36 |
jeblair | can the next announcement be "Gerrit is offline for maintenance, it'll be up when it's up" ? | 00:36 |
clarkb | fungi: maybe? | 00:36 |
Sam-I-Am | clarkb: i'm pretty much in the craft brew capitol, so its easier to walk to a brewery than make beer :/ | 00:36 |
anteaya | jeblair: I'm for that | 00:37 |
clarkb | jeblair: or Gerrit is dead long live Gerrit? | 00:37 |
fungi | "Gerrit is offline for maintenance, drink up!" | 00:37 |
clarkb | Sam-I-Am: me too :) | 00:37 |
anteaya | or happy weekend | 00:37 |
Sam-I-Am | clarkb: denver? | 00:37 |
clarkb | Sam-I-Am: portland oregon | 00:37 |
Sam-I-Am | anteaya: gerrit offline means i should stop working :) | 00:37 |
mordred | it should just be "gerrit is offline" - that's the only thing we can state with certainty | 00:37 |
anteaya | that's what it means | 00:37 |
Sam-I-Am | maybe gerrit should go offline from 6pm to 8am | 00:37 |
fungi | mordred: if you observe it, you change its state | 00:37 |
fungi | so there is no certainty | 00:37 |
mordred | fungi: well played | 00:38 |
anteaya | there's a cat in my lap | 00:38 |
Sam-I-Am | clarkb: plenty of good beer there | 00:38 |
anteaya | no box in sight | 00:38 |
mordred | fungi: to be fair, the chances of me observing gerrit being down causing it to move to the up state are low | 00:38 |
fungi | also, mad props to xkcd for ironic use of the term "local group" | 00:38 |
jeblair | even nodepool is bored | 00:38 |
Sam-I-Am | mordred: is it causing gravity waves? | 00:38 |
jeblair | 2016-02-13 00:38:46,565 DEBUG nodepool.ProviderManager: Manager tripleo-test-cloud-rh1 ran task ListServersTask in 32.8094689846s | 00:39 |
clarkb | Sam-I-Am: apparently breweries per capita goes Vermont 1, Oregon 2, Colorado 3 | 00:39 |
jeblair | okay, not completely bored | 00:39 |
*** EricGonczer_ has joined #openstack-infra | 00:39 | |
Sam-I-Am | clarkb: we need to try harder. | 00:39 |
mordred | jeblair: that is not a good list-servers api response time | 00:39 |
mordred | I tink that db needs to have a table cleaned | 00:39 |
jeblair | clarkb: wow, 3 breweries per capita in co? | 00:39 |
clarkb | jeblair: no no :) that is a ranking within the USA | 00:40 |
jeblair | mordred: i suspect you are right. and maybe an index? | 00:40 |
clarkb | Vermont is 8.6 breweries per 100k residents | 00:40 |
mordred | jeblair: yup. was fixed 2 or 3 releases of openstack agao | 00:40 |
anteaya | tonight is the night for indexing | 00:40 |
anteaya | apparently | 00:40 |
clarkb | sorry thats 100k 21+ residents | 00:40 |
Sam-I-Am | clarkb: riiiight | 00:40 |
clarkb | oregon is 7.4 | 00:40 |
clarkb | colorado 6.1 | 00:40 |
Sam-I-Am | maybe colorado just has a lot more people :) | 00:40 |
fungi | so 0.0086 breweries per person? | 00:41 |
*** mriedem has joined #openstack-infra | 00:41 | |
jeblair | that doesn't seem like nearly enough | 00:41 |
fungi | no, wait, i'm doing percents | 00:41 |
clarkb | fungi: per person 21 and over | 00:41 |
mordred | clarkb: how does that change if you look at burlington vs portland vs fort collins though? | 00:41 |
*** HeOS has joined #openstack-infra | 00:41 | |
fungi | 0.000086 breweries per | 00:41 |
fungi | that seems like far too few | 00:41 |
clarkb | mordred: looking at https://www.brewersassociation.org/statistics/by-state/ let me see if I can get per city | 00:41 |
Sam-I-Am | fort collins was awesome when i lived there. i hear it has even more breweries now. | 00:41 |
anteaya | it has many | 00:41 |
Sam-I-Am | mordred: texas has plenty of great beers too | 00:41 |
fungi | having grown up in/near asheville, it seemed pretty high on the breweries per capita gradient | 00:42 |
mordred | clarkb: there's a thrillist article | 00:42 |
*** markvoelker has joined #openstack-infra | 00:42 | |
*** tzn has joined #openstack-infra | 00:42 | |
mordred | clarkb: it claims portland is #1 | 00:42 |
jeblair | fungi: also on the backyard stills gradient? | 00:42 |
mordred | but I'm not sure it's ranking on per-capita counts | 00:42 |
fungi | jeblair: sure, depending on how far outside city limits you're willing to hike ;) | 00:42 |
clarkb | mordred: portland has 91 breweries in the metro area | 00:43 |
clarkb | mordred: the best part is each neighborhood has its brewery | 00:43 |
fungi | jeblair: keep in mind it's for medicinal purposes, of course | 00:43 |
clarkb | and many have several | 00:43 |
SergeyLukjanov | 3k CRs left | 00:43 |
SergeyLukjanov | still 98% | 00:43 |
SergeyLukjanov | super sad | 00:43 |
anteaya | wooo 98% | 00:43 |
fungi | the used car radiators add some unique flavor profiles | 00:43 |
*** annegentle_ has joined #openstack-infra | 00:43 | |
anteaya | halfway there | 00:43 |
anteaya | do you have a favourite radiator? | 00:44 |
Sam-I-Am | a watched completion % indicator never completes | 00:44 |
calebb | hey looks like the openstackclient that is packaged in apt may be broken, the pip one works for me but that one doesnt | 00:44 |
*** apoorvad has joined #openstack-infra | 00:44 | |
fungi | tasting notes: rubbing alcohol with a hint of '57 chevy | 00:44 |
anteaya | ha ha ha | 00:44 |
*** ybathia has quit IRC | 00:44 | |
clarkb | fungi: I always find it hilarious when people show up to the brew store and buy 200lb of corn and 50lb of barley and rye. Yes that barley and rye is for your tractor I am sure :) | 00:44 |
Sam-I-Am | calebb: not surprising | 00:45 |
jeblair | i'm going to be afk for about 20 mins or so -- if i'm not back in time, gate.sh and check.sh are in ~root on zuul.o.o | 00:45 |
fungi | tasty, tasty biodiesel? | 00:45 |
fungi | thanks jeblair! | 00:45 |
calebb | ok, just making sure it's known, doesnt really bother me since I can just use the pip stuff, but figured someone might want to know | 00:45 |
clarkb | fungi: you can distill alcohol for fuel without all the crazy federal stuff, just minimal fedreal stuff | 00:45 |
Sam-I-Am | calebb: gotta be careful mixing pip with distro packages. can cause various problems. | 00:45 |
*** sridhar_ram has quit IRC | 00:46 | |
clarkb | fungi: but there is bsaically 0 reason to buy barley or rye for fuel purposes because it costs much more | 00:46 |
fungi | calebb: which distro/release? | 00:46 |
*** dims has joined #openstack-infra | 00:46 | |
Sam-I-Am | calebb: the openstackclient in cloud-archive/liberty for 14.04 mostly works, but not sure what your particular problem is | 00:46 |
*** tzn has quit IRC | 00:46 | |
*** markvoelker has quit IRC | 00:46 | |
calebb | fungi: openstack 0.3.0 | 00:47 |
calebb | on ubuntu 14.04 | 00:47 |
*** kzaitsev_mb has quit IRC | 00:47 | |
Sam-I-Am | oh deat | 00:47 |
Sam-I-Am | dear | 00:47 |
fungi | calebb: neat! that's ancient | 00:47 |
Sam-I-Am | cloud-archive/liberty is 1.7.0 | 00:48 |
Sam-I-Am | so that must be the icehouse version? | 00:48 |
fungi | at least in openstack history terms, 0.3.0 is 2 years old | 00:48 |
fungi | https://git.openstack.org/cgit/openstack/python-openstackclient/refs/tags | 00:48 |
calebb | welp | 00:48 |
Sam-I-Am | yeah that makes sense now | 00:48 |
*** piet has joined #openstack-infra | 00:48 | |
fungi | so, basically the one that was current when ubuntu 14.04 was frozen | 00:48 |
Sam-I-Am | 14.04 released with icehouse | 00:48 |
Sam-I-Am | unless you use cloud-archive, thats what you get | 00:48 |
calebb | ok that makes sense | 00:48 |
* Sam-I-Am ponders what osc 0.3.0 was like | 00:49 | |
Sam-I-Am | and now i need another beer | 00:49 |
fungi | i didn't think it _actually_ existed back then | 00:49 |
Sam-I-Am | me neither! lol | 00:49 |
* calebb removes and just runs it out of a virtualenv | 00:49 | |
Sam-I-Am | it maybe supported one command? | 00:49 |
crinkle | 0.3.0 was right before 1.0.0 so it wasn't that bad | 00:49 |
fungi | for all intents and purposes, it wasn't 1.0 until 14 months ago | 00:49 |
clarkb | and I only just started using it in the last week >_> | 00:49 |
*** ddieterly has joined #openstack-infra | 00:49 | |
fungi | there were 3 subsequent releases between 0.3.0 and 1.0.0 | 00:50 |
Sam-I-Am | aight, i need to head out. time to locate edible objects. | 00:50 |
fungi | (0.3.1, 0.4.0, 0.4.1) | 00:50 |
anteaya | Sam-I-Am: good hunting | 00:50 |
*** ybathia has joined #openstack-infra | 00:50 | |
Sam-I-Am | see y'all later | 00:50 |
anteaya | night | 00:50 |
anteaya | I got a copy of Ender's Game the movie the other day, not bad | 00:51 |
smcginnis | anteaya: That was my daughter's favorite for awhile. | 00:52 |
mordred | well, 0.3.0 is pre-os-client-config | 00:52 |
smcginnis | anteaya: I must have seen it a dozen times. | 00:52 |
mordred | so that's sad making | 00:52 |
*** yamamoto_ has quit IRC | 00:52 | |
anteaya | smcginnis: nice, it has some good moments | 00:53 |
*** dims has quit IRC | 00:53 | |
anteaya | I liked the training room scenes | 00:53 |
smcginnis | anteaya: Yeah, not bad. Entertaining. :) | 00:53 |
*** ybathia has quit IRC | 00:53 | |
anteaya | I thought they did a fairly good job | 00:53 |
* anteaya admires the moon | 00:54 | |
anteaya | so beautiful tonight | 00:54 |
*** DericHorn-HP has joined #openstack-infra | 00:55 | |
*** davideagnello has quit IRC | 00:55 | |
*** DericHorn-HP has quit IRC | 00:55 | |
*** rfolco has quit IRC | 00:56 | |
*** DericHorn-HP has joined #openstack-infra | 00:56 | |
*** yamamoto_ has joined #openstack-infra | 00:57 | |
*** ajmiller has quit IRC | 00:57 | |
*** amrith is now known as _amrith_ | 00:58 | |
mordred | so - just so you're all clear on this ... | 00:58 |
mordred | when you decide, in the Dallas area, "tonight I'd like to go eat some Tex-Mex" | 00:58 |
mordred | it's not exactly the easiest task to select the place to go | 00:59 |
asselin | clarkb, you around? full bandersnatch sync finished and left a 'todo' file, Rerunning it says "2016-02-13 00:54:14,400 INFO: 702 packages to sync. | 00:59 |
asselin | " Trying again says the same thing. any ideas? | 00:59 |
fungi | do they call it "tex-mex" there? seems redundant | 00:59 |
clarkb | you cna't walk around the corner and go to your neighbors tex mex restaurant? | 00:59 |
mordred | clarkb: I can | 00:59 |
clarkb | asselin: sorry no I don't know | 00:59 |
mordred | clarkb: but if I always did that I'd miss the cornucopia of options out there | 00:59 |
clarkb | mordred: because tat is what I do except instead of around the corner it is a 10 minute walk | 00:59 |
fungi | i assumed in texas they just called that "food" | 00:59 |
clarkb | there is a tacqueria behind a grocery store that greghaynes approves of that is about a 40 minute wak | 01:00 |
mordred | fungi: nah - gotta distinguish from bbq and tacquerias | 01:00 |
mordred | there are three types of food here | 01:00 |
fungi | oh, great point | 01:00 |
mordred | clarkb: there is a tacqueria in a gas station at the edge of my neighborhood | 01:01 |
greghaynes | mordred: does cachos exist up there? | 01:01 |
*** DericHorn-HP has joined #openstack-infra | 01:01 | |
clarkb | ooh get this 20% of beer sold in oregon is craft beer made in oregon | 01:01 |
fungi | mordred: the sign says "eat here, get gas" | 01:01 |
clarkb | the first state to hit that milestone | 01:01 |
mordred | greghaynes: I do not know this cachos you speak of | 01:01 |
mordred | clarkb: nice! | 01:01 |
greghaynes | 24hr solidly decent texmex that I always get in Houston | 01:01 |
mordred | greghaynes: AH. nope, I do not believe we have that here | 01:02 |
greghaynes | It's good just got off the plane food | 01:02 |
mordred | if I want 24 hour texmex I usually go to taco cabana | 01:02 |
clarkb | thats actually pretty amazing if you think about that | 01:02 |
* anteaya is waiting to hear the 3 types of food | 01:02 | |
clarkb | 1 in 5 beers purchases here were made here by craft brewers | 01:02 |
greghaynes | mordred: hahah, I have so many taco c stories from growing up down there | 01:02 |
mordred | anteaya: texmex, bbq and tacquerias | 01:02 |
anteaya | mordred: ah thank you | 01:02 |
mordred | oh. crappit | 01:02 |
mordred | fungi: I'm wrong. there's four | 01:02 |
fungi | mordred: whaddya mean booze ain't food? | 01:02 |
clarkb | that is competing against anneuhser, coors, miller and crafter from everywhere else | 01:02 |
mordred | anteaya: I also apologize | 01:02 |
anteaya | thank you | 01:02 |
mordred | fungi, anteaya: texmex, bbq, tacquerias ... and STEAK | 01:03 |
anteaya | ah okay that makes sense now | 01:03 |
mordred | yah. I mean | 01:03 |
mordred | wow | 01:03 |
nibalizer | still indexing? | 01:03 |
clarkb | nibalizer: si | 01:03 |
anteaya | my might be evicted for a mistake like that | 01:03 |
fungi | nibalizer: either than or SergeyLukjanov finally went to the cinema | 01:03 |
anteaya | if you made it a second time | 01:03 |
nibalizer | point being, more alerts? | 01:03 |
SergeyLukjanov | 99% -- less than 2k left | 01:04 |
clarkb | nibalizer: only if you say something like "gerrit still indexing will be done hwen it is done. We recommend tacos and beer in the meantime" | 01:04 |
SergeyLukjanov | clarkb ++ | 01:04 |
nibalizer | heh | 01:04 |
anteaya | leave the spelling mistake in | 01:04 |
clarkb | I am trying to think of the last time I bought beer that wasn't brewed here | 01:04 |
clarkb | I cna't remember | 01:04 |
clarkb | it has been so long | 01:04 |
greghaynes | clarkb: way to portland | 01:05 |
clarkb | greghaynes: ddi you see what I said? 20% of beer bought here is brewed here | 01:05 |
clarkb | and I am clearly part of the problem | 01:05 |
mordred | anteaya: oh! I should tell you about this farm | 01:05 |
greghaynes | clarkb: That doesn't even count the beer we brew | 01:06 |
clarkb | greghaynes: exactly | 01:06 |
SergeyLukjanov | Reindexing changes: projects: 100% (910/910), 99% (275810/277491), done | 01:06 |
mordred | anteaya: there's a northern suburb of dallas called Plano - it's currently mostly known for being a sea of mini-mansions and planned communities | 01:06 |
SergeyLukjanov | Reindexed 275810 changes in 9864.1s (28.0/s) | 01:06 |
SergeyLukjanov | real164m33.273s | 01:06 |
SergeyLukjanov | user272m30.175s | 01:06 |
SergeyLukjanov | sys27m29.520s | 01:06 |
SergeyLukjanov | yay! | 01:06 |
fungi | clarkb: mexico? | 01:06 |
clarkb | woot | 01:06 |
mordred | SergeyLukjanov: WOOT! | 01:06 |
clarkb | fungi: does that count? I didn't buy the pacifico in oregon :) | 01:06 |
mordred | anteaya: but many years ago it did not used to be a suburb with tons of mini-mansions - it used to be farms and stuff | 01:06 |
anteaya | mordred: awesome | 01:06 |
pleia2 | SergeyLukjanov: yay! | 01:06 |
fungi | clarkb: well, you didn't specify where you were when you bought beer | 01:07 |
*** DericHorn-HP has quit IRC | 01:07 | |
anteaya | mordred: yup, the farms I would have liked | 01:07 |
mordred | anteaya: in any case, there's this one farm that decided, as the construction happened all around, to just not sell or leave | 01:07 |
fungi | so anyway, time to proceed with turning things back on, starting with gerrit? | 01:07 |
clarkb | fungi: well the stat is specifically people buying beer in oregon | 01:07 |
*** Sukhdev has joined #openstack-infra | 01:07 | |
pleia2 | yep, shall I start gerrit now? | 01:07 |
clarkb | fungi: yes lets do it | 01:07 |
anteaya | fungi: I thought zuul comes online first | 01:07 |
mordred | anteaya: so in the middle of the sprawl, there is a long-lived family farm | 01:07 |
SergeyLukjanov | pleia2 yup | 01:07 |
anteaya | mordred: wow, good for them | 01:07 |
mordred | anteaya: that is the local go-to place to get your raw milk and other things such as that | 01:07 |
pleia2 | gerrit is running | 01:07 |
anteaya | mordred: I want to go there | 01:08 |
anteaya | mordred: do they have bees? | 01:08 |
pleia2 | who's on zuul? | 01:08 |
*** bgmccollum has joined #openstack-infra | 01:08 | |
mordred | anteaya: http://www.luckylayla.com/ | 01:08 |
clarkb | jeblair said the files are there waiting | 01:08 |
SergeyLukjanov | I'll do zuul | 01:08 |
clarkb | SergeyLukjanov: kk | 01:08 |
*** gyee has quit IRC | 01:08 | |
SergeyLukjanov | review.o.o seems to be working | 01:08 |
fungi | let's remember, when we get to the starting zuul again step, to make sure we _don't_ reenqueue any changes until we've confirmed that most/all of our node types are represented registered in gearman again | 01:08 |
pleia2 | renaming gerrit groups | 01:09 |
anteaya | mordred: yum | 01:09 |
clarkb | fungi: good point | 01:09 |
fungi | otherwise it's a carnage of NOT_REGISTERED results all over the joint | 01:09 |
anteaya | SergeyLukjanov: did you get that? | 01:09 |
anteaya | SergeyLukjanov: fungi's point on zuul? | 01:09 |
jeblair | back | 01:09 |
SergeyLukjanov | anteaya I hope so | 01:10 |
SergeyLukjanov | I've just started zuul | 01:10 |
SergeyLukjanov | not yet started gate.sh / check.sh | 01:10 |
anteaya | awesome | 01:10 |
SergeyLukjanov | waiting for zuul alive | 01:10 |
clarkb | we can check nodepool | 01:10 |
SergeyLukjanov | jeblair do you want to continue from this point? | 01:10 |
jeblair | nope i'll watch | 01:11 |
fungi | check the gearman status to make sure rarer node types like bare-precise and devstack-centos7 are showing up with at least one worker | 01:11 |
SergeyLukjanov | okay | 01:11 |
clarkb | I think we are good | 01:11 |
jeblair | clarkb: i told nodepool to reset while we were down, so we should have all types built -- just a matter of making sure they are registered as fungi says | 01:11 |
clarkb | bare-preicse and fedora-23 and devstack-centos7 are all in nodepool at least | 01:11 |
clarkb | ah ok | 01:11 |
fungi | it's more of an issue when we also restart nodepool at the same time | 01:11 |
fungi | since it can take a lot longer in those cases | 01:12 |
SergeyLukjanov | seems we're ok for starting gate/check .sh | 01:12 |
SergeyLukjanov | correct? | 01:12 |
fungi | yeah, gate first obviously | 01:12 |
clarkb | hrm maybe I should plan to do my nodepool restart tomorrow | 01:12 |
fungi | that gives nodepool a chance to start ramping up before we slam it with the check changes | 01:12 |
openstackgerrit | Clark Boylan proposed openstack-infra/nodepool: Dont connect to gearman in cmd unless needed https://review.openstack.org/279829 | 01:13 |
clarkb | jeblair: ^ | 01:13 |
anteaya | should we get our rename patches in the gate | 01:13 |
jeblair | (though, we should have restarted nodepool to clear its jobs out of the gearman queue) | 01:13 |
anteaya | first? | 01:13 |
anteaya | then reenqueue the gate? | 01:13 |
*** ddieterly has quit IRC | 01:13 | |
clarkb | anteaya: either way depending on how quickly we want ot restart puppet | 01:13 |
*** ccrouch1 has quit IRC | 01:13 | |
fungi | oh, now that nodepool gearmans we need to add a step for that to the rename maintenance process i guess | 01:14 |
jeblair | fungi: well, ideally, we'll fix the bugs that are orphaning the jobs | 01:14 |
*** salv-orlando has joined #openstack-infra | 01:14 | |
anteaya | clarkb: okay thanks | 01:14 |
fungi | ahh, these aren't a permanent problem then. i missed this was related to the gearman job leak in nodepool | 01:14 |
jeblair | fungi: right, only reason to restart it atm is because we have 900 nodepool jobs that will never be run | 01:14 |
jeblair | fungi: this distorts the zuul job queue graph so that it's currently meaningless | 01:15 |
jeblair | which is unfortunate, since it's my primary measure of system capacity | 01:15 |
*** ddieterly has joined #openstack-infra | 01:15 | |
*** achanda has joined #openstack-infra | 01:15 | |
fungi | so we should restart nodepool with zuul offline to clean that up, or restarting nodepool while zuul is online should also be fine? | 01:15 |
jeblair | oh, but of course, geard restarted too, so they're gone | 01:16 |
jeblair | so nevermind | 01:16 |
SergeyLukjanov | check.sh is running now | 01:16 |
SergeyLukjanov | renaming changes approved | 01:16 |
SergeyLukjanov | only enable puppet left and ,gitreview CRs | 01:17 |
*** _nadya_ has joined #openstack-infra | 01:17 | |
SergeyLukjanov | I can do both later today to not keep all of you folks awake for a few more hours | 01:17 |
clarkb | jeblair: where do those come from? | 01:17 |
clarkb | nodepool has been decently solid recentlty | 01:17 |
clarkb | with gearman jobs | 01:18 |
jeblair | clarkb: i dunno; we can check in a couple days i guess | 01:18 |
jeblair | clarkb: last i looked the red line did not hit 0 when the nodes graph wasn't maxed out, but maybe it's better now? | 01:19 |
jeblair | the graph on the status page looks like it might be okay now, so maybe is fixed | 01:19 |
jeblair | i'll check status early next week | 01:20 |
clarkb | kk | 01:20 |
anteaya | so when can we clear the topic? | 01:21 |
*** _nadya_ has quit IRC | 01:21 | |
* slogan621 can see his review now in gerrit code review, so things must be progressing | 01:21 | |
jeblair | now i think | 01:21 |
*** salv-orlando has quit IRC | 01:21 | |
fungi | should be safe to #status ok now that changes are reenqueuing | 01:21 |
anteaya | SergeyLukjanov: care to do the honours? | 01:22 |
SergeyLukjanov | #status ok | 01:22 |
openstackstatus | SergeyLukjanov: sending ok | 01:22 |
*** apoorvad has quit IRC | 01:22 | |
*** shashank_hegde has quit IRC | 01:22 | |
fungi | and somehow i doubt we'll forget that we really, really, REALLY want to use the online reindexing next maintenance | 01:22 |
SergeyLukjanov | anteaya, sorry, didn't get it | 01:22 |
jeblair | SergeyLukjanov: for future reference (don't do this now), you can do "#status ok Gerrit is back up" and it will do a channel notice of that | 01:22 |
SergeyLukjanov | jeblair ack | 01:22 |
EmilienM | thank you guys! | 01:23 |
*** ChanServ changes topic to "Discussion of OpenStack Developer and Community Infrastructure | docs http://docs.openstack.org/infra/ | bugs https://storyboard.openstack.org/ | source https://git.openstack.org/cgit/openstack-infra/ | channel logs http://eavesdrop.openstack.org/irclogs/%23openstack-infra/" | 01:23 | |
* slogan621 sends thanks too | 01:23 | |
SergeyLukjanov | check is still re-enqueueing | 01:23 |
jeblair | nodepool is busy | 01:23 |
anteaya | SergeyLukjanov: you got it, thank you | 01:23 |
*** DericHorn-HP has joined #openstack-infra | 01:23 | |
SergeyLukjanov | when check.sh will be finished, I'll run away to avoid wife killing me, will check in a few hours for the merged renaming jobs and enable back the puppetmaster cron | 01:24 |
SergeyLukjanov | anteaya jeblair fungi ^^ | 01:24 |
openstackstatus | SergeyLukjanov: finished sending ok | 01:24 |
*** piet has quit IRC | 01:24 | |
jeblair | SergeyLukjanov: thanks, enjoy your evening! (and if one of us re-enables it first, we'll let you know with a ping in irc) | 01:25 |
SergeyLukjanov | thx! | 01:25 |
jeblair | the mergers look happy | 01:25 |
anteaya | SergeyLukjanov: thank you for all your help | 01:25 |
anteaya | SergeyLukjanov: have a good evening and say hello to DinaBelova | 01:25 |
SergeyLukjanov | yeah, seems like we'll have them merged in a few mins | 01:25 |
jeblair | they will be just a little sluggish at first as they re-clone all the repos, but we have 8 of them so | 01:25 |
SergeyLukjanov | anteaya thx! | 01:26 |
anteaya | nice to work with you again | 01:26 |
DinaBelova | anteaya :) thanks :) | 01:26 |
anteaya | DinaBelova: hey there | 01:26 |
DinaBelova | :D | 01:26 |
anteaya | SergeyLukjanov: is telling stories about you | 01:26 |
SergeyLukjanov | anteaya me too ;) | 01:26 |
anteaya | :) | 01:26 |
DinaBelova | ahaha :D | 01:26 |
jeblair | DinaBelova: sorry we kept him late :) | 01:26 |
*** Sukhdev has quit IRC | 01:26 | |
DinaBelova | jeblair np :D | 01:26 |
fungi | SergeyLukjanov: yes, please get back to your evening! | 01:26 |
anteaya | DinaBelova: you are off to see a movie? | 01:26 |
fungi | SergeyLukjanov: DinaBelova: sorry for the unexpected lengthy maintenance! | 01:26 |
DinaBelova | anteaya we're still going to :D it looks like we have a chance :) | 01:27 |
anteaya | DinaBelova: SergeyLukjanov go go go | 01:27 |
openstackgerrit | Merged openstack-infra/project-config: rename ceilometer-specs to telemetry-specs https://review.openstack.org/248897 | 01:27 |
*** sdake_ is now known as sdake | 01:27 | |
jeblair | when this system is fast, it's fast ^ | 01:28 |
*** yamamoto_ has quit IRC | 01:28 | |
fungi | indeed | 01:29 |
openstackgerrit | Merged openstack-infra/project-config: Rename sahara-scenario -> sahara-tests https://review.openstack.org/269815 | 01:29 |
openstackgerrit | Merged openstack-infra/project-config: Rename fuel-plugin-astra to fuel-plugin-astara https://review.openstack.org/279138 | 01:29 |
jeblair | well, how about i re-enable puppet? | 01:30 |
mordred | ++ | 01:30 |
fungi | that should be perfectly safe now that those have merged | 01:30 |
jeblair | SergeyLukjanov, infra-root: i have re-enabled puppet on puppetmaster | 01:31 |
jeblair | and i cleverly did it just 1 minute past the cron schedule | 01:31 |
SergeyLukjanov | check.sh finished successfully | 01:31 |
jeblair | so 1:45 should be the next run | 01:31 |
SergeyLukjanov | I'm dissappearing ;) | 01:31 |
jeblair | SergeyLukjanov: i thought you were gone! bye :) | 01:31 |
SergeyLukjanov | have a good weekend folks, extended weekend :) | 01:31 |
*** tphummel has quit IRC | 01:32 | |
jeblair | you too | 01:32 |
*** madhu_ak has quit IRC | 01:32 | |
jeblair | anteaya: is https://review.openstack.org/279257 ready? you want to unwip and recheck it? | 01:33 |
*** bpokorny_ has joined #openstack-infra | 01:33 | |
*** tiswanso has quit IRC | 01:34 | |
*** bpokorny_ has quit IRC | 01:35 | |
*** pvaneck has quit IRC | 01:35 | |
*** markvoelker has joined #openstack-infra | 01:37 | |
asselin | any ideas what would cause bandersnatch to hit a url that doesn't exist? http://paste.openstack.org/show/486904/ | 01:37 |
*** bpokorny has quit IRC | 01:37 | |
nibalizer | cool | 01:42 |
fungi | asselin: what's more likely is that there's a stale negative cache in fastly it's tripping over | 01:42 |
*** mriedem has quit IRC | 01:42 | |
clarkb | asselin: mayb that project deleted that file | 01:42 |
asselin | bandersnatch can't handle it? | 01:43 |
clarkb | well it steps through the things | 01:43 |
clarkb | so first it wiould be sdist added | 01:43 |
clarkb | 404 | 01:43 |
clarkb | then sdist removed : noop | 01:43 |
*** mriedem has joined #openstack-infra | 01:44 | |
jeblair | i issued PURGE requests on the index, but they're still showing up | 01:44 |
jeblair | asselin: so you probably need dstufft to look into it | 01:45 |
asselin | there's a few others too that are failing for similar reasons | 01:45 |
asselin | 2016-02-13 01:45:44,826 DEBUG: Getting https://pypi.python.org/packages/3.4/c/cvsslib/cvsslib-0.5.1-py3-none-any.whl (serial None) | 01:46 |
asselin | 2016-02-13 01:45:44,878 ERROR: Error syncing package: cvsslib@1843108 | 01:46 |
*** tiswanso has joined #openstack-infra | 01:46 | |
jeblair | clarkb: are there some new-provider changes i can review before i head out? | 01:47 |
*** openstackgerrit has quit IRC | 01:47 | |
*** openstackgerrit has joined #openstack-infra | 01:47 | |
*** nikhil_k has joined #openstack-infra | 01:48 | |
*** baoli has joined #openstack-infra | 01:50 | |
*** DericHorn-HP has quit IRC | 01:50 | |
*** nikhil has quit IRC | 01:51 | |
*** EricGonczer_ has quit IRC | 01:51 | |
anteaya | jeblair: unwiped, rechecked on 279257 | 01:52 |
clarkb | jeblair: no cleaning up after fires and gerrit restart were distractions | 01:52 |
jeblair | ok | 01:53 |
clarkb | jeblair: crinkle and nibalizer are working on making the ssl stuff for infra cloud match up what was discussed yesterday | 01:53 |
clarkb | then those changes can go in and any osic changes can follow | 01:53 |
jeblair | clarkb: should we just put osic in? | 01:53 |
jeblair | i mean, honestly, they should hardly even conflict | 01:53 |
jeblair | i'm not sure we really need to wait -- that was when i thought we were going to merge them within 5 minutes of each other :) | 01:53 |
clarkb | jeblair: maybe? I am currently semi busy with house purchasing paperwork, but the details are in the passwords file if you are interested in pushing them | 01:53 |
clarkb | otherwise I may find time tomorro | 01:54 |
clarkb | w | 01:54 |
anteaya | clarkb: you have another house? | 01:54 |
clarkb | anteaya: no, but I have to restart the loan process so bank needs new forms signed | 01:54 |
anteaya | clarkb: ah | 01:54 |
anteaya | thought once you were approved for a mortgage you had that until you got a house | 01:54 |
anteaya | maybe that is just up here | 01:54 |
clarkb | anteaya: my experience is that each application is per property as that property acts as collateral and all that | 01:55 |
*** roxanaghe has quit IRC | 01:55 | |
clarkb | so right now it it is pre work to speed process should we find a property | 01:55 |
anteaya | yup | 01:55 |
anteaya | makes sense | 01:55 |
clarkb | (pre approval and all that jazz also last years tax docs just came out so they want those) | 01:55 |
anteaya | sigh | 01:55 |
clarkb | anteaya: the way the market is here if you don't have preapproval as soon as you find something you want there is a good chance it will be gone by the time you do | 01:57 |
clarkb | so this is all about duck lining | 01:57 |
anteaya | yup | 01:57 |
anteaya | was like that in toronto when we sold my grandmother's house | 01:57 |
anteaya | everyone showed up with paperwork and a house inspector | 01:58 |
anteaya | and the three bidders had to outdo each other | 01:58 |
anteaya | when her church, saint james bond united had its final service | 01:59 |
anteaya | we took her past the house | 01:59 |
anteaya | they had knocked it down | 01:59 |
anteaya | :) | 01:59 |
anteaya | :( | 01:59 |
anteaya | I had to make an excuse that we would be late for service | 01:59 |
anteaya | which is why we never stopped | 01:59 |
anteaya | I couldn't show here where her house used to be | 01:59 |
anteaya | :( | 01:59 |
*** annegentle_ has quit IRC | 02:01 | |
*** baoli has quit IRC | 02:01 | |
clarkb | ya its a thing | 02:01 |
clarkb | especially if you have a large lot, you can split them several ways often | 02:02 |
asselin | fungi, jeblair clarkb fyi I deleted those packages that error out from the 'todo' file and now its caught up. (Except for those pacakges of course) | 02:02 |
anteaya | clarkb: :( | 02:02 |
*** wolsen has quit IRC | 02:03 | |
*** sridhar_ram has joined #openstack-infra | 02:03 | |
*** baoli has joined #openstack-infra | 02:04 | |
*** wolsen has joined #openstack-infra | 02:04 | |
slogan621 | clarkb: should I ping you later, add you to the review so you can stay updated if there are changes? | 02:05 |
slogan621 | I'm about to bail. | 02:05 |
clarkb | slogan621: I havne't reviewed it because I won't be around to fix it if the creation fails | 02:05 |
clarkb | slogan621: maybe ping me tomorrow and I can do it then? | 02:05 |
jeblair | asselin: that won't work in the long run | 02:05 |
slogan621 | sure, but it's the weekend, we can wait until next week | 02:05 |
slogan621 | sound good? | 02:06 |
*** yamamoto_ has joined #openstack-infra | 02:06 | |
clarkb | slogan621: yup | 02:06 |
* slogan621 wonders how a new project that no-op'd everything and has no jenkins jobs could possibly fail :-P | 02:06 | |
asselin | jeblair, I know...but will help get stuff working again and minimize weekend issues. I follow-up with dstufft next week. | 02:07 |
slogan621 | please don't tell me :-O | 02:07 |
* slogan621 waves bye | 02:07 | |
*** slogan621 is now known as slogan621-away | 02:07 | |
*** dimtruck is now known as zz_dimtruck | 02:07 | |
*** zz_dimtruck is now known as dimtruck | 02:12 | |
*** achanda has quit IRC | 02:14 | |
*** xyang1 has quit IRC | 02:15 | |
*** annegentle_ has joined #openstack-infra | 02:19 | |
*** slogan621-away has quit IRC | 02:21 | |
*** yamamoto_ has quit IRC | 02:21 | |
*** baoli has quit IRC | 02:22 | |
*** tphummel has joined #openstack-infra | 02:22 | |
*** yamamoto_ has joined #openstack-infra | 02:22 | |
*** yamamoto_ has quit IRC | 02:22 | |
*** yamamoto_ has joined #openstack-infra | 02:22 | |
*** annegentle_ has quit IRC | 02:23 | |
*** cfarquhar has quit IRC | 02:24 | |
anteaya | jeblair: the requirements change failed the grenade test: https://jenkins06.openstack.org/job/gate-grenade-dsvm/26287/console | 02:24 |
anteaya | looks like a volume delete error | 02:27 |
anteaya | I'm getting tired | 02:30 |
anteaya | I'd like to stay and solve this but my eyes are getting heavy | 02:31 |
clarkb | I think that volume test thing is a known issue | 02:31 |
anteaya | which bug is that? | 02:31 |
*** thorst has joined #openstack-infra | 02:31 | |
anteaya | I'm looking at the elastic-recheck page | 02:31 |
clarkb | I don't remember but one of my changes hit it and dims and others seemd to know about it | 02:32 |
anteaya | oh so recheck? | 02:32 |
clarkb | ya | 02:32 |
anteaya | okay thanks | 02:32 |
*** thorst has quit IRC | 02:33 | |
*** thorst has joined #openstack-infra | 02:33 | |
anteaya | hey Registered Users has disappeared in the groups list | 02:34 |
anteaya | from | 02:34 |
fungi | that's what we were hoping | 02:34 |
anteaya | this page still needs a refresh of some sort: https://review.openstack.org/#/admin/projects/openstack-dev/sandbox,access | 02:34 |
*** dimtruck is now known as zz_dimtruck | 02:35 | |
anteaya | as the group disappeared but sandbox still lists the group with a link as its group of preference | 02:35 |
clarkb | anteaya: can you +2 on sandbox? | 02:35 |
anteaya | no | 02:36 |
anteaya | https://review.openstack.org/#/c/279763/ | 02:36 |
anteaya | just -1 +1 | 02:36 |
anteaya | :( | 02:36 |
fungi | grr, that's why i suspected we needed to clean up the groups files on all the affected repos | 02:36 |
fungi | zaro: ^ | 02:36 |
anteaya | so we got rid of the group but not the effect | 02:36 |
anteaya | :( | 02:36 |
fungi | zaro: you said you tested that? | 02:36 |
clarkb | oh right the map will be wrong | 02:37 |
anteaya | we sure had the time | 02:37 |
fungi | well, i asked explicitly if it would be needed, and he assured me that gerrit ignores any group in a groups file for which the uuid is not present in the db | 02:37 |
anteaya | I remember that exchange | 02:37 |
fungi | so maybe this is some other issue? | 02:37 |
anteaya | perhaps? | 02:37 |
anteaya | I only know I'm tired and I fly out tomorrow | 02:38 |
fungi | anteaya: get some rest and have a safe flight | 02:38 |
anteaya | thanks | 02:38 |
anteaya | my requirements change still needs to be merged | 02:38 |
anteaya | it is in check again, keeps failing tests | 02:39 |
anteaya | can you help it through? | 02:39 |
anteaya | I think that is the last thing I needed to do | 02:39 |
anteaya | https://review.openstack.org/#/c/279257/ | 02:40 |
anteaya | sorry I can't stay until its done | 02:41 |
anteaya | g'night | 02:41 |
anteaya | and thank you | 02:41 |
*** mriedem has quit IRC | 02:41 | |
*** thorst has quit IRC | 02:41 | |
*** sdake has quit IRC | 02:43 | |
* Sam-I-Am returns | 02:44 | |
*** achanda has joined #openstack-infra | 02:45 | |
*** salv-orlando has joined #openstack-infra | 02:46 | |
*** dims has joined #openstack-infra | 02:46 | |
*** dims has quit IRC | 02:50 | |
*** chlong has joined #openstack-infra | 02:51 | |
*** nikhil has joined #openstack-infra | 02:52 | |
*** salv-orlando has quit IRC | 02:54 | |
*** nikhil_k has quit IRC | 02:55 | |
*** ggillies has quit IRC | 02:55 | |
*** sridhar_ram has quit IRC | 02:56 | |
*** yamamoto_ has quit IRC | 02:56 | |
*** woodster_ has quit IRC | 02:56 | |
*** ggillies has joined #openstack-infra | 02:57 | |
*** [1]Thelo has joined #openstack-infra | 02:59 | |
*** arxcruz has quit IRC | 02:59 | |
*** Daisy has joined #openstack-infra | 03:00 | |
*** Thelo has quit IRC | 03:01 | |
*** [1]Thelo is now known as Thelo | 03:01 | |
*** zul__ has quit IRC | 03:03 | |
*** zul has joined #openstack-infra | 03:03 | |
*** sdake has joined #openstack-infra | 03:08 | |
*** esikachev has joined #openstack-infra | 03:08 | |
*** Daisy has quit IRC | 03:08 | |
*** Daisy has joined #openstack-infra | 03:09 | |
*** piet_ has joined #openstack-infra | 03:10 | |
*** HenryG has quit IRC | 03:12 | |
*** HenryG has joined #openstack-infra | 03:13 | |
*** esikachev has quit IRC | 03:13 | |
*** cfarquhar has joined #openstack-infra | 03:14 | |
*** cfarquhar has quit IRC | 03:14 | |
*** cfarquhar has joined #openstack-infra | 03:14 | |
*** cfarquhar has quit IRC | 03:14 | |
*** yamahata has quit IRC | 03:15 | |
*** ddieterly has quit IRC | 03:15 | |
*** _nadya_ has joined #openstack-infra | 03:17 | |
*** thorst has joined #openstack-infra | 03:19 | |
*** tphummel has quit IRC | 03:19 | |
*** thorst has quit IRC | 03:19 | |
*** annegentle_ has joined #openstack-infra | 03:19 | |
*** thorst has joined #openstack-infra | 03:20 | |
*** zeih has joined #openstack-infra | 03:21 | |
*** cfarquhar has joined #openstack-infra | 03:21 | |
*** cfarquhar has quit IRC | 03:21 | |
*** cfarquhar has joined #openstack-infra | 03:21 | |
*** achanda has quit IRC | 03:21 | |
*** _nadya_ has quit IRC | 03:22 | |
*** achanda has joined #openstack-infra | 03:22 | |
*** baoli has joined #openstack-infra | 03:25 | |
*** cfarquhar has quit IRC | 03:25 | |
*** zeih has quit IRC | 03:26 | |
*** rfolco has joined #openstack-infra | 03:28 | |
*** thorst has quit IRC | 03:29 | |
*** baoli has quit IRC | 03:30 | |
*** shashank_hegde has joined #openstack-infra | 03:31 | |
*** amotoki has quit IRC | 03:37 | |
*** sdake has quit IRC | 03:37 | |
*** markvoelker has quit IRC | 03:41 | |
*** achanda has quit IRC | 03:48 | |
openstackgerrit | Anne Gentle proposed openstack-infra/infra-specs: Adds specification for serving REST API info on developer.openstack.org https://review.openstack.org/276484 | 03:49 |
*** baoli has joined #openstack-infra | 03:49 | |
*** FallenPegasus has joined #openstack-infra | 03:51 | |
*** chlong has quit IRC | 03:54 | |
*** yamamoto has joined #openstack-infra | 03:56 | |
*** rfolco has quit IRC | 04:01 | |
*** piet_ has quit IRC | 04:02 | |
*** salv-orlando has joined #openstack-infra | 04:03 | |
*** yamamoto has quit IRC | 04:04 | |
*** roxanaghe has joined #openstack-infra | 04:05 | |
*** baoli has quit IRC | 04:07 | |
*** piet has joined #openstack-infra | 04:10 | |
*** roxanaghe has quit IRC | 04:10 | |
*** shashank_hegde has quit IRC | 04:11 | |
*** salv-orlando has quit IRC | 04:13 | |
*** sambetts|afk has quit IRC | 04:15 | |
*** ddieterly has joined #openstack-infra | 04:16 | |
*** sambetts has joined #openstack-infra | 04:16 | |
*** salv-orlando has joined #openstack-infra | 04:17 | |
*** tzn has joined #openstack-infra | 04:18 | |
*** Daisy has quit IRC | 04:21 | |
*** ddieterly has quit IRC | 04:22 | |
*** salv-orlando has quit IRC | 04:22 | |
*** tzn has quit IRC | 04:22 | |
*** annegentle_ has quit IRC | 04:23 | |
*** tiswanso has quit IRC | 04:23 | |
*** dkehn has quit IRC | 04:25 | |
*** thorst has joined #openstack-infra | 04:26 | |
*** dkehn has joined #openstack-infra | 04:27 | |
*** sdake has joined #openstack-infra | 04:30 | |
*** nikhil has quit IRC | 04:32 | |
*** thorst has quit IRC | 04:33 | |
*** nikhil has joined #openstack-infra | 04:33 | |
*** sdake_ has joined #openstack-infra | 04:34 | |
*** piet has quit IRC | 04:35 | |
*** sdake has quit IRC | 04:35 | |
*** markvoelker has joined #openstack-infra | 04:41 | |
*** achanda has joined #openstack-infra | 04:44 | |
*** markvoelker has quit IRC | 04:46 | |
*** esikachev has joined #openstack-infra | 04:48 | |
*** Daisy has joined #openstack-infra | 05:09 | |
*** Daisy has quit IRC | 05:14 | |
lifeless | sigmavirus24_awa: ack; opened will look on monday | 05:17 |
lifeless | ansiwen: pbr builds source distributions, so yes, everything in git should be copied | 05:18 |
*** ddieterly has joined #openstack-infra | 05:18 | |
*** _nadya_ has joined #openstack-infra | 05:18 | |
*** |-paul-| has joined #openstack-infra | 05:20 | |
*** yamahata has joined #openstack-infra | 05:21 | |
*** harlowja_at_home has quit IRC | 05:21 | |
*** salv-orlando has joined #openstack-infra | 05:21 | |
*** piet has joined #openstack-infra | 05:22 | |
*** zeih has joined #openstack-infra | 05:22 | |
*** armax has quit IRC | 05:22 | |
*** ddieterly has quit IRC | 05:22 | |
*** _nadya_ has quit IRC | 05:23 | |
*** zeih has quit IRC | 05:26 | |
zaro | fungi: i thought i did test it. take a look at my comment in https://review.openstack.org/#/c/270537/ | 05:27 |
*** ganso has quit IRC | 05:29 | |
openstackgerrit | Merged openstack/requirements: Change project name from sahara-scenario to sahara-tests https://review.openstack.org/279257 | 05:30 |
*** _nadya_ has joined #openstack-infra | 05:30 | |
*** salv-orlando has quit IRC | 05:31 | |
*** thorst has joined #openstack-infra | 05:31 | |
*** sdake_ has quit IRC | 05:32 | |
*** thorst has quit IRC | 05:38 | |
*** shashank_hegde has joined #openstack-infra | 05:38 | |
*** piet has quit IRC | 05:41 | |
*** piet has joined #openstack-infra | 05:52 | |
*** bpokorny has joined #openstack-infra | 05:53 | |
*** vgridnev has joined #openstack-infra | 05:53 | |
*** piet has quit IRC | 06:07 | |
*** daemontool has joined #openstack-infra | 06:13 | |
*** esikachev has quit IRC | 06:18 | |
*** ddieterly has joined #openstack-infra | 06:19 | |
*** esikachev has joined #openstack-infra | 06:20 | |
*** ddieterly has quit IRC | 06:23 | |
*** hdd has quit IRC | 06:28 | |
*** bpokorny has quit IRC | 06:30 | |
openstackgerrit | OpenStack Proposal Bot proposed openstack/requirements: Updated from generate-constraints https://review.openstack.org/279735 | 06:32 |
openstackgerrit | OpenStack Proposal Bot proposed openstack-infra/project-config: Normalize projects.yaml https://review.openstack.org/279744 | 06:33 |
*** thorst has joined #openstack-infra | 06:37 | |
*** ildikov has quit IRC | 06:42 | |
*** markvoelker has joined #openstack-infra | 06:43 | |
*** thorst has quit IRC | 06:44 | |
*** markvoelker has quit IRC | 06:47 | |
*** _nadya_ has quit IRC | 06:52 | |
*** piet has joined #openstack-infra | 06:57 | |
*** piet has quit IRC | 07:12 | |
*** ddieterly has joined #openstack-infra | 07:19 | |
*** zeih has joined #openstack-infra | 07:22 | |
*** ddieterly has quit IRC | 07:25 | |
*** zeih has quit IRC | 07:27 | |
*** esikachev has quit IRC | 07:30 | |
*** piet has joined #openstack-infra | 07:30 | |
*** mrmartin has joined #openstack-infra | 07:32 | |
*** harlowja_at_home has joined #openstack-infra | 07:33 | |
*** vgridnev has quit IRC | 07:35 | |
*** tobiash_ has joined #openstack-infra | 07:36 | |
*** tobiash has quit IRC | 07:36 | |
*** wolverin_ has joined #openstack-infra | 07:40 | |
*** thorst has joined #openstack-infra | 07:40 | |
*** esikachev has joined #openstack-infra | 07:43 | |
*** wolveri__ has joined #openstack-infra | 07:44 | |
*** wolverin_ has quit IRC | 07:45 | |
*** wolverineav has quit IRC | 07:45 | |
*** piet has quit IRC | 07:45 | |
*** kzaitsev_mb has joined #openstack-infra | 07:45 | |
*** thorst has quit IRC | 07:48 | |
*** wolveri__ has quit IRC | 07:49 | |
*** achanda has quit IRC | 07:56 | |
*** wolverineav has joined #openstack-infra | 07:58 | |
*** kzaitsev_mb has quit IRC | 07:58 | |
*** wolverineav has quit IRC | 08:02 | |
*** mrmartin has quit IRC | 08:07 | |
*** salv-orlando has joined #openstack-infra | 08:09 | |
*** wolverineav has joined #openstack-infra | 08:13 | |
*** hockeynut_afk has joined #openstack-infra | 08:14 | |
*** betherly_ has joined #openstack-infra | 08:14 | |
*** jlk` has joined #openstack-infra | 08:14 | |
*** markusz has joined #openstack-infra | 08:14 | |
*** markus_z has quit IRC | 08:15 | |
*** hockeynut has quit IRC | 08:15 | |
*** betherly has quit IRC | 08:15 | |
*** jlk has quit IRC | 08:15 | |
*** pedroalvarez has quit IRC | 08:15 | |
*** HenryG has quit IRC | 08:15 | |
*** andymaier has quit IRC | 08:15 | |
*** betherly_ is now known as betherly | 08:16 | |
*** pedroalvarez has joined #openstack-infra | 08:17 | |
*** andymaier has joined #openstack-infra | 08:17 | |
*** HenryG has joined #openstack-infra | 08:17 | |
*** Jeffrey4l has joined #openstack-infra | 08:17 | |
openstackgerrit | Merged openstack-infra/project-config: Normalize projects.yaml https://review.openstack.org/279744 | 08:18 |
*** wolverineav has quit IRC | 08:19 | |
*** mrmartin has joined #openstack-infra | 08:20 | |
*** wolverineav has joined #openstack-infra | 08:20 | |
*** ddieterly has joined #openstack-infra | 08:20 | |
*** zhurong has joined #openstack-infra | 08:21 | |
*** zhurong has quit IRC | 08:22 | |
*** shashank_hegde has quit IRC | 08:24 | |
*** ddieterly has quit IRC | 08:25 | |
*** achanda has joined #openstack-infra | 08:25 | |
*** mrmartin has quit IRC | 08:27 | |
*** e0ne has joined #openstack-infra | 08:33 | |
*** esikache1 has joined #openstack-infra | 08:33 | |
*** esikachev has quit IRC | 08:34 | |
*** mrmartin has joined #openstack-infra | 08:38 | |
*** e0ne has quit IRC | 08:41 | |
*** mrmartin has quit IRC | 08:42 | |
*** markvoelker has joined #openstack-infra | 08:44 | |
*** mrmartin has joined #openstack-infra | 08:46 | |
*** |-paul-| has quit IRC | 08:46 | |
*** zhurong has joined #openstack-infra | 08:46 | |
*** achanda has quit IRC | 08:47 | |
*** markvoelker has quit IRC | 08:48 | |
*** thorst has joined #openstack-infra | 08:48 | |
*** achanda has joined #openstack-infra | 08:49 | |
*** esikache1 has quit IRC | 08:50 | |
*** mrmartin has quit IRC | 08:50 | |
*** thorst has quit IRC | 08:54 | |
*** zhurong has quit IRC | 08:56 | |
*** salv-orlando has quit IRC | 08:56 | |
*** zeih has joined #openstack-infra | 08:57 | |
*** infra-red has joined #openstack-infra | 08:59 | |
*** zeih has quit IRC | 09:01 | |
*** achanda has quit IRC | 09:04 | |
*** infra-red has quit IRC | 09:05 | |
*** kzaitsev_mb has joined #openstack-infra | 09:06 | |
*** mikelk has joined #openstack-infra | 09:16 | |
*** harlowja_at_home has quit IRC | 09:16 | |
*** ddieterly has joined #openstack-infra | 09:21 | |
*** ddieterly has quit IRC | 09:25 | |
*** mrmartin has joined #openstack-infra | 09:26 | |
*** mrmartin has quit IRC | 09:26 | |
*** mikelk has quit IRC | 09:28 | |
*** yamamoto has joined #openstack-infra | 09:30 | |
*** infra-red has joined #openstack-infra | 09:30 | |
*** mikelk has joined #openstack-infra | 09:31 | |
*** infra-re_ has joined #openstack-infra | 09:32 | |
*** infra-red has quit IRC | 09:35 | |
*** yamamoto has quit IRC | 09:37 | |
*** yamahata has quit IRC | 09:37 | |
*** yamamoto has joined #openstack-infra | 09:43 | |
*** yamamoto has quit IRC | 09:46 | |
*** rguillebert has joined #openstack-infra | 09:51 | |
*** thorst has joined #openstack-infra | 09:52 | |
*** daemontool has quit IRC | 09:54 | |
*** max_lobur has joined #openstack-infra | 09:55 | |
*** infra-re_ has quit IRC | 09:55 | |
*** infra-red has joined #openstack-infra | 09:56 | |
*** salv-orlando has joined #openstack-infra | 09:57 | |
*** zeih has joined #openstack-infra | 09:57 | |
*** infra-re_ has joined #openstack-infra | 09:57 | |
*** thorst has quit IRC | 09:58 | |
*** infra-red has quit IRC | 10:00 | |
*** zeih has quit IRC | 10:02 | |
*** infra-re_ has quit IRC | 10:06 | |
*** salv-orlando has quit IRC | 10:08 | |
*** mrmartin has joined #openstack-infra | 10:10 | |
*** mrmartin has quit IRC | 10:15 | |
*** infra-red has joined #openstack-infra | 10:17 | |
*** infra-red has quit IRC | 10:19 | |
*** ddieterly has joined #openstack-infra | 10:22 | |
*** ddieterly has quit IRC | 10:26 | |
*** esikache1 has joined #openstack-infra | 10:28 | |
*** kzaitsev_mb has quit IRC | 10:41 | |
*** ildikov has joined #openstack-infra | 10:43 | |
*** markvoelker has joined #openstack-infra | 10:44 | |
*** mrmartin has joined #openstack-infra | 10:46 | |
*** yamamoto has joined #openstack-infra | 10:47 | |
*** markvoelker has quit IRC | 10:49 | |
*** yamamoto has quit IRC | 10:53 | |
*** esikache1 has quit IRC | 10:54 | |
openstackgerrit | yolanda.robla proposed openstack-infra/puppet-infracloud: Add element to bring up bridge on infra cloud https://review.openstack.org/276292 | 10:54 |
*** zeih has joined #openstack-infra | 10:56 | |
*** thorst has joined #openstack-infra | 10:56 | |
*** wolverineav has quit IRC | 10:58 | |
*** thorst has quit IRC | 11:03 | |
*** clayton has quit IRC | 11:03 | |
*** clayton has joined #openstack-infra | 11:04 | |
*** mrmartin has quit IRC | 11:10 | |
*** salv-orlando has joined #openstack-infra | 11:15 | |
*** ddieterly has joined #openstack-infra | 11:22 | |
*** salv-orlando has quit IRC | 11:25 | |
*** ddieterly has quit IRC | 11:27 | |
*** mrmartin has joined #openstack-infra | 11:36 | |
openstackgerrit | Peter Szabo proposed openstack-infra/jenkins-job-builder: Add new parameter for warnings publisher https://review.openstack.org/279878 | 11:40 |
*** _amrith_ is now known as amrith | 11:48 | |
openstackgerrit | yolanda.robla proposed openstack-infra/puppet-infracloud: Setup rabbit user/pass for keystone https://review.openstack.org/278482 | 11:50 |
*** piet has joined #openstack-infra | 11:50 | |
*** HeOS has quit IRC | 11:52 | |
samueldmq | mordred: fungi: clarkb: anteaya: here is the cachaca http://www.cachacasamanau.com.br/index.php/pt/produtos | 11:57 |
*** d0ugal has quit IRC | 11:58 | |
*** wolverineav has joined #openstack-infra | 11:58 | |
*** mrmartin has quit IRC | 11:58 | |
samueldmq | mordred: there is also a version of nega fulo that comes in a clay jar (http://www.emporiobasilico.com.br/wc_content/wc_uploads/cache/97__w320_h320_zc0_ac_m.jpg) | 11:58 |
*** thorst has joined #openstack-infra | 12:01 | |
*** wolverineav has quit IRC | 12:03 | |
*** smarcet has joined #openstack-infra | 12:03 | |
openstackgerrit | Russell Bryant proposed openstack-infra/project-config: Add non-voting rally job for networking-odl. https://review.openstack.org/269174 | 12:07 |
openstackgerrit | Russell Bryant proposed openstack-infra/project-config: Add non-voting rally job for networking-odl. https://review.openstack.org/269174 | 12:07 |
*** thorst has quit IRC | 12:08 | |
openstackgerrit | Russell Bryant proposed openstack-infra/project-config: Add non-voting rally job for networking-odl. https://review.openstack.org/269174 | 12:08 |
openstackgerrit | Peter Szabo proposed openstack-infra/jenkins-job-builder: Add new parameter for warnings publisher https://review.openstack.org/279878 | 12:09 |
*** piet has quit IRC | 12:13 | |
*** ddieterly has joined #openstack-infra | 12:23 | |
*** piet has joined #openstack-infra | 12:25 | |
*** piet has quit IRC | 12:25 | |
*** ddieterly has quit IRC | 12:28 | |
*** tiswanso has joined #openstack-infra | 12:30 | |
*** markvoelker has joined #openstack-infra | 12:30 | |
*** tiswanso has quit IRC | 12:31 | |
*** tiswanso has joined #openstack-infra | 12:32 | |
*** markvoelker has quit IRC | 12:34 | |
*** zeih has quit IRC | 12:34 | |
*** Daisy has joined #openstack-infra | 12:39 | |
*** bastelfreak has quit IRC | 12:40 | |
*** Daisy has quit IRC | 12:44 | |
*** mikelk has quit IRC | 12:49 | |
*** tiswanso has quit IRC | 12:50 | |
*** esikache1 has joined #openstack-infra | 12:50 | |
*** smarcet has quit IRC | 12:50 | |
*** esikache1 has quit IRC | 12:55 | |
*** doug-fish has quit IRC | 12:55 | |
*** doug-fish has joined #openstack-infra | 12:57 | |
*** doug-fish has quit IRC | 12:57 | |
*** [1]Thelo has joined #openstack-infra | 12:59 | |
*** wolverineav has joined #openstack-infra | 13:00 | |
*** Thelo has quit IRC | 13:01 | |
*** [1]Thelo is now known as Thelo | 13:01 | |
*** zhurong has joined #openstack-infra | 13:04 | |
*** wolverineav has quit IRC | 13:05 | |
*** thorst has joined #openstack-infra | 13:07 | |
*** thorst has quit IRC | 13:13 | |
*** zhurong has quit IRC | 13:16 | |
*** zhurong has joined #openstack-infra | 13:19 | |
*** ddieterly has joined #openstack-infra | 13:24 | |
*** ddieterly has quit IRC | 13:28 | |
*** dims has joined #openstack-infra | 13:30 | |
*** markvoelker has joined #openstack-infra | 13:31 | |
*** markvoelker has quit IRC | 13:35 | |
*** rfolco has joined #openstack-infra | 13:38 | |
*** skolekonov_ has joined #openstack-infra | 13:46 | |
*** ddieterly has joined #openstack-infra | 13:49 | |
zhurong | Hi, guys, when i push some commit to greeit, it gives me a error like:`error: unpack failed: error No space left on device`, and my device have much space left, maybe it have some problem with greeit? | 13:49 |
skolekonov_ | Hi, I have the same problem right now | 13:49 |
*** esikache1 has joined #openstack-infra | 13:51 | |
*** rfolco has quit IRC | 13:51 | |
*** d0ugal has joined #openstack-infra | 13:51 | |
*** boris-42 has quit IRC | 13:54 | |
*** njohnston has joined #openstack-infra | 13:54 | |
njohnston | Hi. I want to report an issue, in case y'all haven't gotten a report of it yet. I just did "git review" to push a new patchset, and I got the error: | 13:55 |
njohnston | error: unpack failed: error No space left on device | 13:55 |
njohnston | fatal: Unpack error, check server log | 13:55 |
njohnston | To ssh://natejohnston@review.openstack.org:29418/openstack/neutron.git | 13:55 |
njohnston | ! [remote rejected] HEAD -> refs/publish/master/278088 (n/a (unpacker error)) | 13:56 |
*** esikache1 has quit IRC | 13:56 | |
*** zhurong has quit IRC | 13:57 | |
*** salv-orlando has joined #openstack-infra | 13:59 | |
*** wolverineav has joined #openstack-infra | 14:01 | |
sgordon | njohnston, so it's not just me | 14:02 |
sgordon | :) | 14:02 |
*** rossella_s has quit IRC | 14:03 | |
*** rossella_s has joined #openstack-infra | 14:03 | |
*** wolverineav has quit IRC | 14:06 | |
*** thorst has joined #openstack-infra | 14:11 | |
*** salv-orlando has quit IRC | 14:12 | |
*** zeih has joined #openstack-infra | 14:15 | |
*** thorst has quit IRC | 14:17 | |
*** thorst has joined #openstack-infra | 14:17 | |
*** thorst has quit IRC | 14:19 | |
Jeffrey4l | got the same error when push new PS: error: unpack failed: error No space left on device | 14:24 |
Kiall | yea.. it seems the cloud is full ;) | 14:25 |
*** sdake has joined #openstack-infra | 14:36 | |
*** kun_huang has joined #openstack-infra | 14:38 | |
kun_huang | guys, what's the meaning of extra ATC here? https://governance.openstack.org/reference/projects/nova.html#extra-atcs | 14:38 |
jeblair | njohnston, sgordon, skolekonov_: thanks, looking into it (cc: infra-root) | 14:41 |
*** Daisy has joined #openstack-infra | 14:41 | |
*** Daisy has quit IRC | 14:42 | |
*** Daisy has joined #openstack-infra | 14:42 | |
jeblair | /dev/mapper/main-gerrit 99G 54G 45G 55% /home/gerrit2 | 14:45 |
* Kiall bets it's /var/log - it's always /var/log :) | 14:45 | |
*** zeih has quit IRC | 14:45 | |
jeblair | Kiall: :) nope -- i get the same error trying to create a file in /home/gerrit2. /var is on / and still works | 14:46 |
jeblair | /home/gerrit2 is an LVM'd cinder volume though | 14:46 |
Kiall | out of inodes? | 14:47 |
jeblair | Kiall: oh! good point | 14:47 |
jeblair | /dev/mapper/main-gerrit 6553600 6553600 0 100% /home/gerrit2 | 14:47 |
jeblair | Kiall: thank you :) | 14:47 |
Kiall | that'll do it | 14:47 |
Kiall | no worries | 14:47 |
*** edmondsw has joined #openstack-infra | 14:48 | |
jeblair | #inodes isn't a thing we can change with tune2fs; we need a new filesystem, right? | 14:48 |
*** edmondsw has quit IRC | 14:49 | |
*** Daisy has quit IRC | 14:50 | |
Kiall | ehh, I don't think it can be changed. | 14:51 |
Kiall | Though, you might be able to `git gc` in the repos, to clear out what I imagine would be 10,000's of small packs uploaded... Though, gerrit may do that, or may not like it | 14:51 |
Kiall | https://gerrit-review.googlesource.com/Documentation/cmd-gc.html <-- apparantly, a command to run it via the gerrit shell | 14:52 |
jeblair | Kiall: we do repack, so the pack files should be okay (if large); it's probably the refs, which are not packed, which are the major driver here | 14:53 |
jeblair | Kiall: however, there's a lot of complexity around that that i want to make sure we fully understand before doing it, so i think i'm going to go with the 'fix the filesystem' approach today | 14:53 |
*** max_lobur has quit IRC | 14:53 | |
jeblair | i'm adding a new cinder device and will create a new volume+fs on that, rsync, then restart | 14:55 |
*** zz_dimtruck is now known as dimtruck | 14:56 | |
Kiall | You mentioned LVM, i believe if you add the second volume to the group and extend the fs, you get more inodes (and more risk of a cinder issue..) - but might be quicker given I'm sure you want your Saturday back ;) | 14:56 |
jeblair | Kiall: good point... :) that would probably fix things faster... though, i bet a rsync of 54g on ssds won't take too long | 14:59 |
jeblair | oh | 15:01 |
jeblair | the launchpadlib cache is huge | 15:01 |
jeblair | and full of files | 15:01 |
jeblair | 226355 of them | 15:02 |
*** kun_huang has left #openstack-infra | 15:02 | |
*** max_lobur has joined #openstack-infra | 15:03 | |
*** dims has quit IRC | 15:03 | |
jeblair | also, it's slightly disturbing that 'nova volume-attach' just attached the second volume at the same device as the first one (/dev/xvdb) | 15:03 |
Kiall | the device naming from nova has been somewhat of a "best effort" rather than always accurate ;) | 15:04 |
jeblair | yeesh. scary. though this is the first time i've seen this happen | 15:05 |
jeblair | #status alert Gerrit is offline for filesystem repair | 15:06 |
openstackstatus | jeblair: sending alert | 15:06 |
mordred | morning jeblair | 15:06 |
mordred | anything I can do to help? | 15:06 |
jeblair | mordred: morning! let me know if you think my plan to rysnc to a new fs is okay, or if we should just remove the launchpadlib cache, or extend the fs, either of which might get us up faster. | 15:07 |
mordred | jeblair: I think we should remove the launchpadlib cache for sure - but I also think rsyncing to a new fs is an ok plan | 15:08 |
jeblair | mordred: my thinking is the rsync may take a few mins longer, but is a moderate long-term fix | 15:08 |
mordred | yah | 15:08 |
jeblair | i'm stopping gerrit itself now | 15:08 |
-openstackstatus- NOTICE: Gerrit is offline for filesystem repair | 15:08 | |
*** ChanServ changes topic to "Gerrit is offline for filesystem repair" | 15:09 | |
jeblair | i've unmounted /home/gerrit2 | 15:09 |
jeblair | so back on the cinder front | 15:09 |
jeblair | the kernel has not noticed the new device | 15:09 |
jeblair | which is probably good considering the naming issue from earlier. and possibly related. | 15:09 |
jeblair | there was a command we used to have to run to hot add scsi devices... what was it... | 15:10 |
Kiall | echo "- - -" > /sys/class/scsi/something/scan | 15:10 |
*** dims has joined #openstack-infra | 15:10 | |
openstackstatus | jeblair: finished sending alert | 15:11 |
jeblair | Kiall: yeah that was it | 15:12 |
*** ganso has joined #openstack-infra | 15:12 | |
jeblair | and neat, it made the b device go away | 15:12 |
dims | good morning :) | 15:12 |
mordred | Kiall: dude. you have a good memory brain | 15:12 |
jeblair | oh wait, i'm mistaken... | 15:13 |
jeblair | b is still there, but still no c | 15:13 |
jeblair | so i wonder if we should reboot | 15:13 |
Kiall | mordred: lol, nope... just wasted enough time on figuring out I needed that a while back - it's burned in now. | 15:14 |
Kiall | jeblair: maybe the dup'd attachment name is causing an issue? Does RAX let you specify a device name while attaching? | 15:14 |
jeblair | Kiall: yeah, so i specified /dev/xvdc and cinder show reports that now | 15:15 |
mordred | jeblair: I only see one volume attachment for review.openstack.org in nova | 15:15 |
mordred | | 4ae4bfbf-c90e-4bad-a9d8-4c1f70604ac1 | /dev/xvdb | 87741fa9-92cb-4d1e-8195-6841e2468fb2 | 4ae4bfbf-c90e-4bad-a9d8-4c1f70604ac1 | | 15:15 |
mordred | jeblair: that's from running: nova volume-attachments review.openstack.org | 15:16 |
jeblair | mordred: hrm | 15:16 |
jeblair | | 4ae4bfbf-c90e-4bad-a9d8-4c1f70604ac1 | in-use | review.openstack.org/main01 | 100 | SSD | false | 87741fa9-92cb-4d1e-8195-6841e2468fb2 | | 15:17 |
jeblair | | 986b125a-2629-435b-ba74-3ba089f1cbcc | in-use | review.openstack.org/main02 | 100 | SSD | false | 9d3ff96d-920d-4d4f-854e-7057d268795c | | 15:17 |
jeblair | mordred: that's from cinder list | 15:17 |
mordred | yah | 15:17 |
mordred | I see thta too | 15:17 |
*** zeih has joined #openstack-infra | 15:17 | |
mordred | so it seems cinder thinks it's attached and nova does not | 15:17 |
mordred | which would explain why you don't see it | 15:17 |
jeblair | mordred: i'm using the venvs in http://docs.openstack.org/infra/system-config/sysadmin.html#cinder-volume-management | 15:17 |
jeblair | mordred: my nova doesn't have 'volume-attachments' | 15:17 |
*** thorst has joined #openstack-infra | 15:19 | |
* jeblair resorts to web interface | 15:19 | |
jeblair | webui only says 01 as well | 15:20 |
mordred | jeblair: I'll update the venv - I have also updated the cinder.sh script to include OS_VOLUME_API_VERSION=1 which is needed for recent cinder client with rackspace | 15:20 |
*** salv-orlando has joined #openstack-infra | 15:20 | |
jeblair | mordred: should i detach, wait for you to complete that, then try attach again? | 15:21 |
*** zeih has quit IRC | 15:21 | |
mordred | yah | 15:21 |
mordred | and I'm done | 15:21 |
mordred | oh. WEIRD | 15:21 |
mordred | there's still no volume-attachements command | 15:21 |
mordred | what the heck do I have | 15:21 |
jeblair | mordred: ERROR: publicURL endpoint for volumev2 service in DFW region not found | 15:21 |
jeblair | and now that | 15:21 |
*** EricGonczer_ has joined #openstack-infra | 15:22 | |
mordred | yah - re-source cinder.sh | 15:22 |
*** skolekonov_ has quit IRC | 15:22 | |
jeblair | oh right | 15:22 |
jeblair | better | 15:22 |
jeblair | mordred: shall i try cinder attach one more time, or give you another minute to examine weirdness? | 15:22 |
mordred | nope. go for it | 15:22 |
jeblair | (er, nova attach) | 15:22 |
mordred | jeblair: you can do btw | 15:23 |
jeblair | mordred: attached | 15:23 |
jeblair | webui does not show it | 15:23 |
*** dimtruck is now known as zz_dimtruck | 15:24 | |
mordred | neither does my wonky nova with nova volume-attachments | 15:24 |
mordred | jeblair: maybe try web ui? | 15:24 |
jeblair | i think the next step in this choose your own adventure is either a) webui b) reboot | 15:24 |
mordred | I think webui - becuase this seems to be control plane issues | 15:24 |
jeblair | i'll do webui -- i'll detach in the shell, then try attaching in the webui | 15:24 |
*** rvba has quit IRC | 15:25 | |
jeblair | attaching... | 15:25 |
*** smarcet has joined #openstack-infra | 15:25 | |
*** salv-orlando has quit IRC | 15:25 | |
mordred | oh. also - you know what ... | 15:26 |
jeblair | attached and shows up in webui | 15:26 |
mordred | nova volume-attach is not a synchronous action | 15:26 |
mordred | I agree - it shows up in monty-nova | 15:26 |
jeblair | mordred: but if cinder reports it attached...? | 15:26 |
mordred | so - that's a mystery to explore more later | 15:26 |
mordred | jeblair: that's just cinder's view | 15:26 |
jeblair | mordred: cinder show was saying "available"..."attaching"..."in-use" | 15:26 |
mordred | jeblair: cinder and nova have two different databases for this | 15:27 |
jeblair | so i sort of thought that the attaching/in-use change would reflect whether nova had completed the action... | 15:27 |
mordred | yah - on th enova side if I was using the api, the next step after the attache is to poll the nova server record to see when the attachment shows up | 15:27 |
jeblair | [2086902.923541] blkfront: xvdc: barrier: enabled; persistent grants: disabled; indirect descriptors: disabled; | 15:27 |
jeblair | [2086902.925722] xvdc: unknown partition table | 15:27 |
jeblair | auto-detected (no probe needed) | 15:27 |
*** thorst has quit IRC | 15:28 | |
mordred | awesome | 15:28 |
jeblair | so, resume speed | 15:28 |
mordred | jeblair: this reminds me I need to cook up ansible playbooks for this stuff now that we've got 2.0 and the os_ modules on the server | 15:28 |
*** rvba has joined #openstack-infra | 15:28 | |
*** rvba has quit IRC | 15:28 | |
*** tphummel has joined #openstack-infra | 15:29 | |
jeblair | i'm renaming the old volume to gerritold | 15:30 |
*** rvba has joined #openstack-infra | 15:30 | |
*** rvba has quit IRC | 15:30 | |
*** rvba has joined #openstack-infra | 15:30 | |
*** Jeffrey4l has quit IRC | 15:31 | |
jeblair | okay, new mkfs command... | 15:32 |
*** markvoelker has joined #openstack-infra | 15:32 | |
notmorgan | oh inode issue. fun thing to deal with :( | 15:33 |
jeblair | doubling them would be just about right | 15:33 |
mordred | ++ | 15:33 |
notmorgan | ext4? | 15:34 |
jeblair | notmorgan: yep | 15:34 |
* notmorgan is just curious. | 15:34 | |
notmorgan | jeblair: ah. you know it's been a long time since i ran a FS out of inodes [back when storing < 4k images at $social_media_company_not_facebook$ | 15:34 |
*** markvoelker has quit IRC | 15:36 | |
*** moravec has quit IRC | 15:36 | |
jeblair | i think byte-per-inode defaults to 16384 so i'll set it to 8192 ? | 15:37 |
*** moravec has joined #openstack-infra | 15:37 | |
jeblair | sudo mkfs.ext4 -m 0 -j -i 8192 -L $NAME /dev/main/$NAME | 15:38 |
pabelanger | what happen to gerrit? Result of the move yesterday? | 15:38 |
pabelanger | err, project rename | 15:38 |
*** achanda has joined #openstack-infra | 15:38 | |
notmorgan | pabelanger: inode issue on FS | 15:38 |
notmorgan | pabelanger: they're working on it | 15:38 |
jeblair | pabelanger: probably just bad luck | 15:38 |
pabelanger | Ah | 15:38 |
notmorgan | jeblair: def. bad luck. no way a rename should cause this :P | 15:39 |
jeblair | 13107200 inodes, 26213376 blocks | 15:39 |
jeblair | that's 2x the old fs | 15:39 |
notmorgan | uhm. | 15:39 |
*** jlk` is now known as jlk | 15:39 | |
notmorgan | jeblair: -i or -N? | 15:39 |
notmorgan | -i is bytes-per-inode accoridng to man | 15:39 |
jeblair | notmorgan: i did -i because the math is harder? :) | 15:39 |
jeblair | notmorgan: yeah, i adjusted the ratio instead of the count | 15:40 |
notmorgan | is that what we want? i thought we wanted more actual inodes? | 15:40 |
notmorgan | oh | 15:40 |
notmorgan | okie | 15:40 |
* notmorgan can't brain. | 15:40 | |
jeblair | notmorgan: (you may be thinking of inode size, which i did not change) | 15:40 |
mordred | notmorgan: braining is hard | 15:40 |
notmorgan | with -I vs -i | 15:40 |
notmorgan | yeah | 15:40 |
jeblair | notmorgan: (rather, i changed the ratio mkfs uses to calculate how many inodes to allocate) | 15:40 |
* notmorgan kicks mkfs for using -i and -I and making it hard to read pre-coffee | 15:41 | |
jeblair | but confirmed the same result -- it says we have 13107200 and we had 6553600 before | 15:41 |
notmorgan | thankfully this shouldn't occur too terribly offen :) | 15:42 |
Clint | you reading pre-coffee? | 15:42 |
jeblair | rsyncing | 15:42 |
notmorgan | Clint: yes. | 15:42 |
*** tphummel has quit IRC | 15:43 | |
jeblair | i'm sysadmining pre-coffee | 15:43 |
jeblair | fortunately, Kiall was around (and presumably caffeinated) to tell me what to type :) | 15:43 |
Kiall | jeblair: next step, rm -rf --no-preserve-root / ;) | 15:44 |
pabelanger | Kiall: how often have you had to deal with inodes :) | 15:44 |
Kiall | More often that I probably should have :) | 15:44 |
notmorgan | oh hi Kiall! | 15:45 |
pabelanger | jeblair: distributed sysadmining FTW | 15:45 |
*** flepied1 has joined #openstack-infra | 15:45 | |
jeblair | ++ | 15:46 |
*** achanda has quit IRC | 15:46 | |
mordred | Kiall: you coming to the ops thing in manchester next week? | 15:47 |
*** ddieterly has quit IRC | 15:47 | |
*** tphummel has joined #openstack-infra | 15:47 | |
Kiall | mordred: nope, I've been travelling the last 2 weeks... The thought of more travel.. ugh! | 15:48 |
Kiall | and... speaking of coffee ;) | 15:48 |
mordred | Kiall: I hear that for sure! | 15:48 |
*** flepied has quit IRC | 15:48 | |
jeblair | at about 20% | 15:50 |
*** EricGonczer_ has quit IRC | 15:50 | |
*** ddieterly has joined #openstack-infra | 15:51 | |
*** esikache1 has joined #openstack-infra | 15:53 | |
*** dguitarbite_ has joined #openstack-infra | 15:54 | |
dguitarbite_ | hello, is review.openstack.org down today? | 15:54 |
dguitarbite_ | or I'm stuck in some sort of maintenance limbo? | 15:54 |
pabelanger | dguitarbite_: see topic | 15:54 |
*** skolekonov_ has joined #openstack-infra | 15:54 | |
*** skolekonov_ has quit IRC | 15:54 | |
*** ddieterly has quit IRC | 15:55 | |
dguitarbite_ | pabelanger: thanks, I was kind of panicking | 15:57 |
Clint | don't panic | 15:57 |
*** esikache1 has quit IRC | 15:58 | |
mordred | I have a towel if you need one | 16:00 |
Sam-I-Am | mornings | 16:00 |
jeblair | never could get the hang of them | 16:01 |
*** wolverineav has joined #openstack-infra | 16:03 | |
*** wolverineav has quit IRC | 16:07 | |
mordred | jeblair: I have updated the cinder-venv | 16:08 |
mordred | jeblair: it now has the right things in it | 16:08 |
mordred | jeblair: the existing one was REALLY WEIRD - so I actually made a new one | 16:08 |
jeblair | mordred: did you find the nova attach list thingy command? | 16:08 |
mordred | yes | 16:08 |
mordred | it turns out when I upgraded nova in the venv earlier, I did not actually upgrade nova | 16:09 |
jeblair | mordred: i like your conservative approach | 16:09 |
*** EricGonczer_ has joined #openstack-infra | 16:10 | |
*** zz_dimtruck is now known as dimtruck | 16:11 | |
*** kzaitsev_mb has joined #openstack-infra | 16:15 | |
jeblair | 50% | 16:15 |
*** EricGonczer_ has quit IRC | 16:17 | |
mordred | jeblair: wow. we're having so much fun waiting on things on review.o.o these days arent' we? | 16:20 |
jeblair | ya | 16:20 |
mordred | btw - xenial first alpha is out - so I'm testing building xenial images with disk-image-builder | 16:21 |
jeblair | mordred: see if afs works :) | 16:21 |
jeblair | sometimes they break that | 16:21 |
mordred | jeblair: good call :) | 16:21 |
mordred | jeblair: maybe we can skip trusty for our services still on precise and move straight to xenial :) | 16:22 |
*** zeih has joined #openstack-infra | 16:22 | |
jeblair | maybe so :) | 16:23 |
*** thorst has joined #openstack-infra | 16:27 | |
*** hdd has joined #openstack-infra | 16:30 | |
tristanC | Hello guys, about the zuul memory leak, has it been bisected already ? | 16:33 |
clarkb | jeblair we ran out of inodes on /home/gerrit2 and separately cinder nova couldnt agree on volume attachment? | 16:33 |
*** rkukura has quit IRC | 16:33 | |
clarkb | tristanC: yes fixed too iirc | 16:33 |
mordred | clarkb: yes | 16:33 |
*** thorst has quit IRC | 16:33 | |
mordred | clarkb: I have also updated the versions of things now in cinder-venv - although we do not know if that had anything to do with the problem | 16:34 |
clarkb | and was the problem manifesting beyond just bookeeping discrepancies? | 16:34 |
clarkb | I couldnt tell if it was causing problems by skimming scrollback | 16:35 |
*** kzaitsev_mb has quit IRC | 16:35 | |
jeblair | clarkb: it didn't actually appear on-host | 16:36 |
jeblair | clarkb: i attached using the web ui and that worked | 16:36 |
tristanC | clarkb: I wasn't sure the fix was already merged, thank! | 16:36 |
mordred | yah. so, my hunch is something to do with the old client lib versions on the host - but I have not spent effort to reproduce the issue yet | 16:37 |
*** kzaitsev_mb has joined #openstack-infra | 16:43 | |
*** dguitarbite_ has quit IRC | 16:43 | |
*** ildikov_ has joined #openstack-infra | 16:44 | |
*** ildikov has quit IRC | 16:46 | |
*** kzaitsev_mb has quit IRC | 16:48 | |
*** bpokorny has joined #openstack-infra | 16:52 | |
*** ddieterly has joined #openstack-infra | 16:52 | |
jeblair | 80% | 16:52 |
*** HeOS has joined #openstack-infra | 16:53 | |
*** erikm has joined #openstack-infra | 16:53 | |
*** _nadya_ has joined #openstack-infra | 16:53 | |
jeblair | well crap | 16:54 |
jeblair | gerrit is running | 16:54 |
jeblair | i'm guessing someone didn't agree with the idea that puppet should not ensure -> running | 16:54 |
jeblair | mordred, clarkb: can one of you stop puppet please? | 16:55 |
*** ddieterly has quit IRC | 16:58 | |
*** zeih has quit IRC | 16:58 | |
*** bpokorny has quit IRC | 16:58 | |
jeblair | i'll do it | 16:59 |
jeblair | added review.o.o to emergency hosts file | 16:59 |
jeblair | it appeared that gerrit did not believe it had any projects; hopefully that means that it won't have actually changed anything in the db | 17:00 |
jeblair | on disk we can run the rsync again, but we'll need to add --delete this time | 17:00 |
*** tphummel has quit IRC | 17:00 | |
*** zeih has joined #openstack-infra | 17:01 | |
jeblair | well, unfortunately, there are 15 changes that have a last_updated_on after i stopped gerrit | 17:02 |
*** wolverineav has joined #openstack-infra | 17:04 | |
*** smarcet has quit IRC | 17:04 | |
*** matrohon has joined #openstack-infra | 17:08 | |
*** zeih has quit IRC | 17:08 | |
*** wolverineav has quit IRC | 17:09 | |
clarkb | sorry I am/was juggling children not in a great place to help | 17:10 |
*** Daisy has joined #openstack-infra | 17:10 | |
jeblair | i think it was actually that we re-ran gerrit init | 17:10 |
jeblair | probably some missing file in /home/review site triggered it | 17:10 |
jeblair | 4 patch sets were created after i stopped gerrit | 17:11 |
jeblair | the other change updates are probably just comments | 17:12 |
jeblair | so here's what i'm thinking: fetch those changes into a copy of the git repos; rsync again with --delete; push those changes back into the canonical git repos at their original refs location | 17:14 |
*** Daisy has quit IRC | 17:14 | |
*** jsavak has joined #openstack-infra | 17:14 | |
jeblair | then start gerrit and kick off an online reindex | 17:14 |
jeblair | infra-root: ^ does that sound sensible? | 17:15 |
clarkb | the db will know about them so as long as the refs exist and the shas match it seems like that eould work | 17:16 |
*** armax has joined #openstack-infra | 17:16 | |
*** ganso has quit IRC | 17:19 | |
*** jsavak has quit IRC | 17:20 | |
*** jsavak has joined #openstack-infra | 17:21 | |
*** yamahata has joined #openstack-infra | 17:21 | |
*** zeih has joined #openstack-infra | 17:22 | |
jeblair | okay, refs backed up | 17:22 |
jeblair | running rsync again | 17:22 |
jeblair | rsync has deleted the 4 changes i know about from the git repos | 17:25 |
jeblair | and it's complete with no surprise deletions | 17:26 |
*** zeih has quit IRC | 17:26 | |
*** matrohon has quit IRC | 17:28 | |
jeblair | pushed back into canonical repos | 17:29 |
jeblair | i'm going to start gerrit now | 17:29 |
jeblair | those 4 changes show up in the webui fine | 17:32 |
openstackgerrit | Peter Szabo proposed openstack-infra/jenkins-job-builder: Add new parameter for warnings publisher https://review.openstack.org/279878 | 17:32 |
AJaeger | jeblair: great! | 17:32 |
AJaeger | And Peter Szabo tested for us ;) | 17:32 |
jeblair | searching for them by change-id doesn't work, but that's expected and should be fixed by an online reindex | 17:32 |
*** thorst has joined #openstack-infra | 17:32 | |
*** markvoelker has joined #openstack-infra | 17:32 | |
AJaeger | jeblair: do we know when the online reindex will be finished? Just trying to get some data... | 17:33 |
jeblair | gerrit has 7125 git repo push tasks | 17:33 |
jeblair | AJaeger: we've never run an online reindex before | 17:33 |
jeblair | AJaeger: yesterday, our offline reindex took... what... 3 hours? | 17:34 |
* jeblair checks eavesdrop | 17:34 | |
AJaeger | jeblair: I know - that's why I wonder whether you have a way to measure how long today's will take... | 17:34 |
AJaeger | jeblair: reading backscroll, yes 3 hours ... | 17:34 |
AJaeger | I mean: post mortem | 17:34 |
AJaeger | As data point for the next time... | 17:34 |
jeblair | AJaeger: i _was_ going to suggest that we could copy the index to another host and try an online reindex there. | 17:34 |
jeblair | AJaeger: but now i think i should just run it in production since we know it's out of sync with data on disk (though, only 4 changes) | 17:35 |
AJaeger | jeblair: yeah, go for it! | 17:35 |
jeblair | i will at least wait until all of the git pushes are complete; 5k to go | 17:35 |
jeblair | #status ok Gerrit is back up | 17:36 |
openstackstatus | jeblair: sending ok | 17:36 |
*** markvoelker has quit IRC | 17:37 | |
jeblair | "This command will not start the indexer if it is already running or if the active index is the latest." | 17:37 |
jeblair | hrm. | 17:37 |
jeblair | i'm not sure this command does what we thought it did. | 17:37 |
nibalizer | hi | 17:37 |
jeblair | nibalizer: good morning | 17:38 |
jeblair | 3k pushes | 17:38 |
AJaeger | hi nibalizer! | 17:38 |
AJaeger | jeblair: that sounds indeed wrong ;( | 17:38 |
*** mikelk has joined #openstack-infra | 17:39 | |
*** ChanServ changes topic to "Discussion of OpenStack Developer and Community Infrastructure | docs http://docs.openstack.org/infra/ | bugs https://storyboard.openstack.org/ | source https://git.openstack.org/cgit/openstack-infra/ | channel logs http://eavesdrop.openstack.org/irclogs/%23openstack-infra/" | 17:39 | |
-openstackstatus- NOTICE: Gerrit is back up | 17:39 | |
*** thorst has quit IRC | 17:39 | |
nibalizer | sounds like fun | 17:39 |
jeblair | so, i'll still run that command. but i expect at this point it will simply output something like "nothing to do" to the log and terminate. | 17:39 |
nibalizer | oh its fixed | 17:39 |
jeblair | hopefully we will be pleasantly surprised | 17:40 |
jeblair | 1k pushes; 70% to github | 17:41 |
openstackstatus | jeblair: finished sending ok | 17:42 |
jeblair | 700 pushes all of them to github | 17:42 |
jeblair | we only use 1 thread to push to github, so the last bit will be a little slower | 17:42 |
*** esikache1 has joined #openstack-infra | 17:44 | |
jeblair | 400 | 17:46 |
AJaeger | jeblair: we have some changes that were approved but never made it through the gate. Do you have a saved list to enqueue again - or should we run "recheck" on these? | 17:46 |
AJaeger | or toggle +A? | 17:47 |
jeblair | AJaeger: i don't have a list, so toggle/recheck is the way to go | 17:47 |
AJaeger | jeblair: ok. | 17:49 |
AJaeger | jeblair: I noticed jobs that are merged on the job page - but not in the list, compare https://review.openstack.org/#/c/274117/ with https://review.openstack.org/#/q/project:openstack/api-site | 17:49 |
AJaeger | so, 274117 says "merged" but not in the second list | 17:50 |
AJaeger | And there are a few more of these | 17:50 |
*** armax has quit IRC | 17:50 | |
AJaeger | Are these part of the 4 that went through? | 17:50 |
*** esikache1 has quit IRC | 17:51 | |
jeblair | AJaeger: no; however, 274117 does show up in the git repo | 17:51 |
AJaeger | So, 274117, 274035, 276101, 273353 | 17:51 |
jeblair | AJaeger: so i think the database and git repo are consistent there | 17:51 |
jeblair | and the search results are reflecting an out of date searh index | 17:52 |
*** dimtruck is now known as zz_dimtruck | 17:52 | |
jeblair | incidentally, i specifically noted that all of the changes with updated times during the outage were still in the 'n' (new) state, so that means that there should have been no changes that were merged during that time. (this is a good thing for db/git consistency) | 17:52 |
AJaeger | will the online reindex fix these 4? | 17:53 |
jeblair | AJaeger: hrm, on the list you linked, i see 274117 as merged | 17:53 |
*** _nadya_ has quit IRC | 17:54 | |
jeblair | AJaeger: all 4 of those appear as merged in the list i'm looking at | 17:54 |
openstackgerrit | Stephen Gordon proposed openstack-infra/project-config: Skip magnum functional test jobs on docs changes https://review.openstack.org/277892 | 17:54 |
jeblair | AJaeger: maybe it updated the index since you looked? | 17:54 |
AJaeger | jeblair: might be - which would be nice... Let's see what happens with the rest | 17:54 |
jeblair | okay, it's finished pushes; i'll try index start now | 17:55 |
AJaeger | oh, https://review.openstack.org/#/c/273534/ has now "ready to submit" - never seen before... | 17:55 |
jeblair | Nothing to reindex, index is already the latest version | 17:55 |
jeblair | :( | 17:55 |
jeblair | so, i think a) that means that those 4 changes won't be properly searchable until we do an offline reindex; and b) we may be looking at 3 hour outages for renames now. | 17:56 |
AJaeger | So, no more renames ;) | 17:57 |
jeblair | i'm glad we did the stackforge move when we did | 17:57 |
AJaeger | jeblair: yeah! | 17:57 |
clarkb | could abandon and repush uf not many comments | 17:58 |
clarkb | bad workaround but it is an iption | 17:58 |
jeblair | clarkb: it might even be that pushing a new ps (or maybe even addding a comment) could trigger an index of the change | 17:58 |
clarkb | good point | 17:59 |
AJaeger | should I touch 273534 ? | 17:59 |
jeblair | AJaeger: my guess with 534 is that zuul was trying to merge it right as gerrit was running out of inodes, so the submit action failed | 18:00 |
*** madhu_ak has joined #openstack-infra | 18:00 | |
jeblair | AJaeger: it is not in master (which is good) | 18:00 |
jeblair | AJaeger: so i think just recheck/toggle for that one | 18:01 |
jeblair | clarkb, AJaeger: leaving a comment is enough to get a change indexed | 18:02 |
*** jsavak has quit IRC | 18:02 | |
jeblair | i will comment on those 4 changes | 18:02 |
AJaeger | the change is back in the gate - good! | 18:02 |
*** rossella_s has quit IRC | 18:03 | |
*** rossella_s has joined #openstack-infra | 18:03 | |
jeblair | openstack/api-site refs/changes/86/279886/1:refs/changes/86/279886/1 | 18:03 |
jeblair | openstack/cinder refs/changes/87/279887/1:refs/changes/87/279887/1 | 18:03 |
jeblair | openstack/manila refs/changes/88/279888/1:refs/changes/88/279888/1 | 18:03 |
jeblair | openstack/manila refs/changes/89/279889/1:refs/changes/89/279889/1 | 18:03 |
jeblair | those are the 4 changes uploaded during downtime for reference | 18:04 |
jeblair | i've commented on them all | 18:04 |
AJaeger | looks good for api-site - thanks | 18:04 |
jeblair | okay, i need to go saturday now | 18:06 |
jeblair | i think we're all set here | 18:06 |
AJaeger | jeblair: we have others with out of date index like https://review.openstack.org/#/c/279884/ and https://review.openstack.org/#/q/project:openstack/openstack-doc-tools ;( | 18:06 |
AJaeger | I'll check and add comments | 18:06 |
AJaeger | jeblair: enjoy your Saturday! | 18:06 |
jeblair | AJaeger: yeah, there are probably some changes that were altered in some way after gerrit ran out of inodes but before the maintenance | 18:06 |
jeblair | we could probably look for all changes last updated after 1330 utc but before 1500 and comment on all of them | 18:07 |
jeblair | AJaeger: http://paste.openstack.org/show/486914/ | 18:08 |
jeblair | and actually, if we want to be really sure, this goes back to the last time gerritbot announced something in channel but before the first error report: | 18:10 |
jeblair | http://paste.openstack.org/show/486915/ | 18:10 |
AJaeger | too many for manual change - I went over list of open jobs that have no jenkins vote and will evaluate those... | 18:10 |
jeblair | i might just gerrty this :) | 18:10 |
AJaeger | ;) | 18:11 |
*** mrmartin has joined #openstack-infra | 18:12 | |
*** vmbrasseur has quit IRC | 18:13 | |
jeblair | meh, that's harder to do with a bunch of random changes :/ | 18:14 |
*** vmbrasseur has joined #openstack-infra | 18:14 | |
AJaeger | ;( | 18:16 |
jeblair | i think i can do it with a small change to the auto-sync on search... i'm trying it out | 18:17 |
jeblair | it's syncing them now | 18:17 |
AJaeger | cool | 18:17 |
jeblair | this means quite a few git repo updates | 18:17 |
jeblair | probably would have been faster to just do it in a bash for loop | 18:18 |
jeblair | but this is more fun | 18:18 |
*** mrmartin has quit IRC | 18:18 | |
*** sputnik13 has joined #openstack-infra | 18:18 | |
AJaeger | ;) | 18:19 |
AJaeger | jeblair: I'm leaving now as well, enjoy the weekend! | 18:20 |
jeblair | AJaeger: you too | 18:20 |
*** madhu_ak has quit IRC | 18:21 | |
*** sputnik13 has quit IRC | 18:22 | |
*** zeih has joined #openstack-infra | 18:22 | |
*** shashank_hegde has joined #openstack-infra | 18:23 | |
jeblair | we really do need to gc the git repos... someone was looking into that, right? | 18:24 |
*** jpr has joined #openstack-infra | 18:26 | |
*** dims has quit IRC | 18:30 | |
*** zeih has quit IRC | 18:30 | |
*** mrmartin has joined #openstack-infra | 18:35 | |
*** thorst has joined #openstack-infra | 18:35 | |
*** jpr has quit IRC | 18:39 | |
*** mrmartin has quit IRC | 18:39 | |
fungi | fun, 400 lines of scrollback and multiple highlights on a saturday. catching up! | 18:41 |
*** sdake has quit IRC | 18:42 | |
*** thorst has quit IRC | 18:43 | |
fungi | zaro: the _problem_ was not making the group disappear. deleting it from the database did that just fine. the _problem_ is that the acl is still trying to enforce membership in that group, so any of our acls which reference "registered users" are currently checking against a nonexistent group and preventing any accounts from matching those rules. i _expect_ that removing those lines from the groups | 18:44 |
fungi | files will solve what we wanted solved, and could hopefully be pushed into each effected repo with gerrit online, but your confirmation of this would be helpful | 18:44 |
*** jtomasek_ has joined #openstack-infra | 18:48 | |
fungi | okay, caught up with scrollback now. looks like my decision to spend the first half of the day making marinara caused me to miss all the fun | 18:55 |
fungi | anything still broken that needs immediate attention? | 18:55 |
*** zz_dimtruck is now known as dimtruck | 18:58 | |
*** e0ne has joined #openstack-infra | 18:58 | |
*** wolverineav has joined #openstack-infra | 19:05 | |
*** mrmartin has joined #openstack-infra | 19:06 | |
*** flepied1 has quit IRC | 19:09 | |
*** flepied has joined #openstack-infra | 19:10 | |
*** wolverineav has quit IRC | 19:11 | |
*** hdd has quit IRC | 19:11 | |
*** alkhodos has quit IRC | 19:12 | |
*** harlowja_at_home has joined #openstack-infra | 19:12 | |
fungi | oh, also, as promised, island snow (my wife took some photos of it yesterday and this morning): http://fungi.yuggoth.org/snow/ | 19:16 |
*** ddieterly has joined #openstack-infra | 19:17 | |
*** ddieterly has quit IRC | 19:18 | |
*** ddieterl_ has joined #openstack-infra | 19:18 | |
*** sheel has joined #openstack-infra | 19:20 | |
sheel | Hello All | 19:20 |
sheel | just for informatin | 19:20 |
sheel | It seems duplicate patchsets are generated due to some Gerrit issue | 19:21 |
sheel | If anybody concerned, can check https://review.openstack.org/#/c/274589/ and https://review.openstack.org/279887 | 19:21 |
sheel | This may be due to some manual error but for now it seems due to issue in gerrit indexing.. | 19:21 |
*** sdake has joined #openstack-infra | 19:22 | |
*** harlowja_at_home has quit IRC | 19:24 | |
*** mrmartin has quit IRC | 19:28 | |
*** fedexo has joined #openstack-infra | 19:28 | |
*** _nadya_ has joined #openstack-infra | 19:30 | |
*** markvoelker has joined #openstack-infra | 19:33 | |
*** baoli has joined #openstack-infra | 19:35 | |
*** Sukhdev has joined #openstack-infra | 19:35 | |
*** markvoelker has quit IRC | 19:37 | |
*** thorst has joined #openstack-infra | 19:41 | |
openstackgerrit | James Slagle proposed openstack-infra/tripleo-ci: Wait for pacemaker cluster to settle https://review.openstack.org/279894 | 19:42 |
*** baoli has quit IRC | 19:42 | |
*** shashank_hegde has quit IRC | 19:46 | |
*** thorst has quit IRC | 19:48 | |
*** EricGonczer_ has joined #openstack-infra | 19:49 | |
openstackgerrit | James Slagle proposed openstack-infra/tripleo-ci: Wait for pacemaker cluster to settle https://review.openstack.org/279894 | 19:49 |
*** e0ne has quit IRC | 19:50 | |
*** FallenPegasus has quit IRC | 19:52 | |
*** dimtruck is now known as zz_dimtruck | 19:54 | |
*** alexpilotti_ has joined #openstack-infra | 19:54 | |
*** alexpilotti_ has quit IRC | 19:55 | |
*** zz_dimtruck is now known as dimtruck | 19:57 | |
*** benjaminfarley has joined #openstack-infra | 20:04 | |
*** jtomasek_ has quit IRC | 20:05 | |
*** wolverineav has joined #openstack-infra | 20:07 | |
*** zeih has joined #openstack-infra | 20:09 | |
*** zeih has quit IRC | 20:12 | |
*** sdake has quit IRC | 20:16 | |
*** jsavak has joined #openstack-infra | 20:20 | |
*** shashank_hegde has joined #openstack-infra | 20:24 | |
*** jsavak has quit IRC | 20:25 | |
*** jtomasek_ has joined #openstack-infra | 20:28 | |
*** Sukhdev has quit IRC | 20:30 | |
*** baoli has joined #openstack-infra | 20:32 | |
*** andymaier_ has joined #openstack-infra | 20:33 | |
*** andymaier has quit IRC | 20:36 | |
*** baoli has quit IRC | 20:37 | |
*** sdake has joined #openstack-infra | 20:38 | |
*** gangil has joined #openstack-infra | 20:43 | |
*** gangil has joined #openstack-infra | 20:43 | |
*** thorst has joined #openstack-infra | 20:48 | |
*** thorst has quit IRC | 20:53 | |
*** dimtruck is now known as zz_dimtruck | 20:59 | |
*** EricGonczer_ has quit IRC | 21:00 | |
*** dims has joined #openstack-infra | 21:04 | |
*** gangil has quit IRC | 21:06 | |
*** _nadya_ has quit IRC | 21:10 | |
*** zeih has joined #openstack-infra | 21:10 | |
*** zz_dimtruck is now known as dimtruck | 21:10 | |
*** ddieterl_ has quit IRC | 21:12 | |
openstackgerrit | Davanum Srinivas (dims) proposed openstack/requirements: Updated from generate-constraints https://review.openstack.org/279735 | 21:18 |
*** gangil has joined #openstack-infra | 21:18 | |
*** gangil has joined #openstack-infra | 21:18 | |
*** thorst has joined #openstack-infra | 21:24 | |
*** thorst has quit IRC | 21:25 | |
*** thorst has joined #openstack-infra | 21:26 | |
*** jpr has joined #openstack-infra | 21:26 | |
*** jtomasek_ has quit IRC | 21:28 | |
*** gangil has quit IRC | 21:29 | |
*** gangil has joined #openstack-infra | 21:29 | |
*** gangil has joined #openstack-infra | 21:29 | |
*** gangil has quit IRC | 21:33 | |
*** max_lobur has quit IRC | 21:33 | |
*** markvoelker has joined #openstack-infra | 21:34 | |
*** thorst has quit IRC | 21:35 | |
*** hdd has joined #openstack-infra | 21:36 | |
*** markvoelker has quit IRC | 21:39 | |
*** EricGonczer_ has joined #openstack-infra | 21:43 | |
*** andymaier_ has quit IRC | 21:43 | |
*** sridhar_ram has joined #openstack-infra | 21:44 | |
*** zeih has quit IRC | 21:50 | |
*** shashank_hegde has quit IRC | 21:53 | |
*** ccrouch has joined #openstack-infra | 21:54 | |
*** jsavak has joined #openstack-infra | 21:57 | |
*** baoli has joined #openstack-infra | 21:57 | |
*** jpr has quit IRC | 21:57 | |
openstackgerrit | Tristan Cacqueray proposed openstack-infra/zuul: Add poll.poll() timeout https://review.openstack.org/274445 | 21:58 |
*** rossella_s has quit IRC | 22:03 | |
*** rossella_s has joined #openstack-infra | 22:03 | |
openstackgerrit | James Slagle proposed openstack-infra/tripleo-ci: Wait for pacemaker cluster to settle https://review.openstack.org/279894 | 22:05 |
*** sdake has quit IRC | 22:08 | |
*** ddieterly has joined #openstack-infra | 22:11 | |
*** mriedem has joined #openstack-infra | 22:14 | |
*** ccrouch has quit IRC | 22:17 | |
*** dimtruck is now known as zz_dimtruck | 22:18 | |
*** markvoelker has joined #openstack-infra | 22:18 | |
*** ccrouch has joined #openstack-infra | 22:19 | |
*** baoli has quit IRC | 22:23 | |
*** flepied has quit IRC | 22:26 | |
*** sridhar_ram1 has joined #openstack-infra | 22:27 | |
*** salv-orlando has joined #openstack-infra | 22:29 | |
*** sridhar_ram has quit IRC | 22:29 | |
*** flepied has joined #openstack-infra | 22:31 | |
*** ccrouch has quit IRC | 22:32 | |
*** thorst has joined #openstack-infra | 22:32 | |
*** ccrouch has joined #openstack-infra | 22:35 | |
*** |-paul-| has joined #openstack-infra | 22:35 | |
*** salv-orlando has quit IRC | 22:38 | |
*** thorst has quit IRC | 22:39 | |
*** sdake has joined #openstack-infra | 22:42 | |
*** mikelk has quit IRC | 22:44 | |
*** flepied1 has joined #openstack-infra | 22:47 | |
*** flepied has quit IRC | 22:51 | |
*** [1]Thelo has joined #openstack-infra | 23:00 | |
*** Thelo has quit IRC | 23:02 | |
*** [1]Thelo is now known as Thelo | 23:02 | |
*** jsavak has quit IRC | 23:04 | |
*** hdd has quit IRC | 23:09 | |
*** Daisy has joined #openstack-infra | 23:13 | |
*** sridhar_ram1 has quit IRC | 23:13 | |
*** andymaier_ has joined #openstack-infra | 23:15 | |
*** hdd has joined #openstack-infra | 23:16 | |
*** Daisy has quit IRC | 23:18 | |
*** ddieterly has quit IRC | 23:19 | |
*** andymaier_ has quit IRC | 23:19 | |
*** |-paul-| has quit IRC | 23:32 | |
*** thorst has joined #openstack-infra | 23:37 | |
*** zhurong has joined #openstack-infra | 23:38 | |
openstackgerrit | Frank Kloeker proposed openstack-infra/puppet-translation_checksite: Puppet module to provide a translation checksite https://review.openstack.org/274701 | 23:42 |
openstackgerrit | Frank Kloeker proposed openstack-infra/system-config: Manifest for configuration translation checksite https://review.openstack.org/276466 | 23:43 |
*** thorst has quit IRC | 23:44 | |
*** sheel has quit IRC | 23:47 | |
*** salv-orlando has joined #openstack-infra | 23:52 | |
*** fedexo has quit IRC | 23:57 | |
*** benjaminfarley has quit IRC | 23:57 | |
*** baoli has joined #openstack-infra | 23:57 | |
*** sdake has quit IRC | 23:58 | |
*** fedexo has joined #openstack-infra | 23:58 |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!