| *** shashankhegde has joined #openstack-infra | 00:01 | |
| openstackgerrit | Clark Boylan proposed openstack-infra/system-config: Revert "Initial Fedora 21 nodepool disk-image creation" https://review.openstack.org/136534 | 00:01 |
|---|---|---|
| *** MarkAtwood has joined #openstack-infra | 00:03 | |
| *** andreykurilin_ has quit IRC | 00:04 | |
| *** sdake_ has quit IRC | 00:05 | |
| clarkb | and with that I think nodepool is just about done (at least if you ignore the cleanup for the old one which is still ongoing) | 00:06 |
| clarkb | I will clean up images belonging to the old nodepool monday as we should have a good backlog of images at that point | 00:06 |
| *** davideagnello has quit IRC | 00:06 | |
| *** emagana_ has quit IRC | 00:08 | |
| *** emagana has joined #openstack-infra | 00:09 | |
| *** bitblt has quit IRC | 00:10 | |
| kevinbenton | does the devstack setup run by the gate setup an RPC password on rabbitmq? | 00:10 |
| clarkb | I think it does | 00:11 |
| clarkb | iirc that is one of the things devstack does | 00:11 |
| dimsum__ | it's in devstack-gate/devstack-vm-gate.sh | 00:12 |
| *** SumitNaiksatam has joined #openstack-infra | 00:13 | |
| *** davideagnello has joined #openstack-infra | 00:13 | |
| *** emagana has quit IRC | 00:13 | |
| *** ayoung has quit IRC | 00:13 | |
| dimsum__ | kevinbenton: https://github.com/openstack-infra/devstack-gate/blob/master/devstack-vm-gate.sh#L105 | 00:14 |
| kevinbenton | dimsum__, clarkb: excellent. it's static | 00:14 |
| *** ci-testing_ has joined #openstack-infra | 00:15 | |
| kevinbenton | however, this will make my rally scenario brittle to password changes... | 00:15 |
| *** whoops has quit IRC | 00:15 | |
| clarkb | kevinbenton: you can source it out of devstacks localrc | 00:16 |
| clarkb | then any time it changes you will get the new alue | 00:16 |
| kevinbenton | clarkb: yeah, i will have to do something like that | 00:17 |
| kevinbenton | clarkb: right now it's user-specified in the .yaml file | 00:17 |
| clarkb | also why does rally care about rabbit? I thought rally was hitting openstack APIs and timing them | 00:17 |
| kevinbenton | clarkb: so i'm using rally to measure performance of the neutron RPC apis | 00:18 |
| kevinbenton | clarkb: so it needs to connect to the message bus and run stuff there | 00:18 |
| kevinbenton | clarkb: i'm abusing rally a bit, but it does make nice graphs :-) | 00:19 |
| *** talluri has joined #openstack-infra | 00:21 | |
| *** JayJ has quit IRC | 00:21 | |
| clarkb | fungi: how goes the bread making? | 00:22 |
| adam_g | did logs.openstack.org get flushed? cant seem to find lots of recent logs, ie http://logs.openstack.org/periodic-stable/periodic-nova-python27-juno/be867f9 | 00:23 |
| clarkb | adam_g: its getting fscked. all 12tb of it | 00:24 |
| clarkb | see topic | 00:24 |
| *** arosen has left #openstack-infra | 00:24 | |
| flashgordon | clarkb wow, long does that take? | 00:25 |
| clarkb | flashgordon: forever | 00:25 |
| *** achanda has quit IRC | 00:25 | |
| *** talluri has quit IRC | 00:25 | |
| clarkb | its been a couple hours at this point | 00:25 |
| adam_g | clarkb, ah! doh | 00:26 |
| fungi | clarkb: goes well. i'm getting my sourdough back into full swing... had been just feeding and resting the starter since i'd moved because i was in and out so much. now that it's acclimated to the new locale i expect experimentation to be required for a little while | 00:26 |
| *** mudassirlatif has quit IRC | 00:26 | |
| *** patrickeast has quit IRC | 00:26 | |
| fungi | flashgordon: it takes as long as it takes | 00:27 |
| clarkb | now that summit is over and we are settled into new place I want to get my brewery set up at this house | 00:27 |
| *** SumitNaiksatam has quit IRC | 00:27 | |
| fungi | clarkb: if i had enough room, i'd be doing the same | 00:27 |
| *** mriedem has quit IRC | 00:27 | |
| fungi | i want troughs for koji rice so i can experiment with brewing sake | 00:27 |
| clarkb | I have a natural gas hookup outside on the patio | 00:27 |
| *** patrickeast has joined #openstack-infra | 00:27 | |
| clarkb | should be perfect for boiling all the liquid | 00:28 |
| fungi | but this place is a postage stamp, so it will wait until we find a proper house | 00:28 |
| *** dannywilson has quit IRC | 00:28 | |
| clarkb | fungi: if you have the abilit to steam a lot of rice you reall only need room for a 5 gallon bucket/carboy | 00:28 |
| *** mudassirlatif has joined #openstack-infra | 00:28 | |
| jeblair | how about "Gating and log storage offline due to block device error. Recovery in progress, ETA unknown." ? | 00:28 |
| *** dannywilson has joined #openstack-infra | 00:28 | |
| clarkb | jeblair: ++ | 00:28 |
| anteaya | jeblair: sounds good | 00:28 |
| fungi | jeblair: wfm | 00:28 |
| anteaya | yes we had indoor stoves growing up too | 00:29 |
| jeblair | clarkb: also i want you to get your brewery set up too :) | 00:29 |
| fungi | fsck obviously still proceeding | 00:29 |
| anteaya | my grandmother used a wooden cookstove when I was little | 00:29 |
| anteaya | this is an outdoor stove, less mess | 00:29 |
| jeblair | #status alert Gating and log storage offline due to block device error. Recovery in progress, ETA unknown. | 00:29 |
| openstackstatus | jeblair: sending alert | 00:29 |
| anteaya | and I talked to my parents, who are going to make the trek south starting tomorrow | 00:30 |
| fungi | anteaya: i remember the old saying about "if you can't stand the heat get out of the kitchen" being very appropriate. it was often too warm to hang out in the kitchen, but too cold in the rest of the house unless you were wearing several layers and had a hot water bottle | 00:30 |
| anteaya | they are going with the weather | 00:30 |
| anteaya | fungi: yep | 00:30 |
| clarkb | light just flickered and modem is rebooting | 00:31 |
| clarkb | if I fall off likely a power outage | 00:31 |
| anteaya | I remember my grandfather starting the stove in the morning, he would take his pocket knife and shave cedar shavings to start it | 00:31 |
| anteaya | clarkb: happy power | 00:31 |
| -openstackstatus- NOTICE: Gating and log storage offline due to block device error. Recovery in progress, ETA unknown. | 00:31 | |
| *** ChanServ changes topic to "Gating and log storage offline due to block device error. Recovery in progress, ETA unknown." | 00:31 | |
| fungi | anteaya: also we had terrible lack of insulation and single-pane windows which would accumulate several-inch-thick coating of frozen condensation on the inside when it got especially cold | 00:32 |
| jeblair | i was recently reminded that there's not much better in the world than a dutch oven sitting on top of a wood stove for 3 days | 00:32 |
| anteaya | fungi: I remember that too | 00:32 |
| anteaya | fungi: did you have boatloads of flies in the windows in summer too? | 00:32 |
| fungi | as a kid i used to play with the ice on the inside of the windows | 00:32 |
| anteaya | jeblair: yes | 00:32 |
| fungi | anteaya: yes, black flies. ugh | 00:32 |
| anteaya | fungi: ha ha ha | 00:32 |
| anteaya | fungi: I have similar memories, the upstairs just buzzed in the summer | 00:33 |
| fungi | anteaya: i think the mountains are probably just bits of canada relocated southward | 00:33 |
| clarkb | this wind... | 00:33 |
| anteaya | I think so | 00:33 |
| openstackstatus | jeblair: finished sending alert | 00:33 |
| anteaya | apparently the states good all our diamonds | 00:34 |
| anteaya | where we are are the roots of mountains | 00:34 |
| fungi | stupid glaciers | 00:34 |
| anteaya | and the glaciers shoved all the upper minerals south | 00:34 |
| anteaya | yep | 00:34 |
| anteaya | our minerals are fun but no diamonds | 00:34 |
| anteaya | they went south | 00:34 |
| *** talluri has joined #openstack-infra | 00:34 | |
| anteaya | apatite titanite feldspar | 00:35 |
| anteaya | and lots of granite | 00:35 |
| *** MarkAtwood has quit IRC | 00:35 | |
| *** bitblt has joined #openstack-infra | 00:37 | |
| *** yamamoto has joined #openstack-infra | 00:38 | |
| *** bitblt has left #openstack-infra | 00:39 | |
| clarkb | ice on windows?! see I grew up 6 degrees north of the equator | 00:39 |
| anteaya | sauna | 00:40 |
| anteaya | sauna and heat and sun and waves | 00:41 |
| clarkb | but we also cooked outdoors | 00:41 |
| anteaya | no fond memories of shivering first thing in the morning watching a fire being lit | 00:41 |
| anteaya | ah so fire lighting memories without the shivering? | 00:41 |
| clarkb | well it wasn't every day. But many families there do have a cookhouse | 00:42 |
| anteaya | what happens in a cookhouse? | 00:42 |
| anteaya | is that a kitchen separate from the main house? | 00:42 |
| clarkb | ya | 00:42 |
| *** ryanpetrello has joined #openstack-infra | 00:42 | |
| anteaya | cool | 00:42 |
| clarkb | so that way if your fire burns the building down house is ok :) | 00:42 |
| anteaya | my aunt had one of those, called it a summer kitchen | 00:42 |
| anteaya | smart | 00:42 |
| clarkb | (I actually think the primary use is to keep the smoke in one place) | 00:42 |
| anteaya | ha ha ha | 00:43 |
| *** yamamoto has quit IRC | 00:43 | |
| fungi | yep, when i was very small we had a farmhouse with a detached kitchen | 00:43 |
| fungi | detached bathroom too. you had to _really_ want to go in the middle of a winter night | 00:43 |
| *** shashankhegde has quit IRC | 00:43 | |
| fungi | i have at best vague memories of any of that, as i was still a toddler | 00:44 |
| *** vishy is now known as viscious | 00:45 | |
| clarkb | fungi: did you get a spicy drink yet? | 00:46 |
| fungi | fsck still running. last stdout was about an hour ago "HTREE directory inode 86483799 has an invalid root node. Clear HTree index? yes Inode 86483799 is a zero-length directory. Clear? yes" | 00:46 |
| *** dimsum__ has quit IRC | 00:46 | |
| fungi | spicy rum drink will wait. there's no hurry | 00:47 |
| *** dimsum__ has joined #openstack-infra | 00:47 | |
| fungi | the sourdough sponge is taken care of for now, but it still has a couple days ahead of it on its journey to being bread | 00:48 |
| fungi | compared to beer, that's virtually instant gratification | 00:49 |
| *** patrickeast has quit IRC | 00:50 | |
| *** ChuckC has quit IRC | 00:51 | |
| *** dimsum__ has quit IRC | 00:51 | |
| *** dannywilson has quit IRC | 00:53 | |
| *** dannywilson has joined #openstack-infra | 00:54 | |
| *** shashankhegde has joined #openstack-infra | 00:56 | |
| *** dannywilson has quit IRC | 00:59 | |
| *** ryanpetrello has quit IRC | 00:59 | |
| *** mmaglana has quit IRC | 00:59 | |
| *** kgiusti has left #openstack-infra | 01:00 | |
| *** talluri has quit IRC | 01:04 | |
| clarkb | ok off to make dinner, will check in later. nodepool still looking good. under 200 things to delete at this point | 01:05 |
| anteaya | I've never seen so much green for ready nodes on the graph before | 01:07 |
| anteaya | the green wall | 01:07 |
| *** flashgordon is now known as jogo | 01:08 | |
| *** SumitNaiksatam has joined #openstack-infra | 01:08 | |
| *** gyee_ has quit IRC | 01:10 | |
| *** yamamoto has joined #openstack-infra | 01:10 | |
| *** ChuckC has joined #openstack-infra | 01:14 | |
| clarkb | once new nodepool does real work that will drop | 01:15 |
| *** davideagnello has quit IRC | 01:18 | |
| *** mudassirlatif has quit IRC | 01:20 | |
| anteaya | cool | 01:20 |
| *** tsg has quit IRC | 01:21 | |
| *** shashankhegde has quit IRC | 01:22 | |
| *** otherwiseguy has quit IRC | 01:22 | |
| *** otherwiseguy has joined #openstack-infra | 01:27 | |
| *** dimsum__ has joined #openstack-infra | 01:29 | |
| *** salv-orlando has quit IRC | 01:35 | |
| *** ddieterly has joined #openstack-infra | 01:35 | |
| *** davideagnello has joined #openstack-infra | 01:36 | |
| fungi | still no new output from the fsck | 01:41 |
| *** bdpayne_ has quit IRC | 01:41 | |
| jeblair | no news is no news | 01:42 |
| fungi | still chewing a cpu worth of iowait and ~3gib ram, so it's definitely not idle | 01:42 |
| fungi | but, yaknow, 14tib fs | 01:42 |
| *** otherwiseguy has quit IRC | 01:46 | |
| *** pc_m has quit IRC | 01:47 | |
| *** dimsum__ has quit IRC | 01:50 | |
| *** dimsum__ has joined #openstack-infra | 01:51 | |
| *** dimsum__ has quit IRC | 01:55 | |
| *** rcarrillocruz has quit IRC | 01:58 | |
| *** rcarrillocruz has joined #openstack-infra | 01:58 | |
| *** teran_ has joined #openstack-infra | 01:58 | |
| *** paul-- has quit IRC | 02:00 | |
| *** jhesketh has quit IRC | 02:00 | |
| *** ChuckC has quit IRC | 02:00 | |
| *** groknix has quit IRC | 02:00 | |
| *** markvan_ has quit IRC | 02:00 | |
| *** sdake has quit IRC | 02:00 | |
| *** vryzhenkin has quit IRC | 02:00 | |
| *** viglesias has quit IRC | 02:00 | |
| *** kevinbenton has quit IRC | 02:00 | |
| *** toabctl has quit IRC | 02:00 | |
| *** mtreinish has quit IRC | 02:00 | |
| *** ctlaugh has quit IRC | 02:00 | |
| *** jeblair has quit IRC | 02:00 | |
| *** greghaynes has quit IRC | 02:00 | |
| *** funzo has quit IRC | 02:00 | |
| *** Hunner has quit IRC | 02:00 | |
| *** pcrews has quit IRC | 02:00 | |
| *** ctracey has quit IRC | 02:00 | |
| *** matt__ has quit IRC | 02:00 | |
| *** persia has quit IRC | 02:00 | |
| *** krtaylor has quit IRC | 02:00 | |
| *** Bart__ has quit IRC | 02:00 | |
| *** masayukig has quit IRC | 02:00 | |
| *** Vek has quit IRC | 02:00 | |
| *** kashyap has quit IRC | 02:00 | |
| *** boris-42 has quit IRC | 02:00 | |
| *** Svedrin has quit IRC | 02:00 | |
| *** ilyashakhat has quit IRC | 02:00 | |
| *** erw has quit IRC | 02:00 | |
| *** ameade_ has quit IRC | 02:00 | |
| *** hogepodge has quit IRC | 02:00 | |
| *** mmedvede has quit IRC | 02:00 | |
| *** briancline has quit IRC | 02:00 | |
| *** sc68cal has quit IRC | 02:00 | |
| *** ddieterly has quit IRC | 02:00 | |
| *** yamamoto has quit IRC | 02:00 | |
| *** mestery has quit IRC | 02:00 | |
| *** teran has quit IRC | 02:00 | |
| *** yolanda has quit IRC | 02:00 | |
| *** bradjones has quit IRC | 02:00 | |
| *** doude has quit IRC | 02:00 | |
| *** esmute has quit IRC | 02:00 | |
| *** isq has quit IRC | 02:00 | |
| *** mordred has quit IRC | 02:00 | |
| *** dmsimard_away has quit IRC | 02:00 | |
| *** fandi has quit IRC | 02:00 | |
| *** harlowja has quit IRC | 02:04 | |
| *** harlowja has joined #openstack-infra | 02:04 | |
| openstackgerrit | Michael Krotscheck proposed openstack-infra/storyboard-webclient: Streamlined task edit form. https://review.openstack.org/135739 | 02:05 |
| *** teran_ has quit IRC | 02:05 | |
| openstackgerrit | Michael Krotscheck proposed openstack-infra/storyboard-webclient: Enable HTTP Caching on resources. https://review.openstack.org/136149 | 02:06 |
| openstackgerrit | Michael Krotscheck proposed openstack-infra/storyboard-webclient: Switched use of "Resource.read()" to "Resource.get()" https://review.openstack.org/136148 | 02:06 |
| openstackgerrit | Michael Krotscheck proposed openstack-infra/storyboard-webclient: Enable HTTP Caching on resources. https://review.openstack.org/136149 | 02:06 |
| openstackgerrit | Michael Krotscheck proposed openstack-infra/storyboard-webclient: Switched use of "Resource.read()" to "Resource.get()" https://review.openstack.org/136148 | 02:06 |
| *** teran has joined #openstack-infra | 02:08 | |
| openstackgerrit | Michael Krotscheck proposed openstack-infra/storyboard-webclient: Enable HTTP Caching on resources. https://review.openstack.org/136149 | 02:08 |
| *** mmaglana has joined #openstack-infra | 02:10 | |
| *** shashankhegde has joined #openstack-infra | 02:12 | |
| *** SumitNaiksatam has quit IRC | 02:13 | |
| *** SumitNaiksatam has joined #openstack-infra | 02:14 | |
| *** davideagnello has quit IRC | 02:14 | |
| *** mmaglana has quit IRC | 02:15 | |
| *** jungleboyj has joined #openstack-infra | 02:16 | |
| *** otherwiseguy has joined #openstack-infra | 02:22 | |
| *** adalbas has quit IRC | 02:22 | |
| anteaya | I'm fading and am thinking about logging off for the night | 02:22 |
| fungi | night | 02:23 |
| anteaya | I don't think there is anything of any use I can do | 02:23 |
| anteaya | okay g'night | 02:23 |
| fungi | and thanks for your help! | 02:23 |
| clarkb | fungi anything new? | 02:24 |
| clarkb | anteaya gnite | 02:24 |
| fungi | clarkb: nope | 02:24 |
| fungi | clarkb: i have memories of waiting many, many hours for a 16tib array to fsck on a backup server once, and that was bare metal with local disks | 02:26 |
| clarkb | if its not asking about errors thats good | 02:32 |
| fungi | well, i did fsck -y | 02:32 |
| *** sc68cal has joined #openstack-infra | 02:32 | |
| *** briancline has joined #openstack-infra | 02:32 | |
| *** mmedvede has joined #openstack-infra | 02:32 | |
| *** hogepodge has joined #openstack-infra | 02:32 | |
| *** ameade_ has joined #openstack-infra | 02:32 | |
| *** erw has joined #openstack-infra | 02:32 | |
| *** ilyashakhat has joined #openstack-infra | 02:32 | |
| *** Svedrin has joined #openstack-infra | 02:32 | |
| *** boris-42 has joined #openstack-infra | 02:32 | |
| *** kashyap has joined #openstack-infra | 02:32 | |
| *** Vek has joined #openstack-infra | 02:32 | |
| *** masayukig has joined #openstack-infra | 02:32 | |
| *** Bart__ has joined #openstack-infra | 02:32 | |
| *** krtaylor has joined #openstack-infra | 02:32 | |
| *** persia has joined #openstack-infra | 02:32 | |
| *** matt__ has joined #openstack-infra | 02:32 | |
| *** ctracey has joined #openstack-infra | 02:32 | |
| *** pcrews has joined #openstack-infra | 02:32 | |
| *** Hunner has joined #openstack-infra | 02:32 | |
| *** funzo has joined #openstack-infra | 02:32 | |
| *** ddieterly has joined #openstack-infra | 02:32 | |
| *** yamamoto has joined #openstack-infra | 02:32 | |
| *** mestery has joined #openstack-infra | 02:32 | |
| *** yolanda has joined #openstack-infra | 02:32 | |
| *** bradjones has joined #openstack-infra | 02:32 | |
| *** doude has joined #openstack-infra | 02:32 | |
| *** esmute has joined #openstack-infra | 02:32 | |
| *** isq has joined #openstack-infra | 02:32 | |
| *** mordred has joined #openstack-infra | 02:32 | |
| *** dmsimard_away has joined #openstack-infra | 02:32 | |
| *** ddieterly has quit IRC | 02:32 | |
| *** yamamoto has quit IRC | 02:32 | |
| *** mestery has quit IRC | 02:32 | |
| *** yolanda has quit IRC | 02:32 | |
| *** bradjones has quit IRC | 02:32 | |
| *** doude has quit IRC | 02:32 | |
| *** esmute has quit IRC | 02:32 | |
| *** isq has quit IRC | 02:32 | |
| *** mordred has quit IRC | 02:32 | |
| *** dmsimard_away has quit IRC | 02:32 | |
| fungi | because, really, if it asks about working around an error, am i going to answer in the negative? | 02:32 |
| *** ChuckC has joined #openstack-infra | 02:32 | |
| *** groknix has joined #openstack-infra | 02:32 | |
| *** markvan_ has joined #openstack-infra | 02:32 | |
| *** sdake has joined #openstack-infra | 02:32 | |
| *** vryzhenkin has joined #openstack-infra | 02:32 | |
| *** viglesias has joined #openstack-infra | 02:32 | |
| *** kevinbenton has joined #openstack-infra | 02:32 | |
| *** toabctl has joined #openstack-infra | 02:32 | |
| *** mtreinish has joined #openstack-infra | 02:32 | |
| *** ctlaugh has joined #openstack-infra | 02:32 | |
| *** jeblair has joined #openstack-infra | 02:32 | |
| *** greghaynes has joined #openstack-infra | 02:32 | |
| clarkb | if you want to live dangerously :) | 02:33 |
| *** Longgeek has joined #openstack-infra | 02:33 | |
| fungi | i think we're stuck accepting whattever data loss there is | 02:33 |
| *** markmcclain has quit IRC | 02:33 | |
| clarkb | ya agreed | 02:34 |
| fungi | everything else checked clean, so it's just the log volume in question | 02:34 |
| *** changbl has quit IRC | 02:35 | |
| fungi | chances are we lost some however thousand logfiles were semi-uploaded as the problem was setting in and stuff prior to this evening will be fine | 02:36 |
| *** tsg has joined #openstack-infra | 02:37 | |
| *** leakypipes has quit IRC | 02:38 | |
| *** Longgeek has quit IRC | 02:38 | |
| *** armax has joined #openstack-infra | 02:40 | |
| *** ChuckC has quit IRC | 02:40 | |
| *** groknix has quit IRC | 02:40 | |
| *** markvan_ has quit IRC | 02:40 | |
| *** sdake has quit IRC | 02:40 | |
| *** vryzhenkin has quit IRC | 02:40 | |
| *** viglesias has quit IRC | 02:40 | |
| *** kevinbenton has quit IRC | 02:40 | |
| *** toabctl has quit IRC | 02:40 | |
| *** mtreinish has quit IRC | 02:40 | |
| *** ctlaugh has quit IRC | 02:40 | |
| *** jeblair has quit IRC | 02:40 | |
| *** greghaynes has quit IRC | 02:40 | |
| *** ddieterly has joined #openstack-infra | 02:41 | |
| *** yamamoto has joined #openstack-infra | 02:41 | |
| *** mestery has joined #openstack-infra | 02:41 | |
| *** yolanda has joined #openstack-infra | 02:41 | |
| *** bradjones has joined #openstack-infra | 02:41 | |
| *** doude has joined #openstack-infra | 02:41 | |
| *** esmute has joined #openstack-infra | 02:41 | |
| *** isq has joined #openstack-infra | 02:41 | |
| *** mordred has joined #openstack-infra | 02:41 | |
| *** dmsimard_away has joined #openstack-infra | 02:41 | |
| *** ChuckC has joined #openstack-infra | 02:46 | |
| *** groknix has joined #openstack-infra | 02:46 | |
| *** markvan_ has joined #openstack-infra | 02:46 | |
| *** sdake has joined #openstack-infra | 02:46 | |
| *** vryzhenkin has joined #openstack-infra | 02:46 | |
| *** viglesias has joined #openstack-infra | 02:46 | |
| *** kevinbenton has joined #openstack-infra | 02:46 | |
| *** toabctl has joined #openstack-infra | 02:46 | |
| *** mtreinish has joined #openstack-infra | 02:46 | |
| *** ctlaugh has joined #openstack-infra | 02:46 | |
| *** jeblair has joined #openstack-infra | 02:46 | |
| *** greghaynes has joined #openstack-infra | 02:46 | |
| *** achanda has joined #openstack-infra | 02:46 | |
| *** paul-- has joined #openstack-infra | 02:46 | |
| *** jhesketh has joined #openstack-infra | 02:46 | |
| *** ivar-lazzaro has quit IRC | 02:51 | |
| *** david-lyle is now known as david-lyle_afk | 02:53 | |
| *** achanda has quit IRC | 02:56 | |
| *** bdpayne has joined #openstack-infra | 03:12 | |
| *** cnesa has quit IRC | 03:15 | |
| *** banix has joined #openstack-infra | 03:21 | |
| *** changbl has joined #openstack-infra | 03:22 | |
| *** dmsimard_away is now known as dmsimard | 03:23 | |
| *** ryanpetrello has joined #openstack-infra | 03:24 | |
| *** bdpayne has quit IRC | 03:25 | |
| *** otherwiseguy has quit IRC | 03:30 | |
| *** tnovacik has quit IRC | 03:31 | |
| *** koolhead17 has joined #openstack-infra | 03:33 | |
| *** Longgeek has joined #openstack-infra | 03:34 | |
| *** dmsimard is now known as dmsimard_away | 03:37 | |
| *** ZZelle_ has quit IRC | 03:38 | |
| *** Longgeek has quit IRC | 03:39 | |
| *** dmsimard_away is now known as dmsimard | 03:41 | |
| *** dmsimard is now known as dmsimard_away | 03:43 | |
| *** tsg has quit IRC | 03:46 | |
| *** dmsimard_away is now known as dmsimard | 03:48 | |
| *** yamamoto has quit IRC | 03:56 | |
| *** yamamoto has joined #openstack-infra | 04:03 | |
| *** bhuvan has joined #openstack-infra | 04:05 | |
| *** armax has quit IRC | 04:06 | |
| *** yamamoto has quit IRC | 04:10 | |
| *** armax has joined #openstack-infra | 04:11 | |
| *** armax has quit IRC | 04:12 | |
| *** banix has quit IRC | 04:14 | |
| *** armax has joined #openstack-infra | 04:19 | |
| *** thedodd has joined #openstack-infra | 04:22 | |
| *** dmsimard is now known as dmsimard_away | 04:25 | |
| *** Ryan_Lane has quit IRC | 04:27 | |
| *** mbacchi has quit IRC | 04:32 | |
| *** armax has quit IRC | 04:34 | |
| *** bhunter71 has quit IRC | 04:37 | |
| *** chandankumar has joined #openstack-infra | 04:39 | |
| *** yamamoto has joined #openstack-infra | 04:41 | |
| *** shashankhegde has quit IRC | 04:46 | |
| *** pcrews has quit IRC | 04:46 | |
| stevemar | rough week for infra | 04:51 |
| fungi | woohoo! just reached "Pass 2: Checking directory structure" | 04:52 |
| fungi | stevemar: we've seen worse | 04:52 |
| *** yamamoto has quit IRC | 04:52 | |
| stevemar | fungi, i don't doubt that, but it's just been a weird week | 04:53 |
| stevemar | especially with the stable/icehouse/havana requirements stuff that went on | 04:53 |
| stevemar | is that even sorted out yet? i am tired of rechecking :( | 04:53 |
| fungi | stevemar: i think sdague untangled the most recent round of it | 04:54 |
| stevemar | fungi, that calls for a re-checkin' | 04:57 |
| dtroyer | stevemar: we're blocked by an Icehouse Neutron bug, which was #2 on the hit parade this afternoon…I gave up trying to sort it out, but it's damn consistent now. | 04:59 |
| stevemar | dtroyer, bah humbug! | 05:00 |
| dtroyer | every client I've checked is affected, I don't know about other libs, depends on their jobs | 05:00 |
| *** Ryan_Lane has joined #openstack-infra | 05:00 | |
| stevemar | dtroyer, what was the order for the args... domain > user > project | 05:05 |
| dtroyer | domain -> project -> user | 05:06 |
| dtroyer | general to specific | 05:06 |
| stevemar | gotcha | 05:07 |
| *** uvirtbot has quit IRC | 05:07 | |
| stevemar | dtroyer, we've got 5 patches all chained together :( | 05:08 |
| dtroyer | I've thought about squashing the docs ones to reduce the job count, but at this point I don't think it really matters. | 05:09 |
| dtroyer | I've got 3 more lined up… | 05:09 |
| *** garyh has quit IRC | 05:09 | |
| *** ryanpetrello has quit IRC | 05:10 | |
| *** achanda has joined #openstack-infra | 05:13 | |
| *** davideagnello has joined #openstack-infra | 05:15 | |
| *** achanda has quit IRC | 05:15 | |
| *** SumitNaiksatam has quit IRC | 05:16 | |
| stevemar | dtroyer, what 3? | 05:17 |
| *** SumitNaiksatam has joined #openstack-infra | 05:17 | |
| *** achanda has joined #openstack-infra | 05:18 | |
| *** henditan has joined #openstack-infra | 05:18 | |
| dtroyer | server and region doc updates, os-client-config support | 05:19 |
| *** davideagnello has quit IRC | 05:20 | |
| *** Longgeek has joined #openstack-infra | 05:20 | |
| stevemar | nice | 05:21 |
| stevemar | gotta figure out caching :( | 05:21 |
| dtroyer | I got a quick one sorta working but the nova client needs some attention | 05:22 |
| *** Longgeek has quit IRC | 05:25 | |
| *** rushiagr_away is now known as rushiagr | 05:25 | |
| *** shashankhegde has joined #openstack-infra | 05:29 | |
| *** harlowja is now known as harlowja_away | 05:31 | |
| *** ryanpetrello has joined #openstack-infra | 05:33 | |
| *** ryanpetrello has quit IRC | 05:39 | |
| *** thedodd has quit IRC | 05:40 | |
| *** garyh has joined #openstack-infra | 05:41 | |
| *** dheeraj-gupta-4 has joined #openstack-infra | 05:41 | |
| *** ryanpetrello has joined #openstack-infra | 05:44 | |
| *** achanda has quit IRC | 05:51 | |
| *** koolhead17 has quit IRC | 05:53 | |
| *** yamamoto has joined #openstack-infra | 05:53 | |
| *** viglesias has quit IRC | 05:55 | |
| *** achanda has joined #openstack-infra | 05:57 | |
| *** yamamoto has quit IRC | 05:58 | |
| *** viglesias has joined #openstack-infra | 06:00 | |
| *** ryanpetrello has quit IRC | 06:07 | |
| *** dheeraj-gupta-4 has left #openstack-infra | 06:21 | |
| *** dimsum__ has joined #openstack-infra | 06:29 | |
| *** achanda has quit IRC | 06:34 | |
| *** dimsum__ has quit IRC | 06:34 | |
| *** chandankumar has quit IRC | 06:34 | |
| *** achanda has joined #openstack-infra | 06:41 | |
| *** otherwiseguy has joined #openstack-infra | 06:43 | |
| *** shashankhegde has quit IRC | 06:45 | |
| *** henditan has quit IRC | 06:48 | |
| *** henditan has joined #openstack-infra | 06:48 | |
| *** henditan has quit IRC | 06:49 | |
| *** henditan has joined #openstack-infra | 06:50 | |
| *** stevemar has quit IRC | 06:52 | |
| fungi | fsck is still on phase 2... i'm going to catch a few hours of sleep and hopefully be able to bring it back online as soon as i wake up | 06:59 |
| *** rcarrillocruz has quit IRC | 07:05 | |
| *** Ryan_Lane has quit IRC | 07:05 | |
| *** Longgeek has joined #openstack-infra | 07:06 | |
| *** Ryan_Lane has joined #openstack-infra | 07:07 | |
| *** rcarrillocruz has joined #openstack-infra | 07:09 | |
| *** Longgeek_ has joined #openstack-infra | 07:10 | |
| *** Longgeek has quit IRC | 07:10 | |
| *** Longgeek_ has quit IRC | 07:12 | |
| *** Longgeek has joined #openstack-infra | 07:18 | |
| *** AlexF has joined #openstack-infra | 07:18 | |
| openstackgerrit | Aaron Rosen proposed openstack-infra/nodepool: libffi-dev/libssl-dev/qemu-utils are required https://review.openstack.org/125515 | 07:26 |
| *** achanda has quit IRC | 07:32 | |
| *** AlexF has quit IRC | 07:34 | |
| *** melwitt1 has quit IRC | 07:36 | |
| *** achanda has joined #openstack-infra | 07:53 | |
| *** che-arne has joined #openstack-infra | 07:54 | |
| *** yamamoto has joined #openstack-infra | 07:56 | |
| *** yamamoto has quit IRC | 08:00 | |
| *** ZZelle has quit IRC | 08:02 | |
| *** ZZelle has joined #openstack-infra | 08:02 | |
| *** achanda has quit IRC | 08:07 | |
| *** otherwiseguy has quit IRC | 08:07 | |
| *** uvirtbot has joined #openstack-infra | 08:08 | |
| *** dimsum__ has joined #openstack-infra | 08:18 | |
| *** viglesias has quit IRC | 08:19 | |
| *** nadya has joined #openstack-infra | 08:20 | |
| *** nadya is now known as Guest98480 | 08:21 | |
| *** dimsum__ has quit IRC | 08:22 | |
| *** changbl has quit IRC | 08:26 | |
| *** yamamoto has joined #openstack-infra | 08:34 | |
| *** andreykurilin_ has joined #openstack-infra | 08:46 | |
| *** bhuvan has quit IRC | 08:50 | |
| *** hdd has joined #openstack-infra | 08:56 | |
| *** marun has quit IRC | 08:57 | |
| *** marun has joined #openstack-infra | 08:59 | |
| *** ZZelle_ has joined #openstack-infra | 09:01 | |
| *** Guest98480 has quit IRC | 09:04 | |
| *** marun_ has joined #openstack-infra | 09:05 | |
| *** marun has quit IRC | 09:06 | |
| *** marun_ has quit IRC | 09:10 | |
| *** viglesias has joined #openstack-infra | 09:13 | |
| *** marun_ has joined #openstack-infra | 09:14 | |
| *** marun has joined #openstack-infra | 09:17 | |
| *** marun_ has quit IRC | 09:19 | |
| *** hdd has quit IRC | 09:27 | |
| *** tnovacik has joined #openstack-infra | 09:31 | |
| *** talluri has joined #openstack-infra | 09:33 | |
| *** Ryan_Lane has quit IRC | 09:35 | |
| *** ihrachyshka has joined #openstack-infra | 09:35 | |
| *** marun_ has joined #openstack-infra | 09:37 | |
| *** henditan has quit IRC | 09:37 | |
| *** marun has quit IRC | 09:39 | |
| *** subscope_ has joined #openstack-infra | 09:47 | |
| *** nelsnelson has quit IRC | 09:51 | |
| *** marun_ has quit IRC | 09:56 | |
| *** marun has joined #openstack-infra | 09:56 | |
| *** ihrachyshka has quit IRC | 10:00 | |
| *** marun has quit IRC | 10:01 | |
| *** Longgeek has quit IRC | 10:01 | |
| *** marun_ has joined #openstack-infra | 10:03 | |
| *** marun__ has joined #openstack-infra | 10:04 | |
| *** andreykurilin_ has quit IRC | 10:05 | |
| *** marun_ has quit IRC | 10:07 | |
| *** marun has joined #openstack-infra | 10:07 | |
| *** Longgeek has joined #openstack-infra | 10:09 | |
| *** marun__ has quit IRC | 10:10 | |
| *** marun_ has joined #openstack-infra | 10:13 | |
| *** marun has quit IRC | 10:14 | |
| *** marun has joined #openstack-infra | 10:17 | |
| *** marun_ has quit IRC | 10:19 | |
| *** cnesa has joined #openstack-infra | 10:19 | |
| *** koolhead17 has joined #openstack-infra | 10:22 | |
| *** talluri has quit IRC | 10:24 | |
| *** rcarrillocruz has quit IRC | 10:36 | |
| *** adalbas has joined #openstack-infra | 10:36 | |
| *** rcarrillocruz has joined #openstack-infra | 10:39 | |
| *** MaxV has joined #openstack-infra | 10:51 | |
| *** MaxV has quit IRC | 10:51 | |
| *** MaxV has joined #openstack-infra | 10:54 | |
| *** salv-orlando has joined #openstack-infra | 10:59 | |
| *** adalbas has quit IRC | 11:07 | |
| *** Longgeek has quit IRC | 11:07 | |
| *** funzo has quit IRC | 11:07 | |
| *** Hunner has quit IRC | 11:07 | |
| *** ctracey has quit IRC | 11:07 | |
| *** matt__ has quit IRC | 11:07 | |
| *** persia has quit IRC | 11:07 | |
| *** krtaylor has quit IRC | 11:07 | |
| *** Bart__ has quit IRC | 11:07 | |
| *** masayukig has quit IRC | 11:07 | |
| *** Vek has quit IRC | 11:07 | |
| *** kashyap has quit IRC | 11:07 | |
| *** boris-42 has quit IRC | 11:07 | |
| *** Svedrin has quit IRC | 11:07 | |
| *** ilyashakhat has quit IRC | 11:07 | |
| *** erw has quit IRC | 11:07 | |
| *** ameade_ has quit IRC | 11:07 | |
| *** hogepodge has quit IRC | 11:07 | |
| *** mmedvede has quit IRC | 11:07 | |
| *** briancline has quit IRC | 11:07 | |
| *** sc68cal has quit IRC | 11:07 | |
| *** koolhead17 has quit IRC | 11:07 | |
| *** achanda has joined #openstack-infra | 11:07 | |
| *** adalbas has joined #openstack-infra | 11:10 | |
| *** Longgeek has joined #openstack-infra | 11:10 | |
| *** funzo has joined #openstack-infra | 11:10 | |
| *** Hunner has joined #openstack-infra | 11:10 | |
| *** ctracey has joined #openstack-infra | 11:10 | |
| *** matt__ has joined #openstack-infra | 11:10 | |
| *** persia has joined #openstack-infra | 11:10 | |
| *** krtaylor has joined #openstack-infra | 11:10 | |
| *** Bart__ has joined #openstack-infra | 11:10 | |
| *** masayukig has joined #openstack-infra | 11:10 | |
| *** Vek has joined #openstack-infra | 11:10 | |
| *** kashyap has joined #openstack-infra | 11:10 | |
| *** boris-42 has joined #openstack-infra | 11:10 | |
| *** Svedrin has joined #openstack-infra | 11:10 | |
| *** ilyashakhat has joined #openstack-infra | 11:10 | |
| *** erw has joined #openstack-infra | 11:10 | |
| *** ameade_ has joined #openstack-infra | 11:10 | |
| *** hogepodge has joined #openstack-infra | 11:10 | |
| *** mmedvede has joined #openstack-infra | 11:10 | |
| *** briancline has joined #openstack-infra | 11:10 | |
| *** sc68cal has joined #openstack-infra | 11:10 | |
| *** uvirtbot has quit IRC | 11:11 | |
| *** ChuckC has quit IRC | 11:11 | |
| *** groknix has quit IRC | 11:11 | |
| *** markvan_ has quit IRC | 11:11 | |
| *** sdake has quit IRC | 11:11 | |
| *** vryzhenkin has quit IRC | 11:11 | |
| *** kevinbenton has quit IRC | 11:11 | |
| *** toabctl has quit IRC | 11:11 | |
| *** mtreinish has quit IRC | 11:11 | |
| *** ctlaugh has quit IRC | 11:11 | |
| *** jeblair has quit IRC | 11:11 | |
| *** greghaynes has quit IRC | 11:11 | |
| *** achanda has quit IRC | 11:12 | |
| *** uvirtbot has joined #openstack-infra | 11:13 | |
| *** ChuckC has joined #openstack-infra | 11:13 | |
| *** groknix has joined #openstack-infra | 11:13 | |
| *** markvan_ has joined #openstack-infra | 11:13 | |
| *** sdake has joined #openstack-infra | 11:13 | |
| *** vryzhenkin has joined #openstack-infra | 11:13 | |
| *** kevinbenton has joined #openstack-infra | 11:13 | |
| *** toabctl has joined #openstack-infra | 11:13 | |
| *** mtreinish has joined #openstack-infra | 11:13 | |
| *** ctlaugh has joined #openstack-infra | 11:13 | |
| *** jeblair has joined #openstack-infra | 11:13 | |
| *** greghaynes has joined #openstack-infra | 11:13 | |
| *** talluri has joined #openstack-infra | 11:14 | |
| *** otter768 has quit IRC | 11:29 | |
| *** marun has quit IRC | 11:30 | |
| *** hdd has joined #openstack-infra | 11:50 | |
| *** MaxV has quit IRC | 11:50 | |
| *** mrmartin has joined #openstack-infra | 11:51 | |
| *** HeOS has quit IRC | 11:54 | |
| *** koolhead17 has joined #openstack-infra | 11:54 | |
| *** koolhead17 has joined #openstack-infra | 11:54 | |
| *** salv-orlando has quit IRC | 11:59 | |
| *** hdd has quit IRC | 12:02 | |
| *** MaxV has joined #openstack-infra | 12:04 | |
| *** talluri has quit IRC | 12:08 | |
| fungi | fsck finished while i napped | 12:13 |
| *** talluri has joined #openstack-infra | 12:14 | |
| fungi | rebooting static.o.o now for good measure and then will let loose the hounds of ci | 12:14 |
| *** dimsum__ has joined #openstack-infra | 12:14 | |
| *** talluri has quit IRC | 12:18 | |
| *** koolhead17 has quit IRC | 12:21 | |
| *** dimsum__ has quit IRC | 12:22 | |
| *** koolhead17 has joined #openstack-infra | 12:22 | |
| fungi | also rebooted the jenkins masters since i had the opportunity and they had new kernels installed from security updates since last restart | 12:23 |
| fungi | jobs are running again and nodepool seems to be providing workers | 12:25 |
| *** koolhead17 has quit IRC | 12:26 | |
| fungi | there are still a few changes in zuul pipelines which had some unstable results from before, so those will need to flush out, but the remainder of the queued work should hopefully run fine | 12:28 |
| sdague | fungi: those graphs look very weird | 12:29 |
| fungi | the node graph probably will look weird for a bit since clarkb had two nodepools connected for a while | 12:30 |
| sdague | ah, gotcha | 12:30 |
| sdague | so are things just recovering now? | 12:30 |
| fungi | a job had run to completion in the gate and uploaded logs, so i think we're back on track | 12:30 |
| fungi | sdague: yeah, fsck of a 14tb fs takes a while | 12:30 |
| sdague | yep, no worries | 12:31 |
| fungi | once things look a little cleaner i'll clear the /topic alerts | 12:31 |
| sdague | mostly wanted to figure out if the stuff I had up that's vaguely waiting to test needs any active action | 12:31 |
| sdague | but it looks like they are all mostly in the right queues | 12:32 |
| *** matrohon has joined #openstack-infra | 12:35 | |
| fungi | bbiab | 12:41 |
| openstackgerrit | Merged openstack-infra/jenkins-job-builder: Add support for the matrix-combinations plugin https://review.openstack.org/133698 | 12:45 |
| *** yamamoto has quit IRC | 12:57 | |
| jeblair | fungi: morning | 12:58 |
| jeblair | that last message looks really promising ^ :) | 12:58 |
| *** sdake has quit IRC | 12:59 | |
| *** yamamoto has joined #openstack-infra | 13:01 | |
| *** sdake has joined #openstack-infra | 13:03 | |
| openstackgerrit | Merged openstack-infra/jenkins-job-builder: Add perlcritic capability to violations method https://review.openstack.org/134867 | 13:06 |
| fungi | morning jeblair | 13:19 |
| fungi | and yes, so far so good | 13:19 |
| clarkb | pretend I amnot awake.does nodepool appear to be marking nodes used and delete then deleting nodes properly? | 13:21 |
| *** yamamoto has quit IRC | 13:22 | |
| fungi | yep, though there were some stale ready nodes i'm clearing now | 13:22 |
| fungi | things should hopefully speed up once it replaces those | 13:23 |
| clarkb | cool that was one piece of functionality I could not confirm with running jenkinses | 13:28 |
| clarkb | *without running | 13:29 |
| *** otter768 has joined #openstack-infra | 13:30 | |
| *** mase_x200 has joined #openstack-infra | 13:30 | |
| sdague | question: when the merge check happens, is jenkins -1ing patches for that? | 13:33 |
| fungi | yes | 13:34 |
| sdague | is that true for the zuul merge conflict check as well? | 13:34 |
| clarkb | but no jenkins jobs run in either case | 13:34 |
| clarkb | its just zuul attempting to merge within zuul | 13:35 |
| fungi | now i'm unsure what you're calling "merge check" vs "zuul merge conflict check" | 13:35 |
| *** otter768 has quit IRC | 13:35 | |
| fungi | for the latter, you mean when a patch is in a dependent queue and conflicts with some patch in front of it? | 13:35 |
| sdague | ok, cool | 13:35 |
| sdague | well there seems to be an edge case with this review - https://review.openstack.org/#/c/129689/ | 13:36 |
| sdague | because it shows up in a NOT Verified>=-2,jenkins search | 13:36 |
| sdague | with the abandon logic we're using in nova, I started also looking for things where jenkins didn't seem to have a vote on reviews that were > 2d old | 13:37 |
| sdague | to recheck those | 13:37 |
| sdague | and that one popped up which looks like it should be ok, but it's still showing up in the jenkins didn't vote query | 13:38 |
| fungi | that's weird... it's got a -1 in the verify column from the jenkins account according to the webui | 13:38 |
| sdague | yep | 13:38 |
| sdague | though on the summary page, you don't see the -1 | 13:38 |
| sdague | which makes me think there is an internal sync issue | 13:38 |
| clarkb | or a draft | 13:38 |
| clarkb | but drafts should be mostly dead now | 13:39 |
| sdague | yeh, I'm pretty sure this isn't a draft thing | 13:39 |
| sdague | anyway, it's an anomoly. it's not really important, I can recheck it anyway, but thought I'd at least bring it up | 13:40 |
| clarkb | it doesnt need a recheck | 13:41 |
| sdague | clarkb: true | 13:41 |
| sdague | but if it didn't get rechecked by an automatic query, it wouldn't be the end of the world | 13:42 |
| sdague | there were 5 nova patches that got swept up in vote getting reset going into the gate, then something died, for instance, when I originally ran the query | 13:42 |
| sdague | so they would have fallen out of the common queries people use because they didn't have jenkins results on them | 13:43 |
| clarkb | you can solve that with jenkins=0 queries | 13:44 |
| clarkb | iirc that does work but only if you are explicit about wanting 0 | 13:44 |
| *** mase_x200 has quit IRC | 13:44 | |
| fungi | clarkb: there might be something odd with the zeromq connectivity to some of the jenkins masters... only jenkins 1, 3 and 6 have nodes marked used but i found slaves associated with other masters running jobs while still marked ready in nodepool | 13:45 |
| fungi | wait, it seems to slowly be correcting itself | 13:45 |
| clarkb | fungi :( what does netstat -np | grep 8888 show? | 13:45 |
| fungi | jenkins04 is now in the list of masters with nodes marked used | 13:45 |
| fungi | clarkb: 7 established as expected | 13:46 |
| clarkb | ok | 13:46 |
| sdague | clarkb: no, you can't in this case | 13:47 |
| sdague | NOT Verified>=-2,jenkins | 13:47 |
| sdague | that patch popped up | 13:47 |
| clarkb | you might try toggling the zmq publishing off then on on one of the non happy masters | 13:47 |
| sdague | so from the gerrit search engine jenkins doesn't exist at that patch level | 13:47 |
| clarkb | does that patch come back witg a 0 query? | 13:48 |
| fungi | clarkb: i'll try that in a sec once i get back to a real computer, if they're still exhibiting this behavior | 13:48 |
| clarkb | sdague negating the >=-2 is different | 13:48 |
| clarkb | because gerrit | 13:48 |
| clarkb | basically 0 only matches if asked for | 13:49 |
| sdague | hmmmm | 13:50 |
| sdague | age:2d status:open label:Verified=0,jenkins | 13:50 |
| *** JayJ has joined #openstack-infra | 13:50 | |
| *** JayJ has quit IRC | 13:50 | |
| sdague | seems to have the same results as NOT label:Verified>=-2,jenkins | 13:51 |
| clarkb | including +1 +2? | 13:51 |
| sdague | there are no +1 or +2 | 13:51 |
| clarkb | er wait ya | 13:51 |
| sdague | label:Verified=0,jenkins === NOT label:Verified>=-2,jenkins | 13:52 |
| sdague | from my experiment right now | 13:52 |
| clarkb | there is somethi g different about them. I ran into it when determining self votes | 13:52 |
| sdague | self might be special | 13:52 |
| clarkb | ph maybe | 13:52 |
| sdague | ok, time to stop playing with this stuff and get back to family things. Enjoy the weekend folks. | 13:53 |
| *** teran has quit IRC | 13:56 | |
| fungi | toggled zmq plugin in global config on jenkins07 but so far no change. of course it may not have experienced a node state change yet since then | 14:02 |
| *** ryanpetrello has joined #openstack-infra | 14:06 | |
| clarkb | fungi it might not toggle the connection server side at all instead just prevents writing to it when off | 14:07 |
| clarkb | if it persists a nodepool restart is likely simplest fix | 14:08 |
| jeblair | clarkb: jenkins restart? | 14:09 |
| jeblair | if there's a zmq problem, restarting the affected jenkins is probably the better fix | 14:09 |
| clarkb | oh jenkins may do it too | 14:09 |
| clarkb | ya | 14:09 |
| *** alexpilotti has joined #openstack-infra | 14:18 | |
| *** julim has quit IRC | 14:23 | |
| *** yamamoto has joined #openstack-infra | 14:23 | |
| *** pc_m has joined #openstack-infra | 14:23 | |
| *** yamamoto has quit IRC | 14:28 | |
| *** banix has joined #openstack-infra | 14:33 | |
| *** rushiagr is now known as rushiagr_away | 14:36 | |
| openstackgerrit | Merged openstack-infra/devstack-gate: Fix up sudo wildcard copy https://review.openstack.org/133914 | 14:37 |
| openstackgerrit | Merged openstack-infra/devstack-gate: Copy all apache logs into output https://review.openstack.org/133881 | 14:40 |
| *** rushiagr_away is now known as rushiagr | 14:40 | |
| mordred | morning all | 14:46 |
| nibalizer | ohai | 14:47 |
| *** ryanpetrello has quit IRC | 14:52 | |
| fungi | clarkb: jeblair: interestingly, i restarted all of the jenkinses (with successive reboots spaced a minute or so apart) | 14:59 |
| fungi | rather than just cancelling shutdown in the jenkins service, since they were all complaining that they had kernel updates | 14:59 |
| jeblair | did that make them happier? | 15:00 |
| fungi | no, i mean before | 15:00 |
| jeblair | ah yeah | 15:01 |
| fungi | when i initially brought them back online after i got the logs.o.o site back up and operational | 15:01 |
| jeblair | so if there's a zmq problem, it may be the thing that i think we've seen very rarely before, where zmq just doesn't send any data. in those cases, another jenkins restart usually fixes things | 15:01 |
| mordred | jeblair: that's the saddest sentence I've ever heard | 15:02 |
| fungi | fair enough... at this point looks like jenkins02, 05 and 07 will need that. i'll put them into prepare for shutdown but am about to head out for errands | 15:02 |
| clarkb | its why zmq is fail. we might also try newer jeromq at some point | 15:02 |
| clarkb | as it may be a fixed bug in their implementation | 15:03 |
| mordred | sounds like more a bug in implementation than 0mq itself | 15:03 |
| mordred | I'm pretty sure "not sending any 0mq packets" is not a design feature of 0mq :) | 15:03 |
| jeblair | regardless, the problem goes away when jenkins does (we can have zuul send the messages we need) | 15:03 |
| mordred | yes | 15:04 |
| clarkb | mordred its a known problem with 0mq because it doesnt communicate connectio errors up the stack | 15:04 |
| mordred | ah - ok | 15:04 |
| fungi | okay, jenkins02,05,07 are in prepare for shutdown | 15:05 |
| fungi | i'm heading out to get some exercise but will be back in a couple hours and can restart them them if nobody beats me to it | 15:05 |
| fungi | <- out | 15:05 |
| *** emagana has joined #openstack-infra | 15:11 | |
| *** banix has quit IRC | 15:11 | |
| *** banix has joined #openstack-infra | 15:20 | |
| clarkb | I am cleaning up offlined nodes on 02,05,07 now so that when we restart those masters those nodes don't come back | 15:27 |
| *** matrohon has quit IRC | 15:28 | |
| *** otter768 has joined #openstack-infra | 15:31 | |
| *** banix has quit IRC | 15:33 | |
| clarkb | trying to do it via nodepool so that nodepool knows it can delete those nodes but its kinda slow going | 15:35 |
| *** otter768 has quit IRC | 15:36 | |
| clarkb | old nodepool has had all of its nodes deleted | 15:37 |
| *** pcrews has joined #openstack-infra | 15:55 | |
| clarkb | ok all nodes that were offlined on those three hosts are queued up for deletion | 15:59 |
| clarkb | except for any nodes that were currently running jobs | 15:59 |
| clarkb | so double check before you stop start jenkins | 15:59 |
| *** MarkAtwood has joined #openstack-infra | 16:00 | |
| *** paul-- has quit IRC | 16:07 | |
| *** paul-- has joined #openstack-infra | 16:08 | |
| *** MarkAtwood has quit IRC | 16:10 | |
| clarkb | ok 05 looks clean I am going to restart it now | 16:15 |
| *** mudassirlatif has joined #openstack-infra | 16:17 | |
| clarkb | I see logstash.o.o connected to 05's socket but not nodepool.o.o if I look at the 05 side | 16:18 |
| clarkb | nodepool side says it is connected | 16:18 |
| clarkb | same thing on 02 | 16:19 |
| clarkb | and 07 | 16:19 |
| clarkb | but 01 shows the connection. I think that may be the actual issue | 16:19 |
| clarkb | I am going to put 05 back in shutdown mode now | 16:19 |
| *** changbl has joined #openstack-infra | 16:22 | |
| clarkb | I think we do actually want to restart nodepool or otherwise convince it that its connections don't exist | 16:22 |
| clarkb | but I need to find breakfast and I don't think there is a rush on that so back later | 16:23 |
| clarkb | maybe we tcpkill those connections and see if that forces them to reconnect (that is how 0mq is supposed to behave) | 16:26 |
| *** yjiang5 has quit IRC | 16:28 | |
| *** dmsimard_away is now known as dmsimard | 16:30 | |
| clarkb | theory time before I really afk. when the jenkinses were rebooted 02,05,07 didn't properly close their sockets and left nodepool hanging | 16:30 |
| mordred | really? | 16:31 |
| clarkb | mordred: there is clearly an ESTABLISHED connection from nodepool to 02/05/07 on nodepool.o.o. If you look on 02/05/07 there is nothing | 16:32 |
| clarkb | so ya something like that | 16:32 |
| *** e0ne has joined #openstack-infra | 16:33 | |
| mordred | clarkb: that's just crazypants | 16:33 |
| clarkb | looking at tcp keepalive values I would expect the connection to die after 2 hours | 16:37 |
| clarkb | but its been >2 hours since jenkinses were rebooted | 16:37 |
| clarkb | maybe 0mq modified keepalive times for itself? can you dothat? | 16:37 |
| clarkb | yup looks like it is a sockopt | 16:38 |
| *** subscope_ has quit IRC | 16:39 | |
| *** andreykurilin_ has joined #openstack-infra | 16:40 | |
| *** dmsimard is now known as dmsimard_away | 16:41 | |
| *** AlexF has joined #openstack-infra | 16:42 | |
| *** wenlock has joined #openstack-infra | 16:44 | |
| clarkb | http://lists.zeromq.org/pipermail/zeromq-dev/2013-May/021557.html is likely the issue. I can poke at our tools when not weekend to make the situation better | 16:45 |
| *** hogepodge has quit IRC | 16:47 | |
| *** emagana has quit IRC | 16:48 | |
| *** emagana has joined #openstack-infra | 16:48 | |
| *** hogepodge has joined #openstack-infra | 16:49 | |
| *** emagana_ has joined #openstack-infra | 16:51 | |
| *** julim has joined #openstack-infra | 16:51 | |
| *** shashankhegde has joined #openstack-infra | 16:52 | |
| *** emagana has quit IRC | 16:53 | |
| *** AlexF has quit IRC | 16:55 | |
| *** julim has quit IRC | 16:56 | |
| *** julim has joined #openstack-infra | 17:00 | |
| *** AlexF has joined #openstack-infra | 17:01 | |
| fungi | back | 17:03 |
| fungi | having a look now | 17:04 |
| *** julim has quit IRC | 17:05 | |
| fungi | i'll use killcx on those sockets | 17:05 |
| *** koolhead17 has joined #openstack-infra | 17:05 | |
| *** koolhead17 has joined #openstack-infra | 17:05 | |
| *** julim has joined #openstack-infra | 17:06 | |
| mordred | btw - I HATE places with bad dns | 17:07 |
| mordred | just saying | 17:07 |
| fungi | like... the internet? | 17:07 |
| mordred | well, no, like hotels | 17:07 |
| fungi | oh, even worse than the internet | 17:07 |
| mordred | yah | 17:07 |
| mordred | like, the brokennet | 17:07 |
| fungi | incompenet | 17:08 |
| *** dimsum__ has joined #openstack-infra | 17:08 | |
| *** MaxV has quit IRC | 17:09 | |
| *** armax has joined #openstack-infra | 17:09 | |
| *** garyh has quit IRC | 17:10 | |
| *** dimsum__ has quit IRC | 17:14 | |
| mordred | :) | 17:15 |
| *** pcrews has quit IRC | 17:20 | |
| *** mudassirlatif has quit IRC | 17:21 | |
| *** andreykurilin_ has quit IRC | 17:21 | |
| *** andreykurilin_ has joined #openstack-infra | 17:21 | |
| jeblair | i think we could probably reset the topic now, yeah? | 17:23 |
| fungi | probably. i was hoping to have the nodepool issues worked out first, but i think we've caught up regardless | 17:24 |
| fungi | #status ok | 17:24 |
| openstackstatus | fungi: sending ok | 17:24 |
| *** ChanServ changes topic to "Discussion of OpenStack Developer Infrastructure | docs http://ci.openstack.org | bugs https://launchpad.net/openstack-ci/ | https://git.openstack.org/cgit/openstack-infra/system-config/tree/" | 17:25 | |
| *** yamamoto has joined #openstack-infra | 17:26 | |
| openstackstatus | fungi: finished sending ok | 17:26 |
| *** andreykurilin_ has quit IRC | 17:28 | |
| *** marun has joined #openstack-infra | 17:30 | |
| *** yamamoto has quit IRC | 17:30 | |
| fungi | killcx did nothing (i wonder if it's ineffective on trusty?). even tried disabling and reenabling the plugin on one again after that and still no dice | 17:31 |
| *** otter768 has joined #openstack-infra | 17:32 | |
| clarkb | fungi I dont think the plugin is going to help. nodepool is the broken side | 17:32 |
| fungi | agreed | 17:32 |
| fungi | any objections to a nodepool restart? i can clean up the remaining mess once it's safe to do so | 17:33 |
| clarkb | none from me I think we undsrstand the issue so no important data lost by doing that | 17:33 |
| *** Longgeek has quit IRC | 17:33 | |
| *** armax has quit IRC | 17:35 | |
| *** otter768 has quit IRC | 17:37 | |
| *** AlexF has quit IRC | 17:43 | |
| fungi | yeah, restarting nodepool has solved it | 17:43 |
| fungi | all 7 jenkins masters now show open 0mq sockets inbound from nodepool.o.o | 17:43 |
| *** shashankhegde has quit IRC | 17:45 | |
| *** garyh has joined #openstack-infra | 17:46 | |
| fungi | rackspace's vague and handwavey post-mortem http://paste.openstack.org/show/136487/ | 17:47 |
| fungi | at least they're honest! | 17:47 |
| fungi | okay, none of the jenkins masters are in prepare for shutdown any longer | 17:56 |
| fungi | zuul is also nearly idle | 17:56 |
| fungi | in a couple hours i'll clean up any lingering stale nodepool nodes and then check for alien nodes as well | 17:57 |
| *** garyh has quit IRC | 17:57 | |
| fungi | all jenkins masters are showing used nodes in nodepool now, so aside from needed cleanup things seem to be fine now | 18:05 |
| *** koolhead17 has quit IRC | 18:07 | |
| *** koolhead17 has joined #openstack-infra | 18:07 | |
| *** luqas has joined #openstack-infra | 18:10 | |
| *** e0ne has quit IRC | 18:11 | |
| *** banix has joined #openstack-infra | 18:11 | |
| *** koolhead17 has quit IRC | 18:11 | |
| *** dmsimard_away is now known as dmsimard | 18:14 | |
| *** banix has quit IRC | 18:19 | |
| *** bknudson has joined #openstack-infra | 18:19 | |
| *** wenlock has quit IRC | 18:21 | |
| *** yfried|afk has joined #openstack-infra | 18:23 | |
| *** garyh has joined #openstack-infra | 18:25 | |
| anteaya | so basically what we lost was any logs that were in the process of uploading when the blip occurred plus any subsequent runs | 18:29 |
| anteaya | did I get that right? | 18:29 |
| *** groknix has quit IRC | 18:33 | |
| *** banix has joined #openstack-infra | 18:33 | |
| *** groknix has joined #openstack-infra | 18:33 | |
| *** garyh has quit IRC | 18:35 | |
| *** luqas has quit IRC | 18:36 | |
| *** miqui_ has quit IRC | 18:40 | |
| *** AlexF has joined #openstack-infra | 18:40 | |
| *** miqui_ has joined #openstack-infra | 18:41 | |
| *** dmsimard is now known as dmsimard_away | 18:43 | |
| *** andreykurilin_ has joined #openstack-infra | 18:44 | |
| *** yfried|afk has quit IRC | 18:45 | |
| *** yfried|afk has joined #openstack-infra | 18:45 | |
| *** shashankhegde has joined #openstack-infra | 18:51 | |
| mordred | anteaya: and possibly some previously uploaded logs | 18:52 |
| *** zz_dimtruck is now known as dimtruck | 19:02 | |
| anteaya | ah | 19:03 |
| anteaya | which is unknown | 19:03 |
| *** armax has joined #openstack-infra | 19:04 | |
| *** harlowja_at_home has joined #openstack-infra | 19:04 | |
| *** dmsimard_away is now known as dmsimard | 19:05 | |
| *** bhuvan has joined #openstack-infra | 19:05 | |
| *** garyh has joined #openstack-infra | 19:06 | |
| *** HeOS has joined #openstack-infra | 19:10 | |
| *** ryanpetrello has joined #openstack-infra | 19:14 | |
| *** patrickeast has joined #openstack-infra | 19:15 | |
| *** garyh has quit IRC | 19:16 | |
| *** dmsimard is now known as dmsimard_away | 19:18 | |
| *** dmsimard_away is now known as dmsimard | 19:19 | |
| *** ryanpetrello has quit IRC | 19:21 | |
| *** marun has quit IRC | 19:22 | |
| *** pc_m has quit IRC | 19:23 | |
| *** talluri has joined #openstack-infra | 19:24 | |
| *** ryanpetrello has joined #openstack-infra | 19:24 | |
| *** armax has quit IRC | 19:27 | |
| *** yamamoto has joined #openstack-infra | 19:27 | |
| *** banix has quit IRC | 19:29 | |
| *** yamamoto has quit IRC | 19:32 | |
| *** shashankhegde has quit IRC | 19:32 | |
| *** otter768 has joined #openstack-infra | 19:33 | |
| *** otter768 has quit IRC | 19:38 | |
| tchaypo | evening | 19:38 |
| *** ryanpetrello has quit IRC | 19:43 | |
| harlowja_at_home | mtreinish, is https://bugs.launchpad.net/tempest/+bug/1395368 a known bug in icehouse; seems to be happening quite often | 19:44 |
| uvirtbot | Launchpad bug 1395368 in tempest "ExternalNetworksTestJSON.test_delete_external_networks_with_floating_ip (icehouse) sporadic failures" [Undecided,New] | 19:44 |
| harlowja_at_home | *if u know | 19:44 |
| *** weshay has joined #openstack-infra | 19:45 | |
| *** AlexF has quit IRC | 19:46 | |
| *** yamahata has quit IRC | 19:54 | |
| *** armax has joined #openstack-infra | 19:57 | |
| *** dmsimard is now known as dmsimard_away | 19:59 | |
| *** emagana_ has quit IRC | 20:01 | |
| *** garyh has joined #openstack-infra | 20:07 | |
| *** davideagnello has joined #openstack-infra | 20:08 | |
| *** talluri has quit IRC | 20:10 | |
| *** harlowja_at_home has quit IRC | 20:10 | |
| *** andreykurilin_ has quit IRC | 20:12 | |
| *** andreykurilin_ has joined #openstack-infra | 20:13 | |
| *** garyh has quit IRC | 20:16 | |
| *** patrickeast has quit IRC | 20:19 | |
| *** banix has joined #openstack-infra | 20:28 | |
| *** armax has quit IRC | 20:33 | |
| *** davideagnello has quit IRC | 20:33 | |
| *** rushiagr is now known as rushiagr_away | 20:37 | |
| *** ryanpetrello has joined #openstack-infra | 20:41 | |
| *** emagana has joined #openstack-infra | 20:46 | |
| *** mrmartin has quit IRC | 20:51 | |
| *** patrickeast has joined #openstack-infra | 20:54 | |
| *** emagana has quit IRC | 20:57 | |
| *** Hal_ has joined #openstack-infra | 20:58 | |
| *** emagana has joined #openstack-infra | 20:58 | |
| *** emagana has quit IRC | 21:03 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/nodepool: Add alembic migration support https://review.openstack.org/136589 | 21:06 |
| mordred | clarkb: not that it would have likely helped anything you were doing, but ^^ | 21:06 |
| *** andreykurilin_ has quit IRC | 21:06 | |
| *** garyh has joined #openstack-infra | 21:06 | |
| *** andreykurilin_ has joined #openstack-infra | 21:07 | |
| *** hashar has joined #openstack-infra | 21:08 | |
| *** Hal_ has quit IRC | 21:11 | |
| *** Hal_ has joined #openstack-infra | 21:11 | |
| *** andreykurilin_ has quit IRC | 21:12 | |
| *** andreykurilin_ has joined #openstack-infra | 21:12 | |
| *** SumitNaiksatam has quit IRC | 21:12 | |
| *** pc_m has joined #openstack-infra | 21:13 | |
| *** harlowja_at_home has joined #openstack-infra | 21:14 | |
| *** Hal_ has quit IRC | 21:14 | |
| *** armax has joined #openstack-infra | 21:16 | |
| *** emagana has joined #openstack-infra | 21:17 | |
| *** garyh has quit IRC | 21:17 | |
| *** SumitNaiksatam has joined #openstack-infra | 21:20 | |
| *** harlowja_at_home has quit IRC | 21:28 | |
| *** emagana has quit IRC | 21:28 | |
| *** emagana has joined #openstack-infra | 21:29 | |
| *** hashar has quit IRC | 21:33 | |
| *** otter768 has joined #openstack-infra | 21:34 | |
| *** emagana has quit IRC | 21:34 | |
| *** otter768 has quit IRC | 21:38 | |
| *** dmsimard_away is now known as dmsimard | 21:42 | |
| *** patrickeast has quit IRC | 21:42 | |
| *** matrohon has joined #openstack-infra | 21:43 | |
| openstackgerrit | Spencer Krum proposed openstack-infra/puppet-storyboard: Add metadata.json to puppet-storyboard https://review.openstack.org/136591 | 21:43 |
| *** ryanpetrello has quit IRC | 21:50 | |
| *** ryanpetrello has joined #openstack-infra | 21:50 | |
| openstackgerrit | Spencer Krum proposed openstack-infra/system-config: Experimental: Livegrep implementation https://review.openstack.org/129726 | 21:54 |
| *** emagana has joined #openstack-infra | 21:56 | |
| mordred | nibalizer: it seems you might be online and poking at things | 21:56 |
| mordred | :) | 21:56 |
| mordred | nibalizer: also, you know you added a file called "ls" to that patch, right? | 21:58 |
| *** garyh has joined #openstack-infra | 22:00 | |
| *** dimsum__ has joined #openstack-infra | 22:01 | |
| nibalizer | mordred: erp lol | 22:02 |
| nibalizer | does someone have to sign cla to submit to infra? yes right? | 22:02 |
| mordred | nibalizer: yah | 22:03 |
| *** Ark has joined #openstack-infra | 22:04 | |
| *** Ark is now known as Guest16530 | 22:04 | |
| *** bhuvan has quit IRC | 22:05 | |
| *** hdd has joined #openstack-infra | 22:06 | |
| *** otherwiseguy has joined #openstack-infra | 22:06 | |
| *** armax has quit IRC | 22:07 | |
| openstackgerrit | Spencer Krum proposed openstack-infra/puppet-httpd: Adding readme https://review.openstack.org/135376 | 22:07 |
| *** kevinbenton has quit IRC | 22:08 | |
| *** ryanpetrello has quit IRC | 22:09 | |
| *** bhuvan has joined #openstack-infra | 22:09 | |
| openstackgerrit | Spencer Krum proposed openstack-infra/system-config: Experimental: Livegrep implementation https://review.openstack.org/129726 | 22:19 |
| fungi | nibalizer: for at least some of our infra projects anyway. there are a few (mostly early) projects which do not have cla enforcement set | 22:20 |
| nibalizer | mordred: so yes i am aronud, if you have questions or whatever | 22:20 |
| nibalizer | fungi: okay, im asking in the context of the 'contributing' section of the puppet-httpd README so that will be cla enfrced i think | 22:21 |
| *** davideagnello has joined #openstack-infra | 22:22 | |
| *** otherwiseguy has quit IRC | 22:23 | |
| fungi | nibalizer: then yes | 22:23 |
| *** emagana has quit IRC | 22:25 | |
| *** emagana has joined #openstack-infra | 22:25 | |
| *** davideagnello has quit IRC | 22:26 | |
| *** dimsum__ has quit IRC | 22:26 | |
| openstackgerrit | Spencer Krum proposed openstack-infra/project-config: Use storyboard for puppet-httpd https://review.openstack.org/136594 | 22:27 |
| *** yamamoto has joined #openstack-infra | 22:30 | |
| *** dimsum__ has joined #openstack-infra | 22:30 | |
| *** emagana has quit IRC | 22:30 | |
| *** yamamoto has quit IRC | 22:34 | |
| *** koolhead17 has joined #openstack-infra | 22:38 | |
| openstackgerrit | Thanh Ha proposed openstack-infra/jenkins-job-builder: Add support for Config File Provider to Maven Project module https://review.openstack.org/136225 | 22:38 |
| *** garyh has quit IRC | 22:41 | |
| *** stevemar has joined #openstack-infra | 22:43 | |
| *** hdd has quit IRC | 22:46 | |
| *** andreykurilin_ has quit IRC | 22:49 | |
| *** banix has quit IRC | 22:51 | |
| *** emagana has joined #openstack-infra | 22:53 | |
| *** ivar-lazzaro has joined #openstack-infra | 22:53 | |
| *** pc_m has quit IRC | 22:54 | |
| *** dimsum__ has quit IRC | 22:54 | |
| *** emagana_ has joined #openstack-infra | 22:56 | |
| *** emagana has quit IRC | 22:58 | |
| *** e0ne has joined #openstack-infra | 22:58 | |
| *** emagana_ has quit IRC | 23:00 | |
| openstackgerrit | Joe Gordon proposed openstack-infra/devstack-gate: Set up ssh_known_host based in hostname https://review.openstack.org/136596 | 23:03 |
| *** Guest16530 has quit IRC | 23:07 | |
| *** banix has joined #openstack-infra | 23:07 | |
| *** ivar-lazzaro has quit IRC | 23:08 | |
| mordred | fungi: I'm hacking up a few dib elements to see if we can make our own base cloud images ... and instead of using cloud images am stgarting from debootstrap | 23:09 |
| mordred | fungi: it's FASCINATING to see what things ubuntu has stuck in their 'minimal' thing you get from running debootstrap | 23:09 |
| mordred | libjson-c2, for instance | 23:09 |
| *** mase_x200 has joined #openstack-infra | 23:10 | |
| clarkb | mordred as long as I get lsof and manpages I am happy | 23:11 |
| * clarkb glares at centos | 23:11 | |
| mordred | :) | 23:11 |
| jogo | its nice to have zuul back | 23:11 |
| mordred | clarkb: well, my goal is base images for ubuntu, fedora and centos that are each sane | 23:11 |
| mordred | clarkb: so I'll be sure to keep lsof and manpages on my list :) | 23:12 |
| *** stevemar has quit IRC | 23:12 | |
| clarkb | on monday we need to fix fedora21 image (is that evwn released yet ifnot can we just rm it?) | 23:12 |
| clarkb | ianw I am not sure why we want to be building those images imo | 23:13 |
| clarkb | we have centos7 now too | 23:13 |
| clarkb | which should cover that stuff? | 23:13 |
| mordred | yah. I vote for centos7 | 23:13 |
| clarkb | it builds now on trusty nodepool too :) | 23:13 |
| *** hogepodge has quit IRC | 23:14 | |
| clarkb | the other monday thing I want to do is work on my nodepool patches to support multi image types | 23:14 |
| clarkb | and hack on the 0mq stuff | 23:15 |
| clarkb | but for now back to football. boston college is playing fsu well | 23:15 |
| mordred | clarkb: they ARE indeed | 23:16 |
| jogo | clarkb: FYI https://review.openstack.org/#/c/136596/1 | 23:16 |
| mordred | although I have to admit I'm rooting for FSU - mainly because I don't want the first playoff to have two SEC teams in it | 23:16 |
| jogo | clarkb: I didn't want to touch https://review.openstack.org/#/c/134670/13 since I didn't want to rebase your patch https://review.openstack.org/#/c/131604/13 | 23:16 |
| clarkb | mordred I kinda do because then oregon will play one of them | 23:17 |
| *** mase_x200 has quit IRC | 23:17 | |
| mordred | clarkb: fair | 23:17 |
| clarkb | jogo uhm | 23:19 |
| *** hogepodge has joined #openstack-infra | 23:19 | |
| clarkb | jogo pretty sure you still use IPs in that change | 23:19 |
| clarkb | oh wait | 23:20 |
| *** dmsimard is now known as dmsimard_away | 23:20 | |
| clarkb | jogo thats terrible :P | 23:20 |
| jogo | clarkb: ? | 23:20 |
| clarkb | you already have a list of hosts use that dont do double ssh | 23:20 |
| clarkb | remote command for hostname just ro get hostname to keyscan... | 23:20 |
| jogo | clarkb: heh I do yeah | 23:20 |
| jogo | doh | 23:20 |
| *** e0ne has quit IRC | 23:20 | |
| clarkb | use cut on tmp_hosts | 23:21 |
| jogo | right now just testing if it works or not | 23:21 |
| jogo | if it works will ammend | 23:21 |
| clarkb | (or sed or awk or whatever) | 23:21 |
| clarkb | kk | 23:21 |
| mordred | clarkb: oh - f21 does seem to be out ... | 23:22 |
| clarkb | oh cool. I wonder if it fixes firewalld | 23:23 |
| mordred | and, at least there is a f21 dir on the mirrors that does not say alpha or beta | 23:25 |
| jogo | https://jenkins07.openstack.org/job/gate-nova-docs/8190/console | 23:25 |
| jogo | The remote end hung up upon initial contact | 23:26 |
| clarkb | its that thing rerun | 23:26 |
| clarkb | I think nodepool is deleting subnodes | 23:26 |
| jogo | clarkb: that is a docs job | 23:26 |
| clarkb | oh thats git.o.o | 23:27 |
| clarkb | no clue then | 23:27 |
| jogo | fatal: protocol error: bad pack header | 23:27 |
| *** groknix has quit IRC | 23:28 | |
| *** groknix has joined #openstack-infra | 23:29 | |
| clarkb | the pack should be made on the fly under git:// iirc | 23:30 |
| *** groknix has quit IRC | 23:30 | |
| clarkb | its also how smart http works | 23:30 |
| *** stevemar has joined #openstack-infra | 23:32 | |
| clarkb | mordred dont you wish you were cool enough to be called "Jimbo"? | 23:34 |
| *** otter768 has joined #openstack-infra | 23:35 | |
| *** otter768 has quit IRC | 23:39 | |
| jogo | clarkb: looks like resize uses IP addresses | 23:41 |
| jogo | so I need both | 23:41 |
| *** garyh has joined #openstack-infra | 23:41 | |
| clarkb | yes | 23:41 |
| mordred | clarkb: I do | 23:44 |
| mordred | clarkb: I am not | 23:44 |
| openstackgerrit | Joe Gordon proposed openstack-infra/devstack-gate: Set up ssh_known_host based in hostname https://review.openstack.org/136596 | 23:45 |
| openstackgerrit | Joe Gordon proposed openstack-infra/devstack-gate: Set up ssh_known_host based on hostname https://review.openstack.org/136596 | 23:46 |
| *** emagana has joined #openstack-infra | 23:50 | |
| *** garyh has quit IRC | 23:51 | |
| *** stevemar has quit IRC | 23:52 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/system-config: Add a debootstrap-based Ubuntu element https://review.openstack.org/136597 | 23:52 |
| mordred | clarkb: enjoy ^^ | 23:52 |
| *** dimtruck is now known as zz_dimtruck | 23:53 | |
| *** alexpilotti has quit IRC | 23:53 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/system-config: Add a debootstrap-based Ubuntu element https://review.openstack.org/136597 | 23:54 |
| openstackgerrit | Monty Taylor proposed openstack-infra/system-config: Add debootstrap to nodepool https://review.openstack.org/136598 | 23:54 |
| *** emagana has quit IRC | 23:55 | |
| *** dimsum__ has joined #openstack-infra | 23:55 | |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!