| *** yamahata has quit IRC | 00:02 | |
| *** mwagner_lap has joined #openstack-infra | 00:02 | |
| *** Ryan_Lane1 has quit IRC | 00:02 | |
| *** khyati has quit IRC | 00:03 | |
| *** ArxCruz has quit IRC | 00:03 | |
| *** e0ne has joined #openstack-infra | 00:08 | |
| fungi | hah | 00:09 |
|---|---|---|
| jeblair | ttx, gothicmindfood, mordred, fungi, clarkb: sdague has a proposal on the -dev list to use gerrit for blueprint review and approval | 00:11 |
| *** flaper87 is now known as flaper87|afk | 00:11 | |
| openstackgerrit | Clark Boylan proposed a change to openstack-dev/pbr: Init sphinx config values before accessing them. https://review.openstack.org/78821 | 00:11 |
| jeblair | ttx, gothicmindfood, mordred, fungi, clarkb: s /blueprint/nova blueprint/ | 00:11 |
| clarkb | jeblair: ya I had just caught up to that | 00:11 |
| *** denis_makogon has quit IRC | 00:12 | |
| clarkb | I like the idea. At the very least it seems like it is worth a shot | 00:12 |
| jeblair | yep | 00:12 |
| *** e0ne has quit IRC | 00:12 | |
| clarkb | we could use different branches for each cycle so that things are nicely organized | 00:13 |
| clarkb | or do a dir structure of some sort I suppose | 00:13 |
| jeblair | clarkb: well, since the proposal still had blueprints in lp for release tracking purposes, i don't think you'd have to worry too much about branches or cycles | 00:13 |
| jeblair | clarkb: i think the idea is just to do text iteration in gerrit, but still progress tracking in lp | 00:14 |
| fungi | i think the suggestion has come up at least once before, but the move for the tc to start using gerrit to review policy has set a good precedent since | 00:15 |
| mordred | jeblair: any thoughts on how that interfaces with storyboard in the future? | 00:18 |
| clarkb | mordred: when you get a chance please see recent discussion in -dev | 00:18 |
| jeblair | mordred: not immediately, though i think between editing the story description and then leaving comments, you might get a decent workflow for iteration of the blueprint. | 00:19 |
| * mordred oes to look | 00:20 | |
| fungi | rework storyboard to keep all text in git repositories | 00:21 |
| fungi | problem solved! | 00:22 |
| openstackgerrit | A change was merged to openstack-infra/config: Add GerritBot to openstack-keystone https://review.openstack.org/78071 | 00:22 |
| openstackgerrit | A change was merged to openstack-infra/config: Configure check and gate to stackforge/warm https://review.openstack.org/78231 | 00:23 |
| openstackgerrit | James E. Blair proposed a change to openstack-infra/statusbot: Don't crash on invalid UTF8 https://review.openstack.org/78814 | 00:23 |
| jeblair | sorry everyone needs to review that again. pep8 is INSANE. | 00:23 |
| fungi | yah | 00:24 |
| *** dcramer_ has quit IRC | 00:25 | |
| *** sweston has quit IRC | 00:27 | |
| *** CaptTofu has joined #openstack-infra | 00:27 | |
| openstackgerrit | James E. Blair proposed a change to openstack-infra/config: Add a script to manage IRC perms https://review.openstack.org/78483 | 00:29 |
| *** david-lyle has quit IRC | 00:30 | |
| *** rwsu has quit IRC | 00:31 | |
| *** markwash has quit IRC | 00:31 | |
| *** reed has quit IRC | 00:32 | |
| *** e0ne has joined #openstack-infra | 00:33 | |
| *** thomasem has joined #openstack-infra | 00:33 | |
| openstackgerrit | A change was merged to openstack-infra/config: Add pypy to satori checks (it's already in gates) https://review.openstack.org/78296 | 00:34 |
| *** jswarren has quit IRC | 00:34 | |
| openstackgerrit | A change was merged to openstack-infra/statusbot: Don't crash on invalid UTF8 https://review.openstack.org/78814 | 00:34 |
| *** andreaf2 has quit IRC | 00:34 | |
| *** e0ne has quit IRC | 00:34 | |
| *** openstackgerrit has quit IRC | 00:35 | |
| *** e0ne has joined #openstack-infra | 00:35 | |
| *** openstackgerrit has joined #openstack-infra | 00:35 | |
| fungi | aww, i was in the middle of reviewing the previous patchset of 78483. hopefully the various comments i add are still applicable | 00:35 |
| *** jnoller has joined #openstack-infra | 00:35 | |
| clarkb | fungi: I may have skipped it beause of the iterating :) | 00:36 |
| openstackgerrit | A change was merged to openstack-infra/config: Add statusbot to all known channels https://review.openstack.org/78807 | 00:37 |
| *** bknudson has quit IRC | 00:37 | |
| openstackgerrit | A change was merged to openstack-infra/config: Revert "Temporarily no-vote the requirements check for openstacksdk" https://review.openstack.org/78337 | 00:38 |
| fungi | clarkb: fair point, i should have allowed it to settle longer ;) | 00:38 |
| *** amcrn has quit IRC | 00:38 | |
| *** e0ne has quit IRC | 00:39 | |
| clarkb | sdague: if you haven't disappeared yet I will have comments for the integrated gate template chnage | 00:41 |
| clarkb | sdague: and commented | 00:42 |
| sdague | ok | 00:43 |
| sdague | yeh, I'm actually just pulling a long one today (then hiding after lunch tomorrow | 00:43 |
| * fungi hasn't learned that trick yet | 00:44 | |
| clarkb | fungi: fridays are great for that trick | 00:44 |
| clarkb | fungi: there is at least one new project change outthere that seems to be ready. | 00:44 |
| clarkb | fungi: so we should probably plan to do that and test the remaining jeepyb changes | 00:44 |
| fungi | clarkb: i'm going to have to learn it soon. just found out that i'm moving to the beach in june | 00:44 |
| sdague | clarkb: ok, so all the adds are intentional | 00:44 |
| clarkb | fungi: nice | 00:45 |
| clarkb | sdague: really? a lotof those are noops | 00:45 |
| clarkb | we run the tests but don't actually test the newly tested projects | 00:45 |
| fungi | clarkb: well, landlord decided to terminate the lease and move back in, but at least they gave me a few months notice. wasn't planning to move just yet | 00:45 |
| clarkb | fungi: silly landlords | 00:46 |
| fungi | s/land/over/ | 00:46 |
| openstackgerrit | A change was merged to openstack-infra/elastic-recheck: Delete resolved bugs that have no hits in ES https://review.openstack.org/78452 | 00:46 |
| sdague | clarkb: everything needs to actually be upgrade testing, a bunch of it isn't | 00:47 |
| clarkb | sdague: gotcha, so this is desired state rather than reflected state? | 00:47 |
| sdague | yes | 00:47 |
| clarkb | ok I can go with that | 00:47 |
| sdague | this is driving the commonality that we need | 00:47 |
| sdague | which is why it started with the small job list | 00:47 |
| clarkb | sdague: is this something you want to announce before I merge? | 00:48 |
| clarkb | jeblair: fungi SergeyLukjanov mordred and do you think that is something we should merge not now? | 00:48 |
| openstackgerrit | Joe Gordon proposed a change to openstack-infra/elastic-recheck: Add link to status.o.o/e-r to gerrit comment https://review.openstack.org/78827 | 00:49 |
| fungi | i'm not really expecting to be around later this evening to help fix if it causes issues, but i also wouldn't expect it to really | 00:50 |
| sdague | clarkb: I'm honestly not convinced that it's something we have to announce. your call. my grasp of english is a little light after the long day | 00:50 |
| sdague | the important thing to me is actually the heat change on top of it | 00:50 |
| clarkb | fungi: yeah but it also part of freeze weekstill | 00:50 |
| clarkb | sdague: maybe we cram that in early tomorrow morning? | 00:50 |
| sdague | I mostly did this so the heat change was 6 lines instead of 40 | 00:50 |
| *** gokrokve_ has quit IRC | 00:50 | |
| clarkb | when brains are fresh and we have all day watch the screaming | 00:51 |
| *** unicell has quit IRC | 00:51 | |
| sdague | well if you want me to see the screaming, yuo should do it tonight. Cause I'm cutting at noon :) | 00:51 |
| *** mrodden has joined #openstack-infra | 00:51 | |
| clarkb | oh right noon your time is when I am still sleeping | 00:51 |
| clarkb | :P | 00:51 |
| clarkb | jeblair: have an opinion? otherwise I will approve | 00:51 |
| sdague | clarkb: you just need to wake up earlier | 00:52 |
| *** jcoufal has joined #openstack-infra | 00:52 | |
| clarkb | sdague: gah | 00:52 |
| sdague | earling birds and worms or something | 00:52 |
| clarkb | it is actually getting easier now that the sun is up earlier | 00:52 |
| clarkb | but during the winter I need to hibernate more | 00:52 |
| sdague | completely unrelated, we had a coyote go through the yard at 5pm today. That was new :) | 00:52 |
| *** lcheng has quit IRC | 00:53 | |
| * clarkb takes the plunge | 00:54 | |
| jeblair | fungi, clarkb: i only updated https://review.openstack.org/78483 to add the extra space pep8 wanted | 00:54 |
| clarkb | sdague: approved | 00:54 |
| sdague | clarkb: thanks | 00:54 |
| clarkb | gah right as jeblair returns | 00:54 |
| *** jnoller has quit IRC | 00:54 | |
| jeblair | fungi, clarkb: i would have warned you if it were something more substantial | 00:54 |
| clarkb | oh well he +2'd it | 00:54 |
| clarkb | jeblair: thanks | 00:54 |
| clarkb | sdague: https://review.openstack.org/#/c/78687/ is the one you want? | 00:55 |
| jeblair | fungi: congratulations on the beach! which beach? | 00:55 |
| clarkb | oh you want slow heat | 00:55 |
| sdague | clarkb: yeh, it was on top of mine as well | 00:55 |
| sdague | the large ops one is good as well | 00:55 |
| clarkb | but stevebaker has aksed for other changes to merge first | 00:55 |
| sdague | it will give us back nodes that we aren't using | 00:55 |
| sdague | yeh I have a -1 on that until those merge | 00:55 |
| clarkb | I think I will push largops through and we can work with stevebaker | 00:55 |
| sdague | yep | 00:55 |
| stevebaker | \o | 00:56 |
| sdague | both the tempest patches of his I'm pushing through | 00:56 |
| stevebaker | https://review.openstack.org/#/c/78756/ needs an approve | 00:56 |
| fungi | jeblair: we're not sure yet. looking at rentals on manteo or between nags head and southern shores for a year while we decide where we want to buy | 00:56 |
| sdague | one is in the gate | 00:56 |
| sdague | stevebaker: yeh, let me hit that one | 00:57 |
| clarkb | jogo: you are removing largeops from neutron and nova client but both jobs use nova and one uses neutron | 00:57 |
| *** thomasem has quit IRC | 00:57 | |
| sdague | clean check makes me happy :) | 00:57 |
| clarkb | jogo: so ? | 00:57 |
| sdague | so I can approve early | 00:57 |
| *** hashar has joined #openstack-infra | 00:57 | |
| fungi | jeblair: currently leaning towards kitty hawk/kill devil hills on the sound-facing side of the island | 00:57 |
| stevebaker | sdague: is it safe to approve something that check hasn't finished on these days? | 00:57 |
| *** dansmith has quit IRC | 00:57 | |
| sdague | yes | 00:57 |
| openstackgerrit | A change was merged to openstack-infra/config: create an integrated-gate template https://review.openstack.org/78612 | 00:57 |
| jogo | clarkb: neutron should still have the neutron copy in there | 00:58 |
| sdague | it won't move to gate until it has success in check | 00:58 |
| *** dansmith has joined #openstack-infra | 00:58 | |
| stevebaker | i thought so | 00:58 |
| clarkb | jogo: oh it does. novaclient has both removed though | 00:58 |
| jogo | and not sure how a chane to novaclient can break things | 00:58 |
| jogo | clarkb: I wasn't very sure about that one | 00:58 |
| clarkb | jogo: so I guess my question is why novaclient | 00:58 |
| jogo | clarkb: I don't have an answer, and I am fine eitherway | 00:58 |
| jogo | which means probablly leave it | 00:58 |
| clarkb | jogo: is that because tempest doesn't actually use novaclient? | 00:58 |
| jogo | (put back) | 00:58 |
| jogo | clarkb: oh wait derp | 00:58 |
| jogo | it uses it | 00:59 |
| clarkb | then we definitely need it back | 00:59 |
| sdague | jogo: so can you actually document the failure modes that this detects that aren't found with the other jobs | 00:59 |
| sdague | I think that's a thing we are missing | 00:59 |
| sdague | so when people ask why this job is running, we know | 00:59 |
| jogo | sdague: do that in commit msg or somewhere else? | 01:00 |
| jogo | and I agree with the idea | 01:00 |
| sdague | I think we need it elsewhere | 01:00 |
| sdague | so that we can revisit | 01:00 |
| sdague | exactly where, I'm not sure | 01:00 |
| jeblair | in the layout.yaml file? | 01:00 |
| sdague | jeblair: possibly, if you think there is a place you want large chunks of text | 01:01 |
| sdague | actually, probably in devstack-gate.yaml | 01:01 |
| sdague | where we actually define the jobs | 01:01 |
| jeblair | sdague: that sounds nice too | 01:01 |
| jeblair | (i don't mean both, i mean either) | 01:01 |
| *** hogepodge has quit IRC | 01:01 | |
| sdague | yeh, I think closer to the job definition is better | 01:01 |
| sdague | so I'd lean devstack-gate.yaml | 01:01 |
| *** wenlock has quit IRC | 01:02 | |
| jeblair | wfm | 01:02 |
| *** bknudson has joined #openstack-infra | 01:03 | |
| *** zns has quit IRC | 01:03 | |
| *** rlandy has quit IRC | 01:03 | |
| *** jnoller has joined #openstack-infra | 01:03 | |
| clarkb | jeblair: I am reviewing the irc management changes now. You are comfortable with merging those? | 01:04 |
| *** atiwari has quit IRC | 01:04 | |
| *** bknudson1 has joined #openstack-infra | 01:04 | |
| *** bknudson1 has quit IRC | 01:05 | |
| openstackgerrit | Joe Gordon proposed a change to openstack-infra/config: Don't run large-ops test on repos that it doesn't touch https://review.openstack.org/78687 | 01:05 |
| *** rpodolyaka has quit IRC | 01:07 | |
| *** bknudson has quit IRC | 01:07 | |
| * fungi is trying to scrub some of the various requests which have showed up in bugs or on the infra ml since the weekend, and will probably disappear for the evening once that's caught up | 01:08 | |
| *** xchu has joined #openstack-infra | 01:08 | |
| openstackgerrit | Victor Seva proposed a change to openstack-infra/jenkins-job-builder: use {obj:key} as a way to pass an object and not only strings to templates https://review.openstack.org/73990 | 01:10 |
| jogo | so not sure what best way to document large-ops is | 01:10 |
| jogo | I can see two parts | 01:10 |
| jogo | why its running makes most sense per repo | 01:10 |
| *** bknudson has joined #openstack-infra | 01:11 | |
| *** vkozhukalov has quit IRC | 01:12 | |
| *** Ryan_Lane has quit IRC | 01:13 | |
| *** wchrisj__ has quit IRC | 01:13 | |
| *** blamar has quit IRC | 01:14 | |
| *** markmcclain has quit IRC | 01:14 | |
| sdague | jogo: I think 3 paragraphs. 1 - what it is, 2 - what we expect it to find which is unique compared to other test sets, 3 - what kinds of projects it should run on | 01:14 |
| sdague | kind of like the way we did the tempest field guide - http://docs.openstack.org/developer/tempest/field_guide/api.html | 01:15 |
| sdague | because this is about documenting what we know, so when we make decisions in the future about what jobs are useful, and what aren't, and if this kind of job should run on a new project, that context is useful | 01:16 |
| jogo | sdague: I like the idea but sounds like something for tomorrow | 01:16 |
| sdague | sure | 01:16 |
| fungi | https://launchpad.net/bugs/1287668 brings back up a question i had earlier about our third-party testing account requirements... is there anything special we need to restrict or any particular exceptions we're willing to make when it comes to stackforge projects running an external ci to test and report on changes for only their own code? | 01:18 |
| *** yamahata has joined #openstack-infra | 01:18 | |
| fungi | so much of the policy discussion centered around driver/backend developers voting on changes in openstack projects | 01:19 |
| fungi | on the one hand, i can see the concern that any ci system could go haywire and start voting on projects it didn't intend, but on the other hand do we care enough to go putting in separate acls to restrict access specifically for them on a case-by-case basis? | 01:20 |
| clarkb | fungi: I am leaning towards no | 01:21 |
| *** jnoller has quit IRC | 01:22 | |
| *** sabari has quit IRC | 01:22 | |
| openstackgerrit | Lakshmi N Sampath proposed a change to openstack-infra/config: Add new project graffiti https://review.openstack.org/78829 | 01:22 |
| *** rfolco has quit IRC | 01:23 | |
| fungi | do we still want them to provide a separate published contact address or support the various other things we insist upon for systems voting on openstack projects? do they need to start in the non-voting group and wait for the ptl of their project (if they even have one) to let us know which it should be able to start voting? | 01:23 |
| fungi | er, when it should be able | 01:24 |
| clarkb | fungi: I think the wait in non voting group is a good thing regardless of project because it gets CI and devs all on the same page | 01:24 |
| clarkb | fungi: and separate contact addresses are probably useful regardless of project as well | 01:24 |
| *** gokrokve has joined #openstack-infra | 01:25 | |
| *** Alexandra is now known as alex-luncheon | 01:25 | |
| fungi | that's where i was leaning on those too. cool | 01:26 |
| *** hashar has quit IRC | 01:27 | |
| *** wchrisj has joined #openstack-infra | 01:29 | |
| *** gokrokve has quit IRC | 01:31 | |
| *** unicell has joined #openstack-infra | 01:33 | |
| *** e0ne has joined #openstack-infra | 01:35 | |
| *** gokrokve has joined #openstack-infra | 01:36 | |
| fungi | clarkb: is https://launchpad.net/bugs/1288035 a duplicate of one you added earlier in the week? gerrit's claiming it already exists | 01:36 |
| *** dprince has quit IRC | 01:36 | |
| clarkb | I swear I added it | 01:37 |
| * clarkb looks at the groups | 01:37 | |
| clarkb | it is | 01:37 |
| clarkb | fungi: its there | 01:37 |
| clarkb | IBM ZMV CI | 01:37 |
| clarkb | *ZVM | 01:37 |
| clarkb | fungi: https://review.openstack.org/#/admin/groups/270,members | 01:38 |
| fungi | yep! looks right in the db too. i'll find the original request and close that bug out then. thanks! | 01:38 |
| clarkb | I should've responded to the list with info too | 01:38 |
| clarkb | gah running out of time to finish irc stuff review | 01:39 |
| clarkb | I will leave the comment I have so far but I need food then user group thing | 01:39 |
| *** thuc has quit IRC | 01:39 | |
| *** e0ne has quit IRC | 01:39 | |
| fungi | have fun! | 01:39 |
| *** thuc has joined #openstack-infra | 01:39 | |
| openstackgerrit | Lakshmi N Sampath proposed a change to openstack-infra/config: Add new project graffiti https://review.openstack.org/78829 | 01:42 |
| *** wchrisj has quit IRC | 01:42 | |
| *** Ryan_Lane has joined #openstack-infra | 01:43 | |
| *** derekh has quit IRC | 01:43 | |
| *** thuc has quit IRC | 01:43 | |
| *** Ryan_Lane has quit IRC | 01:47 | |
| *** Ryan_Lane has joined #openstack-infra | 01:48 | |
| *** wchrisj has joined #openstack-infra | 01:48 | |
| *** hogepodge has joined #openstack-infra | 01:49 | |
| fungi | wow, flashbacks of another ibm.cn service account request. they asked for this one twice on the same day in the infra ml a couple weeks ago, and then when you added their account and followed up, they just blindly replied to your response with another copy of the request | 01:50 |
| fungi | and also opened a bug with the same content | 01:53 |
| clarkb | :( | 01:54 |
| *** dzimine has joined #openstack-infra | 01:55 | |
| *** harlowja has quit IRC | 01:56 | |
| sdague | fungi: did they also have out of office messages sent to the same bug? | 01:57 |
| *** prad_ has quit IRC | 01:58 | |
| *** Ryan_Lane1 has joined #openstack-infra | 01:59 | |
| *** dzimine has left #openstack-infra | 01:59 | |
| *** nosnos has joined #openstack-infra | 02:00 | |
| fungi | sdague: i guess we'll find out soon | 02:00 |
| *** prad has joined #openstack-infra | 02:01 | |
| sdague | ok... this has been a very long day. So I'm going to go crash | 02:02 |
| *** jcoufal has quit IRC | 02:03 | |
| sdague | clarkb: if the tempest reviews in - https://review.openstack.org/#/c/78698/ make it to the gate tonight, please push go on the heat add | 02:03 |
| *** antonym has quit IRC | 02:03 | |
| *** dkehn has quit IRC | 02:03 | |
| *** dkehn has joined #openstack-infra | 02:03 | |
| *** Ryan_Lane1 has quit IRC | 02:03 | |
| sdague | possibly thats 41 minutes away | 02:04 |
| sdague | based on zuul estimates | 02:04 |
| *** banix has joined #openstack-infra | 02:04 | |
| *** dhellmann_ has joined #openstack-infra | 02:05 | |
| *** hogepodge has quit IRC | 02:05 | |
| *** changbl has joined #openstack-infra | 02:05 | |
| *** krotscheck has quit IRC | 02:06 | |
| *** dachary has quit IRC | 02:06 | |
| *** bogdando has quit IRC | 02:06 | |
| *** SpamapS has quit IRC | 02:06 | |
| *** dhellmann has quit IRC | 02:06 | |
| *** weshay has quit IRC | 02:06 | |
| *** flaper87|afk has quit IRC | 02:07 | |
| *** kiall has quit IRC | 02:07 | |
| *** shortstop has quit IRC | 02:07 | |
| *** SpamapS has joined #openstack-infra | 02:07 | |
| *** dhellmann_ is now known as dhellmann | 02:07 | |
| *** bogdando has joined #openstack-infra | 02:07 | |
| *** SpamapS has quit IRC | 02:07 | |
| *** SpamapS has joined #openstack-infra | 02:07 | |
| *** sirushti has joined #openstack-infra | 02:08 | |
| *** weshay has joined #openstack-infra | 02:08 | |
| *** krotscheck has joined #openstack-infra | 02:08 | |
| *** antonym has joined #openstack-infra | 02:08 | |
| *** dachary has joined #openstack-infra | 02:08 | |
| *** antonym is now known as Guest31645 | 02:08 | |
| *** Guest31645 has joined #openstack-infra | 02:08 | |
| *** kiall has joined #openstack-infra | 02:08 | |
| *** morganfainberg is now known as morganfainberg_Z | 02:09 | |
| *** flaper87|afk has joined #openstack-infra | 02:11 | |
| *** nicedice__ has joined #openstack-infra | 02:11 | |
| *** flaper87|afk is now known as flaper87 | 02:11 | |
| *** yamahata has quit IRC | 02:12 | |
| *** yamahata has joined #openstack-infra | 02:12 | |
| *** Guest31645 has quit IRC | 02:12 | |
| *** kiall has quit IRC | 02:13 | |
| *** SpamapS has quit IRC | 02:13 | |
| *** gema has quit IRC | 02:14 | |
| *** nicedice_ has quit IRC | 02:14 | |
| *** Guest31645 has joined #openstack-infra | 02:15 | |
| *** SpamapS has joined #openstack-infra | 02:15 | |
| *** gema has joined #openstack-infra | 02:15 | |
| *** nicedice_ has joined #openstack-infra | 02:16 | |
| *** gokrokve has quit IRC | 02:17 | |
| fungi | https://launchpad.net/bugs/1279016 is teetering on the edge of a flame war over a half-dozen line patch to add a (what seems to me to be a completely unnecessary) feature to git-review | 02:18 |
| fungi | and i really, really don't want it to turn out that way | 02:18 |
| *** jp_at_hp1 has joined #openstack-infra | 02:19 | |
| *** yonglihe_ has joined #openstack-infra | 02:21 | |
| *** yonglihe_ has quit IRC | 02:22 | |
| *** yonglihe_ has joined #openstack-infra | 02:22 | |
| *** dolphm_ has joined #openstack-infra | 02:24 | |
| *** rajeshlal has joined #openstack-infra | 02:24 | |
| *** proffalk1n has joined #openstack-infra | 02:25 | |
| *** arosen1 has joined #openstack-infra | 02:27 | |
| *** nati_ueno has quit IRC | 02:29 | |
| *** Ryan_Lane has quit IRC | 02:32 | |
| *** e0ne has joined #openstack-infra | 02:33 | |
| *** dkehn has quit IRC | 02:35 | |
| *** openstackgerrit has quit IRC | 02:35 | |
| *** rcarrillocruz1 has quit IRC | 02:35 | |
| *** jungleboyj has quit IRC | 02:35 | |
| *** dims has quit IRC | 02:35 | |
| *** yamahata_ has quit IRC | 02:35 | |
| *** jeremyb has quit IRC | 02:35 | |
| *** sbadia has quit IRC | 02:35 | |
| *** mtreinish has quit IRC | 02:35 | |
| *** dripton has quit IRC | 02:35 | |
| *** asadoughi has quit IRC | 02:35 | |
| *** mattoliverau has quit IRC | 02:35 | |
| *** che-arne has quit IRC | 02:35 | |
| *** SpamapS has quit IRC | 02:35 | |
| *** yamahata has quit IRC | 02:35 | |
| *** dachary has quit IRC | 02:35 | |
| *** changbl has quit IRC | 02:35 | |
| *** mfink has quit IRC | 02:35 | |
| *** dstanek has quit IRC | 02:35 | |
| *** rossella_s has quit IRC | 02:35 | |
| *** gyee has quit IRC | 02:35 | |
| *** saper has quit IRC | 02:35 | |
| *** jaypipes has quit IRC | 02:35 | |
| *** morgabra has quit IRC | 02:35 | |
| *** gaelL_ has quit IRC | 02:35 | |
| *** maurosr has quit IRC | 02:35 | |
| *** Shrews has quit IRC | 02:35 | |
| *** sahumada has quit IRC | 02:35 | |
| *** clarkb has quit IRC | 02:35 | |
| *** jpeeler has quit IRC | 02:35 | |
| *** kevinbenton has quit IRC | 02:35 | |
| *** marun has quit IRC | 02:35 | |
| *** kmartin has quit IRC | 02:35 | |
| *** mestery has quit IRC | 02:35 | |
| *** rajeshlal has quit IRC | 02:35 | |
| *** jp_at_hp1 has quit IRC | 02:35 | |
| *** flaper87 has quit IRC | 02:35 | |
| *** bogdando has quit IRC | 02:35 | |
| *** CaptTofu has quit IRC | 02:35 | |
| *** mwagner_lap has quit IRC | 02:35 | |
| *** ociuhandu has quit IRC | 02:35 | |
| *** bradm has quit IRC | 02:35 | |
| *** koolhead17 has quit IRC | 02:35 | |
| *** cp16net has quit IRC | 02:35 | |
| *** locke105 has quit IRC | 02:35 | |
| *** sdake has quit IRC | 02:35 | |
| *** moted has quit IRC | 02:35 | |
| *** Guest31645 has quit IRC | 02:35 | |
| *** nicedice__ has quit IRC | 02:35 | |
| *** krotscheck has quit IRC | 02:35 | |
| *** sirushti has quit IRC | 02:35 | |
| *** dhellmann has quit IRC | 02:35 | |
| *** bknudson has quit IRC | 02:35 | |
| *** xchu has quit IRC | 02:35 | |
| *** mrodden has quit IRC | 02:35 | |
| *** jp_at_hp has quit IRC | 02:35 | |
| *** jhesketh_ has quit IRC | 02:35 | |
| *** mgagne has quit IRC | 02:35 | |
| *** mkerrin has quit IRC | 02:35 | |
| *** gmoro has quit IRC | 02:35 | |
| *** tianst20 has quit IRC | 02:35 | |
| *** yongli has quit IRC | 02:35 | |
| *** proffalken has quit IRC | 02:35 | |
| *** arosen has quit IRC | 02:35 | |
| *** sileht has quit IRC | 02:35 | |
| *** dolphm has quit IRC | 02:35 | |
| *** jamielennox has quit IRC | 02:35 | |
| *** cyeoh has quit IRC | 02:35 | |
| *** AaronGr has quit IRC | 02:35 | |
| *** oubiwann has quit IRC | 02:35 | |
| *** sdague has quit IRC | 02:35 | |
| *** dolphm_ has quit IRC | 02:35 | |
| *** nicedice_ has quit IRC | 02:35 | |
| *** weshay has quit IRC | 02:35 | |
| *** banix has quit IRC | 02:35 | |
| *** nosnos has quit IRC | 02:35 | |
| *** dansmith has quit IRC | 02:35 | |
| *** oubiwann-ef has quit IRC | 02:35 | |
| *** chandan_kumar has quit IRC | 02:35 | |
| *** juice has quit IRC | 02:35 | |
| *** zul has quit IRC | 02:35 | |
| *** ryanpetrello has quit IRC | 02:35 | |
| *** stevebaker has quit IRC | 02:35 | |
| *** krtaylor has quit IRC | 02:35 | |
| *** spredzy has quit IRC | 02:35 | |
| *** luisg has quit IRC | 02:35 | |
| *** soren has quit IRC | 02:35 | |
| *** ilyashakhat_ has quit IRC | 02:35 | |
| *** jraim has quit IRC | 02:35 | |
| *** miqui has quit IRC | 02:35 | |
| *** saschpe has quit IRC | 02:35 | |
| *** annegentle has quit IRC | 02:35 | |
| *** dragondm has quit IRC | 02:35 | |
| *** tristanC has quit IRC | 02:35 | |
| *** bookwar has quit IRC | 02:35 | |
| *** skraynev_afk has quit IRC | 02:35 | |
| *** rainya has quit IRC | 02:35 | |
| *** zhiyan_ has quit IRC | 02:35 | |
| *** wayneeseguin has quit IRC | 02:35 | |
| *** mrda_away has quit IRC | 02:35 | |
| *** vipul has quit IRC | 02:35 | |
| *** davidlenwell has quit IRC | 02:35 | |
| *** EntropyWorks has quit IRC | 02:35 | |
| *** loquacities has quit IRC | 02:35 | |
| *** jeblair has quit IRC | 02:35 | |
| *** Mithrandir has quit IRC | 02:35 | |
| *** jd__ has quit IRC | 02:35 | |
| *** amrith has quit IRC | 02:35 | |
| *** jgriffith has quit IRC | 02:35 | |
| *** katyafervent has quit IRC | 02:35 | |
| *** nijaba has quit IRC | 02:35 | |
| *** pabelanger has quit IRC | 02:35 | |
| *** greghaynes has quit IRC | 02:35 | |
| *** mancdaz has quit IRC | 02:35 | |
| *** gema has quit IRC | 02:35 | |
| *** wchrisj has quit IRC | 02:35 | |
| *** unicell has quit IRC | 02:35 | |
| *** SumitNaiksatam has quit IRC | 02:35 | |
| *** sdake_ has quit IRC | 02:35 | |
| *** afazekas has quit IRC | 02:35 | |
| *** pcrews has quit IRC | 02:35 | |
| *** lahoucine has quit IRC | 02:35 | |
| *** homeless has quit IRC | 02:35 | |
| *** salv-orlando has quit IRC | 02:35 | |
| *** simonmcc has quit IRC | 02:35 | |
| *** alaski has quit IRC | 02:35 | |
| *** NikitaKonovalov has quit IRC | 02:35 | |
| *** michchap has quit IRC | 02:35 | |
| *** alex-luncheon has quit IRC | 02:35 | |
| *** westmaas has quit IRC | 02:35 | |
| *** medieval1 has quit IRC | 02:35 | |
| *** DuncanT- has quit IRC | 02:35 | |
| *** dangers_away has quit IRC | 02:35 | |
| *** SergeyLukjanov has quit IRC | 02:35 | |
| *** ruhe has quit IRC | 02:35 | |
| *** DinaBelova has quit IRC | 02:35 | |
| *** ogelbukh has quit IRC | 02:35 | |
| *** jasond` has quit IRC | 02:35 | |
| *** gothicmindfood has quit IRC | 02:35 | |
| *** notmyname has quit IRC | 02:35 | |
| *** pleia2 has quit IRC | 02:35 | |
| *** comstud has quit IRC | 02:35 | |
| *** Guest6432 has quit IRC | 02:35 | |
| *** niska has quit IRC | 02:35 | |
| *** lyxus has quit IRC | 02:35 | |
| *** branen has quit IRC | 02:35 | |
| *** dkehn_ has quit IRC | 02:35 | |
| *** samalba has quit IRC | 02:35 | |
| *** geekinutah has quit IRC | 02:35 | |
| *** rahmu has quit IRC | 02:35 | |
| *** nibalizer has quit IRC | 02:35 | |
| *** plomakin_ has quit IRC | 02:35 | |
| *** rdwrer has quit IRC | 02:35 | |
| *** dosaboy has quit IRC | 02:35 | |
| *** vladan has quit IRC | 02:35 | |
| *** spiffxp_ has quit IRC | 02:35 | |
| *** jcooley has quit IRC | 02:35 | |
| *** belliott has quit IRC | 02:35 | |
| *** adarazs_ has quit IRC | 02:35 | |
| *** jogo has quit IRC | 02:35 | |
| *** guitarzan has quit IRC | 02:35 | |
| *** mburned has quit IRC | 02:35 | |
| *** aude has quit IRC | 02:35 | |
| *** Steap has quit IRC | 02:35 | |
| *** gilliard has quit IRC | 02:35 | |
| *** lifeless has quit IRC | 02:35 | |
| *** mika has quit IRC | 02:35 | |
| *** jesusaurus has quit IRC | 02:35 | |
| *** primeministerp has quit IRC | 02:35 | |
| *** gpocentek has quit IRC | 02:35 | |
| *** GheRivero has quit IRC | 02:35 | |
| *** BobBall has quit IRC | 02:35 | |
| *** mordred has quit IRC | 02:35 | |
| *** ihrachys|afk has quit IRC | 02:35 | |
| *** dtroyer has quit IRC | 02:35 | |
| *** Daviey has quit IRC | 02:35 | |
| *** Alex_Gaynor has quit IRC | 02:35 | |
| *** tteggel has quit IRC | 02:35 | |
| *** boris-42 has quit IRC | 02:35 | |
| *** Adri2000 has quit IRC | 02:35 | |
| *** StevenK has quit IRC | 02:35 | |
| *** JayF has quit IRC | 02:35 | |
| *** EmilienM has quit IRC | 02:35 | |
| *** chmouel has quit IRC | 02:35 | |
| *** fungi has quit IRC | 02:35 | |
| *** zigo has quit IRC | 02:35 | |
| *** jroll has quit IRC | 02:35 | |
| *** pvo has quit IRC | 02:35 | |
| *** Guest34377 has quit IRC | 02:35 | |
| *** wendar has quit IRC | 02:35 | |
| *** jbryce has quit IRC | 02:35 | |
| *** russellb has quit IRC | 02:35 | |
| *** jlk has quit IRC | 02:35 | |
| *** adam_g has quit IRC | 02:35 | |
| *** ttx has quit IRC | 02:35 | |
| *** gondoi has quit IRC | 02:35 | |
| *** msuriar_ has quit IRC | 02:35 | |
| *** mikal has quit IRC | 02:35 | |
| *** rockstar has quit IRC | 02:35 | |
| *** jeckersb has quit IRC | 02:35 | |
| *** hughsaunders has quit IRC | 02:35 | |
| *** isviridov has quit IRC | 02:35 | |
| *** fbo_away has quit IRC | 02:35 | |
| *** ChanServ has quit IRC | 02:35 | |
| *** morganfainberg_Z has quit IRC | 02:35 | |
| *** ildikov_ has quit IRC | 02:35 | |
| *** achuprin has quit IRC | 02:35 | |
| *** Ng has quit IRC | 02:35 | |
| *** ekarlso has quit IRC | 02:35 | |
| *** Hunner has quit IRC | 02:35 | |
| *** bodepd has quit IRC | 02:35 | |
| *** zaro has quit IRC | 02:35 | |
| *** psedlak has quit IRC | 02:35 | |
| *** malini_afk has quit IRC | 02:35 | |
| *** bnemec has quit IRC | 02:35 | |
| *** therve has quit IRC | 02:35 | |
| *** JoshNang has quit IRC | 02:35 | |
| *** tian has quit IRC | 02:35 | |
| *** anteaya has quit IRC | 02:35 | |
| *** tru_tru has quit IRC | 02:35 | |
| *** tchaypo has quit IRC | 02:35 | |
| *** jomara has quit IRC | 02:35 | |
| *** portante has quit IRC | 02:35 | |
| *** hdd has quit IRC | 02:35 | |
| *** hub_cap has quit IRC | 02:35 | |
| *** phschwartz has quit IRC | 02:35 | |
| *** med_ has quit IRC | 02:35 | |
| *** jamespage has quit IRC | 02:35 | |
| *** shardy has quit IRC | 02:35 | |
| *** NobodyCam has quit IRC | 02:35 | |
| *** devananda has quit IRC | 02:35 | |
| *** SlickNik has quit IRC | 02:35 | |
| *** tnurlygayanov has quit IRC | 02:35 | |
| *** gokrokve has joined #openstack-infra | 02:36 | |
| *** dstufft has quit IRC | 02:37 | |
| *** e0ne has quit IRC | 02:38 | |
| *** mtreinish has joined #openstack-infra | 02:40 | |
| *** amotoki has quit IRC | 02:55 | |
| *** dkranz has joined #openstack-infra | 02:55 | |
| *** Ryan_Lane has joined #openstack-infra | 03:00 | |
| *** prad has quit IRC | 03:00 | |
| *** mrda_away has joined #openstack-infra | 03:01 | |
| *** mkerrin has joined #openstack-infra | 03:02 | |
| *** mgagne has joined #openstack-infra | 03:02 | |
| *** bknudson has joined #openstack-infra | 03:02 | |
| *** sirushti has joined #openstack-infra | 03:02 | |
| *** cyeoh has joined #openstack-infra | 03:02 | |
| *** AaronGr has joined #openstack-infra | 03:02 | |
| *** Guest31645 has joined #openstack-infra | 03:02 | |
| *** mrodden has joined #openstack-infra | 03:02 | |
| *** mattoliverau has joined #openstack-infra | 03:02 | |
| *** asadoughi has joined #openstack-infra | 03:02 | |
| *** thomasem has joined #openstack-infra | 03:02 | |
| *** tnurlygayanov has joined #openstack-infra | 03:03 | |
| *** thomasem has quit IRC | 03:03 | |
| *** Ryan_Lane has quit IRC | 03:04 | |
| *** tnurlygayanov has quit IRC | 03:04 | |
| *** asadoughi has quit IRC | 03:06 | |
| *** mattoliverau has quit IRC | 03:06 | |
| *** mrodden has quit IRC | 03:06 | |
| *** Guest31645 has quit IRC | 03:06 | |
| *** AaronGr has quit IRC | 03:06 | |
| *** cyeoh has quit IRC | 03:06 | |
| *** sirushti has quit IRC | 03:06 | |
| *** bknudson has quit IRC | 03:06 | |
| *** mgagne has quit IRC | 03:06 | |
| *** mkerrin has quit IRC | 03:06 | |
| *** khyati has joined #openstack-infra | 03:08 | |
| *** khyati has quit IRC | 03:10 | |
| *** tnurlygayanov has joined #openstack-infra | 03:12 | |
| *** juice has joined #openstack-infra | 03:12 | |
| *** mkerrin has joined #openstack-infra | 03:12 | |
| *** mgagne has joined #openstack-infra | 03:12 | |
| *** bknudson has joined #openstack-infra | 03:12 | |
| *** sirushti has joined #openstack-infra | 03:12 | |
| *** cyeoh has joined #openstack-infra | 03:12 | |
| *** AaronGr has joined #openstack-infra | 03:12 | |
| *** Guest31645 has joined #openstack-infra | 03:12 | |
| *** mrodden has joined #openstack-infra | 03:12 | |
| *** mattoliverau has joined #openstack-infra | 03:12 | |
| *** asadoughi has joined #openstack-infra | 03:12 | |
| *** krotscheck has joined #openstack-infra | 03:17 | |
| *** dolphm has joined #openstack-infra | 03:21 | |
| *** davidlenwell has joined #openstack-infra | 03:25 | |
| *** kiall has joined #openstack-infra | 03:25 | |
| *** jraim has joined #openstack-infra | 03:27 | |
| *** annegentle has joined #openstack-infra | 03:31 | |
| *** rainya has joined #openstack-infra | 03:31 | |
| *** stevebaker has joined #openstack-infra | 03:31 | |
| *** dstufft has joined #openstack-infra | 03:31 | |
| *** weshay has joined #openstack-infra | 03:31 | |
| *** banix has joined #openstack-infra | 03:31 | |
| *** dansmith has joined #openstack-infra | 03:31 | |
| *** oubiwann-ef has joined #openstack-infra | 03:31 | |
| *** zul has joined #openstack-infra | 03:31 | |
| *** ryanpetrello has joined #openstack-infra | 03:31 | |
| *** krtaylor has joined #openstack-infra | 03:31 | |
| *** miqui has joined #openstack-infra | 03:31 | |
| *** zhiyan has joined #openstack-infra | 03:31 | |
| *** dragondm has joined #openstack-infra | 03:31 | |
| *** tristanC has joined #openstack-infra | 03:31 | |
| *** bookwar has joined #openstack-infra | 03:31 | |
| *** wayneeseguin has joined #openstack-infra | 03:31 | |
| *** dstufft has quit IRC | 03:31 | |
| *** annegentle has quit IRC | 03:31 | |
| *** weshay has quit IRC | 03:31 | |
| *** dansmith has quit IRC | 03:31 | |
| *** zul has quit IRC | 03:31 | |
| *** weshay has joined #openstack-infra | 03:31 | |
| *** dansmith has joined #openstack-infra | 03:31 | |
| *** zul has joined #openstack-infra | 03:32 | |
| *** sdague has joined #openstack-infra | 03:32 | |
| *** Daisy has joined #openstack-infra | 03:32 | |
| *** EntropyWorks has joined #openstack-infra | 03:33 | |
| *** annegentle has joined #openstack-infra | 03:33 | |
| *** jgriffith has joined #openstack-infra | 03:37 | |
| *** krotscheck has quit IRC | 03:40 | |
| *** basha has joined #openstack-infra | 03:56 | |
| *** Ryan_Lane has joined #openstack-infra | 04:01 | |
| *** mgagne has quit IRC | 04:02 | |
| *** basha has quit IRC | 04:12 | |
| *** e0ne has joined #openstack-infra | 04:33 | |
| *** mgagne has joined #openstack-infra | 04:34 | |
| *** Daisy has quit IRC | 04:35 | |
| *** e0ne has quit IRC | 04:37 | |
| *** talluri has joined #openstack-infra | 04:38 | |
| *** talluri has quit IRC | 04:40 | |
| *** talluri has joined #openstack-infra | 04:46 | |
| *** fifieldt has joined #openstack-infra | 04:48 | |
| *** krotscheck has joined #openstack-infra | 04:52 | |
| *** banix has quit IRC | 04:53 | |
| *** thuc has joined #openstack-infra | 04:56 | |
| *** nosnos has joined #openstack-infra | 05:00 | |
| *** sirushti is now known as shortstop | 05:04 | |
| *** sabari has joined #openstack-infra | 05:04 | |
| *** mgagne has quit IRC | 05:11 | |
| *** wenlock has joined #openstack-infra | 05:28 | |
| *** reed has joined #openstack-infra | 05:28 | |
| *** fifieldt has quit IRC | 05:30 | |
| *** reed has quit IRC | 05:30 | |
| *** amotoki has joined #openstack-infra | 05:39 | |
| *** thuc has quit IRC | 05:45 | |
| *** thuc has joined #openstack-infra | 05:45 | |
| *** wenlock has quit IRC | 05:47 | |
| *** thuc has quit IRC | 05:50 | |
| *** markmcclain has joined #openstack-infra | 06:00 | |
| *** persia__ has quit IRC | 06:03 | |
| *** oubiwann-ef has quit IRC | 06:05 | |
| *** CaptTofu has joined #openstack-infra | 06:07 | |
| *** CaptTofu has quit IRC | 06:12 | |
| *** rpodolyaka has joined #openstack-infra | 06:32 | |
| *** e0ne has joined #openstack-infra | 06:33 | |
| *** sabari has quit IRC | 06:35 | |
| *** e0ne has quit IRC | 06:38 | |
| *** jhesketh__ has quit IRC | 06:51 | |
| *** saju_m has joined #openstack-infra | 07:01 | |
| *** mkoderer has joined #openstack-infra | 07:04 | |
| *** rpodolyaka has quit IRC | 07:04 | |
| *** zhiyan is now known as zhiyan_ | 07:04 | |
| *** andreaf has joined #openstack-infra | 07:17 | |
| *** nosnos_ has joined #openstack-infra | 07:27 | |
| *** nosnos has quit IRC | 07:27 | |
| *** denis_makogon has joined #openstack-infra | 07:30 | |
| *** unicell has joined #openstack-infra | 07:54 | |
| *** yolanda_ has joined #openstack-infra | 07:54 | |
| *** ildikov_ has joined #openstack-infra | 07:54 | |
| *** pblaho has joined #openstack-infra | 07:54 | |
| *** derekh has joined #openstack-infra | 07:54 | |
| *** yaguang has joined #openstack-infra | 07:54 | |
| *** wenlock has joined #openstack-infra | 07:54 | |
| *** jungleboyj has joined #openstack-infra | 07:54 | |
| *** david-lyle has joined #openstack-infra | 07:54 | |
| *** dstufft has joined #openstack-infra | 07:54 | |
| *** jeremyb has joined #openstack-infra | 07:54 | |
| *** skraynev has joined #openstack-infra | 07:54 | |
| *** ilyashakhat has joined #openstack-infra | 07:54 | |
| *** vipul has joined #openstack-infra | 07:54 | |
| *** chandan_kumar has joined #openstack-infra | 07:54 | |
| *** soren has joined #openstack-infra | 07:54 | |
| *** saschpe has joined #openstack-infra | 07:54 | |
| *** yamahata_ has joined #openstack-infra | 07:54 | |
| *** spredzy_ has joined #openstack-infra | 07:54 | |
| *** dims_ has joined #openstack-infra | 07:54 | |
| *** sbadia has joined #openstack-infra | 07:54 | |
| *** tianst20 has joined #openstack-infra | 07:54 | |
| *** sweston has joined #openstack-infra | 07:54 | |
| *** rcarrillocruz has joined #openstack-infra | 07:54 | |
| *** dkehn has joined #openstack-infra | 07:54 | |
| *** dripton_ has joined #openstack-infra | 07:54 | |
| *** oubiwann has joined #openstack-infra | 07:54 | |
| *** sileht has joined #openstack-infra | 07:54 | |
| *** gmoro has joined #openstack-infra | 07:54 | |
| *** dhellmann has joined #openstack-infra | 07:54 | |
| *** jhesketh_ has joined #openstack-infra | 07:54 | |
| *** jamielennox|away has joined #openstack-infra | 07:54 | |
| *** xchu has joined #openstack-infra | 07:54 | |
| *** rajeshlal has joined #openstack-infra | 07:54 | |
| *** gema has joined #openstack-infra | 07:54 | |
| *** SpamapS has joined #openstack-infra | 07:54 | |
| *** yamahata has joined #openstack-infra | 07:54 | |
| *** flaper87 has joined #openstack-infra | 07:54 | |
| *** dachary has joined #openstack-infra | 07:54 | |
| *** bogdando has joined #openstack-infra | 07:54 | |
| *** changbl has joined #openstack-infra | 07:54 | |
| *** mwagner_lap has joined #openstack-infra | 07:54 | |
| *** mfink has joined #openstack-infra | 07:54 | |
| *** dstanek has joined #openstack-infra | 07:54 | |
| *** ociuhandu has joined #openstack-infra | 07:54 | |
| *** SumitNaiksatam has joined #openstack-infra | 07:54 | |
| *** rossella_s has joined #openstack-infra | 07:54 | |
| *** sdake_ has joined #openstack-infra | 07:54 | |
| *** afazekas has joined #openstack-infra | 07:54 | |
| *** saper has joined #openstack-infra | 07:54 | |
| *** jaypipes has joined #openstack-infra | 07:54 | |
| *** lahoucine has joined #openstack-infra | 07:54 | |
| *** achuprin has joined #openstack-infra | 07:54 | |
| *** Ng has joined #openstack-infra | 07:54 | |
| *** morgabra has joined #openstack-infra | 07:54 | |
| *** jroll has joined #openstack-infra | 07:54 | |
| *** JoshNang has joined #openstack-infra | 07:54 | |
| *** JayF has joined #openstack-infra | 07:54 | |
| *** malini_afk has joined #openstack-infra | 07:54 | |
| *** homeless has joined #openstack-infra | 07:54 | |
| *** jomara has joined #openstack-infra | 07:54 | |
| *** bradm has joined #openstack-infra | 07:54 | |
| *** Alex_Gaynor has joined #openstack-infra | 07:54 | |
| *** simonmcc has joined #openstack-infra | 07:54 | |
| *** gaelL_ has joined #openstack-infra | 07:54 | |
| *** maurosr has joined #openstack-infra | 07:54 | |
| *** alaski has joined #openstack-infra | 07:54 | |
| *** Shrews has joined #openstack-infra | 07:54 | |
| *** NikitaKonovalov has joined #openstack-infra | 07:54 | |
| *** gondoi has joined #openstack-infra | 07:54 | |
| *** che-arne has joined #openstack-infra | 07:54 | |
| *** ekarlso has joined #openstack-infra | 07:54 | |
| *** jd__ has joined #openstack-infra | 07:54 | |
| *** gilliard has joined #openstack-infra | 07:54 | |
| *** michchap has joined #openstack-infra | 07:54 | |
| *** sahumada has joined #openstack-infra | 07:54 | |
| *** clarkb has joined #openstack-infra | 07:54 | |
| *** jpeeler has joined #openstack-infra | 07:54 | |
| *** koolhead17 has joined #openstack-infra | 07:54 | |
| *** tteggel has joined #openstack-infra | 07:54 | |
| *** westmaas has joined #openstack-infra | 07:54 | |
| *** kevinbenton has joined #openstack-infra | 07:54 | |
| *** medieval1 has joined #openstack-infra | 07:54 | |
| *** marun has joined #openstack-infra | 07:54 | |
| *** DuncanT- has joined #openstack-infra | 07:54 | |
| *** dangers_away has joined #openstack-infra | 07:54 | |
| *** kmartin has joined #openstack-infra | 07:54 | |
| *** boris-42 has joined #openstack-infra | 07:54 | |
| *** SergeyLukjanov has joined #openstack-infra | 07:54 | |
| *** ruhe has joined #openstack-infra | 07:54 | |
| *** DinaBelova has joined #openstack-infra | 07:54 | |
| *** mestery has joined #openstack-infra | 07:54 | |
| *** Adri2000 has joined #openstack-infra | 07:54 | |
| *** Hunner has joined #openstack-infra | 07:54 | |
| *** ogelbukh has joined #openstack-infra | 07:54 | |
| *** bodepd has joined #openstack-infra | 07:54 | |
| *** zaro has joined #openstack-infra | 07:54 | |
| *** jasond` has joined #openstack-infra | 07:54 | |
| *** gothicmindfood has joined #openstack-infra | 07:54 | |
| *** notmyname has joined #openstack-infra | 07:54 | |
| *** pleia2 has joined #openstack-infra | 07:54 | |
| *** psedlak has joined #openstack-infra | 07:54 | |
| *** comstud has joined #openstack-infra | 07:54 | |
| *** Guest6432 has joined #openstack-infra | 07:54 | |
| *** niska has joined #openstack-infra | 07:54 | |
| *** lyxus has joined #openstack-infra | 07:54 | |
| *** branen has joined #openstack-infra | 07:54 | |
| *** dkehn_ has joined #openstack-infra | 07:54 | |
| *** samalba has joined #openstack-infra | 07:54 | |
| *** geekinutah has joined #openstack-infra | 07:54 | |
| *** bnemec has joined #openstack-infra | 07:54 | |
| *** therve has joined #openstack-infra | 07:54 | |
| *** tian has joined #openstack-infra | 07:54 | |
| *** loquacities has joined #openstack-infra | 07:54 | |
| *** jeblair has joined #openstack-infra | 07:54 | |
| *** rdwrer has joined #openstack-infra | 07:54 | |
| *** cp16net has joined #openstack-infra | 07:54 | |
| *** phschwartz has joined #openstack-infra | 07:54 | |
| *** locke105 has joined #openstack-infra | 07:54 | |
| *** Mithrandir has joined #openstack-infra | 07:54 | |
| *** amrith has joined #openstack-infra | 07:54 | |
| *** sdake has joined #openstack-infra | 07:54 | |
| *** moted has joined #openstack-infra | 07:54 | |
| *** katyafervent has joined #openstack-infra | 07:54 | |
| *** nijaba has joined #openstack-infra | 07:54 | |
| *** Steap has joined #openstack-infra | 07:54 | |
| *** mancdaz has joined #openstack-infra | 07:54 | |
| *** greghaynes has joined #openstack-infra | 07:54 | |
| *** pabelanger has joined #openstack-infra | 07:54 | |
| *** StevenK has joined #openstack-infra | 07:54 | |
| *** rahmu has joined #openstack-infra | 07:54 | |
| *** nibalizer has joined #openstack-infra | 07:54 | |
| *** plomakin_ has joined #openstack-infra | 07:54 | |
| *** dosaboy has joined #openstack-infra | 07:54 | |
| *** tchaypo has joined #openstack-infra | 07:54 | |
| *** EmilienM has joined #openstack-infra | 07:54 | |
| *** vladan has joined #openstack-infra | 07:54 | |
| *** spiffxp_ has joined #openstack-infra | 07:54 | |
| *** belliott has joined #openstack-infra | 07:54 | |
| *** jcooley has joined #openstack-infra | 07:54 | |
| *** adarazs_ has joined #openstack-infra | 07:54 | |
| *** jogo has joined #openstack-infra | 07:54 | |
| *** guitarzan has joined #openstack-infra | 07:54 | |
| *** mburned has joined #openstack-infra | 07:54 | |
| *** anteaya has joined #openstack-infra | 07:54 | |
| *** tru_tru has joined #openstack-infra | 07:54 | |
| *** aude has joined #openstack-infra | 07:54 | |
| *** portante has joined #openstack-infra | 07:54 | |
| *** lifeless has joined #openstack-infra | 07:54 | |
| *** mika has joined #openstack-infra | 07:54 | |
| *** jesusaurus has joined #openstack-infra | 07:54 | |
| *** primeministerp has joined #openstack-infra | 07:54 | |
| *** gpocentek has joined #openstack-infra | 07:54 | |
| *** GheRivero has joined #openstack-infra | 07:54 | |
| *** BobBall has joined #openstack-infra | 07:54 | |
| *** mordred has joined #openstack-infra | 07:54 | |
| *** ihrachys|afk has joined #openstack-infra | 07:54 | |
| *** Daviey has joined #openstack-infra | 07:54 | |
| *** dtroyer has joined #openstack-infra | 07:54 | |
| *** hdd has joined #openstack-infra | 07:54 | |
| *** hub_cap has joined #openstack-infra | 07:54 | |
| *** chmouel has joined #openstack-infra | 07:54 | |
| *** fungi has joined #openstack-infra | 07:54 | |
| *** zigo has joined #openstack-infra | 07:54 | |
| *** pvo has joined #openstack-infra | 07:54 | |
| *** Guest34377 has joined #openstack-infra | 07:54 | |
| *** wendar has joined #openstack-infra | 07:54 | |
| *** jbryce has joined #openstack-infra | 07:54 | |
| *** russellb has joined #openstack-infra | 07:54 | |
| *** jlk has joined #openstack-infra | 07:54 | |
| *** adam_g has joined #openstack-infra | 07:54 | |
| *** ttx has joined #openstack-infra | 07:54 | |
| *** med_ has joined #openstack-infra | 07:54 | |
| *** jamespage has joined #openstack-infra | 07:54 | |
| *** msuriar_ has joined #openstack-infra | 07:54 | |
| *** mikal has joined #openstack-infra | 07:54 | |
| *** rockstar has joined #openstack-infra | 07:54 | |
| *** jeckersb has joined #openstack-infra | 07:54 | |
| *** hughsaunders has joined #openstack-infra | 07:54 | |
| *** morganfainberg has joined #openstack-infra | 07:54 | |
| *** isviridov has joined #openstack-infra | 07:54 | |
| *** fbo_away has joined #openstack-infra | 07:54 | |
| *** ChanServ has joined #openstack-infra | 07:54 | |
| *** shardy has joined #openstack-infra | 07:54 | |
| *** dickson.freenode.net sets mode: +o ChanServ | 07:54 | |
| *** NobodyCam has joined #openstack-infra | 07:54 | |
| *** devananda has joined #openstack-infra | 07:54 | |
| *** SlickNik has joined #openstack-infra | 07:54 | |
| *** jraim has quit IRC | 07:56 | |
| *** jraim has joined #openstack-infra | 07:56 | |
| *** mkoderer has quit IRC | 07:56 | |
| *** mkoderer has joined #openstack-infra | 07:56 | |
| *** unicell has quit IRC | 07:56 | |
| *** kiall is now known as Guest23417 | 07:56 | |
| *** Ryan_Lane is now known as Guest73507 | 07:56 | |
| *** tteggel has quit IRC | 07:56 | |
| *** CaptTofu has joined #openstack-infra | 07:56 | |
| *** denis_makogon has quit IRC | 07:56 | |
| *** Guest23417 has quit IRC | 07:56 | |
| *** Alex_Gaynor has quit IRC | 07:56 | |
| *** Alex_Gaynor has joined #openstack-infra | 07:57 | |
| *** kiall_ has joined #openstack-infra | 07:58 | |
| *** flaper87 has quit IRC | 07:59 | |
| *** flaper87 has joined #openstack-infra | 07:59 | |
| *** gokrokve has quit IRC | 07:59 | |
| *** CaptTofu has quit IRC | 08:01 | |
| *** tteggel has joined #openstack-infra | 08:03 | |
| *** jlibosva has joined #openstack-infra | 08:04 | |
| ttx | jeblair: will read | 08:04 |
| *** morganfainberg is now known as morganfainberg_Z | 08:07 | |
| *** afazekas has quit IRC | 08:08 | |
| *** wenlock has quit IRC | 08:18 | |
| *** jgallard has joined #openstack-infra | 08:21 | |
| jgallard | hi all | 08:24 |
| *** pblaho has quit IRC | 08:25 | |
| jgallard | we would like to contribute with the multi node aspects on the infra | 08:25 |
| jgallard | we want to build our own infra, and plane to look deeper into nodepool | 08:25 |
| jgallard | this is the right way to do that? | 08:25 |
| *** pblaho has joined #openstack-infra | 08:25 | |
| jgallard | may be fungi , sdague , jeblair ^ (when you have few minutes :-) ) | 08:29 |
| *** gokrokve has joined #openstack-infra | 08:30 | |
| *** dizquierdo has joined #openstack-infra | 08:32 | |
| *** gokrokve_ has joined #openstack-infra | 08:32 | |
| *** gokrokve has quit IRC | 08:35 | |
| *** gokrokve_ has quit IRC | 08:37 | |
| clarkb | jgallard: I think nodepool will definitely be involved | 08:37 |
| jgallard | hi clarkb | 08:38 |
| clarkb | jgallard: I think I would like to see nodepool create heat stacks that can be used for multinode testing. but our public clouds don't have heat endpoints yet and we aren't running our own so this may not be an immediately doable thing | 08:39 |
| jgallard | ah ok! | 08:39 |
| jgallard | ah ok | 08:39 |
| jgallard | but, it's not possible to have nodepool able to manage "several vms" and register only one VM as a jenkins slave | 08:40 |
| jgallard | ? | 08:40 |
| jgallard | i'm not sure if i'm clear, sorry | 08:40 |
| clarkb | not today | 08:40 |
| clarkb | nodepool is currently built to register every node with jenkins | 08:40 |
| jgallard | can we propose to modify it in order to manage "set of VM"? | 08:41 |
| *** e0ne has joined #openstack-infra | 08:42 | |
| clarkb | definitely and that is basically what a heat stack is whcih is why I think heat is a good way to do it | 08:42 |
| clarkb | jgallard: this might be a good discussion for the mail list though. Since I need to head to bed and fungi, jeblair, and sdague are most likely sleeping | 08:42 |
| clarkb | jgallard: want to kick off the discussion on the -infra list? | 08:43 |
| jgallard | héhé ok :-) | 08:43 |
| jgallard | ok! | 08:43 |
| jgallard | I will do it | 08:43 |
| jgallard | thanks a lot for your time :-) | 08:43 |
| *** andreaf has quit IRC | 08:43 | |
| clarkb | great, I look forward to reading it in the morning | 08:43 |
| clarkb | jgallard: no problem | 08:43 |
| *** nosnos has joined #openstack-infra | 08:45 | |
| *** nosnos_ has quit IRC | 08:46 | |
| *** sarob_ has joined #openstack-infra | 08:49 | |
| *** sarob_ has quit IRC | 08:55 | |
| *** e0ne_ has joined #openstack-infra | 08:56 | |
| *** e0ne has quit IRC | 08:59 | |
| *** markmcclain has quit IRC | 09:02 | |
| *** ildikov_ has quit IRC | 09:02 | |
| *** jpich has joined #openstack-infra | 09:03 | |
| *** che-arne has quit IRC | 09:03 | |
| *** johnthetubaguy has joined #openstack-infra | 09:06 | |
| *** yassine has joined #openstack-infra | 09:06 | |
| *** johnthetubaguy1 has joined #openstack-infra | 09:08 | |
| *** johnthetubaguy has quit IRC | 09:09 | |
| *** basha has joined #openstack-infra | 09:11 | |
| *** basha has quit IRC | 09:13 | |
| *** basha has joined #openstack-infra | 09:14 | |
| *** dizquierdo has quit IRC | 09:16 | |
| *** ildikov_ has joined #openstack-infra | 09:18 | |
| *** basha_ has joined #openstack-infra | 09:18 | |
| *** basha has quit IRC | 09:18 | |
| *** basha_ is now known as basha | 09:18 | |
| *** jooools has joined #openstack-infra | 09:20 | |
| *** malini_afk is now known as malini | 09:29 | |
| *** mkerrin has quit IRC | 09:30 | |
| *** hashar has joined #openstack-infra | 09:30 | |
| *** gokrokve has joined #openstack-infra | 09:33 | |
| *** derekh has quit IRC | 09:35 | |
| *** andreaf has joined #openstack-infra | 09:36 | |
| *** gokrokve has quit IRC | 09:38 | |
| *** andreaf has quit IRC | 09:42 | |
| *** e0ne_ has quit IRC | 09:42 | |
| *** hashar has quit IRC | 09:43 | |
| *** fbo_away is now known as fbo | 09:44 | |
| *** CaptTofu has joined #openstack-infra | 09:44 | |
| *** CaptTofu has quit IRC | 09:49 | |
| *** sarob_ has joined #openstack-infra | 09:52 | |
| *** mkerrin has joined #openstack-infra | 09:55 | |
| *** sarob_ has quit IRC | 09:56 | |
| *** e0ne has joined #openstack-infra | 09:57 | |
| *** krotscheck has quit IRC | 09:59 | |
| jd__ | hey guys, https://review.openstack.org/#/c/77209/ has been merged but it seems it only works on RHEL node (py26), can't find memcached in the PATH on py27 nodes | 10:02 |
| *** jp_at_hp has joined #openstack-infra | 10:03 | |
| SergeyLukjanov | jd__, hey | 10:12 |
| SergeyLukjanov | jd__, probably, images aren't rebuilt yet | 10:12 |
| SergeyLukjanov | jd__, because nodepool rebuild slave images ones per day | 10:13 |
| jd__ | SergeyLukjanov: ack | 10:13 |
| *** johnthetubaguy1 is now known as johnthetubaguy | 10:17 | |
| *** sweston has quit IRC | 10:18 | |
| *** saju_m has quit IRC | 10:19 | |
| sdague | sigh, something in tempest is failing to compile, and testr is making it impossible to understand what | 10:25 |
| sdague | in the gate | 10:25 |
| *** CaptTofu has joined #openstack-infra | 10:26 | |
| *** andre__ has joined #openstack-infra | 10:30 | |
| *** e0ne_ has joined #openstack-infra | 10:30 | |
| *** Daisy has joined #openstack-infra | 10:32 | |
| *** andreaf has joined #openstack-infra | 10:32 | |
| *** gokrokve has joined #openstack-infra | 10:33 | |
| *** e0ne has quit IRC | 10:34 | |
| *** hashar has joined #openstack-infra | 10:35 | |
| *** gokrokve has quit IRC | 10:38 | |
| *** Daisy has quit IRC | 10:41 | |
| sdague | oh... it's all the rax nodes | 10:49 |
| sdague | yay, all rax nodes can't run tempest | 10:49 |
| sdague | :( | 10:49 |
| *** sarob_ has joined #openstack-infra | 10:53 | |
| *** mfink has quit IRC | 10:53 | |
| *** CaptTofu has quit IRC | 10:53 | |
| *** CaptTofu has joined #openstack-infra | 10:54 | |
| *** CaptTofu has quit IRC | 10:58 | |
| *** sarob_ has quit IRC | 11:00 | |
| *** e0ne_ has quit IRC | 11:02 | |
| *** jgallard has quit IRC | 11:10 | |
| jd__ | I sense a disturbance in the force | 11:11 |
| jd__ | I see from eventlet import greenthread everywhere | 11:11 |
| jd__ | +error | 11:11 |
| *** CaptTofu has joined #openstack-infra | 11:12 | |
| sdague | all rax nodes | 11:12 |
| sdague | the tests pass fine on hp nodes | 11:12 |
| sdague | but something is wrong with all rax nodes | 11:12 |
| sdague | which means, we're dead in the water | 11:12 |
| sdague | rax nodes are the majority | 11:13 |
| *** ociuhandu has quit IRC | 11:18 | |
| *** xchu has quit IRC | 11:19 | |
| sdague | I'm going to try a hack to see if I can get tempest out of this issue | 11:19 |
| sdague | https://review.openstack.org/#/c/78935/ | 11:20 |
| *** medieval1 has quit IRC | 11:22 | |
| *** e0ne has joined #openstack-infra | 11:22 | |
| *** rlandy has joined #openstack-infra | 11:22 | |
| *** medieval1 has joined #openstack-infra | 11:23 | |
| *** lcestari_ has joined #openstack-infra | 11:26 | |
| *** medieval1 has quit IRC | 11:27 | |
| *** medieval1 has joined #openstack-infra | 11:28 | |
| SergeyLukjanov | sdague, jd__, as a temp option we can disable rax nodes, but I think it's really useless... | 11:29 |
| *** saju_m has joined #openstack-infra | 11:30 | |
| jd__ | I'm not even sure why sitepackage is True in the first place, that sounds like a bad idea to me | 11:30 |
| *** gokrokve has joined #openstack-infra | 11:33 | |
| sdague | jd__: because tempest should use the system packages | 11:33 |
| sdague | we shouldn't be reinstalling this all over again | 11:34 |
| sdague | actually doing that will effectively stop doing cli testing | 11:34 |
| *** lcestari_ has quit IRC | 11:34 | |
| *** gokrokve_ has joined #openstack-infra | 11:35 | |
| sdague | so site packages might let people merge code, but it's terrible | 11:35 |
| sdague | honestly, we need to stop using tox for tempest entirely | 11:36 |
| sdague | SergeyLukjanov: so johnthetubaguy says that rax updated images yesterday | 11:38 |
| *** gokrokve has quit IRC | 11:38 | |
| SergeyLukjanov | sdague, looks like that's the reason... | 11:38 |
| *** gokrokve_ has quit IRC | 11:40 | |
| mkoderer | sdague: unfortunately subunit discover doesn't returns the actual issue. Or did you find more in the logs than me? | 11:41 |
| sdague | mkoderer: this is something much more fundamental | 11:41 |
| *** e0ne has quit IRC | 11:42 | |
| *** lcestari_ has joined #openstack-infra | 11:48 | |
| sdague | SergeyLukjanov: do you have the ability to pull rax out of nodepool allocation? | 11:48 |
| sdague | for devstack nodes | 11:49 |
| SergeyLukjanov | sdague, I have no access to the nodepool server, but I think that we can remove rax provider for devstack nodes from the nodepool config and it should work IMO | 11:49 |
| SergeyLukjanov | let me check the config file | 11:49 |
| sdague | yeh, because while it would drop capacity, stuff would pass | 11:50 |
| SergeyLukjanov | sdague, exactly | 11:50 |
| sdague | especially with people working on FFE | 11:50 |
| sdague | also - https://review.openstack.org/#/c/78939/ - we really need to do that | 11:51 |
| *** nosnos has quit IRC | 11:51 | |
| SergeyLukjanov | sdague, probably I should ask for access to the nodepool server due to the rate of issues with our cloud providers :) | 11:52 |
| *** banix has joined #openstack-infra | 11:53 | |
| SergeyLukjanov | sdague, is it broken on all rax regions? | 11:53 |
| *** hashar has quit IRC | 11:56 | |
| *** jhesketh_ has quit IRC | 11:58 | |
| *** CaptTofu has quit IRC | 12:00 | |
| sdague | SergeyLukjanov: iad and dfw for sure | 12:01 |
| SergeyLukjanov | and looks like openstackgerrit bot isn't alive too ;( | 12:01 |
| SergeyLukjanov | sdague, https://review.openstack.org/78942 | 12:01 |
| sdague | SergeyLukjanov: cool, so if you push that, will nodepool do all the right things? | 12:02 |
| *** VINOD has joined #openstack-infra | 12:02 | |
| sdague | or does it need a restart that you don't have the ability to do | 12:02 |
| SergeyLukjanov | sdague, I think yes, it's written to reload config | 12:02 |
| *** basha has quit IRC | 12:02 | |
| sdague | ok, can you push it through? | 12:03 |
| SergeyLukjanov | sdague, I think yes, hope fungi will not kill us for it :) | 12:04 |
| SergeyLukjanov | def run(self): | 12:04 |
| SergeyLukjanov | while not self._stopped: | 12:04 |
| SergeyLukjanov | try: | 12:04 |
| SergeyLukjanov | config = self.loadConfig() | 12:04 |
| sdague | yeh, well I'll take the wrath for it | 12:05 |
| SergeyLukjanov | due to this code nodepool will re-read config on each iteration | 12:05 |
| sdague | great | 12:05 |
| sdague | lets do it | 12:05 |
| SergeyLukjanov | sdague, done | 12:06 |
| *** malini is now known as malini_afk | 12:06 | |
| sdague | SergeyLukjanov: basically, with all patches failing, I don't think we can make things worse :) | 12:06 |
| SergeyLukjanov | sdague, :) | 12:07 |
| sdague | oh, how long until puppet puts it in place? | 12:10 |
| SergeyLukjanov | sdague, I think that puppet will update the config file in a half of hour and then we should see something | 12:10 |
| sdague | ok | 12:10 |
| sdague | yeh, I did a recheck early | 12:10 |
| SergeyLukjanov | sdague, revert is already prepared :) https://review.openstack.org/#/c/78943/ | 12:10 |
| saper | fungi: oh I definitely do have, thanks :) | 12:11 |
| *** david-lyle has quit IRC | 12:12 | |
| sdague | SergeyLukjanov: cool | 12:12 |
| sdague | biab | 12:14 |
| SergeyLukjanov | sdague, heh, I'm trying to catch all issue to rename savanna smoothly... | 12:14 |
| *** sdague has quit IRC | 12:18 | |
| *** lcostantino has joined #openstack-infra | 12:20 | |
| *** lcestari_ has quit IRC | 12:21 | |
| *** sdague has joined #openstack-infra | 12:22 | |
| *** ArxCruz has joined #openstack-infra | 12:25 | |
| *** aysyanne has joined #openstack-infra | 12:28 | |
| *** gokrokve has joined #openstack-infra | 12:33 | |
| *** lcestari_ has joined #openstack-infra | 12:35 | |
| *** gokrokve has quit IRC | 12:38 | |
| *** smarcet has joined #openstack-infra | 12:38 | |
| *** malini_afk is now known as malini | 12:41 | |
| *** e0ne has joined #openstack-infra | 12:41 | |
| *** malini is now known as malini_afk | 12:41 | |
| *** talluri has quit IRC | 12:45 | |
| *** CaptTofu has joined #openstack-infra | 12:48 | |
| *** saju_m has quit IRC | 12:49 | |
| SergeyLukjanov | sdague, we still have some devstack-precise slaves on rax https://jenkins04.openstack.org | 12:50 |
| SergeyLukjanov | sdague, but I think that config should be already applied... and no new slaves should be added | 12:50 |
| *** CaptTofu has quit IRC | 12:53 | |
| sdague | ok | 12:54 |
| SergeyLukjanov | sdague, hm https://jenkins01.openstack.org/job/check-tempest-dsvm-savanna-neutron/228/ | 12:54 |
| sdague | so our fail rate should drop a lot? | 12:54 |
| SergeyLukjanov | sdague, Build has been executing for 16 min on devstack-precise-rax-iad-2311051 ^^ | 12:54 |
| SergeyLukjanov | sdague, I hope yes | 12:54 |
| *** CaptTofu has joined #openstack-infra | 12:55 | |
| SergeyLukjanov | sdague, do we have an issue for rechecks? | 12:56 |
| sdague | SergeyLukjanov: no | 12:56 |
| SergeyLukjanov | sdague, ok, will do | 12:57 |
| *** yaguang has quit IRC | 12:57 | |
| sdague | SergeyLukjanov: hmm... I see to still be getting rax nodes | 13:00 |
| sdague | oh well, fungi will be up soon | 13:00 |
| *** yamahata has quit IRC | 13:00 | |
| *** proffalk1n is now known as ProfFalken | 13:00 | |
| SergeyLukjanov | sdague, yup, probably, I'm not right that it's enough to disable rax slaves | 13:01 |
| SergeyLukjanov | the other option is that puppet just was disabled on nodepool server due to the problems with providers | 13:03 |
| SergeyLukjanov | sdague, I've opened an issue for it https://bugs.launchpad.net/openstack-ci/+bug/1289345 | 13:03 |
| SergeyLukjanov | the only thing that we now is that images on rax updated and nothing works now, yup? | 13:04 |
| *** banix has quit IRC | 13:04 | |
| *** julim has joined #openstack-infra | 13:06 | |
| *** malini_afk is now known as malini | 13:08 | |
| *** weshay has quit IRC | 13:08 | |
| *** ociuhandu has joined #openstack-infra | 13:08 | |
| *** malini is now known as malini_afk | 13:09 | |
| *** rfolco has joined #openstack-infra | 13:09 | |
| sdague | SergeyLukjanov: yeh | 13:10 |
| SergeyLukjanov | sdague, mostly all tempest-dsvm jobs are starting on hpcloud last 10 mins | 13:10 |
| sdague | well we know the error, it's an import of greenthreads from eventlet | 13:10 |
| *** weshay has joined #openstack-infra | 13:10 | |
| SergeyLukjanov | sdague, probably, there are some problems with installation of them... lack of python-dev in os | 13:11 |
| SergeyLukjanov | for ex. | 13:11 |
| sdague | https://jenkins07.openstack.org/job/check-tempest-dsvm-full/3181/console | 13:11 |
| *** kiall_ is now known as Kiall | 13:12 | |
| *** jgallard has joined #openstack-infra | 13:12 | |
| *** sandywalsh has joined #openstack-infra | 13:16 | |
| rcarrillocruz | hey guys, i'm working in some stuff on gerritbot , for some reason it won't start | 13:20 |
| rcarrillocruz | http://paste.openstack.org/show/72860/ | 13:20 |
| rcarrillocruz | any hints? | 13:20 |
| rcarrillocruz | i've stepped into the DaemonContext call and the process does not fork | 13:20 |
| rcarrillocruz | pleia2: you were working on gerritbot these days, are you able to run 'python bot.py <blah config>' just fine? | 13:21 |
| *** pdmars has joined #openstack-infra | 13:21 | |
| *** pdmars has quit IRC | 13:24 | |
| *** mbacchi has joined #openstack-infra | 13:25 | |
| *** CaptTofu has quit IRC | 13:25 | |
| *** mfink has joined #openstack-infra | 13:28 | |
| *** pbelanyi has joined #openstack-infra | 13:33 | |
| *** gokrokve has joined #openstack-infra | 13:33 | |
| *** sandywalsh has quit IRC | 13:35 | |
| *** gokrokve has quit IRC | 13:37 | |
| *** jcoufal has joined #openstack-infra | 13:38 | |
| *** medieval1_ has joined #openstack-infra | 13:39 | |
| sdague | hmm... so rax network is kind of gone to wonkiness eh? | 13:39 |
| sdague | I'm trying to replicate this on my rax credit, and complete fail | 13:40 |
| *** medieval1 has quit IRC | 13:41 | |
| *** alexpilotti has joined #openstack-infra | 13:42 | |
| *** medieval1_ has quit IRC | 13:43 | |
| SergeyLukjanov | sdague, quite strange... probably latest nodepool slave image is just broken | 13:45 |
| SergeyLukjanov | sdague, I still see rax slaves... | 13:46 |
| sdague | yeh | 13:46 |
| sdague | SergeyLukjanov: you have nodepool logs from rax rebuild to point at? | 13:46 |
| *** rcarrillocruz1 has joined #openstack-infra | 13:46 | |
| sdague | I wonder if with the weird network issues that rax seems to have if something broke in it, and it didn't get caught | 13:46 |
| *** sandywalsh has joined #openstack-infra | 13:47 | |
| SergeyLukjanov | sdague, I don't see any errors in http://nodepool.openstack.org/image.log | 13:47 |
| SergeyLukjanov | sdague, except installation of perl-Error module ;) | 13:48 |
| *** rcarrillocruz has quit IRC | 13:49 | |
| *** medieval1 has joined #openstack-infra | 13:49 | |
| *** reaper has joined #openstack-infra | 13:50 | |
| *** jcoufal has quit IRC | 13:51 | |
| sdague | yeh, the image is broken, on a base PHVM image I'm fine | 13:53 |
| sdague | so we probably need to actually test the nodepool images before they get put into rotation | 13:54 |
| sdague | oh, wait, fungi on vacation today? | 13:54 |
| *** thuc has joined #openstack-infra | 13:55 | |
| *** CaptTofu has joined #openstack-infra | 13:55 | |
| *** rcarrillocruz has joined #openstack-infra | 13:59 | |
| *** thomasem has joined #openstack-infra | 13:59 | |
| sileht | sdague, SergeyLukjanov does https://bugs.launchpad.net/tempest/+bug/1289255 is a duplicate of the rax image issue ? | 14:00 |
| sdague | sileht: yes | 14:00 |
| *** CaptTofu has quit IRC | 14:00 | |
| sileht | sdague, thx, marked as duplicate | 14:01 |
| *** banix has joined #openstack-infra | 14:01 | |
| *** thuc_ has joined #openstack-infra | 14:01 | |
| *** rcarrillocruz1 has quit IRC | 14:01 | |
| *** thuc_ has quit IRC | 14:04 | |
| *** thuc_ has joined #openstack-infra | 14:04 | |
| *** thuc has quit IRC | 14:04 | |
| *** rcarrillocruz1 has joined #openstack-infra | 14:04 | |
| *** rcarrillocruz has quit IRC | 14:05 | |
| *** flaper87 is now known as flaper87|afk | 14:05 | |
| *** bowdengl has joined #openstack-infra | 14:06 | |
| *** thuc_ has quit IRC | 14:08 | |
| *** thuc has joined #openstack-infra | 14:09 | |
| fungi | all broken? | 14:09 |
| fungi | no, not on vacation | 14:09 |
| sdague | yes | 14:09 |
| fungi | wow, lookitthat... so what got released this time? | 14:10 |
| sdague | https://bugs.launchpad.net/openstack-ci/+bug/1289345 | 14:10 |
| sdague | nothing that I can figure out | 14:10 |
| sdague | everything I know I just put in that bug | 14:10 |
| fungi | yep, reading | 14:10 |
| sdague | SergeyLukjanov and I tried ineffectively to disable rax nodes in nodepool | 14:10 |
| *** wchrisj has joined #openstack-infra | 14:11 | |
| *** flaper87|afk is now known as flaper87 | 14:11 | |
| fungi | or killing and restarting my browser since it seems to ignore me. so the theory is that rackspace did something? | 14:12 |
| sdague | johnthetubaguy: confirmed the monthly image refresh happened late on wed | 14:12 |
| *** thuc has quit IRC | 14:12 | |
| fungi | fun! | 14:13 |
| *** pblaho has quit IRC | 14:13 | |
| sdague | which means I'm fairly confident this is the first nodepool image based on that | 14:13 |
| *** pdmars has joined #openstack-infra | 14:13 | |
| johnthetubaguy | sdague: that image certainly got updated anyways | 14:13 |
| fungi | i will take a quick look and then delete this morning's nodepool images if so | 14:13 |
| *** andreaf has quit IRC | 14:13 | |
| *** CaptTofu has joined #openstack-infra | 14:13 | |
| sdague | fungi: if you can get me on a node, that would be helpful | 14:14 |
| fungi | yep | 14:14 |
| fungi | wow, our bug is the only hit on google for "ImportError: cannot import name greenthread" too. nice! | 14:14 |
| sdague | hah | 14:14 |
| *** VINOD has quit IRC | 14:14 | |
| *** thuc has joined #openstack-infra | 14:15 | |
| sdague | so my only guess, which I can't confirm, is there are stale pyc files in that image | 14:15 |
| sdague | that are f'ing something up | 14:15 |
| fungi | did anybody diff the pip freeze yet? | 14:15 |
| sdague | I couldn't find anything obvious | 14:15 |
| sdague | also | 14:15 |
| sdague | if it was upstream, it would fail on my rax node | 14:16 |
| *** VINOD has joined #openstack-infra | 14:16 | |
| fungi | yeah | 14:16 |
| sdague | so I'm pretty confident this is a bad image | 14:16 |
| sdague | realistically, it took us multiple spins on the PVHVM image before there was a working one | 14:17 |
| sdague | so it would not surprise me that someone made a change they thought was innocent, and it wasn't | 14:17 |
| *** thuc has quit IRC | 14:17 | |
| fungi | sdague: ssh jenkins@162.242.229.73 | 14:18 |
| *** yamahata has joined #openstack-infra | 14:18 | |
| fungi | sdague: wait, that ones probably a bad example | 14:18 |
| fungi | py3k node. let me grab a better one | 14:19 |
| *** mriedem has joined #openstack-infra | 14:19 | |
| fungi | we want a devstack-precise | 14:19 |
| sdague | yeh | 14:19 |
| *** thuc has joined #openstack-infra | 14:19 | |
| *** rajeshlal has quit IRC | 14:20 | |
| fungi | sdague: ssh jenkins@23.253.54.35 | 14:20 |
| *** thuc has quit IRC | 14:21 | |
| *** thuc_ has joined #openstack-infra | 14:21 | |
| sdague | is that done doing things? | 14:22 |
| fungi | looks like it's still running https://jenkins01.openstack.org/job/gate-murano-devstack-dsvm/2/console | 14:22 |
| sdague | yeh, I see dpkg in top | 14:22 |
| *** thuc_ has quit IRC | 14:22 | |
| fungi | jenkins estimates it's got another 6 minutes to go | 14:22 |
| fungi | short devstack job | 14:22 |
| *** che-arne has joined #openstack-infra | 14:23 | |
| fungi | but the node is held indefinitely so it's not getting deleted | 14:23 |
| SergeyLukjanov | ooops, sorry, my irc client crushed some time ago :( | 14:24 |
| SergeyLukjanov | fungi, morning | 14:24 |
| *** thuc has joined #openstack-infra | 14:24 | |
| *** thuc has quit IRC | 14:24 | |
| fungi | SergeyLukjanov: howdy | 14:24 |
| SergeyLukjanov | fungi, was https://review.openstack.org/#/c/78942/ totally incorrect? | 14:25 |
| SergeyLukjanov | fungi, I already have a revert for it ;) https://review.openstack.org/#/c/78943/ | 14:25 |
| fungi | well, there's no need to reduce the min-ready on those | 14:26 |
| fungi | but mainly the reason it did nothing is because puppet's still disabled on nodepool yesterday since i wanted to get https://review.openstack.org/78639 reviewed and merged so that the local configuration changes yesterday's emergency brought about wouldn't get automatically undone | 14:27 |
| sdague | oh, ok | 14:27 |
| sdague | so it would have worked | 14:27 |
| sdague | except we were already in a degraded state | 14:27 |
| *** thuc has joined #openstack-infra | 14:27 | |
| SergeyLukjanov | fungi, oh, that was one of the my options :() | 14:27 |
| fungi | yeah, it would have also lowered the reserve node count unnecessarily, but that probably wouldn't have been a substantial impact to overall performance | 14:28 |
| SergeyLukjanov | so, I'm going to return back min-ready | 14:28 |
| *** hashar has joined #openstack-infra | 14:28 | |
| *** jungleboyj has quit IRC | 14:28 | |
| fungi | SergeyLukjanov: well, if we get confirmation that it's a problem with the base image this morning's images were built from, i can just delete those images and nodepool will fall back to using the good ones from yesterday | 14:28 |
| sdague | fungi: so 2 things | 14:29 |
| fungi | so at this point we don't necessarily need to disable rackspace in the config | 14:29 |
| sdague | 1 - can we test the images before we promote them? | 14:29 |
| SergeyLukjanov | fungi, and new savanna name was clarified, so, I'd like to ask you for renaming this weekend, /me will make needed CRs tonight | 14:29 |
| *** thuc has quit IRC | 14:29 | |
| sdague | because this has happened a few times in the past | 14:29 |
| fungi | sdague: it's certainly possible. would just need to be an additional feature in nodepool | 14:29 |
| sdague | 2 - can we +A this - https://review.openstack.org/#/c/78939/ | 14:29 |
| sdague | so at least the failure in node pool will happen when people are around | 14:30 |
| sdague | I realize jeblair also needs to sign off on that | 14:30 |
| fungi | sdague: reasoning seems sound to me. not approving it right this moment since that's not something that's going to help innediately in the present situation | 14:30 |
| sdague | but I'd really like to stop assuming nodepool image update doesn't need people around when it happens | 14:31 |
| fungi | yep, makes sense | 14:31 |
| sdague | fungi: and yeh, the +A was more a +2 question. I just don't want to lose that. | 14:31 |
| sdague | fungi: which review is running on this node? | 14:31 |
| fungi | sdague: https://review.openstack.org/78969 | 14:32 |
| *** pbelanyi_ has joined #openstack-infra | 14:32 | |
| fungi | it completed a couple minutes ago | 14:32 |
| *** wchrisj has quit IRC | 14:33 | |
| sdague | fungi: gah, that's murano | 14:33 |
| *** gokrokve has joined #openstack-infra | 14:33 | |
| *** VINOD has quit IRC | 14:33 | |
| sdague | so it's not anything like a real devstack setup :) | 14:33 |
| ruhe | sdague: yes, murano devstack is currently broken | 14:34 |
| ruhe | problem is on murano side | 14:34 |
| *** pbelanyi has quit IRC | 14:34 | |
| sdague | ruhe: yeh, http://logs.openstack.org/69/78969/1/check/gate-murano-devstack-dsvm/80196e7/console.html#_2014-03-07_14_21_08_394 | 14:34 |
| sdague | you don't have a functional test dir | 14:34 |
| fungi | okay, i'll try to find one which is closer to integrated gate tests. problem is nodepool just tells me node type, so i have to cross-reference the node id to one of 7 jenkins masters to find out what job it got assigned and hold it before the job completes | 14:34 |
| sdague | fungi: sure | 14:34 |
| ruhe | sdague: patch with functests is still on review. there are some internal blockers | 14:35 |
| sdague | fungi: yeh, there is no devstack setup on that | 14:35 |
| *** dims_ has quit IRC | 14:36 | |
| miqui | gerrit question: how do you prevent cloning via https ? | 14:36 |
| fungi | sdague: how about the cells job for this devstack change? https://review.openstack.org/78971 | 14:36 |
| *** VINOD has joined #openstack-infra | 14:36 | |
| sdague | cells doesn't setup tempest | 14:37 |
| sdague | https://jenkins03.openstack.org/job/check-tempest-dsvm-full/4716/console | 14:37 |
| sdague | can yuo do that one ? | 14:37 |
| *** prad_ has joined #openstack-infra | 14:37 | |
| *** gokrokve has quit IRC | 14:38 | |
| SergeyLukjanov | fungi, https://review.openstack.org/78973 - revert of min-ready for devstack-precise slaves | 14:38 |
| sdague | fungi: oh, that was one other data point | 14:38 |
| sdague | removing site_packages from tox.ini on tempest made things pass | 14:38 |
| sdague | https://jenkins03.openstack.org/job/check-tempest-dsvm-full/4716/console - yeh, that's failing, so should be doing cleanup | 14:39 |
| sdague | if we catch it quick | 14:39 |
| fungi | sdague: ssh jenkins@23.253.111.206 | 14:39 |
| fungi | that's the one you asked for | 14:40 |
| sdague | fungi: great | 14:40 |
| *** CaptTofu has quit IRC | 14:41 | |
| fungi | and yeah, if taking site_packages out of tox.ini "fixes" it, then it's almost certainly either something already installed on the system or something in rackspace's ubuntu mirror which isn't the same in hp's | 14:41 |
| fungi | have we been seeing unit tests failing similarly, or just devstack/tempest jobs? | 14:43 |
| *** rlandy has quit IRC | 14:46 | |
| *** yamahata has quit IRC | 14:46 | |
| fungi | oh, though in positive news, hpcloud-az2 just allowed us to get further on image builds and not fail out almost immediately, so maybe they finally fixed the issue in the past few minutes which has been plaguing us there for weeks | 14:46 |
| sdague | so my normal work around, which is to run nose to find the import error, doesn't work | 14:47 |
| sdague | because nose is working | 14:47 |
| sdague | how I wish testr had an even vaguely sane UI for these issues | 14:47 |
| *** VINOD has quit IRC | 14:48 | |
| *** dims_ has joined #openstack-infra | 14:48 | |
| fungi | with the default python there, under my account, i can 'from eventlet import greenthread' successfully, so there must be a deeper issue in the virtualenv<->system libs interaction | 14:49 |
| *** VINOD has joined #openstack-infra | 14:49 | |
| sdague | fungi: ok, I have a trace | 14:50 |
| fungi | oh! | 14:50 |
| sdague | http://paste.openstack.org/show/72873/ | 14:50 |
| sdague | thats with the venv | 14:50 |
| fungi | cffi! | 14:51 |
| sdague | which works fine not in the venv | 14:51 |
| fungi | yeah, it's using system-installed cffi | 14:51 |
| fungi | does moving /usr/local/lib/python2.7/dist-packages/cryptography/hazmat/bindings/__pycache__/_cffi__x5eaa210axf0ae7e21.c out of the way fix it? | 14:52 |
| sdague | right, so the issue is that tempest is using tox | 14:52 |
| sdague | which we really shouldn't be | 14:52 |
| sdague | going to have to unwind that whole thing | 14:52 |
| sdague | because we should use system libraries | 14:52 |
| sdague | fungi: it's a No Such File issue | 14:53 |
| fungi | so... does devstack pip-install things like cffi into the system? | 14:53 |
| fungi | because pip freeze in the system context claims cffi is installed, and it's definitely not there from a deb | 14:53 |
| *** dcramer_ has joined #openstack-infra | 14:54 | |
| sdague | why are you sure it's not there from a deb? | 14:54 |
| *** malini_afk is now known as malini | 14:54 | |
| *** wchrisj has joined #openstack-infra | 14:54 | |
| sdague | fungi: yeh, it's in /usr/local | 14:54 |
| sdague | so pip installed it | 14:54 |
| sdague | but it's 0.8.1 in /usr/local | 14:55 |
| sdague | and 0.8.2 in tox | 14:55 |
| fungi | sdague: first appears in raring... precise doesn't seem to have a python-cffi package (even in backports), so unless it used to be called something else, not there from the deb | 14:56 |
| fungi | and dpkg doesn't think it's installed, and it's rooting around in /usr/local, yeah | 14:56 |
| *** pblaho has joined #openstack-infra | 14:56 | |
| fungi | so all signs point to pip installation | 14:56 |
| fungi | just wondering if the base image from rackspace has pip-installed things or if devstack did that on its own | 14:57 |
| *** dansmith is now known as damnsmith | 14:57 | |
| *** jpeeler has quit IRC | 14:57 | |
| *** wenlock has joined #openstack-infra | 14:57 | |
| *** VINOD has quit IRC | 14:57 | |
| sdague | yeh 0.8.2 is just broken | 14:58 |
| *** VINOD has joined #openstack-infra | 14:58 | |
| *** mfer has joined #openstack-infra | 14:58 | |
| fungi | i suspect that's the one they rushed to release to deal with setuptools 3.x? | 14:59 |
| fungi | uploaded on: 2014-03-07 | 14:59 |
| *** khyati has joined #openstack-infra | 15:00 | |
| fungi | so we need to cffi!=0.8.2 in requirements? | 15:00 |
| *** CaptTofu has joined #openstack-infra | 15:01 | |
| fungi | i can also try to roll us back to yesterday's nodepool images in rax if you think it will help | 15:01 |
| *** VINOD has quit IRC | 15:01 | |
| *** jnoller has joined #openstack-infra | 15:01 | |
| sdague | yeh, honestly, I'd cap them for now, as I have no faith they'll release a new good one later | 15:01 |
| *** VINOD has joined #openstack-infra | 15:02 | |
| *** dizquierdo has joined #openstack-infra | 15:02 | |
| sdague | though block is probably fine | 15:02 |
| jeblair | g'morning | 15:02 |
| *** jungleboyj has joined #openstack-infra | 15:02 | |
| fungi | top o' the mornin' to ya jeblair! | 15:03 |
| SergeyLukjanov | jeblair, howdy | 15:03 |
| fungi | can you smell the broken cffi release all up in our servers? | 15:03 |
| *** rcarrillocruz has joined #openstack-infra | 15:04 | |
| *** wenlock has quit IRC | 15:04 | |
| fungi | sdague: new question... can you explain how we're using cffi 0.8.2 in tox when it's not in our pypi mirror? http://pypi.openstack.org/openstack/cffi/ | 15:04 |
| sdague | fungi: nope | 15:04 |
| *** rpodolyaka has joined #openstack-infra | 15:05 | |
| fungi | i'm digging into that one myself | 15:05 |
| sdague | tox -e full | 15:05 |
| jeblair | fungi: i assume they fixed the broken setuptools thing by releasing a broken thing to distract us | 15:05 |
| fungi | jeblair: that seems to be the long and short of it | 15:05 |
| sdague | fungi: you know, we're hopping users for tempest | 15:05 |
| sdague | so the environment is probably reset | 15:05 |
| sdague | the mirror is controlled by environment, right? | 15:05 |
| jeblair | sdague: what broke with nodepool images? | 15:05 |
| fungi | sdague: i *think* we override the mirror config for jenkins, stack and tempest accounts, but worth double-checking in devstack-gate | 15:06 |
| sdague | jeblair: well, now I'm slightly less convinced it's the image | 15:06 |
| sdague | fungi: where would I see that on this node | 15:06 |
| sdague | fungi: yeh, I'm definitely not hitting the mirror | 15:06 |
| sdague | (full)tempest@devstack-precise-rax-dfw-2316847:~$ pip install -U cffi | 15:06 |
| sdague | Downloading/unpacking cffi from https://pypi.python.org/packages/source/c/cffi/cffi-0.8.2.tar.gz#md5=37fc88c62f40d04e8a18192433f951ec | 15:06 |
| *** rcarrillocruz1 has quit IRC | 15:06 | |
| jeblair | sdague: d-g writes a bunch of pip config files in the accounts' homedirs | 15:07 |
| fungi | sdague: yep, i don't see the dotfiles i'd expect for that in ~tempest | 15:07 |
| jeblair | ~/.pip/pip.conf | 15:07 |
| sdague | there is no .pip | 15:07 |
| fungi | and ~/.pydistutils.cfg | 15:07 |
| jeblair | in jenkins, stack, and tepest | 15:07 |
| fungi | and right, ~tempest is missing them on this host i held | 15:08 |
| sdague | jeblair: nope, definitely not for tempest | 15:08 |
| *** sarob_ has joined #openstack-infra | 15:08 | |
| fungi | so maybe something is recreating or moving the tempest user's home after we add those? | 15:08 |
| sdague | the tempest home dir is a git clone | 15:08 |
| jeblair | sdague: link to the original run for the host? | 15:08 |
| *** pbelanyi_ has quit IRC | 15:08 | |
| sdague | fungi: yuo have it? | 15:08 |
| *** pbelanyi_ has joined #openstack-infra | 15:08 | |
| jeblair | sdague: oh. that might explain that. | 15:08 |
| sdague | jeblair: it's up the backlog somewhere | 15:09 |
| jeblair | http://logs.openstack.org/10/77710/3/gate/gate-tempest-dsvm-large-ops/1608dbf/logs/devstacklog.txt.gz ? | 15:09 |
| fungi | https://jenkins03.openstack.org/job/check-tempest-dsvm-full/4716/console | 15:09 |
| jeblair | fungi: thx | 15:09 |
| fungi | http://logs.openstack.org/71/78971/1/check/check-tempest-dsvm-full/a961c0e/ | 15:10 |
| fungi | http://logs.openstack.org/71/78971/1/check/check-tempest-dsvm-full/a961c0e/logs/devstack-gate-setup-host.txt.gz#_2014-03-07_14_26_34_524 | 15:10 |
| jeblair | sdague: so i suspect that's getting eradicated when the tempest git repo updates | 15:10 |
| sdague | there are other non tracked files in there | 15:11 |
| fungi | sudo -u root cp /home/jenkins/.pip/pip.conf /root/.pip/pip.conf | 15:11 |
| fungi | oh, wait, wrong one | 15:11 |
| fungi | sudo cp /home/jenkins/.pip/pip.conf /opt/stack/new/tempest/.pip/pip.conf | 15:11 |
| *** bnemec is now known as beekneemech | 15:11 | |
| *** jpeeler has joined #openstack-infra | 15:12 | |
| *** jpeeler has quit IRC | 15:12 | |
| *** jpeeler has joined #openstack-infra | 15:12 | |
| fungi | does something delete and re-clone that directory though? | 15:12 |
| jeblair | 2014-03-07 14:27:39.157 | + git clean -x -f -d -q | 15:12 |
| jeblair | sdague: ^ | 15:12 |
| fungi | ahh, yep | 15:12 |
| jeblair | not after that | 15:12 |
| jeblair | sdague: so untracked files will have been created after that | 15:12 |
| jeblair | http://logs.openstack.org/71/78971/1/check/check-tempest-dsvm-full/a961c0e/logs/devstack-gate-setup-workspace-new.txt.gz#_2014-03-07_14_27_39_157 | 15:12 |
| *** lcostantino has quit IRC | 15:12 | |
| jeblair | that's the log filen for that | 15:12 |
| *** sarob_ has quit IRC | 15:12 | |
| sdague | oh http://logs.openstack.org/71/78971/1/check/check-tempest-dsvm-full/a961c0e/logs/devstack-gate-setup-workspace-new.txt.gz#_2014-03-07_14_27_34_112 | 15:13 |
| fungi | so it turns out that tempest has not been installing its virtualenvs restricted to our mirror for... probably quite a long time now | 15:13 |
| sdague | right, I just find the clean | 15:13 |
| sdague | correct | 15:13 |
| *** pbelanyi_ is now known as pbelanyi | 15:13 | |
| sdague | so, honestly, long term fix is tempest needs to not use a virtual env at all | 15:13 |
| jeblair | what expects the git repo to be at ~tempest? how hard would it be to make ~tempest /home/tempest? | 15:13 |
| sdague | this has been bothering me for a while | 15:13 |
| sdague | jeblair: honestly, I don't think anything expects it | 15:14 |
| fungi | short-term fix, tempest can move and restore those two files (or exclude them in the clean, even better) | 15:14 |
| sdague | so creating tempest home as /home/tempest I think is fine | 15:14 |
| sdague | where is that done? puppet or d-g? | 15:14 |
| jeblair | d-g; i'll take a stab at it | 15:15 |
| fungi | yeah, if nothing in tempest is expecting to call things relative to its own homedir, that should be a trivial solution | 15:15 |
| fungi | i'll go ahead and get a cffi!=0.8.2 going | 15:15 |
| sdague | fungi: I pushed it up already | 15:15 |
| sdague | https://review.openstack.org/#/c/78982/ | 15:15 |
| fungi | oh, all's the better | 15:15 |
| sdague | Alex_Gaynor: correctly asked if we reported it upstream | 15:16 |
| fungi | i guess that means gerritbot is on walkabout again | 15:16 |
| fungi | i'll go find him | 15:16 |
| *** VINOD has quit IRC | 15:16 | |
| sdague | As I've been debugging this for the last 5 hrs, that's not going to happen for me | 15:16 |
| *** lcostantino has joined #openstack-infra | 15:16 | |
| *** VINOD has joined #openstack-infra | 15:16 | |
| *** Guest31645 is now known as antonym | 15:17 | |
| sdague | I'm done in about 2 more hours for the day, yesterday was far too long of a day | 15:17 |
| *** antonym has quit IRC | 15:17 | |
| *** antonym has joined #openstack-infra | 15:17 | |
| Alex_Gaynor | sdague: I dont' hvae any of the details on this issue, can you please file it? | 15:17 |
| fungi | Alex_Gaynor: we'll get someone to put together a short writeup of how to reproduce it and find the cffi bug reporter | 15:18 |
| Alex_Gaynor | bitbucket.org/cffi/cffi | 15:18 |
| *** coolsvap has joined #openstack-infra | 15:19 | |
| fungi | Alex_Gaynor: thanks! | 15:19 |
| *** jeblair is now known as jegerritbot | 15:19 | |
| jegerritbot | James E. Blair proposed a change to devstack-gate: https://review.openstack.org/78987 | 15:19 |
| *** jegerritbot is now known as jeblair | 15:19 | |
| fungi | bwahaha | 15:19 |
| damnsmith | jeblair: nice | 15:19 |
| fungi | jerritbot would also be acceptable | 15:19 |
| fungi | so he went missing on a netsplit and never returned... | 15:20 |
| fungi | 2014-03-07 02:35:23 <-- openstackgerrit (~openstack@review.openstack.org) has quit (*.net *.split) | 15:20 |
| *** dripton_ is now known as dripton | 15:20 | |
| *** openstackgerrit has joined #openstack-infra | 15:22 | |
| fungi | AttributeError: 'NoneType' object has no attribute 'send' | 15:22 |
| fungi | our old friend | 15:22 |
| openstackgerrit | James E. Blair proposed a change to openstack-infra/gerritbot: Don't crash on UTF8 errors https://review.openstack.org/78989 | 15:22 |
| jeblair | fungi: ah, yeah, that one should be fixed by the upgrade to 8.5.4 | 15:23 |
| jeblair | fungi: and that ^ should make it safe to upgrade to 8.5.4 | 15:23 |
| sdague | fungi: so you know what it might actually be, which isn't 0.8.2 per say. It's the fact that it's 2 different cffi, and pycrypto was built with the old one | 15:23 |
| jeblair | so hopefully we'll be in a better place soon. but definitely a different place. :) | 15:23 |
| fungi | sdague: sounds plausible | 15:24 |
| fungi | sdague: that would also explain why disabling site packages "fixes" the error | 15:24 |
| sdague | fungi: yep | 15:24 |
| sdague | which actually means the requirements block wouldn't help | 15:25 |
| sdague | if this is true, this is actually a much bigger issue | 15:25 |
| sdague | I'm going to start corrupting that failed node to test something | 15:25 |
| fungi | sdague: feel free--it's all yours until you're done with it | 15:25 |
| *** pcrews has joined #openstack-infra | 15:26 | |
| *** gokrokve has joined #openstack-infra | 15:26 | |
| sdague | yeh | 15:27 |
| fungi | jeblair: does irclib perform its own pidfile management? does that explain why the pidfile goes missing from /var/run after a minute or two? | 15:27 |
| sdague | so if I install cffi 0.8.2 on the system, same issue | 15:28 |
| sdague | so basically the repeat is the following | 15:28 |
| sdague | pip install cffi==0.8.1 | 15:28 |
| fungi | sdague: so it's that pycrypto needs to == the version it built with | 15:28 |
| sdague | pip install pycrypto | 15:28 |
| sdague | pip install eventlet | 15:28 |
| sdague | pip install cffi==0.8.2 | 15:28 |
| sdague | python | 15:28 |
| *** rcarrillocruz has quit IRC | 15:28 | |
| sdague | from eventlet import greenthread | 15:28 |
| *** rcarrillocruz has joined #openstack-infra | 15:29 | |
| fungi | _boom_ | 15:29 |
| sdague | which may be just a general structural problem with cffi | 15:30 |
| sdague | that we only started hitting since our required pycrypto required it | 15:30 |
| jeblair | fungi: no i think it's our standard daemon module stuff, however, it's possible that 'bot.start' actually returns; i'll look into it | 15:30 |
| *** rcarrillocruz1 has joined #openstack-infra | 15:31 | |
| sdague | jeblair: the d-g patch running? | 15:31 |
| jeblair | sdague: https://jenkins06.openstack.org/job/check-tempest-dsvm-full/3341/ | 15:31 |
| *** ryanpetrello has quit IRC | 15:31 | |
| *** luisg has joined #openstack-infra | 15:32 | |
| *** ryanpetrello has joined #openstack-infra | 15:32 | |
| *** wenlock has joined #openstack-infra | 15:33 | |
| *** rcarrillocruz has quit IRC | 15:33 | |
| fungi | looks like it's made it past where we were failing previously | 15:35 |
| fungi | sdague: i'm still sort of curious why it was only exhibiting on rax nodes... coincidence you think? | 15:36 |
| SergeyLukjanov | fungi, clarkb, jeblair, mordred, sorry... could you please remove one wiki page for me at wiki.o.o? | 15:36 |
| *** mgagne has joined #openstack-infra | 15:36 | |
| sdague | fungi: no idea actually | 15:36 |
| fungi | SergeyLukjanov: sure, which one? | 15:36 |
| sdague | so.... pypi is a CDN now right? | 15:37 |
| SergeyLukjanov | fungi, https://wiki.openstack.org/wiki/Sahara please | 15:37 |
| SergeyLukjanov | fungi, it blocks me to just move the Savanna page with all subpages :( | 15:37 |
| openstackgerrit | Antoine Musso proposed a change to openstack-infra/jenkins-job-builder: Migrate to pbr https://review.openstack.org/76200 | 15:37 |
| sdague | which means maybe the HP cloud's hitting a CDN mirror not yet updated | 15:37 |
| sdague | and it would just explode later | 15:37 |
| jeblair | hashar: ooh nice! :) | 15:38 |
| sdague | fungi: you want to find me an HP node to log into? | 15:38 |
| fungi | sdague: yes. however i suspect that the several rax and hp regions/azs we're using would be traversing different parts of the internet and hitting different parts of the cdn network | 15:38 |
| fungi | sdague: sure. just a sec | 15:38 |
| sdague | fungi: yeh, not sure | 15:38 |
| *** bowdengl has quit IRC | 15:38 | |
| hashar | jeblair: I found out I will never understand setuptools/distutils/whatever.. But can manage to write a package from scratch using pbr :-] | 15:38 |
| SergeyLukjanov | hashar, heh, I understand you :) | 15:39 |
| *** VINOD has quit IRC | 15:39 | |
| fungi | hashar: you should tweeter that on the twitternets and blame mordred for unleashing your packages upon the internet ;) | 15:39 |
| jeblair | mordred has made the world a better place with pbr! | 15:40 |
| hashar | yeah it is all monty fault | 15:40 |
| hashar | I am no more writing PHP as a result :/ | 15:40 |
| jeblair | hashar: that's pretty much our motto around here | 15:40 |
| fungi | SergeyLukjanov: deleted | 15:40 |
| SergeyLukjanov | fungi, thank you! | 15:40 |
| jeblair | hashar: that's the best thing i've heard all day! | 15:40 |
| *** david-lyle has joined #openstack-infra | 15:40 | |
| mordred | hashar: yay! | 15:41 |
| jeblair | i mean, i've only been up for 40 minutes but still | 15:41 |
| fungi | hashar: i take that as a major improvement. the less php on the internet the better ;) | 15:41 |
| hashar | so Wikimedia as an internal cabal which is all about getting rid of PHP | 15:42 |
| hashar | it is fighting with the HipHop Virtual Machine Cabal (which is pushing the Facebook implementation of PHP) | 15:43 |
| hashar | and the CoolKid I do Javascript on the backend cabal | 15:43 |
| fungi | how can i funnel my wikimedia foundation donations directly to that cabal? particularly to its beer fund? | 15:43 |
| hashar | fungi: easiest is to meet us face to face I guess :-] | 15:43 |
| hashar | more seriously gotta rebase that pbr change | 15:43 |
| fungi | hashar: i'll just have to make sure to do that some time | 15:43 |
| hashar | fungi: if you are in the bay area, I can give you a few names and you can then show up at the office :] | 15:44 |
| lifeless | russellb: whats up with reviewstats? | 15:44 |
| fungi | sdague: ssh jenkins@15.185.250.188 | 15:45 |
| SergeyLukjanov | fungi, can I have access to slaves too? | 15:46 |
| fungi | sdague: https://jenkins04.openstack.org/job/check-tempest-dsvm-neutron-full/1522/console is running on that one in hpcloud-az3 | 15:47 |
| fungi | hashar: i'm not in the san-francisco bay area (other side of that continent) | 15:47 |
| fungi | otherwise i definitely would ;) | 15:47 |
| fungi | SergeyLukjanov: what are you wanting to test? | 15:49 |
| fungi | i'm currently working on reproducing it on my debian/testing workstation here at home, just for confirmation | 15:50 |
| lifeless | fungi: btw I thought of another way to solve that hung socket problem | 15:51 |
| lifeless | fungi: change the nodepool config to remove the cloud; let it reconfigure; put the cloud back | 15:51 |
| *** mgagne has quit IRC | 15:51 | |
| fungi | lifeless: perhaps. does nodepool shut down hung connections when it reconfigures and sees the provider missing in its config? | 15:52 |
| *** mgagne has joined #openstack-infra | 15:52 | |
| lifeless | fungi: I'm thinking the socket might be left leaked but a new provider object with noew client and new socket | 15:52 |
| lifeless | fungi: new nova client, that is | 15:52 |
| fungi | lifeless: worth trying next time, if there is a next time | 15:52 |
| fungi | sdague: i'm also unable to reproduce this in a venv if there's no cffi installed system-wide | 15:54 |
| fungi | sdague: which is consistent with the site packages thing | 15:54 |
| *** cody-somerville has joined #openstack-infra | 15:54 | |
| sdague | fungi: so the issue is 2 need to be installed? | 15:54 |
| fungi | sdague: possibly--i'm testing that now | 15:55 |
| sdague | aha! | 15:55 |
| sdague | hp nodes have python-openssl installed at a package level | 15:57 |
| mordred | why? | 15:57 |
| sdague | no idea | 15:57 |
| sdague | but it protects us some how | 15:57 |
| mordred | also - has anybody looked at the new split-out libvirt stuff yet? like, so that we can turn off site-packages in nova? | 15:57 |
| fungi | mordred: i believe that hinges on uca, which dims is still trying to get working | 15:58 |
| mordred | uca | 15:59 |
| fungi | not sure if his current uca headaches are regressions in new libvirt or something unrelated at this point though | 15:59 |
| mordred | gotcha. just making sure that wasn't waiting on me in some crazy way | 15:59 |
| sdague | fungi: so the hp nodes install python-openssl | 16:01 |
| fungi | mordred: no, there's been plenty of back-and-forth with openstack devs, ubuntu devs and libvirt devs trying to find a working combination i think | 16:01 |
| sdague | which is at a sufficient level for the rest of openstack | 16:01 |
| sdague | so we don't pip install it | 16:01 |
| sdague | rax doesn't | 16:01 |
| sdague | so we're pip installing it | 16:01 |
| sdague | apparently binding it to the specific cffi | 16:01 |
| *** sarob_ has joined #openstack-infra | 16:01 | |
| fungi | sdague: yeah, the lack of glance coupled with providers all liking to roll their own images leads to frustrating inconsistencies like that | 16:02 |
| openstackgerrit | Antoine Musso proposed a change to openstack-infra/jenkins-job-builder: Migrate to pbr https://review.openstack.org/76200 | 16:02 |
| mordred | sdague: I don't see a reference to python-openssl in our repo | 16:02 |
| sdague | mordred: in which repo? | 16:02 |
| mordred | config | 16:02 |
| fungi | mordred: pre-installed on the base images, i think is the assertion | 16:02 |
| mordred | yes | 16:02 |
| sdague | mordred: yeh, I'm assuming base image | 16:02 |
| mordred | I'm agreeing with that | 16:02 |
| fungi | ahh, right-o | 16:02 |
| mordred | what we could do is _add_ it to our base slave config so that the nodes match | 16:03 |
| sdague | sure | 16:03 |
| sdague | or even devstack | 16:04 |
| *** thedodd has joined #openstack-infra | 16:04 | |
| mordred | yah | 16:04 |
| fungi | why does virtualenv --system-site-packages imply --no-pip or --no-setuptools in 1.11? | 16:04 |
| mordred | btw - rax has requestable glance endpoints now too | 16:04 |
| sdague | but that's not a root issue | 16:04 |
| sdague | it just hides the fail | 16:04 |
| mordred | yah | 16:04 |
| mordred | fungi: because you probably have pip and setuptools in system-site-packages? (/me is guessing) | 16:04 |
| sdague | the real issue is something about python openssl & cffi doesn't actually venv correctly | 16:04 |
| fungi | mordred: probably... oh--i need to activate the venv then to use them. now i get it | 16:04 |
| sdague | and someone with more python-fu would need to sort that | 16:05 |
| fungi | i'm used to directly referencing stuff in ./venv/bin/ | 16:05 |
| jeblair | mordred: ooh nice! does hpcloud 1.1? | 16:05 |
| mordred | jeblair: yes | 16:05 |
| jeblair | mordred: wow, so we should be able to use glance rsn | 16:05 |
| mordred | yup | 16:05 |
| mordred | I'm quite excited about that - for several reasons - not the least of which being not needing to care about our cloud providers batshit ideas of what shoudl be in base images | 16:05 |
| *** sarob_ has quit IRC | 16:05 | |
| jeblair | assuming hpcloud has enough the 12tb of ram needed for all our nodes. | 16:05 |
| jeblair | s/enough// | 16:05 |
| *** hashar has quit IRC | 16:05 | |
| mordred | yah | 16:05 |
| fungi | gah! trying to install pycrypto inside a venv with --system-site-packages enabled also breaks horribly | 16:05 |
| fungi | "error: could not create '/usr/local/lib/python2.7/dist-packages/Crypto': Permission denied" | 16:05 |
| fungi | well, yeah, DUH | 16:05 |
| *** hashar has joined #openstack-infra | 16:05 | |
| *** oubiwan__ has joined #openstack-infra | 16:05 | |
| fungi | that looks familiar too. perhaps debian/jessee has a broken virtualenv version packaged | 16:06 |
| fungi | 1.11.1 may have fixed that | 16:06 |
| *** russellb is now known as rustlebee | 16:06 | |
| damnsmith | lifeless: I think it was running out of memory on the VM it was using and he took it down until he can fix it for real | 16:06 |
| damnsmith | should we be keeping the gate queue full for when this gets corrected, or is it better to wait? | 16:06 |
| *** markwash has joined #openstack-infra | 16:06 | |
| *** dizquierdo has quit IRC | 16:09 | |
| *** Guest73507 has quit IRC | 16:09 | |
| sdague | jeblair: apparently your fix didn't work - https://jenkins07.openstack.org/job/check-grenade-dsvm/2520/console | 16:09 |
| sdague | that's on grenade | 16:09 |
| sdague | damnsmith: so this is actually something we need to fix in d-g | 16:09 |
| *** VINOD_ has joined #openstack-infra | 16:09 | |
| sdague | so everything's going to bounce until we get the right fix in | 16:09 |
| fungi | sdague: jeblair: looks like that run failed out, maybe on something unrelated | 16:09 |
| sdague | fungi: no, it's actually the same error | 16:09 |
| sdague | at least on the grenade run | 16:09 |
| sdague | so that didn't apparently change the home dir correctly | 16:09 |
| sdague | or something | 16:09 |
| *** dizquierdo_ has joined #openstack-infra | 16:09 | |
| fungi | "FAIL: tempest.api.compute.servers.test_server_actions.ServerActionsTestJSON.test_create_backup[gate]" | 16:09 |
| fungi | that? | 16:09 |
| *** Ryan_Lane has joined #openstack-infra | 16:09 | |
| *** Ryan_Lane has joined #openstack-infra | 16:09 | |
| sdague | no | 16:09 |
| sdague | https://jenkins07.openstack.org/job/check-grenade-dsvm/2520/console | 16:09 |
| damnsmith | sdague: okay, just want to know what the proper action on our part is.. I'll spread the word, thanks | 16:09 |
| fungi | sdague: oh, maybe we have a similar problem in grenade then? the check-tempest-dsvm-full made it much further | 16:10 |
| jeblair | http://logs.openstack.org/87/78987/1/check/check-grenade-dsvm/ef0b700/logs/ | 16:10 |
| jeblair | the useradd cmd looks correct there | 16:10 |
| jeblair | http://logs.openstack.org/87/78987/1/check/check-grenade-dsvm/ef0b700/logs/devstack-gate-setup-host.txt.gz#_2014-03-07_15_34_14_074 | 16:11 |
| *** markmcclain has joined #openstack-infra | 16:11 | |
| jeblair | 2014-03-07 15:34:14.260 | + sudo cp /home/jenkins/.pip/pip.conf /home/tempest/.pip/pip.conf | 16:12 |
| jeblair | and that looks good | 16:12 |
| *** vkozhukalov has joined #openstack-infra | 16:12 | |
| sdague | jeblair: are you sure that HOME is getting set correctly when the sudo happens? | 16:13 |
| jeblair | sdague: oh, possibly not | 16:13 |
| *** alaski is now known as lascii | 16:13 | |
| *** markmcclain1 has joined #openstack-infra | 16:13 | |
| openstackgerrit | Antoine Musso proposed a change to openstack-infra/jenkins-job-builder: Migrate to pbr https://review.openstack.org/76200 | 16:13 |
| jeblair | sudo -H -u stack ./grenade.sh | 16:14 |
| jeblair | that should do it | 16:14 |
| sdague | no, it's the tempest call | 16:14 |
| jeblair | sudo -H -u tempest tox -eall -- --concurrency=$TEMPEST_CONCURRENCY | 16:15 |
| jeblair | looks like -H is set for all the tempest variants | 16:15 |
| *** markmcclain has quit IRC | 16:16 | |
| sdague | ok, I'm out of ideas. If yuo can get a grenade node on rax held, can look | 16:16 |
| sdague | fungi: or trigger a mirror update, then at least we'd be dealing with 1 cffi version | 16:17 |
| sdague | we'd break again on this in the future, but it would let code flow | 16:17 |
| sdague | and it is FFE time | 16:17 |
| fungi | of course the pip freeze at the end of the grenade run is not for the virtualenv tempest used, so says cffi==0.8.1 | 16:18 |
| sdague | fungi: yep | 16:18 |
| sdague | I'm sort of at the point that what we have is a ton of different work arounds, and the right fix, which is stop using tox for tempest | 16:18 |
| sdague | which is going to take a week | 16:18 |
| *** cody-somerville has quit IRC | 16:19 | |
| *** nicedice has joined #openstack-infra | 16:19 | |
| sdague | so we should just implement whatever work around makes stuff work again fastest if it's going to hold for a week | 16:19 |
| *** markwash_ has joined #openstack-infra | 16:21 | |
| fungi | the remaining two jobs are wrapping up for that change now, so i can recheck it and hold the grenade job workers | 16:21 |
| fungi | should only be a couple minutes | 16:21 |
| *** markwash has quit IRC | 16:21 | |
| *** markwash_ is now known as markwash | 16:21 | |
| openstackgerrit | James E. Blair proposed a change to openstack-infra/devstack-gate: Use the default home directory location for tempest. https://review.openstack.org/78987 | 16:22 |
| jeblair | spamming for nodes ^ | 16:23 |
| *** freyes has joined #openstack-infra | 16:24 | |
| *** rcleere has joined #openstack-infra | 16:24 | |
| fungi | i'll grab them in nodepool as soon as they get assigned and get sdague's keys added | 16:24 |
| *** thuc_ has joined #openstack-infra | 16:24 | |
| *** thuc_ has quit IRC | 16:24 | |
| jeblair | we only want the grenade node, right? | 16:24 |
| fungi | the non-grenade jobs didn't seem to fail from this issue, right | 16:25 |
| *** basha has joined #openstack-infra | 16:25 | |
| fungi | yeah i guess we only have one grenade job for d-g changes anyway | 16:25 |
| *** jlibosva has quit IRC | 16:26 | |
| *** thuc has joined #openstack-infra | 16:28 | |
| *** thuc has quit IRC | 16:29 | |
| *** thuc has joined #openstack-infra | 16:29 | |
| *** thuc has quit IRC | 16:29 | |
| fungi | sdague: ssh jenkins@15.185.165.19 | 16:30 |
| *** zns has joined #openstack-infra | 16:30 | |
| *** thuc has joined #openstack-infra | 16:31 | |
| *** thuc has quit IRC | 16:31 | |
| fungi | oh, also, the bare-precise and devstack-precise loops i had running in hpcloud-az2 finally successfully built images, both completed successfully just over an hour ago, so the nodes there should be using fresh scripts now | 16:31 |
| jeblair | fungi, sdague: that's an hpcloud node; is that useful for us, or does it need to be rax? | 16:31 |
| jeblair | fungi: and yay about az2! | 16:31 |
| fungi | jeblair: it may not break because of the deb-installed pycrypto, right | 16:31 |
| fungi | jeblair: on az2, yeah, suspicious that they both started working at the same moment... i strongly suspect clarkb has an update on that ticket waiting in his inbox | 16:32 |
| *** thuc has joined #openstack-infra | 16:32 | |
| jeblair | fungi: should we roll again? | 16:32 |
| *** zns has quit IRC | 16:32 | |
| fungi | jeblair: i think so | 16:32 |
| fungi | you want to re-push? | 16:32 |
| openstackgerrit | James E. Blair proposed a change to openstack-infra/devstack-gate: Use the default home directory location for tempest https://review.openstack.org/78987 | 16:32 |
| fungi | watching for the new one | 16:33 |
| *** thuc has quit IRC | 16:33 | |
| fungi | looks like it should get a node assignment at any moment | 16:33 |
| fungi | oh, maybe not. it's still about 25 nodes behind | 16:33 |
| *** markmcclain1 has quit IRC | 16:34 | |
| sdague | jeblair: yeh, it needs to be rax | 16:34 |
| *** jgallard has quit IRC | 16:34 | |
| sdague | jeblair: so, honestly, why don't we kick off the mirror script instead? | 16:35 |
| sdague | that will fix this | 16:35 |
| sdague | and as long as cffi doesn't release again in the next 7 days, once we can gut tox from tempest, we'll be fine | 16:35 |
| *** thuc has joined #openstack-infra | 16:35 | |
| *** thuc has quit IRC | 16:35 | |
| fungi | sdague: i still think this is a good opportunity to get a better understanding of what's happening with grenade which isn't respecting the mirror overrides | 16:36 |
| fungi | that worries me a lot more than cffi | 16:36 |
| sdague | fungi: sure, except we're about to stop having tempest user download packages | 16:36 |
| *** e0ne has quit IRC | 16:37 | |
| jeblair | sdague: will that affect our ability to do cross-release testing? | 16:37 |
| sdague | jeblair: I don't think so | 16:37 |
| *** zns has joined #openstack-infra | 16:37 | |
| fungi | i can certainly start the mirror update, but would like to be sure we've got a solid example held first | 16:38 |
| sdague | we install the new requirements globally | 16:38 |
| jeblair | fungi: az2 again :( | 16:38 |
| fungi | that one landed in hpcloud too, jeblair | 16:38 |
| rainya | general question: where is the storyboard project information / repo / dev team? | 16:38 |
| *** pblaho has quit IRC | 16:38 | |
| openstackgerrit | James E. Blair proposed a change to openstack-infra/devstack-gate: Use the default home directory location for tempest. https://review.openstack.org/78987 | 16:38 |
| rainya | and good regional time of the day to all | 16:38 |
| fungi | rainya: in #storyboard, and at http://ci.openstack.org/storyboard/ | 16:38 |
| rainya | fungi, thank you sir | 16:38 |
| jeblair | rainya: and http://git.openstack.org/cgit/openstack-infra/storyboard | 16:39 |
| fungi | rainya: any time! | 16:39 |
| rainya | gorgeous | 16:39 |
| jeblair | rainya: and most exciting of all: http://storyboard.openstack.org/ | 16:39 |
| sdague | fungi: sure, there it's just that we're in the FFE period, and we have thus far failed over 1000 jobs on this | 16:39 |
| *** amotoki has quit IRC | 16:39 | |
| *** thuc has joined #openstack-infra | 16:39 | |
| sdague | and basically blocked a full .eu day of development | 16:40 |
| rainya | jeblair, fungi: not sure if you all have any info yet from the operator meetup this week or if you all were there, but the #1 thing I have heard is about blueprint reviews in gerrit and that the nova team is on board with that as well; it seemed I heard / read / saw that storyboard was goign to tackle blueprints first | 16:40 |
| jeblair | sdague: 'we' did no such thing. | 16:40 |
| openstackgerrit | David Ostrovsky proposed a change to openstack-infra/config: Gerrit-2.8: Add secondary index support https://review.openstack.org/60080 | 16:40 |
| *** thuc has quit IRC | 16:40 | |
| *** mrodden has quit IRC | 16:40 | |
| sdague | we in the royal sense | 16:40 |
| sdague | because a mirror update will make this go away | 16:40 |
| fungi | i admit it... it was me and the mouse in my pocket | 16:40 |
| *** dizquierdo_ has quit IRC | 16:41 | |
| jeblair | sdague: i'm absolutely not taking any responsibility for problems caused by python projects releasing new versions of things. | 16:41 |
| *** gyee has joined #openstack-infra | 16:41 | |
| sdague | jeblair: that's fine | 16:41 |
| jeblair | sdague: i'm happy to help fix this, as i think i'm demonstrating | 16:41 |
| fungi | i would at least like to know if we have a larger hole in the homedir management in grenade jobs before we go papering over it just to get everything rolling again | 16:41 |
| jeblair | fungi: ++ | 16:41 |
| sdague | sure, that's cool, I'm just done with my work day in about 10 minutes. Was hoping to know the gate was flowing before that | 16:42 |
| *** thuc has joined #openstack-infra | 16:42 | |
| fungi | rax! holding now | 16:42 |
| *** thuc has quit IRC | 16:42 | |
| sdague | so what other info do you need from me before I disappear | 16:42 |
| *** reaper has quit IRC | 16:43 | |
| fungi | 23.253.73.168 is the new node | 16:43 |
| fungi | sdague: i can stick your keys on there real quick if you want to dig around on it in the next 10 minutes | 16:43 |
| openstackgerrit | David Ostrovsky proposed a change to openstack-infra/config: Activate scondary index support https://review.openstack.org/79005 | 16:43 |
| *** reed has joined #openstack-infra | 16:44 | |
| fungi | added you to the jenkins user just in case | 16:44 |
| fungi | i'll go ahead and start up a mirror run now if there are no objections | 16:44 |
| *** rpodolyaka has quit IRC | 16:44 | |
| jeblair | fungi: should be okay | 16:45 |
| fungi | triggered | 16:46 |
| *** thuc has joined #openstack-infra | 16:46 | |
| *** sweston has joined #openstack-infra | 16:46 | |
| *** thuc has quit IRC | 16:46 | |
| *** thuc has joined #openstack-infra | 16:46 | |
| fungi | the overrides are definitely in /home/tempest and that's reflected as the proper homedir in /etc/passwd too | 16:47 |
| fungi | jenkins says we're about 25 minutes out from updated cffi on the mirror | 16:49 |
| *** basha has quit IRC | 16:50 | |
| *** yamahata has joined #openstack-infra | 16:50 | |
| *** vkozhukalov has quit IRC | 16:50 | |
| *** chandan_kumar has quit IRC | 16:51 | |
| *** thuc has quit IRC | 16:51 | |
| *** mrodden has joined #openstack-infra | 16:52 | |
| *** MarkAtwood has joined #openstack-infra | 16:54 | |
| *** yassine has quit IRC | 16:56 | |
| *** jooools has quit IRC | 16:56 | |
| *** blamar has joined #openstack-infra | 16:59 | |
| *** mriedem1 has joined #openstack-infra | 16:59 | |
| *** comstud is now known as bearhands | 17:00 | |
| *** e0ne has joined #openstack-infra | 17:01 | |
| *** mriedem has quit IRC | 17:02 | |
| sdague | fungi: things look like they are passing gate now | 17:02 |
| sdague | so new cffi must have landed | 17:02 |
| openstackgerrit | Antoine Musso proposed a change to openstack-infra/jenkins-job-builder: Fix doc error in the testng publisher https://review.openstack.org/79012 | 17:02 |
| openstackgerrit | Antoine Musso proposed a change to openstack-infra/jenkins-job-builder: Migrate to pbr https://review.openstack.org/76200 | 17:02 |
| sdague | ok, and with that I'm off. Hopefully the rest of the day is less eventful :) | 17:02 |
| fungi | sdague: looks like it... http://pypi.openstack.org/openstack/cffi/ | 17:03 |
| fungi | i see 0.8.2 there now | 17:03 |
| fungi | though my mirror run wasn't the trigger | 17:03 |
| fungi | oh, or maybe it was and the console just hasn't reflected that | 17:04 |
| fungi | still thinks the job has 11 minutes left, so i doubt it's to the rsync stage yet | 17:04 |
| fungi | yeah, the sqlalchemy-migrade 0.9 release triggered an earlier run | 17:05 |
| *** rpodolyaka has joined #openstack-infra | 17:05 | |
| fungi | sqlalchemy-migrate | 17:05 |
| *** ildikov_ has quit IRC | 17:07 | |
| mordred | rainya: we had some brief discussions about storyboard/gerrit/blueprint design yesterday - but it's still a new thought so we don't have a specific plan | 17:08 |
| mordred | rainya: I believe that if we use gerrit for the discussion of and approval of the design of a blueprint | 17:08 |
| mordred | rainya: we can still use storyboard to track the work needed on that | 17:08 |
| *** mrodden has quit IRC | 17:08 | |
| mordred | rainya: so, for instance, a blueprint approval in gerrit could trigger a story creation in storyboard - since now that it's been approved, someone ahs to work on the darned thing :) | 17:09 |
| *** mrodden has joined #openstack-infra | 17:10 | |
| *** jogo is now known as flashgordon | 17:10 | |
| *** dangers_away is now known as dangers | 17:11 | |
| *** pbelanyi has quit IRC | 17:12 | |
| *** rwsu has joined #openstack-infra | 17:12 | |
| fungi | i could imagine storyboard growing the ability to have its display embed content from specially-named files in certain repositories for things like that | 17:13 |
| *** harlowja has joined #openstack-infra | 17:13 | |
| fungi | if it went that route | 17:13 |
| zaro | jeblair: question about gerrit upgrade. | 17:14 |
| jeblair | mordred: i think storyboard should attempt to solve the problem people are trying to use gerrit for with blueprints | 17:15 |
| *** Hefeweizen has joined #openstack-infra | 17:16 | |
| zaro | jeblair: thought came into my head. not sure if we have decided to got with WIP plugin or WIP vote. if it's the vote then should we think about converting the WIP changes to Workflow -1 ? | 17:16 |
| mordred | jeblair: really? I kind of really like the gerrit blueprint idea myself | 17:16 |
| mordred | jeblair: but I'm not wedded to anything obviously | 17:17 |
| *** gokrokve has quit IRC | 17:17 | |
| jeblair | mordred: i think the gerrit blueprint idea is more about the fact that you can't discuss and iteratively develop something in launchpad blueprints. if storyboard ends up not being able to discuss and iteratively develop stories, i'm not sure we've made something useful. | 17:17 |
| fungi | one up-side is that you don't have to grow your own revision tracking and approval mechanisms if you just tie to files in a code review system | 17:17 |
| *** SumitNaiksatam has quit IRC | 17:18 | |
| mordred | jeblair: it's the voting/approval aspect I like about re-using gerrit for this | 17:18 |
| fungi | however, i can understand people wanting to consume storyboard without wanting to have a code review system (outside of openstack's use case) | 17:18 |
| jeblair | fungi: revision tracking true, though some kind of approval system needs to be there | 17:18 |
| *** hashar has quit IRC | 17:18 | |
| *** oubiwan__ has quit IRC | 17:19 | |
| jeblair | becaues if you can't set a blueprint to approved and target a milestone (substituto appropriate storyboard words there), or set a bug to invalid or complete (ditto), then, again, it's not useful | 17:19 |
| SergeyLukjanov | jeblair, my dream is to have a button "release this milestone" | 17:20 |
| jeblair | SergeyLukjanov: ttx's dream too, i believe. | 17:20 |
| SergeyLukjanov | I think ttx dreams about it too | 17:20 |
| *** VINOD_ has quit IRC | 17:20 | |
| SergeyLukjanov | yup :) | 17:20 |
| *** VINOD_ has joined #openstack-infra | 17:21 | |
| smarcet | fungi: hi, where i cant get the full list of supported api version per release as depicted here https://ask.openstack.org/en/question/2096/openstack-api-versions-and-release-content/ but for all releases/components/versions ? | 17:21 |
| jeblair | mordred: i mean, at some point, if we wanted to put everything in git, maybe we should use http://syncwith.us/sd/ instead of storyboard | 17:22 |
| fungi | smarcet: i'm not really a guru on openstack apis, but dtroyer might have some good suggestions | 17:22 |
| *** tchaypo is now known as jammyjam | 17:22 | |
| smarcet | fungi: ok tnx , will ask him :) | 17:22 |
| mordred | I don't want to put everythign in git | 17:23 |
| mordred | I think that's a bad idea | 17:23 |
| mordred | :) | 17:23 |
| jeblair | (actually i think sd is a really good idea) | 17:23 |
| smarcet | dtroyer: hi, where i cant get the full list of supported api version per release as depicted here https://ask.openstack.org/en/question/2096/openstack-api-versions-and-release-content/ but for all releases/components/versions ? | 17:23 |
| jeblair | (but not what we need) | 17:23 |
| *** rpodolyaka is now known as malor | 17:23 | |
| *** jgrimm has joined #openstack-infra | 17:24 | |
| mordred | I just think there is an interesting idea about the process aroudn writing a thing and the process of tracking work on that thing being different | 17:24 |
| * mordred back ina sec | 17:24 | |
| *** NobodyCam is now known as BlartVersenwald | 17:24 | |
| *** _david_ has joined #openstack-infra | 17:24 | |
| *** vkozhukalov has joined #openstack-infra | 17:25 | |
| *** thuc has joined #openstack-infra | 17:25 | |
| jeblair | mordred: i agree; however if storyboard is merely a status tracking system, i think it's far less compelling. we want to use it for three distinct tasks that we do now: bug tracking, feature design, and ui/ux design. (and part of the goal is to somewhat merge those activities) | 17:26 |
| dtroyer | smarcet: I don't know of a single place that has been gathered, although it would be nice to have. | 17:26 |
| openstackgerrit | gordon chung proposed a change to openstack-infra/config: enable doc generation for pycadf https://review.openstack.org/75998 | 17:26 |
| *** thuc has quit IRC | 17:26 | |
| jeblair | mordred: all of those activities involve collaboration and evolution | 17:26 |
| jeblair | mordred: so to be effective, i think storyboard needs to support that all around | 17:27 |
| smarcet | dtroyer: well trying to do that :), but need some references, i got the former link as reference for keystone | 17:27 |
| dtroyer | TBH, I think it is more useful to be able to query the version from a particular cloud by hitting the '/' route for an endpoint. IIRC that works for all of the integrated projects now and has for a few releases | 17:27 |
| smarcet | dtroyer: needs the rest of the components | 17:27 |
| *** thuc has joined #openstack-infra | 17:27 | |
| dtroyer | as for references, the change logs may be your best bet wihtout resorting to reading source code | 17:27 |
| jeblair | mordred: in particular, consider that while it's very easy to write a text-only blueprint in gerrit, it's extremely difficult to attach a diagram. and it's completely useless for ui mockups | 17:27 |
| fungi | dtroyer: i'm going to guess smarcet's question ties into the defcore/refstack planning, so possibly figuring out how to tell what apis a given cloud deployment *should* support, possibly introspectively but probably more proscriptively | 17:28 |
| jeblair | mordred: so if we punt all of the collaboration aspects into gerrit, then we leave ux and anything visual out in the cold. and i'm pretty sure we decided that supporting ux as first-class was an important goal. | 17:28 |
| rainya | jeblair, i moved the conversation to #storyboard, sorry | 17:29 |
| *** nati_ueno has joined #openstack-infra | 17:29 | |
| *** salv-orlando has joined #openstack-infra | 17:29 | |
| dtroyer | good to know…is that work tied to specific releases? and how does that apply to deployments tracking trunk? That's why I was thinking to validate what the deployment claims to support… | 17:29 |
| smarcet | fungi: yeah basically its for the marketplace feature, trying to establishing some rules so user cant enter garbage when they select components->releases->api_version | 17:29 |
| *** gokrokve has joined #openstack-infra | 17:29 | |
| rainya | jeblair, and while i have so much "free" time with life and stuff, it's a compelling system to be designed and implemented with gerrit and storyboard and <other things we haven't thought of yet> | 17:30 |
| dtroyer | smarcet: so that is different, it sounds like you really just need a recorded history | 17:30 |
| rainya | i am going to the beach this week (spring break!) and then south korea next week (conference) and then woot! it's on | 17:30 |
| jeblair | rainya: i'm not at all suggesting that it shouldn't be closely tied. i advocate that quite strongly | 17:30 |
| smarcet | dtroyer: yeah something like that | 17:30 |
| *** fbo is now known as fbo_away | 17:31 | |
| *** gokrokve has quit IRC | 17:31 | |
| rainya | jeblair, yep! got that completely :) i'm interesting in seeing how the pieces fit together on paper and crafting the story for the system because that's the part of software I like best | 17:31 |
| rainya | interested | 17:31 |
| dtroyer | change logs/release notes would be my start point for that | 17:31 |
| jeblair | rainya: however, i think it should fully support the most important use cases we identified it needs to cover; blueprints and ux design/review are among those | 17:32 |
| *** krotscheck has joined #openstack-infra | 17:32 | |
| dtroyer | or buy dolphm some $COLD_BEVERAGE and ask him to do the other projects too ;) | 17:32 |
| smarcet | dtroyer: got it, tnx :) | 17:33 |
| rainya | jeblair, phil from rackspace who you spoke with last week and Vek (kevin) have some ideas around an openstack-service-like system that would help us internally at Rackspace, we've decided we need to set up gerrit to get our internal iteration cycle working properly, the operaters and nova project are on board with gerrit for blueprints, and storyboard is already in progress and something the community can own | 17:33 |
| fungi | dtroyer: smarcet: so if the goal is mainly input validation/restriction for selection widgets in the marketplace interface, perhaps limiting them to the lists of api versions which actually exist or have existed is good enough, rather than trying to match them up against openstack integrated releases (because as you've seen, the api versions tend to overlap, change at different times for different | 17:33 |
| fungi | components, and don't necessarily follow releases, nor do deployers necessarily deploy releases) | 17:33 |
| rainya | jeblair: finding the best way through all of that to, as you say, support the most important use cases is a fascinating tangle for me | 17:33 |
| smarcet | fungi: got it | 17:33 |
| jeblair | rainya: yeah, i think using gerrit for blueprints is a great idea, but it's important to understand that is being proposed because of significant failures in launchpad's implementation of blueprints, many of which we have already identified and are a significant motivation for creating storyboard | 17:34 |
| jeblair | rainya: i think we should proceed with that, and in parallel, learn what we can from it and apply it to storyboard to make sure that storyboard can do what it needs to do. | 17:35 |
| pleia2 | rcarrillocruz1: might be a shot in the dark, but does the PID from the last time the bot was running still exist? | 17:36 |
| *** jpich has quit IRC | 17:37 | |
| *** vponomaryov has quit IRC | 17:37 | |
| pleia2 | for the peanut gallery: rcarrillocruz1 is having trouble with his gerritbot http://paste.openstack.org/show/72860/ | 17:37 |
| rainya | jeblair, +2! it's taken me two years to finally get to a point where I feel like i will have cycles to spend in the openstack-infra program, but i feel like i am finally here; i am not a coder or a particular strong operator, but i don't suck at system design, understanding people, understanding their problems, and helping bring it all together and communicating it to all sorts | 17:38 |
| jeblair | rainya: rock on! | 17:38 |
| reed | jeblair, agreed, I'm afraid that 'moving' the bp to gerrit will take time/energy/motivation off of storyboard | 17:38 |
| fungi | pleia2: we should try running a current gerritbot sometime... is rcarrillocruz1 running from the master branch, or the 1.5-year-old "latest" release? | 17:38 |
| reed | and using gerrit for bp still looks like a hack to me, and a bad one too | 17:39 |
| jeblair | reed: i think the need that nova has is great enough that we should support the effort; i don't think many of the people actually involved in storyboard will need to spend much time on it | 17:39 |
| pleia2 | fungi: didn't say | 17:39 |
| jeblair | reed: and i think we can learn a lot from the process | 17:39 |
| rainya | jeblair, it also helps that rackspace's infrastrcuture deployments team (my team) is finally reachign the point where the scale of the openstack infra in the public cloud is making life miserable and since you guys already solve a lot of those problems, good time to (1) ask for input / guidance / advice (2) contribute back as we iterate and improve | 17:40 |
| jeblair | reed, ttx: btw if you see fifieldt around, ask him to register with nickserv. :) | 17:40 |
| reed | jeblair, yeah, that's why I haven't chimed in the thread :) it's painful | 17:40 |
| *** markmcclain has joined #openstack-infra | 17:40 | |
| reed | jeblair, ACK | 17:40 |
| *** VINOD_ has quit IRC | 17:41 | |
| jeblair | pleia2: also possibly an issue writing to the pidfiel location | 17:42 |
| _david_ | jeblair, moving blueprint to gerrit? What does it mean exactly? | 17:42 |
| pleia2 | jeblair: nod | 17:43 |
| fungi | rainya: i'm sorry to hear that the scale of our infrastructure/testing is making life miserable for rackspace's public cloud! | 17:43 |
| fungi | rainya: we'd like to know how we can help mitigate that too | 17:43 |
| rainya | fungi, not YOUR scale | 17:43 |
| fungi | OH! | 17:43 |
| *** Ryan_Lane1 has joined #openstack-infra | 17:43 | |
| fungi | hee | 17:43 |
| reed | _david_, check the openstack-dev list, there is a loooong thread there | 17:43 |
| rainya | fungi, nope, not even an issue for (that's VW's problem ;P) | 17:44 |
| fungi | nice ;) | 17:44 |
| reed | and Upstream Training in Atlanta is sold out ... wow | 17:44 |
| rainya | fungi, our jenkins infra is overloaded with jobs, upstream dev is still a "however long it takes" process (even though I know the PTLs are doing a LOT of great work to keep it moving with the reviews) and I still have a production product to update as frequently as we can get code ready because people like to get new stuff | 17:45 |
| *** mgagne has quit IRC | 17:45 | |
| *** nati_ueno has quit IRC | 17:46 | |
| *** mgagne has joined #openstack-infra | 17:46 | |
| rainya | fungi, my version of miserable mitigation is cheering / bullying on the object work in nova so control plane and computes can be upgraded indepedently and taking the "let's see if we can run openstack as a public cloud" infrastructure that we've used for the last years and making it into something i don't lose sleep over at night | 17:46 |
| rainya | you guys MIGHT have a few thoughts around how to do that | 17:46 |
| rainya | or, how NOT to do that | 17:46 |
| fungi | rainya: we do some of that, for sure. and for what we don't, we at least get to see some of the pain points | 17:47 |
| *** thomasem has quit IRC | 17:47 | |
| SergeyLukjanov | am I right that problem with rax slaves is solved and it was cffi? | 17:48 |
| mordred | jeblair: I don't think I was suggesting we do all of the collaboration aspects in gerrit at all - I think your point about the ui mockups is well taken | 17:48 |
| fungi | rainya: but the fact that rackspace is pitching in on the drive to non-jenkins test infrastructure is a huge help too, and is much welcomed | 17:48 |
| fungi | SergeyLukjanov: i think the problem is subtle and while triggered by the cffi release isn't necessarily its fault directly | 17:49 |
| mordred | jeblair: purely that, gerrit or not, I think it's an interesting point that the process of developing and deciding upon a potentially complex design is much different than the workflow aroudn accepting or updating the status on a story that needs to be accomplished | 17:49 |
| rainya | fungi, can you clarity "drive to non-jenkins test infrastructure" means to you? too many words and about 25% of the time, i only sound like i know what I'm talking about | 17:49 |
| mordred | rainya: we'll be deleting jenkins as soon as we can | 17:50 |
| jeblair | mordred: i agreed. launchpad completely ignores that for blueprints and does a 'meh' job on bugs. hopefully storyboard will at least do an acceptable job there. :) | 17:50 |
| rainya | mordred, zuul and gearman? | 17:50 |
| mordred | jeblair: ++ | 17:50 |
| mordred | rainya: yes | 17:50 |
| rainya | kk | 17:50 |
| fungi | rainya: mikal and jhesketh's stuff with turbo-hipster (gearman-based job workers with no jenkins) is pretty cool even if just a s a proof-of-concept | 17:50 |
| mordred | rainya: zuul is the important piece of the puzzle | 17:50 |
| rainya | i've got to transition us to "jenkins at scale" in the near term so we have the breathing space to look at zuul and how it fits | 17:51 |
| mordred | rainya: once you get gerrit -> zuul + nodepool - jenkins is just a mostly big dumb box that's not worth a ton | 17:51 |
| *** bhuvan has joined #openstack-infra | 17:51 | |
| *** bhuvan_ has joined #openstack-infra | 17:51 | |
| mordred | rainya: I'd humbly suggest that that's the wrong direction to think of things in | 17:51 |
| mordred | rainya: you will not be able to scale jenkins if you don't get to zuul first | 17:51 |
| mordred | if you get to zuul, then you can scale jenkins out | 17:51 |
| SergeyLukjanov | fungi, looks like we should revert my attempt to disable rax slaves to not forget about it later - https://review.openstack.org/#/c/78943/ | 17:51 |
| mordred | by adding multiple jenkins masters | 17:51 |
| mordred | if you try to scale jenkins first, you will fail | 17:52 |
| fungi | rainya: right, we scaled jenkins by using zuul | 17:52 |
| *** dprince has joined #openstack-infra | 17:52 | |
| rainya | mordred, i'd humbly suggest you haven't lived through my 2013 and while it isn't optimal, where I am at right now is just "be able to add a jenkins slave" | 17:52 |
| mordred | right | 17:52 |
| rainya | so my scale issues are very tiny right now compared | 17:52 |
| mordred | fair | 17:52 |
| jeblair | SergeyLukjanov: oh, why didn't your attempt work? | 17:52 |
| mordred | rainya: you should also look at nodepool then | 17:52 |
| rainya | and the people that would look at zuul are the people that are chasing overloaded jenkins slave | 17:52 |
| rainya | s | 17:52 |
| mordred | rainya: because that's how we manage adding jenkins slaves :) | 17:52 |
| SergeyLukjanov | jeblair, puppet was disabled on nodepool server ;) | 17:53 |
| *** mrodden has quit IRC | 17:53 | |
| jeblair | SergeyLukjanov: :( | 17:53 |
| jeblair | fungi: is it running again now? | 17:53 |
| fungi | jeblair: i've still got puppet disabled because https://review.openstack.org/78780 and https://review.openstack.org/78639 are applied locally | 17:53 |
| fungi | can we merge those and i'll start it back up? | 17:53 |
| rainya | mordred, noted and thanks and will do; once i can add a few extra slaves (we are stupidly under provisioned right now though i don't really have the numbers yet) and get the various services off my back, gooder things will happpppen | 17:53 |
| *** zns has quit IRC | 17:54 | |
| mordred | jeblair: agree - and I agree storyboard should do good on both - I think supporting nova blueprints-in-gerrit for a period of time should not impact storyboard much at all - and might not even be a terrible integration | 17:54 |
| rainya | mordred, my enfp is gonna be all up in that | 17:54 |
| mordred | rainya: awesome | 17:54 |
| fungi | jeblair: basically still riding the coattails of yesterday morning's firedrill | 17:54 |
| fungi | jeblair: so they bled together | 17:54 |
| jeblair | fungi, clarkb: i think if you're going to make a configuration tweak on a server, it's better to just self-approve the change rather than let puppet sit disabled | 17:54 |
| fungi | jeblair: will do from now on | 17:54 |
| mordred | jeblair: ++ | 17:54 |
| bearhands | rainya: ya, i'd love to get us to match upstream... but first things first | 17:54 |
| jeblair | (since you've already made the call that it _must_ be changed anyway, better to have it in the history and accessible to/fixable by everyone) | 17:55 |
| mordred | ++ | 17:55 |
| bearhands | (also: happy to see you in here :) | 17:55 |
| clarkb | morning | 17:55 |
| clarkb | jeblair: sounds good | 17:56 |
| fungi | i've approved SergeyLukjanov's revert instead so those other two patches don't merge-conflict | 17:56 |
| _david_ | fungi, jeblair, clarkb what is wrong here: https://review.openstack.org/#/c/60080/5 | 17:56 |
| fungi | er, approved his revert first (not instead) | 17:56 |
| *** bhuvan__ has joined #openstack-infra | 17:56 | |
| *** bhuvan___ has joined #openstack-infra | 17:56 | |
| rainya | bearhands, yuppers | 17:57 |
| openstackgerrit | A change was merged to openstack-infra/config: Revert "Disable rax provider for devstack-precise slaves" https://review.openstack.org/78943 | 17:57 |
| jeblair | _david_: i think it's because 'secondary_index' and '_type' aren't actually defined as class arguments | 17:59 |
| openstackgerrit | A change was merged to openstack-infra/config: Lower rax-ord max-servers in nodepool to 56 https://review.openstack.org/78780 | 17:59 |
| *** bhuvan has quit IRC | 18:00 | |
| *** bhuvan_ has quit IRC | 18:00 | |
| *** markmcclain has quit IRC | 18:01 | |
| *** bhuvan has joined #openstack-infra | 18:01 | |
| *** bhuvan_ has joined #openstack-infra | 18:01 | |
| *** bhuvan___ has quit IRC | 18:01 | |
| *** bhuvan__ has quit IRC | 18:01 | |
| openstackgerrit | James E. Blair proposed a change to openstack-infra/statusbot: Sleep when joining channels https://review.openstack.org/79033 | 18:03 |
| openstackgerrit | James E. Blair proposed a change to openstack-infra/statusbot: Handle lots of channels better https://review.openstack.org/79034 | 18:04 |
| openstackgerrit | David Ostrovsky proposed a change to openstack-infra/config: Gerrit-2.8: Add secondary index support https://review.openstack.org/60080 | 18:04 |
| openstackgerrit | David Ostrovsky proposed a change to openstack-infra/config: Activate scondary index support https://review.openstack.org/79005 | 18:04 |
| jeblair | clarkb, mordred, fungi, SergeyLukjanov: ^ with those, I think we can start up statusbot again (the first is required because it can't currently join 54 channels; the second will make being on 54 channels a lot nicer) | 18:05 |
| SergeyLukjanov | jeblair, looking on em now | 18:05 |
| jeblair | oh, oops i think i left a debug thing in that one too | 18:05 |
| openstackgerrit | James E. Blair proposed a change to openstack-infra/statusbot: Handle lots of channels better https://review.openstack.org/79034 | 18:06 |
| jeblair | fixed ^ | 18:06 |
| *** SumitNaiksatam has joined #openstack-infra | 18:06 | |
| openstackgerrit | James E. Blair proposed a change to openstack-infra/gerritbot: Don't crash on UTF8 errors https://review.openstack.org/78989 | 18:06 |
| *** thomasem has joined #openstack-infra | 18:06 | |
| jeblair | fungi: the grenade job reported back success :( it must have caught the earlier mirror update | 18:07 |
| *** harlowja has quit IRC | 18:08 | |
| fungi | jeblair: yeah, we'd probably need to artificially recreate those conditions to get the same behavior, though we may be able to construct a simpler example. i'll ponder that | 18:08 |
| *** thomasem has quit IRC | 18:09 | |
| clarkb | I do have an email. It was apparently related to the large number of deleted node records... | 18:09 |
| fungi | clarkb: good to know that we at least helped them spot a potential error condition, even if it took them that long to find it | 18:10 |
| *** harlowja has joined #openstack-infra | 18:15 | |
| *** mrodden has joined #openstack-infra | 18:15 | |
| clarkb | fwiw I disagree about removing tox from tempest. Tempest should run out of a virtualenv | 18:17 |
| clarkb | it isn't necessary in the gate but should be work on principle | 18:18 |
| fungi | i tend to agree, and would be more in favor of getting tempest working without site_packages=true | 18:18 |
| fungi | seems like it would be a serviceable use case, being able to pip install tempest into a virtualenv and then point it at your private cloud or your devstack server in your lab or whatever | 18:20 |
| clarkb | exactly | 18:20 |
| *** _david_ has quit IRC | 18:20 | |
| *** nati_ueno has joined #openstack-infra | 18:20 | |
| jeblair | wth: http://logs.openstack.org/33/79033/1/check/gate-statusbot-pep8/503e3dd/console.html | 18:20 |
| fungi | filesystem error? | 18:21 |
| clarkb | TokenError EOF in multiline statement <- fun | 18:21 |
| fungi | oh, wait | 18:21 |
| *** johnthetubaguy has quit IRC | 18:21 | |
| fungi | so maybe mismatched braces/parens somewhere | 18:22 |
| clarkb | there is a lot of scrollback for friday morning /me reviews the list of changes that were just collected | 18:22 |
| fungi | jeblair: see inline comment | 18:23 |
| jeblair | that doesn't happen for me locally; i'm going to recheck | 18:23 |
| clarkb | after getting through most of the sb, would https://review.openstack.org/#/c/78939/1 have helped anything? | 18:23 |
| *** moted has quit IRC | 18:23 | |
| gyee | anyone seeing this when running tox behind a corporate proxy? Download error on https://pypi.python.org/simple/: [Errno 1] _ssl.c:509: error:140770FC:SSL routines:SSL23_GET_SERVER_HELLO:unknown protocol -- Some packages may not be found! | 18:23 |
| *** moted has joined #openstack-infra | 18:23 | |
| jeblair | fungi: oh, it was the previous change. gotcha, thanks | 18:23 |
| clarkb | I don't necessarily disagree with what that change does but I also thing it is reacting to what we now know was a non issue | 18:23 |
| gyee | though running pip install outside of tox works fine | 18:24 |
| *** e0ne has quit IRC | 18:25 | |
| fungi | clarkb: 78939 would not have helped this particular fire drill, no | 18:25 |
| openstackgerrit | James E. Blair proposed a change to openstack-infra/statusbot: Sleep when joining channels https://review.openstack.org/79033 | 18:25 |
| openstackgerrit | James E. Blair proposed a change to openstack-infra/statusbot: Handle lots of channels better https://review.openstack.org/79034 | 18:25 |
| fungi | clarkb: though i too think the suggestion has merit | 18:25 |
| jeblair | moreso since it now includes things like updates to slave scripts | 18:25 |
| fungi | clarkb: the main down side is rebuilding images during our peak utilization has a higher chance of failing, maybe | 18:26 |
| *** bhuvan_ has quit IRC | 18:26 | |
| *** bhuvan has quit IRC | 18:26 | |
| jeblair | (which before would have been tested on long running hosts) | 18:26 |
| clarkb | jeblair: good point | 18:26 |
| clarkb | fungi: ya especially if there are quota issues | 18:26 |
| SergeyLukjanov | oh, folks, I have question about renaming - will we have a redirect from savanna to sahara on github and git.o.o? | 18:26 |
| clarkb | SergeyLukjanov: we haven't done that in the past | 18:27 |
| fungi | SergeyLukjanov: generally we just send an announcement instructing developers to update their git remotes | 18:28 |
| *** zns has joined #openstack-infra | 18:29 | |
| clarkb | jeblair: how does pep8 play into https://review.openstack.org/#/c/78989/2/gerritbot/bot.py ? | 18:29 |
| jeblair | clarkb: you're not allowed to have urls that are longer than 79 chars | 18:30 |
| jeblair | clarkb: comments even | 18:30 |
| clarkb | oh ha | 18:30 |
| jeblair | yeah, ha. | 18:30 |
| clarkb | indeed | 18:30 |
| fungi | the whole internet breaks | 18:30 |
| fungi | didn't you know? | 18:30 |
| SergeyLukjanov | clarkb, fungi, ok, got it, I just see that http://github.com/openstack/quantum redirects :) | 18:31 |
| jeblair | i'm going to write a letter to google and ask them to make the urls shorter | 18:31 |
| clarkb | 1400 UTC is probably early enough in the day that it is before the major loading | 18:31 |
| openstackgerrit | A change was merged to openstack-infra/gerritbot: Don't crash on UTF8 errors https://review.openstack.org/78989 | 18:31 |
| clarkb | SergeyLukjanov: oh we may have done it for github but we definitely didn'tfor git.o.o | 18:31 |
| fungi | SergeyLukjanov: i think github may do that automatically if we move the repository instead of deleting and letting it get created anew | 18:31 |
| SergeyLukjanov | fungi, yup, I think so | 18:31 |
| fungi | SergeyLukjanov: though that only works for renames within the same organization (so for stackforge->openstack for example it won't happen anyway) | 18:32 |
| SergeyLukjanov | fungi, I hope we'll never need to move savanna to stackforge back :) | 18:32 |
| SergeyLukjanov | sahara* | 18:32 |
| fungi | same here! | 18:33 |
| SergeyLukjanov | I'm thinking about overriding SAVANNA_REPO var in devstack jobs to not break the gate, is in acceptable? | 18:33 |
| fungi | SergeyLukjanov: so on github i think that will end up happening if we follow our usual steps... at http://ci.openstack.org/gerrit.html#renaming-a-project step 12 is "Rename the project in GitHub..." | 18:33 |
| SergeyLukjanov | fungi, nice | 18:34 |
| clarkb | I think I have gotten through all of the emergency related chagne at this point. Do let me know if I missed anything | 18:35 |
| fungi | SergeyLukjanov: i think as long as your job configuration is corrected as part of the config project change for this, we end up applying it before you test anything anyway | 18:35 |
| fungi | SergeyLukjanov: or will that not help because you also have to coordinate a hook modification inside savanna? | 18:35 |
| SergeyLukjanov | fungi, our tests are now in tempest | 18:36 |
| clarkb | fungi: we need to send mail to zhiwei yes? | 18:36 |
| SergeyLukjanov | fungi, I'm afraid that we'll need to coordinate with updating devstack integration | 18:36 |
| SergeyLukjanov | and devstack-gate repo | 18:36 |
| *** amcrn has joined #openstack-infra | 18:37 | |
| fungi | SergeyLukjanov: okay, so we could merge the config change first and the tempest change right after it? | 18:37 |
| fungi | oh, so we need changes to devstack-gate, tempest and config to make it work? | 18:37 |
| SergeyLukjanov | fungi, devstack's one I think and devstack-gate | 18:37 |
| SergeyLukjanov | fungi, tempest should just work, it depends on endpoint that is data_processing | 18:38 |
| fungi | it's possible we could deadlock between tempest and devstack-gate changes in that case, and if so we should force in whichever is the simplest/shortest change to make the other one work | 18:38 |
| SergeyLukjanov | the main issue is that we have SAVANNA_REPO=${SAVANNA_REPO:-${GIT_BASE}/openstack/savanna.git} in devstack | 18:38 |
| *** branen_ has joined #openstack-infra | 18:38 | |
| fungi | SergeyLukjanov: oh, so no tempest change needed, just config and devstack-gate? | 18:38 |
| SergeyLukjanov | fungi, yup, I think it should be enough | 18:38 |
| SergeyLukjanov | fungi, allow new sahara repos in devstack-gate and override SAVANNA_REPO in savanna jobs | 18:39 |
| fungi | SergeyLukjanov: in that case, we merge the config change, then we merge the devstack-gate change, then your change in savanna which is needed to support that, if any | 18:39 |
| SergeyLukjanov | fungi, it'll make me able to update devstack integration and remove override from job then | 18:39 |
| flashgordon | fungi: if you have a minute https://review.openstack.org/#/c/75960/ so I can test out https://review.openstack.org/#/c/78828/ | 18:40 |
| fungi | SergeyLukjanov: a two-step change is acceptable for that if it's the easiest path, in my opinion | 18:40 |
| SergeyLukjanov | fungi, yup, I think I'll add aliases for cli commands in savanna and update devstack integration | 18:40 |
| openstackgerrit | A change was merged to openstack-infra/config: Add bare-centos6 and py3k-precise nodes to hpcloud https://review.openstack.org/78639 | 18:40 |
| SergeyLukjanov | fungi, ok, thanks | 18:40 |
| *** reed has quit IRC | 18:41 | |
| fungi | now that 78639 is in, after 18:45 i should be safe to go ahead and reenable puppet agent on the nodepool server | 18:41 |
| fungi | i'll check with --noop first just to be sure | 18:41 |
| fungi | flashgordon: looking | 18:41 |
| flashgordon | fungi: thanks | 18:42 |
| *** branen has quit IRC | 18:42 | |
| *** markmcclain has joined #openstack-infra | 18:42 | |
| clarkb | flashgordon: re for comment on 75960 nova being broken sounds like a perfect reason for experimental queue :) | 18:42 |
| flashgordon | clarkb: well we know how to fix it | 18:43 |
| flashgordon | clarkb: I hope to have this gating ASAP | 18:43 |
| *** bhuvan has joined #openstack-infra | 18:43 | |
| *** bhuvan_ has joined #openstack-infra | 18:44 | |
| clarkb | flashgordon: yup understood, I will leave my +2 in place | 18:44 |
| openstackgerrit | James E. Blair proposed a change to openstack-infra/statusbot: Handle lots of channels better https://review.openstack.org/79034 | 18:44 |
| flashgordon | clarkb: if this drags out I am all for moving to experimental queue | 18:44 |
| clarkb | jeblair: have you had a chance to see my comments on https://review.openstack.org/#/c/78483/ and its parent? | 18:44 |
| clarkb | fungi: https://review.openstack.org/#/c/78810/1 nice catch on that. Does it need any particular hand holdhing? (I am approving it now) | 18:47 |
| jeblair | clarkb: hrm, i'm not sure irclib supports ssl | 18:47 |
| fungi | clarkb: no hand-holding needed. i'll keep tabs on the gerrit logs to see if it finally starts --dry-run logging that it would have commented | 18:48 |
| jeblair | oh, yeah it does, i'll look into that | 18:48 |
| clarkb | jeblair: they have an example, it is a bit rudiementary but should work | 18:48 |
| *** dprince has quit IRC | 18:48 | |
| fungi | clarkb: i manually tested an equivalent of that edit on review.o.o and it seemed to solve the issue | 18:48 |
| clarkb | jeblair: great | 18:48 |
| clarkb | fungi: I figured you had :) | 18:48 |
| *** MarkAtwood has quit IRC | 18:49 | |
| fungi | i certainly wouldn't have found it otherwise ;) | 18:49 |
| clarkb | fungi: I am going to start prepping for the friday project creation party | 18:50 |
| openstackgerrit | A change was merged to openstack-infra/jeepyb: Welcome message hook query result is an int https://review.openstack.org/78810 | 18:50 |
| clarkb | I think anteaya started an etherpad, do you have that handy? | 18:50 |
| clarkb | anteaya: ^ | 18:51 |
| fungi | clarkb: https://etherpad.openstack.org/p/new-projects-2014-02-28 | 18:51 |
| jeblair | i think anteaya is far far away now | 18:52 |
| clarkb | oh right the flight was last night instead of tonight | 18:53 |
| fungi | clarkb: worth noting, we may want to re-re-re-revert the automatic manage-projects running removal in puppet | 18:53 |
| openstackgerrit | James E. Blair proposed a change to openstack-infra/statusbot: Support SSL https://review.openstack.org/79047 | 18:53 |
| fungi | clarkb: last time i was floating a cherry-pick of it on top of a dev env, but that was a bit of a pain in retrospect | 18:53 |
| clarkb | fungi: to make it not run? | 18:53 |
| * clarkb is trying to count negations and see where we end up | 18:53 | |
| fungi | clarkb: right, so we could merge them all and then run it once | 18:53 |
| fungi | manually | 18:54 |
| clarkb | fungi: I think that is a good idea I will throw it on the etherpad (trying to collect all the infos first) | 18:54 |
| fungi | k | 18:54 |
| fungi | i'll quick-approve that revert so it can get in well ahead | 18:54 |
| clarkb | fungi: https://review.openstack.org/#/c/74130/ is one I think we should just approve so that we don't have to worry about stacking it | 18:55 |
| rainya | oh, fungi, missed the comment about turbo-hipster; even if it is a proof-of-concept, it has saved my team and i from more than one "what were you thinking?" db migrations to our production regions that would have been painful and unnecessary so i'm a fan | 18:55 |
| *** zhiyan_ is now known as zhiyan | 18:55 | |
| fungi | rainya: awesome! | 18:55 |
| * clarkb fixes mordred's change really quickly so that it can go in too | 18:56 | |
| *** oubiwann has quit IRC | 18:57 | |
| openstackgerrit | James E. Blair proposed a change to openstack-infra/gerritbot: Support SSL https://review.openstack.org/79049 | 18:57 |
| openstackgerrit | Clark Boylan proposed a change to openstack-infra/jeepyb: Support resetting back to a non-master branch https://review.openstack.org/77311 | 18:58 |
| *** zhiyan is now known as zhiyan_ | 18:58 | |
| *** bhuvan_ has quit IRC | 19:00 | |
| *** bhuvan has quit IRC | 19:00 | |
| *** bhuvan_ has joined #openstack-infra | 19:01 | |
| *** bhuvan has joined #openstack-infra | 19:01 | |
| *** oubiwann has joined #openstack-infra | 19:02 | |
| *** bada has joined #openstack-infra | 19:02 | |
| openstackgerrit | Clark Boylan proposed a change to openstack-infra/config: Revert "Revert "Stop running manage-projects"" https://review.openstack.org/79050 | 19:03 |
| clarkb | fungi: ^ | 19:03 |
| *** jp_at_hp has quit IRC | 19:03 | |
| jeblair | that means we're stopping, right? | 19:03 |
| clarkb | jeblair: yes | 19:04 |
| gondoi | are there any docs on how to configure gerritbot for our project? | 19:04 |
| fungi | jeblair: it removes a bunch of lines, so i hope so! ;) | 19:04 |
| clarkb | jeblair: I wrote stuff in the commit message to make it clearer | 19:04 |
| jeblair | gondoi: http://ci.openstack.org/stackforge.html#configure-stackforge-project-to-use-gerritbot | 19:04 |
| clarkb | * more clear | 19:04 |
| gondoi | jeblair: perfect! thanks | 19:04 |
| clarkb | now the hard part, hunting down and rereviewing changes | 19:05 |
| fungi | clarkb: keep in mind that 77311 is not related to fixing anything urgent in manage-projects (though i'm not adverse to it receiving some additional testing while you're in there running anyway) | 19:06 |
| *** dcramer_ has quit IRC | 19:06 | |
| fungi | *sigh* https://jenkins02.openstack.org/job/gate-savanna-pep8/879/console | 19:06 |
| clarkb | fungi: ya I can go either way, but it was a one char change to fix so I did it | 19:07 |
| gondoi | jeblair: what about a list of possible events to send? or are the 3 in the example the list? | 19:07 |
| clarkb | fungi: we won't actually hit that code best I can tell | 19:07 |
| fungi | every time i notice a random network connectivity, file i/o or jenkins slave agent communication failure these days it's inevitably in hpcloud-az2 | 19:07 |
| clarkb | gondoi: right now only patchset created and change merged because there is a bug breaking the others | 19:07 |
| *** blamar has quit IRC | 19:07 | |
| gondoi | thanks | 19:07 |
| *** che-arne has quit IRC | 19:07 | |
| *** bhuvan__ has joined #openstack-infra | 19:08 | |
| *** bhuvan___ has joined #openstack-infra | 19:08 | |
| fungi | clarkb: oh, i saw someone randomly picked up the release notification feature request bug for gerritbot too and did some local testing with a dev gerrit to work out the details | 19:08 |
| fungi | clarkb: maybe we should dupe them into fixing the cdrv-2 issue while they're at it ;) | 19:08 |
| *** hogepodge has joined #openstack-infra | 19:09 | |
| *** bhuvan_ has quit IRC | 19:11 | |
| *** bhuvan has quit IRC | 19:12 | |
| fungi | oho! "WARNING: Making bare-precise-hpcloud-az2-2335514 offline temporarily due to the lack of disk space" | 19:12 |
| SpamapS | Ok.. I'm trying to spin up a manual run of check-tripleo-overcloud-precise | 19:12 |
| fungi | why are we still hitting that? and on a pep8 job for savanna of all places? | 19:13 |
| SpamapS | where does /opt/stack/new come from? | 19:13 |
| *** lenachek has joined #openstack-infra | 19:13 | |
| *** elasticio has joined #openstack-infra | 19:13 | |
| openstackgerrit | Ziad Sawalha proposed a change to openstack/requirements: Add PEP-257 Testing Tools https://review.openstack.org/79054 | 19:13 |
| openstackgerrit | A change was merged to openstack-infra/jeepyb: Change return code and command said log.info https://review.openstack.org/74130 | 19:13 |
| *** oubiwan__ has joined #openstack-infra | 19:13 | |
| fungi | SpamapS: devstack-gate makes it and clones devstack components into it, i believe. i'll get you the file/lineno | 19:13 |
| SpamapS | nodepool.yaml.erb ? | 19:14 |
| SpamapS | which is happily segfaulting my vim :( | 19:14 |
| SergeyLukjanov | I see savanna word :) | 19:14 |
| fungi | SpamapS: so i hear. sounds like a bad yaml colorizer in saucy? | 19:14 |
| SpamapS | yeah | 19:15 |
| fungi | SergeyLukjanov: https://jenkins02.openstack.org/job/gate-savanna-pep8/879/console died on lack of disk space | 19:15 |
| SpamapS | 'bout time I upgrade to trusty anyway :p | 19:15 |
| clarkb | fungi: they are 10GB root devices | 19:15 |
| clarkb | we may be running into those limits again :( | 19:15 |
| SergeyLukjanov | fungi, oh, we're just running flake8 and check_autoupdate under the pep8 tox env | 19:15 |
| SpamapS | fungi: ah, prepare_node_tripleo.sh is what I was missing | 19:16 |
| fungi | SergeyLukjanov: i thought as much, which is shy i found it a very strange job to hit a disk space error | 19:16 |
| *** ociuhandu has quit IRC | 19:17 | |
| fungi | SpamapS: oh, i thought you were asking about an equivalent in our normal devstack jobs | 19:17 |
| SergeyLukjanov | fungi, yup, that's really strange... | 19:17 |
| SpamapS | fungi: no, I'm debugging consistent failures of check-tripleo-overcloud-precise | 19:17 |
| *** SumitNaiksatam has quit IRC | 19:18 | |
| SpamapS | fungi: pretty sure we're running out of space too actually :-P | 19:18 |
| fungi | SpamapS: for devstack-gate, it gets created here unless it already exists.. http://git.openstack.org/cgit/openstack-infra/devstack-gate/tree/functions.sh#n249 | 19:18 |
| fungi | clarkb: yeah, if i get a free moment i'll hold an az2 node and look at the disk utilization | 19:18 |
| fungi | maybe we've done something silly which undid some of our earlier workarounds | 19:19 |
| SergeyLukjanov | fungi, there are 21 same failures for the last 24h - http://logstash.openstack.org/#eyJzZWFyY2giOiJtZXNzYWdlOlwiTG9va3MgbGlrZSB0aGUgbm9kZSB3ZW50IG9mZmxpbmUgZHVyaW5nIHRoZSBidWlsZC4gQ2hlY2sgdGhlIHNsYXZlIGxvZyBmb3IgdGhlIGRldGFpbHMuXCIiLCJmaWVsZHMiOltdLCJvZmZzZXQiOjAsInRpbWVmcmFtZSI6Ijg2NDAwIiwiZ3JhcGhtb2RlIjoiY291bnQiLCJ0aW1lIjp7InVzZXJfaW50ZXJ2YWwiOjB9LCJzdGFtcCI6MTM5NDIxOTk1MTU4OX0= | 19:19 |
| fungi | SergeyLukjanov: sounds likely. i'd be willing to bet almost all are in hpcloud and correspond to full disk offline reasons in corresponding jenkins master logs | 19:20 |
| SergeyLukjanov | one more - http://logstash.openstack.org/#eyJzZWFyY2giOiJtZXNzYWdlOlwiTG9va3MgbGlrZSB0aGUgbm9kZSB3ZW50IG9mZmxpbmUgZHVyaW5nIHRoZSBidWlsZC4gQ2hlY2sgdGhlIHNsYXZlIGxvZyBmb3IgdGhlIGRldGFpbHMuXCIgYW5kIG1lc3NhZ2U6XCJqYXZhLmlvLkZpbGVOb3RGb3VuZEV4Y2VwdGlvbjpcIiIsImZpZWxkcyI6W10sIm9mZnNldCI6MCwidGltZWZyYW1lIjoiODY0MDAiLCJncmFwaG1vZGUiOiJjb3VudCIsInRpbWUiOnsidXNlcl9pbnRlcnZhbCI6MH0sInN0YW1wIjoxMzk0MjIwMDExNTM1fQ== | 19:20 |
| SergeyLukjanov | fungi, yup, *all* of them are hpcloud | 19:21 |
| openstackgerrit | James E. Blair proposed a change to openstack-infra/config: Add a job to check IRC channel access https://review.openstack.org/78061 | 19:21 |
| openstackgerrit | James E. Blair proposed a change to openstack-infra/config: Add a script to manage IRC perms https://review.openstack.org/78483 | 19:21 |
| fungi | SergeyLukjanov: that reinforces my suspicions then. it's endemic | 19:21 |
| SergeyLukjanov | heh | 19:21 |
| *** mriedem has joined #openstack-infra | 19:21 | |
| openstackgerrit | BK Box proposed a change to openstack-infra/config: Join gerritbot to the satori IRC channel https://review.openstack.org/79058 | 19:22 |
| openstackgerrit | James E. Blair proposed a change to openstack-infra/statusbot: Support SSL https://review.openstack.org/79047 | 19:22 |
| openstackgerrit | James E. Blair proposed a change to openstack-infra/gerritbot: Support SSL https://review.openstack.org/79049 | 19:22 |
| *** mriedem1 has quit IRC | 19:25 | |
| jeblair | fungi: unit tests on hpcloud nodes are new, right? | 19:25 |
| jeblair | since yesterday | 19:25 |
| *** sarob_ has joined #openstack-infra | 19:25 | |
| fungi | jeblair: nope, we've had bare-precise nodes there for a while | 19:26 |
| *** morganfainberg_Z is now known as morganfainberg | 19:26 | |
| fungi | jeblair: bare-centos6 and py3k-centos6 are new there since yesterday | 19:26 |
| fungi | er, py3k-precise | 19:26 |
| jeblair | oh ok | 19:26 |
| fungi | heh, py3k-centos6 would be quite the feat ;) | 19:26 |
| fungi | so i'm thinking maybe run-unittests.sh or run-tox.sh should spew a df / as well as the ip address show we were discussing earlier... concerns with that? | 19:27 |
| jeblair | fungi: so devstack-gate moves git repos around, but i don't think we do the same thing for unit tests | 19:27 |
| *** SumitNaiksatam has joined #openstack-infra | 19:28 | |
| fungi | jeblair: currently we do not, no | 19:28 |
| fungi | jeblair: on the node i held, 1.6G in /opt/git | 19:28 |
| fungi | which is 16% of our root filesystem | 19:28 |
| *** thomasem has joined #openstack-infra | 19:29 | |
| jeblair | fungi: there's still something like 3.5G free | 19:29 |
| fungi | yeah, which is why i think there's got to be some other contributing factor | 19:30 |
| fungi | 1.1G in /var/cache/apt | 19:30 |
| *** reed has joined #openstack-infra | 19:30 | |
| *** yassine has joined #openstack-infra | 19:32 | |
| fungi | the reason i suggest a df in the console log is i'm wondering whether some of these might be coming with something smaller than a 10g rootfs | 19:32 |
| fungi | just, you know, at random. for laughs | 19:33 |
| fungi | though the majority i've seen happen during log copying | 19:34 |
| clarkb | fungi: jeblair SergeyLukjanov I think https://etherpad.openstack.org/p/new-projects-2014-02-28 is up to date now | 19:34 |
| fungi | so maybe there's something which just spins wildly out of control during that phase now and then? | 19:34 |
| jeblair | fungi: log copying only happens in d-g and we moved that | 19:34 |
| clarkb | should I propose patches to change the group with tag push access? | 19:34 |
| fungi | clarkb: 03-07? | 19:34 |
| clarkb | fungi: I used the one you linked me | 19:34 |
| jeblair | fungi: or do you mean the scp step? | 19:34 |
| fungi | clarkb: did i link to the old one? i didn't mean to, sossy :( | 19:35 |
| clarkb | thats fine I will update the new one with the content on the old one | 19:35 |
| fungi | jeblair: that's the next thing which would have appeared in the log | 19:35 |
| fungi | jeblair: so it looks from the log like jenkins offlined the slave for a full disk immediately before or in the middle of the console log scp | 19:36 |
| jeblair | fungi: i think the disk space is a red herring | 19:36 |
| *** andre__ has quit IRC | 19:36 | |
| jeblair | fungi: i think this is the same 'jenkins offlined a node very close to some gearman-plugin action' bug we've seen recently | 19:36 |
| fungi | jeblair: you think maybe jenkins got a write error and just assumed? | 19:37 |
| jeblair | fungi: http://paste.openstack.org/show/72923/ | 19:37 |
| jeblair | https://bugs.launchpad.net/openstack-ci/+bug/1284371 | 19:38 |
| clarkb | jeblair: fungi SergeyLukjanov mordred anteaya https://etherpad.openstack.org/p/new-projects-2014-03-07 is up to date now | 19:38 |
| openstackgerrit | Michael Krotscheck proposed a change to openstack-infra/storyboard: Updated oslo https://review.openstack.org/78496 | 19:39 |
| fungi | jeblair: i do find it interesting that all the hits for it in logstash are hpcloud nodes though | 19:39 |
| jeblair | fungi: i see a rax node in there | 19:39 |
| fungi | oh! | 19:40 |
| clarkb | if folks can review those changes and point out any issues I may have missed that would be great. Once that is done I will rebase them to avoid any conflicts and s/core/ptl/ as appropriate | 19:40 |
| jeblair | clarkb: can i link to a single line in logstash? | 19:40 |
| clarkb | jeblair: yes, however kibana doesn't make it easy iirc. every document has a unique id you can query for | 19:40 |
| clarkb | trying to remember if that unique id can be found via kibana | 19:41 |
| clarkb | ya I don't think you can with kibana, if you do it via REST you can | 19:41 |
| clarkb | :/ | 19:41 |
| jeblair | bare-precise-rax-iad-2313399 shows up in http://logstash.openstack.org/#eyJzZWFyY2giOiJtZXNzYWdlOlwiTG9va3MgbGlrZSB0aGUgbm9kZSB3ZW50IG9mZmxpbmUgZHVyaW5nIHRoZSBidWlsZC4gQ2hlY2sgdGhlIHNsYXZlIGxvZyBmb3IgdGhlIGRldGFpbHMuXCIgYW5kIG1lc3NhZ2U6XCJqYXZhLmlvLkZpbGVOb3RGb3VuZEV4Y2VwdGlvbjpcIiIsImZpZWxkcyI6W10sIm9mZnNldCI6MCwidGltZWZyYW1lIjoiODY0MDAiLCJncmFwaG1vZGUiOiJjb3VudCIsInRpbWUiOnsidXNlcl9pbnRlcnZhbCI6MH0sInN0YW1wIjoxMzk0MjIwMDExNTM1fQ== | 19:42 |
| *** blamar has joined #openstack-infra | 19:42 | |
| fungi | ahh, expanding the search from a day to a week does find one | 19:42 |
| jeblair | clarkb: is 20 seconds of polling long enough you think? | 19:42 |
| *** esmute has joined #openstack-infra | 19:42 | |
| clarkb | jeblair: you could search for AND NOT build_node:bare-precise-hpcloud* | 19:43 |
| clarkb | jeblair: good question, since it is all local filesystem stuff I figured it would be | 19:43 |
| clarkb | jeblair: I can certainly bump that number if you think we need to be more conservative | 19:43 |
| jeblair | just wondering if that might be slower if gerrit is merging and replicating a nova change | 19:43 |
| clarkb | oh good point | 19:43 |
| jeblair | clarkb: i aprvd that change for now, we can up it later | 19:43 |
| clarkb | jeblair: ok | 19:44 |
| openstackgerrit | A change was merged to openstack-infra/config: Revert "Revert "Stop running manage-projects"" https://review.openstack.org/79050 | 19:44 |
| clarkb | I am going to grab food here shortly. Will hop back on the project stuff after that | 19:45 |
| jeblair | clarkb: oh, did you not want to approve those but instead test manually? | 19:45 |
| jeblair | just noticed the comment on https://review.openstack.org/#/c/71323/3 | 19:46 |
| fungi | looks like the rebase never got jenkins tests enqueued | 19:47 |
| jeblair | fungi: rebase of what? | 19:48 |
| fungi | 71323 | 19:49 |
| *** malor has quit IRC | 19:50 | |
| fungi | probably got uploaded during the zuul restart a couple days ago | 19:50 |
| fungi | heh, we both rechecked it | 19:51 |
| jeblair | shouldn't https://review.openstack.org/#/c/76270/3 be an openstack-infra project? | 19:51 |
| fungi | yeah, i thought that was the student project ttx was wanting | 19:52 |
| fungi | i agree it probably shouldn't have been in stackforge in that case | 19:53 |
| fungi | unless that's a separate effort | 19:54 |
| *** zul has quit IRC | 19:55 | |
| *** vkozhukalov has quit IRC | 19:55 | |
| jeblair | i hate to delay student work (2 weeks like like 1/8th of a semester); unfortunately, this is the first time i've seen this. we might want to get confirmation from ttx. | 19:57 |
| jeblair | is there a mentor or project manager for openstack on this? | 19:57 |
| *** yolanda_ has quit IRC | 19:57 | |
| fungi | good question... i thought ttx was but maybe not | 19:58 |
| openstackgerrit | Joe Gordon proposed a change to openstack-infra/config: Add non-voting *-partial-ncpu to grenade, devstack and tempest https://review.openstack.org/75960 | 19:58 |
| jeblair | i just added him to the review | 19:58 |
| flashgordon | rebase conflict on the integrated gate patch ^ | 19:58 |
| flashgordon | jeblair: https://review.openstack.org/#/c/71652/ I just saw jenkisn fail and then gate jobs start | 19:59 |
| fungi | jeblair: ldbragst (ldbragst@us.ibm.com) replied to clarkb's connect on 76270 in such a way that makes me think he might be involved on it | 20:00 |
| fungi | s/connect/comment/ | 20:01 |
| flashgordon | fungi: ^ | 20:01 |
| flashgordon | and that patch will just fail in gate, because its a valid fail | 20:01 |
| *** ryanpetrello has quit IRC | 20:02 | |
| *** ryanpetrello has joined #openstack-infra | 20:04 | |
| *** dangers is now known as dangers_away | 20:04 | |
| openstackgerrit | A change was merged to openstack-infra/jeepyb: Poll manage projects config fetches. https://review.openstack.org/77314 | 20:05 |
| jeblair | flashgordon: that's annoying | 20:05 |
| jeblair | fungi: these new machines really have a useless amount of diskspace | 20:05 |
| jeblair | zuul's root fs just filled | 20:05 |
| jeblair | flashgordon: unfortunately, i don't think i can debug that further because of the missing logs | 20:06 |
| flashgordon | jeblair: I pulled my patch the bad patch (mine) out of gate, but yeah very strange | 20:07 |
| flashgordon | jeblair: ack, this was just an FYI more then anything else | 20:07 |
| jeblair | flashgordon: that's important to get right if we ever pull jobs out of gate and count on check instead, so we need to figure it out; if you see it again, please let us know | 20:07 |
| jeblair | fungi: i think we can remove zuul's git repos on zuul.o.o now, that will give us 2.2G | 20:08 |
| *** zehicle_at_dell has joined #openstack-infra | 20:08 | |
| fungi | gah | 20:09 |
| fungi | yes, definitely | 20:09 |
| fungi | should be entirely safe at this point if a find says no files have been updated there | 20:10 |
| openstackgerrit | Morgan Fainberg proposed a change to openstack-infra/config: Add in non-voting sample-config check https://review.openstack.org/78038 | 20:11 |
| flashgordon | jeblair: will do | 20:11 |
| fungi | jeblair: oh, though we had mounted /var/lib/zuul/git on tmpfs anyway | 20:12 |
| fungi | jeblair: had the underlying ones not been cleared out? | 20:13 |
| clarkb | jeblair: merging is fine it means I get to propose fixes instead of new patchsets :) | 20:14 |
| mordred | jeblair 40G should be enough for anyone | 20:14 |
| jeblair | mordred: it's 10 | 20:15 |
| jeblair | oh, no it's 40 you're right | 20:15 |
| jeblair | i was confusing rax and hp :) | 20:15 |
| fungi | jeblair: what did you end up deleting? | 20:15 |
| jeblair | fungi: logs from feb, and just now the git repos | 20:16 |
| fungi | jeblair: though i do agree that we can probably blow away and unconfigure that tmpfs for teh git repos now | 20:16 |
| *** hashar has joined #openstack-infra | 20:16 | |
| *** ociuhandu has joined #openstack-infra | 20:16 | |
| jeblair | oh that is a tmpfs | 20:16 |
| jeblair | when did that happen? | 20:16 |
| openstackgerrit | BK Box proposed a change to openstack-infra/config: Join gerritbot to the satori IRC channel https://review.openstack.org/79058 | 20:16 |
| clarkb | after lca | 20:16 |
| clarkb | because git is slow | 20:16 |
| jeblair | is that in puppet or was that a snowflake on z.o.o? | 20:17 |
| fungi | jeblair: when we were scrambling for temporary ways to boost zuul's performance until the merger split | 20:17 |
| clarkb | but zm0X mostly solves that | 20:17 |
| jeblair | (am wondering if zm0x have tmpfs's) | 20:17 |
| fungi | they do not | 20:17 |
| jeblair | so i think the main thing we need to do is compress zuul's logs | 20:17 |
| fungi | it was done byhand during the replacement server build | 20:17 |
| *** jungleboyj has quit IRC | 20:18 | |
| gondoi | pleia2: is there a was to reset my change to get it reviewed again, or do I just need to ping you directly? | 20:18 |
| gondoi | irt: 79058 | 20:18 |
| fungi | we had to make zuul.o.o a much larger flavor to accommodate the additional ram filesystem basically, so set that tempfs up temporarily as a stop-gap | 20:18 |
| *** mrodden has quit IRC | 20:18 | |
| fungi | i expect zuul might even be able to be downgraded to a smaller server at this point | 20:18 |
| clarkb | ya probably | 20:19 |
| fungi | it's definitely looking fairly underutilized from the graphs | 20:19 |
| fungi | could probably go down to a 16g or even 8g flavor safely in the future | 20:20 |
| *** Ryan_Lane1 has quit IRC | 20:20 | |
| *** mkoderer has quit IRC | 20:21 | |
| fungi | the load average on it has barely broken 1 for weeks now | 20:21 |
| *** mrodden has joined #openstack-infra | 20:22 | |
| fungi | it's funny--you can essentially see the demarcation on the monthly load average graph where the mergers came online | 20:22 |
| SpamapS | can anyone tell me what caused this fail? https://review.openstack.org/#/c/78305/ | 20:24 |
| SpamapS | something about can't import greenthread? | 20:24 |
| clarkb | yup. cffi had an interaction with system openssl | 20:25 |
| clarkb | it was a really really weird situation | 20:25 |
| SpamapS | clarkb: bug for reverify? | 20:25 |
| clarkb | Im not sure if one was created this happened while I was asleep | 20:25 |
| *** dcramer_ has joined #openstack-infra | 20:25 | |
| fungi | SpamapS: cffi 0.8.2 in a virtualenv with site_packages=true and 0.8.1 on the system was causing issues with pycrypto under eventlet for some reason | 20:25 |
| SpamapS | ah | 20:25 |
| fungi | i think pycrypto was compiling against one and then tryign to use the other | 20:26 |
| SpamapS | fungi: perhaps pycrypto decided to demand that the sysadmin stop two-timing cffi? | 20:26 |
| fungi | sure, why not | 20:26 |
| fungi | i have the bug number here somewhere | 20:26 |
| *** lenachek has quit IRC | 20:28 | |
| fungi | SpamapS: 1289345 looks like what they opened when it first started, though it may need some updating | 20:30 |
| *** gokrokve has joined #openstack-infra | 20:30 | |
| *** malini is now known as malini_afk | 20:32 | |
| *** andreaf has joined #openstack-infra | 20:37 | |
| openstackgerrit | Michael Krotscheck proposed a change to openstack-infra/storyboard: Updated oslo https://review.openstack.org/78496 | 20:37 |
| *** smarcet has quit IRC | 20:38 | |
| *** rossella_s has quit IRC | 20:39 | |
| *** rossella_s has joined #openstack-infra | 20:40 | |
| fungi | i've updated it with a summary of everything we discovered after that bug fell silent | 20:43 |
| *** blamar_ has joined #openstack-infra | 20:45 | |
| fungi | ttx: if you're lurking, we're trying to sort out whether logan paschke's change to add a "gerrit-powered-agenda" project to stackforge is the git-backed schedule student project you were talking about previously | 20:45 |
| openstackgerrit | A change was merged to openstack-infra/config: Add non-voting *-partial-ncpu to grenade, devstack and tempest https://review.openstack.org/75960 | 20:46 |
| *** blamar has quit IRC | 20:46 | |
| *** blamar_ is now known as blamar | 20:46 | |
| clarkb | fungi: jeblair: do we want to avoid putting it into stackforge if it is official? | 20:47 |
| jeblair | clarkb: yes | 20:47 |
| fungi | i updated the review mentioning https://launchpad.net/bugs/1250758 (once i was able to find it) | 20:48 |
| fungi | if they are related, it would be nice for the bug to get a mention in the commit message, and assign it to whoever's working on it | 20:50 |
| fungi | but i'm not ruling out the possibility that this is some completely unrelated thing | 20:50 |
| *** SumitNaiksatam has quit IRC | 20:50 | |
| jeblair | fungi: ttx's last comment in the bug also strongly suggests that they _are_ the same thing | 20:50 |
| *** sarob__ has joined #openstack-infra | 20:51 | |
| clarkb | jeblair: ya I think they must be | 20:51 |
| *** Ryan_Lane1 has joined #openstack-infra | 20:51 | |
| *** blamar has quit IRC | 20:51 | |
| clarkb | mordred: https://review.openstack.org/#/c/78821/ could use some love | 20:51 |
| clarkb | mordred: would probably be good to have working man page builds prior to release :) | 20:52 |
| *** sarob_ has quit IRC | 20:52 | |
| fungi | the upstream url says "NDSU Capstone" which does sound like a universityish thing too | 20:52 |
| mordred | clarkb: +A | 20:53 |
| *** rpodolyaka has joined #openstack-infra | 20:54 | |
| clarkb | maybe reed knows? | 20:54 |
| clarkb | reed: ^ is the NDSU capstone thing ttx's official calendaring project stuff? | 20:54 |
| reed | no clue | 20:54 |
| *** vkozhukalov has joined #openstack-infra | 20:55 | |
| ttx | fungi, jeblair, clarkb: yes, that's the student project thing | 20:55 |
| *** sarob__ has quit IRC | 20:55 | |
| clarkb | ttx: so that should live in the openstack*/ org? | 20:55 |
| *** Ryan_Lane1 has quit IRC | 20:55 | |
| clarkb | s/the/an/ | 20:55 |
| reed | sure if that's the calendaring stuff it may win the prize for most accurately obfuscated name | 20:55 |
| jeblair | ttx: yay! can we put it in openstack-infra and consider it part of the infra program? | 20:55 |
| *** dmsimard has joined #openstack-infra | 20:55 | |
| ttx | clarkb: well, amybe we want to see if they end up with something usable first | 20:56 |
| fungi | ttx: unusable things go in stackforge? | 20:56 |
| jeblair | we don't really have 'usable' as a criteria for acceptance into the infra program | 20:56 |
| jeblair | we'd have to get rid of most of it | 20:56 |
| fungi | hee | 20:56 |
| ttx | jeblair: then openstack/* it is | 20:57 |
| fungi | anyway, if we're okay putting it into openstack-infra, i can draft a quick update to that change (and link the bug number) so that we can roll it in today | 20:58 |
| clarkb | the log gearman client OOMd | 20:58 |
| clarkb | I have restarted it | 20:58 |
| reed | ttx, from NDSU Capstone ... something about calendar? did you pick the name using a random generator? :) | 20:58 |
| clarkb | north dakota state university I presume | 20:58 |
| clarkb | fungi: want to make the stack of changes when you do that? | 20:59 |
| clarkb | fungi: or I can do it (but need to s/core/ptl/ on a couple changes too) | 20:59 |
| fungi | i hate to deprive a student of an opportunity for more interaction on this change, but on the other hand the sooner their project is in gerrit the more of that they'll get anyway | 20:59 |
| clarkb | ++ | 20:59 |
| jeblair | ya | 20:59 |
| fungi | clarkb: i didn't stack any of the ones i did last week. there were way more and none ended up merge-conflicting anyway | 20:59 |
| clarkb | fungi: oh cool I suppose I won't worry about it then | 21:00 |
| fungi | hooray for alphabetization of things | 21:00 |
| clarkb | fungi: I can push the -ptl updates if you do NDSU | 21:00 |
| fungi | on it | 21:00 |
| ttx | reed: I think that's the project name. Maybe openstack/capstone would be better | 21:01 |
| ttx | haven't seen the review | 21:01 |
| openstackgerrit | Clark Boylan proposed a change to openstack-infra/config: Add kickstack https://review.openstack.org/78936 | 21:01 |
| jeblair | well, capstone refers to a final project that students do as part of their study | 21:01 |
| fungi | ttx: https://review.openstack.org/76270 | 21:01 |
| jeblair | so it doesn't really describe what the project itself is | 21:01 |
| fungi | ttx: but i'm about to s/stackforge/openstack-infra/ that real quick now | 21:02 |
| * reed jeblair writes and it all makes sense | 21:02 | |
| openstackgerrit | Clark Boylan proposed a change to openstack-infra/config: Add new project graffiti https://review.openstack.org/78829 | 21:02 |
| fungi | ttx: "gerrit-powered-agenda" seems to be the name they've chosen | 21:02 |
| fungi | reed: ^ | 21:02 |
| fungi | (for better or for worse) | 21:02 |
| * reed facepalm | 21:03 | |
| jeblair | it's not the worst, that's for sure | 21:03 |
| *** Ryan_Lane1 has joined #openstack-infra | 21:03 | |
| fungi | it's cheeky and somewhat related to the subject matter, so not as bad as some of ours | 21:03 |
| jeblair | i mean our most famous project is 'jenkins-job-builder'. they probably figured we'd _love_ 'gerrit-powered-agenda'. | 21:04 |
| reed | i torture myself editing unanswered questions on ask on friday afternoons | 21:04 |
| *** sarob_ has joined #openstack-infra | 21:04 | |
| ttx | reed: beware or we appoint you project name generator | 21:04 |
| jeblair | ttx: 'project-name-generator' | 21:04 |
| reed | ttx, Ill generate names, you edit questions on Ask :) | 21:05 |
| krotscheck | Has anyone recently run into issues with an 'from babel import localedata' error? | 21:06 |
| reed | I'm throwing some extra ''''''''''''' here and there since I seem to miss them once in a while | 21:06 |
| ttx | reed: You'll generate names, I'll snowboard :P | 21:06 |
| reed | oh, cool | 21:06 |
| reed | i'll follow you on alpine record :) | 21:06 |
| jeblair | krotscheck: i have not seen that | 21:06 |
| *** bhuvan___ has quit IRC | 21:07 | |
| *** bhuvan__ has quit IRC | 21:07 | |
| jeblair | krotscheck: according to logstash, only you. | 21:07 |
| krotscheck | Harumph. | 21:08 |
| jeblair | oh wait, | 21:08 |
| jeblair | ceilometer has seen it too | 21:08 |
| krotscheck | I thought i saw it in savanna and celiometer | 21:08 |
| krotscheck | But in those cases, moving the dependency to runtime requirements worked. | 21:08 |
| *** rpodolyaka has quit IRC | 21:08 | |
| krotscheck | It's already in our runtime requirements :/ | 21:08 |
| * clarkb gets nerd sniped and responds to ML | 21:09 | |
| clarkb | once fungi gets the new patchset for NDSU up I think all changes are ready and will dive in | 21:10 |
| fungi | well, i have the updated patchset ready, but the change author just replied to our comments to the contrary | 21:10 |
| clarkb | ah | 21:10 |
| *** sandywalsh has quit IRC | 21:11 | |
| openstackgerrit | Dan Smith proposed a change to openstack/requirements: Sets the minimum python-novaclient version to 2.17.0 https://review.openstack.org/79078 | 21:12 |
| jeblair | clarkb: which topic? | 21:12 |
| clarkb | jeblair: testr questions | 21:13 |
| *** ArxCruz has quit IRC | 21:13 | |
| openstackgerrit | Antoine Musso proposed a change to openstack-infra/jenkins-job-builder: Support the Post Build Script plugin https://review.openstack.org/77628 | 21:13 |
| *** _david_ has joined #openstack-infra | 21:13 | |
| jeblair | krotscheck: why is postgres still required? | 21:14 |
| openstackgerrit | Dan Smith proposed a change to openstack/requirements: Sets the minimum python-novaclient version to 2.17.0 https://review.openstack.org/79078 | 21:14 |
| *** dcramer_ has quit IRC | 21:14 | |
| _david_ | jeblair, thanks for the hint one puppet problem was fixed, the other still exists: | 21:15 |
| damnsmith | who can vote on requirements changes? | 21:15 |
| _david_ | jeblair, https://review.openstack.org/#/c/60080/ | 21:15 |
| krotscheck | jeblair: Because we haven't removed it yet. | 21:15 |
| fungi | damnsmith: https://review.openstack.org/#/admin/groups/131 | 21:16 |
| _david_ | WARNING: top-scope variable being used without an explicit namespace on line 185 | 21:16 |
| jeblair | krotscheck: unassailable logic! | 21:16 |
| damnsmith | fungi: thanks | 21:16 |
| *** rpodolyaka has joined #openstack-infra | 21:16 | |
| krotscheck | jeblair: Correction- I don' tthink anyone's actually proposed that we not support postgres. | 21:16 |
| jeblair | krotscheck: oh, i thought that came up at the meeting | 21:16 |
| fungi | damnsmith: the openstack-ci-core and openstack-release groups are in there mostly for emergency control when we need to fix stuff, but prefer to lean on the others listed directly as members of requirements-core for normal reviews | 21:17 |
| damnsmith | fungi: okay | 21:17 |
| fungi | damnsmith: also, i'm sure that group would love another volunteer core reviewer! ;) | 21:17 |
| damnsmith | fungi: no way am I looking for that kind of responsibility :) | 21:17 |
| fungi | hah | 21:18 |
| damnsmith | fungi: breakage penalty is high :) | 21:18 |
| krotscheck | jeblair: Tell ya what: Once I figure out this oslo thing, and land the 2 gazillion patches that are preventing us from dogfooding because of the bloody migration failures, we can reopen that discussion. | 21:18 |
| * krotscheck is a bit frustrated at the moment. | 21:18 | |
| jeblair | krotscheck: take it easy, i was just asking to fill in time while i was installing postgres so i could run the tests locally while working on a constructive solution. | 21:19 |
| jeblair | krotscheck: ...and i can't reproduce it locally. | 21:19 |
| jeblair | oh wait, i may not have been testing the right code.. | 21:20 |
| openstackgerrit | Michael Krotscheck proposed a change to openstack-infra/storyboard: Updated oslo https://review.openstack.org/78496 | 21:20 |
| *** ryanpetrello_ has joined #openstack-infra | 21:21 | |
| *** bhuvan has joined #openstack-infra | 21:21 | |
| *** bhuvan_ has joined #openstack-infra | 21:21 | |
| *** SumitNaiksatam has joined #openstack-infra | 21:21 | |
| jeblair | krotscheck: what did you change there ^? | 21:21 |
| jeblair | and yes, i can reproduce locally | 21:22 |
| krotscheck | jeblair: added posix_ipc to test requirements | 21:22 |
| fungi | clarkb: i'm going to make a quick pass through the other new project reviews and i guess we can start approving those. if we don't hear back in the affirmative from logan before we're done, it can wait until next friday i guess | 21:23 |
| *** aysyanne has quit IRC | 21:23 | |
| *** ryanpetrello has quit IRC | 21:23 | |
| *** ryanpetrello_ is now known as ryanpetrello | 21:23 | |
| clarkb | fungi: ok | 21:24 |
| *** lcostantino has quit IRC | 21:25 | |
| fungi | clarkb: what did "verified" mean next to the "has upstream"? on the previous iteration i was using verified as a quick tag on the projects i had confirmed cloned successfully from every git server | 21:26 |
| clarkb | fungi: I just loaded the url in a web browser | 21:27 |
| clarkb | I suppose I should've done a proper clone | 21:27 |
| *** mbacchi has quit IRC | 21:27 | |
| fungi | oh, git it. i meant i tested cloning them after they were manage-projects created to make sure they'd replicated successfully to all the members of the server farm | 21:28 |
| fungi | since we'd previously seen some potential issues around that | 21:28 |
| openstackgerrit | David Ostrovsky proposed a change to openstack-infra/config: Gerrit-2.8: Add secondary index support https://review.openstack.org/60080 | 21:30 |
| clarkb | fungi: ah that isn't what I meant. I was just checking that the upstream existed | 21:31 |
| fungi | got it. i usually do that on all new project reviews anyway | 21:32 |
| openstackgerrit | Morgan Fainberg proposed a change to openstack-infra/config: Add in non-voting sample-config check https://review.openstack.org/78038 | 21:32 |
| *** ArxCruz has joined #openstack-infra | 21:32 | |
| *** ArxCruz has quit IRC | 21:32 | |
| fungi | clarkb: i'm good with all the changes other than the one we're still confused on | 21:32 |
| clarkb | great | 21:33 |
| clarkb | why don't we start with those (we can always add in the remaining one later and do another manual run) | 21:33 |
| fungi | sure | 21:33 |
| fungi | did you want to batch all four of these together or run manage-projects more than once? | 21:34 |
| clarkb | should probably do one first, then based on those results either batch or go one at a time | 21:34 |
| fungi | sounds good. give more chance to adjust patches and iterate | 21:35 |
| fungi | i can approve one of the very simple config changes first | 21:35 |
| fungi | groups for oslo.rootwrap? | 21:35 |
| clarkb | sounds good | 21:35 |
| *** oubiwan__ has quit IRC | 21:35 | |
| fungi | in it goes | 21:36 |
| krotscheck | jeblair: Moving conversation back here - I've tried to migrate down all the way to 0.9.6, with no luck. | 21:36 |
| clarkb | fungi: how were you updating jeepyb on review.o.o? was that on a separate branch whcih you switched back to master when done? | 21:36 |
| fungi | clarkb: i was in-place patching the .py file in /usr/local | 21:37 |
| *** datsun180b has joined #openstack-infra | 21:37 | |
| fungi | because that was the most expedient way to test, since it's otherwise cd from git master via puppet agent updates | 21:37 |
| jeblair | krotscheck: i think i found something | 21:37 |
| clarkb | fungi: wow ok | 21:37 |
| krotscheck | jeblair: Oh? | 21:38 |
| fungi | clarkb: though you could just clone it locally in your homedir and run out of there too | 21:38 |
| clarkb | I know, I can clone it into my homedir, mess with it there and pip install from there | 21:38 |
| jeblair | krotscheck: http://paste.openstack.org/show/72932/ | 21:38 |
| clarkb | oh or run it out of a virtualenv | 21:38 |
| * clarkb does that | 21:38 | |
| *** SnowDust has joined #openstack-infra | 21:38 | |
| fungi | yup | 21:38 |
| jeblair | krotscheck: minimum reproducible test case ^ | 21:38 |
| jeblair | krotscheck: that's from storyboard/db/models.py | 21:38 |
| SnowDust | fungi : https://15.185.162.128/job/Trove-Gate/2831/console ( can u see why this may happen ? ) | 21:38 |
| jeblair | krotscheck: does that come from oslo? | 21:38 |
| jeblair | krotscheck: basically the "turn warnings into errors" is extremely broad | 21:39 |
| jeblair | the comment says "turn sqlalchemy warnings into errors" but in actuality it turns all warnings in all modules into errors | 21:40 |
| jeblair | krotscheck: also, this is _very_ sensitive to import ordering, so could be masked in other projects using something similar | 21:40 |
| krotscheck | Wait, what? Import ordering is a thing in python? | 21:40 |
| *** dstanek is now known as dstanek_afk | 21:40 | |
| fungi | SnowDust: you probably want to log into that machine (or a similar one) and run 'apt-cache policy git' but it looks like for some reason it doesn't think the git package exists | 21:41 |
| jeblair | krotscheck: yep. module level code like that 'warnings' line is generally frowned upon too because of this sort of thing (it gets executed on import) | 21:41 |
| fungi | SnowDust: so basically you want to look into why | 21:41 |
| datsun180b | could have just been a fluke with a mirror, for example | 21:41 |
| fungi | datsun180b: possibly yes | 21:42 |
| *** bhuvan has quit IRC | 21:42 | |
| *** bhuvan_ has quit IRC | 21:42 | |
| datsun180b | we can retrigger it all day but i didn't want to just hammer it if there was a real problem | 21:42 |
| *** bhuvan has joined #openstack-infra | 21:43 | |
| *** bhuvan_ has joined #openstack-infra | 21:43 | |
| *** oubiwan__ has joined #openstack-infra | 21:43 | |
| krotscheck | jeblair: Hrm. Fascinating. | 21:43 |
| clarkb | EnvironmentError: mysql_config not found <- from installing jeepyb in a venv on review.o.o | 21:43 |
| krotscheck | jeblair: Hey look, I can yell at mordred for that one! | 21:43 |
| *** harlowja has quit IRC | 21:43 | |
| clarkb | fungi: jeblair have you seen that before? | 21:44 |
| jeblair | clarkb: i think it means that you aren't using site packages and so you are trying to build the mysql module without the -dev package installed | 21:44 |
| krotscheck | jeblair: Thanks for helping debug this. | 21:44 |
| krotscheck | jeblair: I think that also fixes our migration pain | 21:44 |
| fungi | clarkb: i agree with jeblair | 21:44 |
| clarkb | jeblair: thanks, I will allow site packages then | 21:45 |
| *** SnowDust has quit IRC | 21:45 | |
| fungi | clarkb: make your venv with --system-site-packages and see if it helps | 21:45 |
| fungi | ahh, you just said that | 21:45 |
| * fungi curses the lag | 21:45 | |
| clarkb | yup site packages enabled fixed it | 21:46 |
| jeblair | krotscheck: no problem; oh, yeah, that does look like it came from mordred. i wonder where he got that from... | 21:46 |
| openstackgerrit | Michael Krotscheck proposed a change to openstack-infra/storyboard: Updated oslo https://review.openstack.org/78496 | 21:46 |
| clarkb | now I am waiting for https://review.openstack.org/#/c/78667/ to merge and propogate | 21:47 |
| *** lcheng has joined #openstack-infra | 21:47 | |
| jeblair | krotscheck, mordred: if you want to put that back, this is the right way to do it: https://www.gitorious.org/mediagoblin/mediagoblin-pyconfigure/commit/601e6e0cce3af9a2521e6bf4e6d50740c84dc653 | 21:48 |
| jeblair | krotscheck, mordred: (note that it's both inside of a function, and scopes the warnings it affects to SAWarning) | 21:48 |
| clarkb | fungi: were you running manage-projects against individual projects or the whole thing? | 21:49 |
| clarkb | fungi: I should probably run it against everything I suppose | 21:49 |
| openstackgerrit | A change was merged to openstack-infra/statusbot: Sleep when joining channels https://review.openstack.org/79033 | 21:50 |
| krotscheck | jeblair: Is this a useful thing to do at runtime, or should that just be in tests? | 21:50 |
| * krotscheck can see pro/con :/ | 21:50 | |
| fungi | clarkb: we have at least one issue which running it wide-open underscores... logic errors when determining whether to do the upstream tracking branch updates | 21:51 |
| fungi | clarkb: i think that should be resolved now from one of the patches which merged earlier in the week, but doing it at least once to double-check would be good | 21:51 |
| clarkb | ok | 21:52 |
| jeblair | krotscheck: tbh, i've never done it; i'm not sure what kinds of things it would catch. probably worth setting it and running the app through its paces to see. | 21:52 |
| fungi | clarkb: also running with -v or -d will probably result in you wanting to redirect or tee to a file | 21:52 |
| SergeyLukjanov | do we have a recheck bug for http://logs.openstack.org/00/78200/2/gate/gate-savanna-pep8/0937881/console.html#_2014-03-07_18_16_23_993 ? | 21:52 |
| clarkb | fungi: jeblair: any chance you guys can review https://review.openstack.org/#/c/77311 before I run that code? | 21:52 |
| fungi | clarkb: looking | 21:52 |
| jeblair | SergeyLukjanov: yeah, it's on the elastic recheck page | 21:52 |
| fungi | i already half-reviewed it last week | 21:52 |
| fungi | i should just finish doing that | 21:52 |
| krotscheck | jeblair: Well, right now it's converting SQLite migration warnings into errors that are making our migrations break, which isn't really useful. | 21:53 |
| jeblair | krotscheck: oh, what are the warnings? | 21:53 |
| SergeyLukjanov | jeblair, oh, find it, thx, need to grep short strings | 21:54 |
| jeblair | fungi, clarkb: oh yeah i had also half-reviewed that. lgtm. ish. | 21:54 |
| krotscheck | jeblair: UserWarning: Skipping unsupported ALTER for creation of implicit constraint | 21:54 |
| fungi | same. not approving since clarkb is going to hand-test it first anyway | 21:54 |
| jeblair | krotscheck: oh, yeah. gotcha. | 21:54 |
| jeblair | krotscheck: not much you can do about sqlite not supporting alter. | 21:54 |
| clarkb | fungi: jeblair: I just noticed that code may be wrong... | 21:54 |
| clarkb | fungi: jeblair: I think mordred flopped the default return values | 21:55 |
| fungi | oh, ew | 21:55 |
| krotscheck | jeblair: Yeah, but since that gets converted into an error, and it's a transactionless atabase, alembic will try to re-run the migration and run into duplicate column errors. | 21:55 |
| krotscheck | jeblair: Anyway, this should fix it. | 21:55 |
| clarkb | fungi: also basename won't work there as expected | 21:55 |
| * clarkb comments | 21:56 | |
| openstackgerrit | Jason-oldos proposed a change to openstack-infra/config: Add openstack/ironic-python-agent https://review.openstack.org/79088 | 21:56 |
| *** SumitNaiksatam has quit IRC | 21:57 | |
| clarkb | fungi: jeblair: can you double check my inline comments? I may be wrong and if I am would like to continue testing that patch | 21:58 |
| fungi | i'm curious to see the comment. i usually don't like misusing os.path semantics for something which isn't actually a local file path, because that's not portable (assuming that this snippet of code will always run on systems using / as a path.set) | 21:58 |
| *** SnowDust has joined #openstack-infra | 21:58 | |
| fungi | clarkb: i think you're right on the basename. i hadn't considered repository names with slashes in them | 21:59 |
| fungi | probably better to trim the string | 21:59 |
| jeblair | i would have thought that patch would have been tested. | 21:59 |
| *** SumitNaiksatam has joined #openstack-infra | 22:00 | |
| jeblair | i don't know why i keep believing that. | 22:00 |
| clarkb | or split(foo, '/', 1) | 22:00 |
| clarkb | jeblair: lol | 22:00 |
| clarkb | I am going to pop it off my stack and reinstall jeepyb in my local virtualenv | 22:00 |
| fungi | k | 22:00 |
| fungi | it's irrelevant to this task anyway, so not a big loss for now | 22:00 |
| *** elasticio has quit IRC | 22:01 | |
| *** sarob_ has quit IRC | 22:01 | |
| clarkb | lol I can't pip install --upgrade . because that tries to reinstall mysql python | 22:01 |
| fungi | i'm going to need to disappear here shortly to go get some dinner' | 22:01 |
| * clarkb creates a new virtualenv | 22:02 | |
| clarkb | and done. https://review.openstack.org/#/c/78667/ still hasn't merged :/ | 22:03 |
| clarkb | waiting on test slaves | 22:03 |
| *** ryanpetrello has quit IRC | 22:04 | |
| lifeless | fungi: could you delete the current tripleo ci slaves ? | 22:05 |
| *** mfink has quit IRC | 22:05 | |
| openstackgerrit | David Ostrovsky proposed a change to openstack-infra/config: Gerrit-2.8: Add secondary index support https://review.openstack.org/60080 | 22:05 |
| fungi | lifeless: i take it something happened? | 22:05 |
| lifeless | fungi: I've bumped the flavor to say 16G not 8 | 22:06 |
| lifeless | fungi: to fix what we believe is the cause of our consistent overcloud failures | 22:06 |
| clarkb | fungi: have we pinged zhiwei? | 22:06 |
| lifeless | fungi: I'm prepping a patch to halve the quota size in nodepool for it | 22:06 |
| fungi | clarkb: not yet. i keep trying to find my way to e-mail. he's in apac so doesn't usually come around until nearly midnight my time | 22:06 |
| *** luisg has quit IRC | 22:07 | |
| clarkb | ya | 22:07 |
| fungi | lifeless: if i delete the nodes, will we get any new ones? nodepool is looking for nodes with at least 8192mb ram so i guess it will work. needs to have "h1" in the flavor name too | 22:08 |
| lifeless | yes flavor name is identical | 22:09 |
| *** SnowDust has quit IRC | 22:09 | |
| *** thomasem has quit IRC | 22:09 | |
| *** luisg has joined #openstack-infra | 22:09 | |
| *** SumitNaiksatam has quit IRC | 22:10 | |
| *** mfer has quit IRC | 22:11 | |
| *** fcarpenter has joined #openstack-infra | 22:11 | |
| *** pdmars has quit IRC | 22:11 | |
| openstackgerrit | A change was merged to openstack-infra/config: Create ACL groups for oslo.rootwrap https://review.openstack.org/78667 | 22:12 |
| fungi | there we go... just in time for the 22:15 pulse | 22:12 |
| clarkb | I think I will start with running just on oslo.rootwrap | 22:13 |
| *** dmsimard has left #openstack-infra | 22:13 | |
| *** SumitNaiksatam has joined #openstack-infra | 22:13 | |
| clarkb | as any bugs with one project should show up for others and subsequent runs can be larger | 22:13 |
| fungi | sounds good. shall i go ahead and approve more, do you want them one-by-one or in a lump? | 22:14 |
| clarkb | lets see how this next one goes | 22:14 |
| clarkb | s/next/first/ | 22:14 |
| fungi | sure thing | 22:14 |
| openstackgerrit | lifeless proposed a change to openstack-infra/config: Reduce max servers for tripleo-cloud. https://review.openstack.org/79091 | 22:15 |
| lifeless | fungi: and ^ | 22:15 |
| lifeless | despite | 22:15 |
| lifeless | Vim: Caught deadly signal SEGV | 22:15 |
| lifeless | Vim: Finished. | 22:15 |
| lifeless | Segmentation fault (core dumped) | 22:15 |
| StevenK | lifeless: 190 / 2 = 95, not 80? | 22:16 |
| lifeless | StevenK: EIDONTCARE | 22:16 |
| StevenK | Haha | 22:16 |
| fungi | i took it as extra breathing room, rather than assuming lifeless doesn't know math | 22:16 |
| lifeless | fungi: exactly | 22:16 |
| clarkb | lifeless: that is apparently fixed in trusty vim | 22:16 |
| clarkb | saucy vim is terrible | 22:16 |
| lifeless | I was using nano and my brain was focused on WTF vim is SEGVing | 22:16 |
| fungi | clarkb: logan agreed, so i'm pushing that updated patchset and adding it to the list now | 22:17 |
| StevenK | If I can't troll with the simple things, how can I troll with the complex ... | 22:17 |
| clarkb | I installed the vim -dbg package and looked at the core dump in gdb briefly but I think it didn't have complete tracebacks fro some reason. I gave up at that point because building vim by hand is meh | 22:17 |
| StevenK | clarkb: You probably need theentireworld-dbg, given what vim links against | 22:18 |
| *** hashar has quit IRC | 22:18 | |
| * fungi hears vim actually links against emacs | 22:19 | |
| StevenK | steven@vanquished:~% ldd /usr/bin/vim | grep emacs | 22:19 |
| StevenK | zsh: done ldd /usr/bin/vim | | 22:19 |
| StevenK | zsh: exit 1 grep emacs | 22:19 |
| * fungi ducks | 22:19 | |
| jeblair | fungi: i don't think it would have crashed if it did. :) | 22:19 |
| fungi | indeed | 22:19 |
| *** sarob_ has joined #openstack-infra | 22:19 | |
| StevenK | I wonder if it still SEGVs if you set syntax off | 22:19 |
| *** bhuvan has quit IRC | 22:19 | |
| *** bhuvan_ has quit IRC | 22:19 | |
| *** bhuvan has joined #openstack-infra | 22:20 | |
| *** hemanthravi has joined #openstack-infra | 22:20 | |
| *** bhuvan_ has joined #openstack-infra | 22:21 | |
| openstackgerrit | Jeremy Stanley proposed a change to openstack-infra/config: Create gerrit-powered-agenda project https://review.openstack.org/76270 | 22:21 |
| *** arosen1 has left #openstack-infra | 22:21 | |
| *** bhuvan_ has quit IRC | 22:23 | |
| *** bhuvan has quit IRC | 22:23 | |
| * clarkb waits for puppet to run on review.o.o | 22:23 | |
| *** harlowja has joined #openstack-infra | 22:23 | |
| StevenK | Huh. vi -u NONE nodepool.yaml.erb works | 22:23 |
| fungi | StevenK: yeah i've assumed it was just a crash bug introduced in the yaml syntax highlighter, but since it's not happening to me on debian i don't care much | 22:24 |
| *** jnoller has quit IRC | 22:24 | |
| clarkb | fungi: ok puppet has updated review.o.o, running manage-projects -d oslo.rootwrap | tee foolog no | 22:25 |
| clarkb | *now | 22:25 |
| *** bhuvan has joined #openstack-infra | 22:25 | |
| clarkb | fungi: ^ that look right to you? | 22:25 |
| StevenK | fungi: Just commenting out syntax enable in my .vimrc doesn't stop the SEGV, though | 22:25 |
| jeblair | clarkb, fungi: i think we could update review.o.o manually so we're not spending 15 minutes waiting between steps... | 22:25 |
| StevenK | After two tries, since I had forgot the comment character is " | 22:25 |
| clarkb | jeblair: yeah, I wanted to avoid messing with puppet but will probably do that for the next round | 22:26 |
| *** bhuvan_ has joined #openstack-infra | 22:26 | |
| fungi | clarkb: looks right, though i just redirected to a file and tailed it | 22:26 |
| * fungi is a slacker | 22:26 | |
| fungi | i find it interesting that https://jenkins03.openstack.org/job/gate-nova-pep8/2531/console made it into the gate | 22:26 |
| *** hemanthravi has quit IRC | 22:27 | |
| clarkb | er openstack/oslo.rootwrap is the name. trying again | 22:27 |
| jeblair | fungi: oh that's like the error flashgordon showed us earlier | 22:27 |
| fungi | yup | 22:27 |
| jeblair | i'll check the zuul logs | 22:27 |
| clarkb | looks like that worked | 22:28 |
| fungi | jeblair: yeah, jenkins failed it in check | 22:28 |
| fungi | jeblair: repeatedly, it seems | 22:28 |
| jeblair | fungi: hrm, and enqueued it into gate 40 minutes later with apparently no intervening activity? | 22:29 |
| fungi | that's extremely strange | 22:29 |
| clarkb | I went ahead and put ttx in the oslo-rootwrap-groups | 22:29 |
| fungi | clarkb: awesome | 22:30 |
| jeblair | fungi: oh: 2014-03-07 20:59:29,242 DEBUG zuul.DependentPipelineManager: Change <Change 0x7f95d10d78d0 72346,10> needs <Change 0x7f95753993d0 71479,18> and is ready to merge | 22:30 |
| dhellmann | clarkb: could you add me, too? ttx is going to be ooo next week | 22:30 |
| jeblair | it got pulled in by a dependency | 22:30 |
| fungi | dependency | 22:30 |
| fungi | yep | 22:30 |
| fungi | i missed it had one which was approved later | 22:30 |
| clarkb | dhellmann: sure | 22:30 |
| dhellmann | clarkb: doug-hellmann | 22:30 |
| fungi | jeblair: so the dependency factor seems to have bypassed the need for a +1 check | 22:31 |
| clarkb | dhellmann: done | 22:31 |
| fungi | or could just be coincidence, but that seems like a corner case code path | 22:31 |
| dhellmann | clarkb: thanks! | 22:31 |
| *** yassine has quit IRC | 22:32 | |
| fungi | clarkb: so did we want to approve the rest and you disable puppet and reset to them one at a time or whatever in a dev env with --test? | 22:32 |
| jeblair | fungi: i believe that's correct. our +1 check is all about triggers, but zuul pulls changes into the queue that are otherwise ready | 22:32 |
| jeblair | regardless of what the triggers say | 22:32 |
| clarkb | fungi: yup | 22:33 |
| clarkb | fungi: that sounds good | 22:33 |
| fungi | i'm probably going to have to skip out on the remainder of today's manage-projects testing exercise due to hungry otherperson hovering | 22:33 |
| clarkb | fungi: remind me though | 22:33 |
| clarkb | will we have a race with cgit? | 22:33 |
| clarkb | eg approve, puppet runs on cgit prior to my manual run on gerrit, that project breaks? | 22:34 |
| fungi | clarkb: yeah, just re-trigger replication once you're sure puppet agent has had a chance to see the projects.yaml update | 22:34 |
| clarkb | thanks | 22:34 |
| fungi | those other three changes still wanted a +2, and the one i added the new patchset for hasn't gotten check results yet | 22:35 |
| fungi | so it's possible i screwed up the alphabetizing | 22:35 |
| *** alexpilotti has quit IRC | 22:35 | |
| openstackgerrit | Khai Do proposed a change to openstack-infra/config: update All-Project configuration for gerrit 2.8 https://review.openstack.org/70818 | 22:36 |
| openstackgerrit | Khai Do proposed a change to openstack-infra/config: update All-Project configuration for gerrit 2.8 https://review.openstack.org/70818 | 22:37 |
| fungi | we've got an unusually high volume of jobs running for this time on a friday | 22:37 |
| jeblair | clarkb: let me know if you want me to do something | 22:37 |
| clarkb | fungi: I went ahead and approved them | 22:37 |
| fungi | clarkb: saw that--thanks | 22:37 |
| clarkb | jeblair: nothing right now, going to let those merge then I will manually pick them off one by one into a dev env and apply puppet agent --test then manually run the script | 22:37 |
| clarkb | jeblair: will ping you if something comes up | 22:37 |
| openstackgerrit | Sergey Lukjanov proposed a change to openstack-infra/config: First step of renaming Savanna to Sahara https://review.openstack.org/79097 | 22:39 |
| *** hogepodge_ has joined #openstack-infra | 22:40 | |
| *** cody-somerville has joined #openstack-infra | 22:40 | |
| *** hemanthravi has joined #openstack-infra | 22:40 | |
| *** hogepodge has quit IRC | 22:40 | |
| *** hogepodge_ is now known as hogepodge | 22:40 | |
| openstackgerrit | A change was merged to openstack-infra/config: Add kickstack https://review.openstack.org/78936 | 22:41 |
| *** julim has quit IRC | 22:42 | |
| *** mriedem has quit IRC | 22:42 | |
| openstackgerrit | A change was merged to openstack-infra/config: Add new project graffiti https://review.openstack.org/78829 | 22:42 |
| clarkb | hrm actually I can just puppet agent --test the top of the stack and get all of them into the server then run manage project on them individually | 22:43 |
| * clarkb does this instead | 22:43 | |
| *** oubiwan__ has quit IRC | 22:43 | |
| * fungi disappears for now... will ping zhiwei and cc the infra ml about possibly renaming tomorrow when doing savanna->sahara | 22:44 | |
| openstackgerrit | Sergey Lukjanov proposed a change to openstack-infra/devstack-gate: Allow Sahara (ex. Savanna) repos to be cloned. https://review.openstack.org/79099 | 22:44 |
| clarkb | fungi: anytime after 10am should be fine with me | 22:44 |
| clarkb | I can do earlier maybe :) | 22:44 |
| clarkb | uh local PST time | 22:44 |
| *** CaptTofu has quit IRC | 22:44 | |
| jeblair | wait, what's going on tomorrow? | 22:44 |
| * fungi will ping zhiwei and cc the infra ml when he returns in a few hours, *about* the rename tomorrow | 22:44 | |
| *** krotscheck has quit IRC | 22:44 | |
| *** CaptTofu has joined #openstack-infra | 22:45 | |
| openstackgerrit | A change was merged to openstack-dev/pbr: Init sphinx config values before accessing them. https://review.openstack.org/78821 | 22:45 |
| clarkb | jeblair: renaming savanna to sahara | 22:45 |
| *** sarob_ has quit IRC | 22:45 | |
| fungi | jeblair: SergeyLukjanov had gotten last-minute clearance for the new project name and wanted to know if we could still do the rename this weekend if he went ahead and drafted the changes | 22:45 |
| * fungi is really gone, really | 22:45 | |
| *** luisg has quit IRC | 22:45 | |
| *** sarob_ has joined #openstack-infra | 22:46 | |
| jeblair | how did i miss scheduling an outage? | 22:46 |
| SergeyLukjanov | jeblair, we'd like to rename earlier to have enough time to complete renaming and test before the RC/GA | 22:46 |
| clarkb | jeblair: because it only came up like not long ago and has been mostly informal I think | 22:46 |
| clarkb | eg we haven't properly scheduled it yet | 22:46 |
| jeblair | SergeyLukjanov: that's great, but we like to schedule gerrit outages and announce them in advance | 22:46 |
| jeblair | clarkb: if it's happening at 10am tomorrow, when is it going to be scheduled? | 22:47 |
| clarkb | jeblair: I think that is what fungi is doing | 22:47 |
| jeblair | clarkb: that's what's happenening in a few hours | 22:47 |
| jeblair | clarkb: fungi is sending _something_ to the ml in a few hours... | 22:48 |
| jeblair | clarkb: and something is apparently happening at 10am... | 22:48 |
| jeblair | tomorrow | 22:48 |
| clarkb | no | 22:48 |
| clarkb | I am saying I can help after that time period | 22:48 |
| clarkb | because fungi hasn't set a time yet | 22:48 |
| *** fcarpenter has left #openstack-infra | 22:48 | |
| jeblair | i don't think this is something that is ready to happen this weekend | 22:49 |
| clarkb | jeblair: thats fair | 22:49 |
| *** CaptTofu has quit IRC | 22:49 | |
| SergeyLukjanov | apologize for last-minute request... we've been waiting name verification for two weeks... | 22:49 |
| *** sarob_ has quit IRC | 22:50 | |
| *** sarob_ has joined #openstack-infra | 22:50 | |
| clarkb | well the other config changes are being slow. I am going to start with graffiti and kickstack now | 22:51 |
| *** luis_ has joined #openstack-infra | 22:51 | |
| SergeyLukjanov | jeblair, heh, that means postpone for a week? :( | 22:52 |
| *** datsun180b has left #openstack-infra | 22:52 | |
| andreaf | sdague, dtroyer: could you plz re-approve https://review.openstack.org/#/c/78808/? I had to rebase it. thanks | 22:52 |
| jeblair | SergeyLukjanov: not necessarily a week; we can schedule downtime whenever we want; but i think we should schedule it and announce it in advance, and it would also be good to have time to review the changes and discuss if there are any complications | 22:54 |
| *** sarob_ has quit IRC | 22:55 | |
| jeblair | SergeyLukjanov: (for instance, will the renames break gating jobs, especially cross-project gating, devstack, etc...) | 22:55 |
| SergeyLukjanov | jeblair, it'll be great for savanna to rename repos asap due to the lack of time before release, names verification took too much time... | 22:55 |
| SergeyLukjanov | jeblair, currently, only async gating is enabled for savanna, so, anyway, it'll affect only one project | 22:56 |
| SergeyLukjanov | jeblair, I think that step-by-step renaming w/o breaking the jobs is possible by overriding repos vars in devstack jobs for savanna | 22:56 |
| clarkb | jeblair: puppet agent --test --environment development --noop is showing changes that don't match what I expect. trying to figure that out now | 22:56 |
| jeblair | SergeyLukjanov: isn't there a client and a server? | 22:57 |
| SergeyLukjanov | jeblair, yup, a mean one project as whole - server, client, etc. | 22:57 |
| *** krtaylor has quit IRC | 22:57 | |
| SergeyLukjanov | jeblair, https://review.openstack.org/#/c/79097/1/modules/openstack_project/files/jenkins_job_builder/config/devstack-gate.yaml | 22:57 |
| *** sweston has quit IRC | 22:58 | |
| *** oubiwan__ has joined #openstack-infra | 22:58 | |
| SergeyLukjanov | jeblair, after this change, we'll be able to update devstack and remove overrides | 22:58 |
| SergeyLukjanov | jeblair, https://review.openstack.org/79099 will make dg clone needed repos | 22:59 |
| *** andreaf has quit IRC | 22:59 | |
| openstackgerrit | A change was merged to openstack-infra/config: Create gerrit-powered-agenda project https://review.openstack.org/76270 | 22:59 |
| jeblair | SergeyLukjanov: i would love to have time to review all of those changes and discuss it with others. i don't have time for that, and i don't think we should schedule downtime this weekend with no notice. | 23:00 |
| jeblair | don't have time for that *today* | 23:00 |
| _david_ | Hah, Gerrit support dark Themes now in CodeMirror: http://i.imgur.com/MT0ynSw.png | 23:01 |
| *** dcramer_ has joined #openstack-infra | 23:01 | |
| SergeyLukjanov | jeblair, ok, thx, I hope that we'll be able to land it till the mid of the next week... | 23:02 |
| openstackgerrit | A change was merged to openstack-infra/config: Add back old projects to replicate to git.o.o https://review.openstack.org/78490 | 23:02 |
| StevenK | \o/ | 23:02 |
| _david_ | How cool is that? ;-) | 23:02 |
| clarkb | jeblair: trying it without the environment now | 23:03 |
| StevenK | How long will that take to hit git.o.o? | 23:03 |
| clarkb | jeblair: its weird and I don't know what I am doing wrong with the dev env | 23:03 |
| clarkb | StevenK: 25 minutes | 23:03 |
| openstackgerrit | Khai Do proposed a change to openstack-infra/config: update trigger to match new workflow label https://review.openstack.org/79103 | 23:03 |
| clarkb | jeblair: nope, normal env does it too | 23:03 |
| clarkb | jeblair: you don't have time to take a look do you? | 23:03 |
| _david_ | StevenK, this is 2.9 ;-) | 23:04 |
| openstackgerrit | Sergey Lukjanov proposed a change to openstack-infra/config: First step of renaming Savanna to Sahara https://review.openstack.org/79097 | 23:04 |
| clarkb | jeblair: puppet was running, I stopped it, then puppet agent --test --noop shows a huge diff of stuff that is ugly | 23:04 |
| *** krotscheck has joined #openstack-infra | 23:04 | |
| jeblair | clarkb: where? | 23:04 |
| clarkb | jeblair: review.o.o | 23:04 |
| clarkb | jeblair: and it doesn't show the new projects | 23:04 |
| *** sarob_ has joined #openstack-infra | 23:05 | |
| jeblair | apparently we download mysql connector and bouncy castle every 10 minutes | 23:05 |
| clarkb | jeblair: maybe, but I think something is wrong | 23:05 |
| jeblair | i'm looking; reading log files | 23:06 |
| clarkb | like why is that delta for the gerrit config showing up and non of the new acl files or project files updating | 23:06 |
| *** luis_ has quit IRC | 23:06 | |
| *** zehicle_at_dell has quit IRC | 23:06 | |
| *** bknudson has quit IRC | 23:06 | |
| *** hemanthravi has quit IRC | 23:07 | |
| *** mfink has joined #openstack-infra | 23:08 | |
| jeblair | clarkb: i think the diff is thanks to gerrit | 23:08 |
| *** zns has quit IRC | 23:08 | |
| *** david-lyle has quit IRC | 23:08 | |
| *** jhesketh has joined #openstack-infra | 23:08 | |
| jeblair | clarkb: i believe it rewrites its own config files after we run the installation step | 23:08 |
| clarkb | ah | 23:09 |
| jeblair | notice the changes are all just syntax things (no quotes -> quotes) | 23:09 |
| *** sarob_ has quit IRC | 23:09 | |
| jeblair | that doesn't answer the other half of your question, but are you sure the new projects aren't already in place? | 23:09 |
| SergeyLukjanov | jeblair, how many days before gerrit maintenance are you expect? | 23:09 |
| clarkb | interesting ya | 23:09 |
| jeblair | SergeyLukjanov: at a minimum we'd want to try to give 24 hour notice | 23:09 |
| openstackgerrit | Michael Krotscheck proposed a change to openstack-infra/storyboard: Updated oslo https://review.openstack.org/78496 | 23:09 |
| jeblair | SergeyLukjanov: usually a few days | 23:09 |
| clarkb | jeblair: so that can explain the diff, now what about the lack of acl and project file updates | 23:10 |
| jeblair | clarkb: what's a new project? | 23:10 |
| *** jhesketh has quit IRC | 23:10 | |
| clarkb | graffiti | 23:10 |
| clarkb | and kickstack | 23:10 |
| SergeyLukjanov | jeblair, ok, got it, so, if we'll be agreed on the changes at monday we can schedule renaming to Tue/Wed | 23:11 |
| SergeyLukjanov | jeblair, thank you | 23:11 |
| jeblair | clarkb: /home/gerrit2/acls/stackforge/graffiti.config exists | 23:12 |
| clarkb | jeblair: oh, gah I bet it was a race | 23:12 |
| jeblair | Mar 7 22:45:40 review puppet-agent[21517]: (/File[/home/gerrit2/acls/stackforge/graffiti.config]/ensure) defined content as '{md5}e66eea76a62213ce9cc566b556941249' | 23:12 |
| clarkb | I stopped puppet after the file went int | 23:13 |
| jeblair | Mar 7 22:53:56 review puppet-agent[21517]: Caught TERM; calling stop | 23:13 |
| jeblair | clarkb: yep ^ | 23:13 |
| jeblair | clarkb: last puppetd run before you stopped it | 23:13 |
| clarkb | jeblair: are you seeing that in /var/log/puppet or syslog? | 23:13 |
| jeblair | syslog | 23:13 |
| clarkb | jeblair: thank you for looking my brain is apparently derping | 23:13 |
| jeblair | clarkb: np. very good thing to check. don't want to accidentally upgrade to 2.8. | 23:13 |
| openstackgerrit | Michael Krotscheck proposed a change to openstack-infra/storyboard: Added DELETE method for projects, stories, and tasks. https://review.openstack.org/77763 | 23:14 |
| jeblair | (or maybe we do, i dunno. ;) | 23:14 |
| jeblair | krotscheck: that's very exciting. ^ :) | 23:14 |
| *** hogepodge has quit IRC | 23:14 | |
| *** rwsu has quit IRC | 23:15 | |
| openstackgerrit | Khai Do proposed a change to openstack-infra/config: update trigger to match new workflow label https://review.openstack.org/79103 | 23:15 |
| *** rpodolyaka has quit IRC | 23:16 | |
| *** mrodden has quit IRC | 23:16 | |
| *** zigo has quit IRC | 23:17 | |
| openstackgerrit | Khai Do proposed a change to openstack-infra/config: update trigger to match new workflow label https://review.openstack.org/79103 | 23:17 |
| *** rpodolyaka has joined #openstack-infra | 23:18 | |
| *** hogepodge has joined #openstack-infra | 23:20 | |
| *** banix has quit IRC | 23:20 | |
| *** rcleere has quit IRC | 23:21 | |
| openstackgerrit | A change was merged to openstack-infra/storyboard: Add superuser check https://review.openstack.org/77859 | 23:23 |
| *** sarob_ has joined #openstack-infra | 23:24 | |
| jeblair | i have completed gerrit-powered-agenda. the acl rules did not seem to take effect. i don't have logs because the tee command i used was not correct | 23:25 |
| openstackgerrit | David Ostrovsky proposed a change to openstack-infra/config: Activate scondary index support https://review.openstack.org/79005 | 23:25 |
| *** hogepodge has quit IRC | 23:26 | |
| *** sarob_ has quit IRC | 23:26 | |
| *** sarob_ has joined #openstack-infra | 23:28 | |
| *** _david_ has quit IRC | 23:29 | |
| *** flaper87 is now known as flaper87|afk | 23:30 | |
| *** rcarrillocruz has joined #openstack-infra | 23:32 | |
| *** sarob_ has quit IRC | 23:32 | |
| *** SumitNaiksatam has quit IRC | 23:33 | |
| clarkb | jeblair: ok, I think I am back. How things on project creation front? | 23:34 |
| *** rpodolyaka has quit IRC | 23:34 | |
| jeblair | clarkb: still nothing here | 23:34 |
| jeblair | https://review.openstack.org/#/admin/projects/openstack-infra/gerrit-powered-agenda,access | 23:34 |
| jeblair | i tried making a local change to the acl file to force m-p to push a new commit but that hasn't caused it to take effect in gerrit | 23:34 |
| *** rcarrillocruz1 has quit IRC | 23:34 | |
| jeblair | however... | 23:35 |
| jeblair | [project] state = active | 23:35 |
| jeblair | is missing from that file | 23:35 |
| jeblair | that wasn't it | 23:36 |
| clarkb | jeblair: do you have logs on disk somewhere? | 23:36 |
| clarkb | oh I see them | 23:36 |
| jeblair | i copied zuul.config to gpa.config and it worked | 23:36 |
| *** amcrn has quit IRC | 23:37 | |
| clarkb | https://review.openstack.org/#/admin/projects/openstack-infra/gerrit-powered-agenda,access I see stuff | 23:37 |
| clarkb | ah | 23:37 |
| * clarkb looks at config | 23:37 | |
| jeblair | i'm reverting gpa.config back | 23:37 |
| jeblair | - requireChangeId = true | 23:37 |
| jeblair | + requiredChangeId = true | 23:37 |
| *** vkozhukalov has quit IRC | 23:38 | |
| clarkb | was ^ the problem? | 23:38 |
| jeblair | well, maybe not the only one; there's no config now | 23:39 |
| *** tcheglov has joined #openstack-infra | 23:39 | |
| *** homeless has quit IRC | 23:40 | |
| clarkb | https://review.openstack.org/gitweb?p=openstack-infra/gerrit-powered-agenda.git;a=blob;f=project.config;h=fe00306f56116abf095ca3ad5cc9fe8dd74e52c1;hb=dff7ca4db6f5e7f68e571a487d52591bc5f3e60f is the current version of the project.config | 23:42 |
| clarkb | which gerrit accepted if it is there | 23:42 |
| jeblair | clarkb: https://review.openstack.org/#/admin/projects/openstack-infra/gerrit-powered-agenda,access | 23:42 |
| jeblair | clarkb: i expect that not to be empty. | 23:42 |
| clarkb | me too | 23:42 |
| clarkb | https://review.openstack.org/gitweb?p=openstack-infra/gerrit-powered-agenda.git;a=blob;f=groups;h=0fe395794f8bc1b6596ab981db2efee14332749d;hb=dff7ca4db6f5e7f68e571a487d52591bc5f3e60f groups file seems correct too | 23:43 |
| *** rpodolyaka has joined #openstack-infra | 23:43 | |
| *** vkozhukalov has joined #openstack-infra | 23:43 | |
| clarkb | nothing in the gerrit error log looks promising | 23:44 |
| *** hogepodge has joined #openstack-infra | 23:44 | |
| jeblair | extra '/'s | 23:44 |
| clarkb | jeblair: the leading slashes? | 23:45 |
| StevenK | Hm. Now I can clone melange, but git does love it at all: | 23:45 |
| jeblair | yep | 23:45 |
| StevenK | warning: You appear to have cloned an empty repository. | 23:45 |
| StevenK | steven@vanquished:~/openstack/openstack/melange% git log | 23:45 |
| StevenK | fatal: bad default revision 'HEAD' | 23:45 |
| jeblair | StevenK: i'll kick replication | 23:45 |
| clarkb | StevenK: we need to force replication for that repo probably | 23:45 |
| jeblair | done, should show up in a sec | 23:45 |
| jeblair | clarkb: it's there now | 23:45 |
| jeblair | StevenK: try now | 23:46 |
| clarkb | jeblair: thanks, I can push a patch for that so that puppet matches reality | 23:46 |
| openstackgerrit | James E. Blair proposed a change to openstack-infra/config: Fix GPA typos https://review.openstack.org/79106 | 23:46 |
| jeblair | clarkb: nm ^ | 23:47 |
| StevenK | jeblair: Looks like success, thanks. | 23:47 |
| StevenK | Also the README.md for gerrit-powered-agenda looks like it's for something else? | 23:47 |
| clarkb | jeblair: perfect | 23:47 |
| clarkb | jeblair: approved | 23:48 |
| jeblair | clarkb: graffiti is done | 23:48 |
| jeblair | StevenK: it could be better, but the project is a 'capstone' project for a compsci university program | 23:49 |
| jeblair | StevenK: so it's not like completely the wrong project or something :) | 23:49 |
| StevenK | jeblair: Right. "NDSU Capstone" just made me think 'uuh ...' | 23:49 |
| jeblair | clarkb: graffiti and kickstack are empty repos, right? | 23:50 |
| clarkb | jeblair: graffiti is, kickstack isnt | 23:50 |
| *** echohead has left #openstack-infra | 23:50 | |
| jeblair | clarkb: http://git.openstack.org/cgit/stackforge/graffiti/ lgtm | 23:50 |
| jeblair | as does https://review.openstack.org/#/admin/projects/stackforge/graffiti,access | 23:50 |
| clarkb | jeblair: yup, agree | 23:50 |
| jeblair | kickstack now | 23:51 |
| clarkb | then we should do a full run | 23:51 |
| jeblair | http://git.openstack.org/cgit/stackforge/kickstack and https://review.openstack.org/#/admin/projects/stackforge/kickstack,access look good | 23:51 |
| clarkb | agree | 23:52 |
| jeblair | clarkb: /home/clarkb/venvjeepyb/bin/manage-projects 2>&1 | tee full.log | 23:52 |
| jeblair | clarkb: ^? | 23:52 |
| clarkb | yeah let me double check | 23:52 |
| jeblair | oh di i need '-d' | 23:52 |
| *** rossella_s has quit IRC | 23:52 | |
| clarkb | -d would make it a lot more verbose | 23:52 |
| jeblair | clarkb: /home/clarkb/venvjeepyb/bin/manage-projects -d 2>&1 | tee full.log | 23:52 |
| clarkb | yeah | 23:52 |
| jeblair | clarkb: your call. -d yes or no? | 23:53 |
| *** Ryan_Lane1 has quit IRC | 23:53 | |
| clarkb | may as well -d it | 23:53 |
| jeblair | running | 23:53 |
| jeblair | clarkb: i have to afk for a bit | 23:53 |
| *** dims_ has quit IRC | 23:53 | |
| *** CaptTofu has joined #openstack-infra | 23:54 | |
| clarkb | jeblair: ok | 23:54 |
| clarkb | jeblair: thank you for the help | 23:54 |
| *** thuc has quit IRC | 23:54 | |
| clarkb | jeblair: before you go if you aren't leaving immediately | 23:54 |
| *** thuc has joined #openstack-infra | 23:54 | |
| clarkb | jeblair: do you think we should go ahead and approve the jeepyb config change? it is being tested ^ | 23:55 |
| clarkb | but it also touches some of the other jeepyb stuff | 23:55 |
| clarkb | meh sounds like a good thing for monday | 23:55 |
| *** Ryan_Lane1 has joined #openstack-infra | 23:57 | |
| *** CaptTofu has quit IRC | 23:58 | |
| *** thuc has quit IRC | 23:59 | |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!
