| *** daya_k has joined #openstack-infra | 00:00 | |
| mordred | clarkb: oh - btw ... | 00:01 |
|---|---|---|
| mordred | speaking of upstreams | 00:01 |
| mordred | the patches I made against ansible and that are sitting in that review in the config repo have all landed upstream | 00:01 |
| clarkb | or as a hack, we can add an option to allauth to do the wrong thing with openid and get username from the openid provider | 00:01 |
| openstackgerrit | Russell Bryant proposed a change to openstack-infra/config: add experimental job for nova with f20 virt-preview https://review.openstack.org/113350 | 00:01 |
| clarkb | mordred: cool | 00:01 |
| *** sdake_ has joined #openstack-infra | 00:02 | |
| *** ianw has quit IRC | 00:02 | |
| *** dkliban is now known as dkliban_afk | 00:03 | |
| *** ianw has joined #openstack-infra | 00:03 | |
| nibalizer | muahahahha upstreaming is super fun | 00:03 |
| *** esker has quit IRC | 00:03 | |
| * anteaya looks for the muahahahha repo | 00:04 | |
| *** e0ne has quit IRC | 00:06 | |
| anteaya | okay just as a reality check, we have some jobs in post sitting at 7 hours | 00:08 |
| *** MarkAtwood has quit IRC | 00:08 | |
| anteaya | now I might have missed an important piece explaning this since i was out for a chunk of the day | 00:08 |
| pleia2 | yep, node starving | 00:08 |
| pleia2 | (ran out) | 00:09 |
| anteaya | but if someone can scrutinize I think we need some scrutiny | 00:09 |
| anteaya | so early fungi had a theory something might be going straight from build to delete | 00:09 |
| anteaya | anyone with root access able to do some splunking? | 00:09 |
| *** e0ne has joined #openstack-infra | 00:10 | |
| pleia2 | anteaya: see clarkb's comments about 20 minutes ago | 00:10 |
| *** rloo_afk has left #openstack-infra | 00:10 | |
| fungi | well, there are about 45 rackspace nodes and 6 hpcloud nodes stuck in error+deleting right now, though in the past few days i've noticed rax is periodically cleaning them up (this may be phschwartz's script in action) | 00:10 |
| *** camunoz has joined #openstack-infra | 00:10 | |
| clarkb | yup and hpcloud appears to have a high error rate | 00:11 |
| fungi | however that's not really enough to make a huge difference in what we're seeing | 00:11 |
| fungi | the error rate however, would | 00:11 |
| clarkb | fungi: the difference is hpcloud is high error rate too | 00:11 |
| *** otherwiseguy has joined #openstack-infra | 00:11 | |
| clarkb | they don't have anything about it on https://community.hpcloud.com/status so maybe I should look closer | 00:11 |
| clarkb | will be a nice break from staring at pootle allauth code | 00:12 |
| *** flwang_ has quit IRC | 00:13 | |
| *** e0ne has quit IRC | 00:13 | |
| anteaya | clarkb: ha ha ha | 00:13 |
| anteaya | well there's that | 00:13 |
| fungi | oh yeah... http://paste.openstack.org/show/93595/ | 00:14 |
| clarkb | mordred: pleia2: I just tried removing django.contrib.auth.backends.ModelBackend from authentication backends and that didn't remove local auth | 00:15 |
| fungi | we burned 6485 nodes on launch errors in hpcloud yesterday, according to the logs | 00:15 |
| fungi | and it's continuing in the current freshly-rotated log as well | 00:16 |
| clarkb | mordred: pleia2: I wonder if horizon devs would be of help /me context switches to nodepool now | 00:16 |
| fungi | 117 in the past 16 minutes | 00:16 |
| anteaya | fungi: go us | 00:16 |
| *** funzo has quit IRC | 00:17 | |
| *** e0ne has joined #openstack-infra | 00:17 | |
| mordred | fungi: what sort of errors are we getting? timeouts? or hard errors? | 00:17 |
| *** dims_ has joined #openstack-infra | 00:18 | |
| mordred | krotscheck, clarkb: I'm starting to hack on launchpad import | 00:18 |
| clarkb | fungi: looks like it is failing to allocate floating ips | 00:18 |
| *** nelsnelson has quit IRC | 00:18 | |
| anteaya | looks like something changed round about 18 hours ago with hp cloud | 00:18 |
| clarkb | fungi: we could be leaking them again /me looks | 00:18 |
| jogo | so sanity check: https://review.openstack.org/#/admin/groups/188,members should list the QA ptl in groups? | 00:18 |
| clarkb | mordred: ^ novaclient throws exceptions when doing floating ips | 00:19 |
| mordred | I do not think there is a _great_ way to maintain multi-project bug+task info ... I'll come back to taht | 00:19 |
| *** daya_k has quit IRC | 00:19 | |
| mordred | clarkb: are we not reaping floating ips again? | 00:19 |
| anteaya | jogo: no idea, that is up to the grenade ptl | 00:19 |
| clarkb | mordred: looking | 00:19 |
| mordred | rememer when we had that problem a while back? | 00:19 |
| jogo | anteaya: grenade is in QA | 00:19 |
| jogo | and there is no QA ptl only tempest ptl https://review.openstack.org/#/admin/groups/355,members | 00:19 |
| anteaya | jogo: so I'd go with yes on that | 00:19 |
| * mordred thinks we should try to re-use floating ips if they exist on the account | 00:19 | |
| jogo | fungi mordred: thoughts | 00:20 |
| anteaya | mtreinish isn't listed there and should be | 00:20 |
| fungi | mordred: http://paste.openstack.org/show/93596/ | 00:20 |
| mordred | in fact, I wrote the code to do that already in the ansible patches - I could backport to nodepool ... | 00:20 |
| jogo | is it easy to rename tempest-ptl to QA ptl? | 00:20 |
| jogo | or something like that | 00:20 |
| fungi | mordred: clarkb: could be fip leaking | 00:20 |
| clarkb | mordred: I still disagree on that | 00:20 |
| mordred | clarkb: ok. let's argue about taht later | 00:20 |
| clarkb | mordred: the scale back is hard | 00:20 |
| clarkb | but ya looks like leaky floating ips | 00:21 |
| anteaya | jogo: or add tempest-ptl to grenade-core group | 00:21 |
| anteaya | jogo: basically it is up to mtreinish | 00:21 |
| mordred | clarkb: agree re: leaky - you want to get them or youwant me to? | 00:21 |
| fungi | last time this came up, tteggel gave us a quick-n-dirty script we could use to find and remove disassociated floating ips | 00:22 |
| *** nelsnelson has joined #openstack-infra | 00:22 | |
| jogo | anteaya: and the naming tempest-ptl in grenade isn't an issue? | 00:22 |
| mtreinish | anteaya, jogo: it's that way now mostly because I haven't been an active reviewer on grenade historically | 00:22 |
| clarkb | mordred: I am trying to remember how you got htem last time? | 00:22 |
| *** mkerrin has quit IRC | 00:22 | |
| anteaya | he can decide and tell us what he wants and we refer to the email and make it so | 00:22 |
| anteaya | jogo: not if mtreinish is happy | 00:22 |
| jogo | mtreinish: and that there is no unified review team in qa | 00:22 |
| anteaya | mtreinish: so what if any renames do you want in gerrit groups | 00:23 |
| *** shashankhegde has quit IRC | 00:23 | |
| mordred | clarkb: nova floating-ip-list | grep ' \- ' | awk '{print $2}' | 00:23 |
| mtreinish | jogo: yeah I guess that's the other half :) | 00:23 |
| mordred | so I'm going to run : nova floating-ip-list | grep ' \- ' | awk '{print $2}' | xargs -n1 nova floating-ip-delete | 00:23 |
| mordred | any objections? | 00:23 |
| anteaya | jogo mtreinish so have a qa meeting item and discuss how you want teh groups to be set up | 00:23 |
| clarkb | mordred: iirc it is faster to use neutron | 00:23 |
| clarkb | mordred: but no objection here | 00:23 |
| mordred | oh - right. why was it faster? | 00:24 |
| fungi | mordred: that will get just floating ips which aren't associated with an instance, i take it? | 00:24 |
| anteaya | mtreinish jogo then tell us and we will move things around to reflect how you want it to be | 00:24 |
| mtreinish | anteaya: hmm, I'll have to think of a structure that makes sense | 00:24 |
| jogo | mtreinish: maybe it would be best with the other grenade cores around | 00:24 |
| jogo | anteaya: thank you | 00:24 |
| anteaya | mtreinish: very good | 00:24 |
| mtreinish | anteaya: I'm not sure it warrants a meeting item | 00:24 |
| clarkb | mordred: because it doesn't proxy through nova which is more hops or some such | 00:24 |
| anteaya | jogo mtreinish np | 00:24 |
| mordred | kk | 00:24 |
| mordred | one sec | 00:24 |
| clarkb | mordred: except neutron output is less useful | 00:24 |
| clarkb | and the command name is different | 00:24 |
| anteaya | mtreinish: however you want to to have some air time | 00:24 |
| mtreinish | jogo: yeah I don't want to fiat this without dtroyer or sdague's input | 00:24 |
| *** xchu has joined #openstack-infra | 00:24 | |
| *** melwitt has quit IRC | 00:24 | |
| clarkb | krotscheck: if the UX folks want a list of grievances they should hang out here :) | 00:24 |
| jogo | mtreinish: hehe | 00:24 |
| mtreinish | because this will be the same discussion for devstack probably pretty soon | 00:25 |
| fungi | mordred: if so, i'm all for it. i think tteggel's script used neutronclient and novaclient to map them up and then generate a list of which ones were orphaned, but it was relatively nontrivial to do that way from python | 00:25 |
| *** gyee has quit IRC | 00:25 | |
| mordred | neutron floatingip-list | grep -v 10.0 | awk '{print $2}' | 00:25 |
| krotscheck | mordred: Thanks | 00:25 |
| fungi | (nontrivial compared to a one-liner at any rate) | 00:25 |
| krotscheck | clarkb: context? | 00:25 |
| *** dimtruck is now known as zz_dimtruck | 00:25 | |
| mordred | krotscheck: the UX folks want to know what it's like to use openstack ... | 00:26 |
| clarkb | krotscheck: `nova floating-ip-list` and `neutron floatingip-list` do the same thing with two different command names and two different output formats | 00:26 |
| mordred | krotscheck: we use it and fail constantly | 00:26 |
| clarkb | krotscheck: its just terrible UX | 00:26 |
| clarkb | mordred: you need to chomp the header | 00:26 |
| clarkb | mordred: or I guess you can just let that floating ip delate of 'id' fail | 00:26 |
| mordred | meh. it'll just fail :) - that was my plan :) | 00:27 |
| clarkb | mordred: sounds good | 00:27 |
| krotscheck | Oh man, I wonder how many have thought about commandline UX. | 00:27 |
| clarkb | mordred: can you add that to the nodepool page on ci.openstack.org? | 00:27 |
| clarkb | krotscheck: is there another UX? | 00:27 |
| krotscheck | clarkb: Yes, there’s also hipster UX, but you clearly haven’t heard of it yet. | 00:27 |
| fungi | krotscheck: i think that's the one they're always talking about on reddit? with the handlebar mustache? | 00:28 |
| krotscheck | fungi: reddit? That’s way too mainstream. | 00:29 |
| clarkb | fungi: on hacker news | 00:29 |
| krotscheck | Since everyone’s feeling punchy, how do y’all feel about a puppet CR for storyboard? https://review.openstack.org/#/c/98474/ | 00:29 |
| clarkb | krotscheck: I am not thrilled about rabbitmq ;) | 00:30 |
| krotscheck | clarkb: Too mainstream for you Portlandians? :) | 00:31 |
| clarkb | krotscheck: no, its super heavy weight for local queues and its logs are bad | 00:31 |
| clarkb | krotscheck: also if you try to HA it split brains are lossy | 00:32 |
| fungi | clarkb: also, your detailed exit codes nodepool prep fix merged, if you wanted to test a manual image update | 00:32 |
| *** armax has left #openstack-infra | 00:32 | |
| clarkb | the split brain handling is basically "Oh I am joining a larger side of the split let me delete all of my data' | 00:32 |
| *** wenlock has quit IRC | 00:32 | |
| clarkb | fungi: thanks | 00:32 |
| *** mkerrin has joined #openstack-infra | 00:33 | |
| fungi | clarkb: also bug 1355167 is perplexing, if you have any ideas | 00:33 |
| uvirtbot | Launchpad bug 1355167 in openstack-ci "run-pep8.sh usage now being output" [Critical,In progress] https://launchpad.net/bugs/1355167 | 00:33 |
| *** e0ne has quit IRC | 00:33 | |
| *** vermarah_ has quit IRC | 00:34 | |
| *** gokrokve has quit IRC | 00:35 | |
| clarkb | fungi: weird | 00:35 |
| openstackgerrit | A change was merged to openstack-infra/jenkins-job-builder: add include/exclude regions/users in svn scm (doc and tests) https://review.openstack.org/108152 | 00:35 |
| fungi | clarkb: it's like there are occasionally slaves getting booted from old snapshots in rax-dfw or something | 00:36 |
| mattoliverau | Maybe delete any old snapshots in DFW just to be safe | 00:37 |
| mattoliverau | old images I mean | 00:37 |
| fungi | mattoliverau: oh, good idea... i should see if there are older ones nodepool doesn't have indexed | 00:37 |
| fungi | mayhaps dfw is just handing us a random old snapshot instead of the one we're asking for on some compute nodes... bad local cache maybe? could that really happen? | 00:38 |
| clarkb | krotscheck: commented. let me know what you think of those things | 00:38 |
| phschwartz | fungi: I have pinged the admins to take a look to push through cleaning them faster | 00:38 |
| krotscheck | clarkb: Thanks | 00:38 |
| *** ZZelle has quit IRC | 00:38 | |
| phschwartz | fungi: Most likely won't be till tomorrow that they get them all cleaned | 00:39 |
| mattoliverau | well its run_pep8.. we should be seeing alot more if it was a bad current image. | 00:39 |
| mattoliverau | so maybe | 00:39 |
| fungi | phschwartz: appreciated. i don't think it's currently crippling us the way our floating ip leak in hpcloud is | 00:39 |
| clarkb | fungi: did run-pep8 regress via some other change maybe? I woudl expect a lot of failures if that was the case | 00:39 |
| phschwartz | :( | 00:39 |
| fungi | clarkb: nope, i checked that first thing | 00:39 |
| clarkb | fungi: cool | 00:40 |
| fungi | clarkb: i even logged into an instance booted from the snapshots there and they had the right (current) version of the script | 00:40 |
| fungi | it's too rare to be all pep8 jobs in rax-dfw, but it's *only* happening in dfw, which is pretty bizarre | 00:40 |
| krotscheck | clarkb: I think I agree with you on the password thing everywhere except for init.pp - I want that file to be as dead-simple to use as possible, i.e. require as little configuration as possible. | 00:41 |
| fungi | and it's been going on since friday, seems | 00:41 |
| krotscheck | clarkb: I’ll figure out the admin thing and get back to you. | 00:41 |
| clarkb | krotscheck: ok | 00:42 |
| clarkb | krotscheck: we can do the passwords in a follow up if the admin and erlang things don't need changes | 00:42 |
| clarkb | krotscheck: since that is how mysql is done I am not super worried about it, but would be good to clean up at some point | 00:42 |
| *** SumitNaiksatam has quit IRC | 00:43 | |
| openstackgerrit | Steve Kowalik proposed a change to openstack-infra/config: Add python3-jobs to os-refresh-config https://review.openstack.org/112493 | 00:44 |
| mordred | fungi: all of the floating ips are cleared now | 00:45 |
| clarkb | oh also apparently the precise image on hpcloud changed? jesusaurus did your patch merge yet? | 00:46 |
| *** marun has quit IRC | 00:46 | |
| *** eharney has joined #openstack-infra | 00:46 | |
| *** marun has joined #openstack-infra | 00:47 | |
| jesusaurus | clarkb: lemme check | 00:47 |
| jesusaurus | clarkb: https://review.openstack.org/#/c/112923/ | 00:48 |
| jesusaurus | not yet | 00:48 |
| *** otter768 has joined #openstack-infra | 00:49 | |
| fungi | mordred: i'll keep an eye out and see if we continue getting any similar tracebacks. hopefully our velocity will pack back up so we can burn through the current pile | 00:49 |
| fungi | s/pack/pick/ | 00:49 |
| mordred | krotscheck, clarkb: so - I don't think there is any way to maintain story/task associations if we import projects at different times | 00:49 |
| mordred | or at least, not a GOOD way | 00:49 |
| krotscheck | mordred: details? | 00:50 |
| fungi | mordred: i think if we import infra projects as a lump, that will probably be mostly fine. when it comes to more general openstack projects, i agree they probably have to all happen in one shot | 00:50 |
| mordred | well, we don't have a place in the storyboard db to store external ids | 00:50 |
| krotscheck | mordred: Why not make the DB ID in storyboard == the ID from launchpad? I seem to recall that we weren’t about to run into ID conflict problems... | 00:51 |
| mordred | krotscheck: so, if we import, say, openstack-ci project, which has several bugs that have an openstack-ci task and tasks for other projects like nova | 00:51 |
| clarkb | jesusaurus: woo /me grabs gpg file to confirm new image name | 00:51 |
| mordred | and we make a story per bug with one task in each for openstack-ci | 00:51 |
| mordred | when we come back and import nova, I don't know that there is a 'good' way to let nova know to attach its task to an existing story | 00:51 |
| fungi | unless our import methodology is far smarter than i'm envisioning, i don't see how we deal with a bug which affects task series for nova, glance and cinder for example, if those projects' bugs are imported at different times. changes could continue to the bug in lp between when nova and cinder got imported, for example | 00:52 |
| *** ramashri has quit IRC | 00:52 | |
| mordred | well, I think we have to do all of openstack in a lump | 00:52 |
| mordred | it's mainly how we deal with us | 00:52 |
| mordred | and I'm mostly thinking "ignor it" | 00:52 |
| fungi | yeah, that was my feeling too | 00:52 |
| mordred | or - we could keep a little local db hash thing | 00:53 |
| mordred | that keeps the mapping for next time | 00:53 |
| fungi | right, for infra bugs, i think we pick one or two fairly disconnected projects to import first and live with any separation of tasks which might occur. then do config and other more active (bug-wise) infra projects in one go | 00:54 |
| fungi | where the cross-project tasks are more likely to be present in greater numbers | 00:55 |
| fungi | probably the biggest schism will be between the openstack-ci bugs and devstack/tempest | 00:56 |
| fungi | though we do generally see a lot of openstack-ci+some_server_project bugs too | 00:56 |
| *** ominakov has joined #openstack-infra | 00:56 | |
| jesusaurus | clarkb: youre a better man than i, i just did a `nova list` | 00:56 |
| jesusaurus | er, image-list | 00:57 |
| clarkb | jesusaurus: fungi has us checking :) | 00:57 |
| fungi | jesusaurus: canonical publishes those signed lists for a reason--may as well not let them feel like it's a wasted effort right? ;) | 00:57 |
| jesusaurus | fungi: fair enough :) | 00:58 |
| *** mkerrin has quit IRC | 00:58 | |
| *** bdpayne has quit IRC | 00:58 | |
| clarkb | fungi: one of my image builds has fialed in puppet due to a pip derp | 00:59 |
| clarkb | fungi: so we should know real soon now if my thing to catch that works | 00:59 |
| fungi | the fact that hpcloud doesn't actually indicate who the partners providing their partner images are makes me a little paranoid. "some dude in the waterfront park provided these images. we think they may be safe" | 00:59 |
| *** yaguang has joined #openstack-infra | 00:59 | |
| * jesusaurus imagines some back-alley cloud images sold out of a trenchcoat | 01:00 | |
| fungi | admittedly it's far better than before when they just rolled their own and you had to hope they were basically standardized builds | 01:00 |
| fungi | but i appreciate that there's a trust chain now | 01:00 |
| * anteaya runs away to find some dinner | 01:01 | |
| *** SumitNaiksatam has joined #openstack-infra | 01:02 | |
| *** amcrn has quit IRC | 01:02 | |
| clarkb | jesusaurus: fungi ok image file signature checked and verified | 01:03 |
| clarkb | it seems happy but I am not in that chain of trust so ya | 01:03 |
| *** packet has joined #openstack-infra | 01:03 | |
| clarkb | fungi: any chance yo uwant to review https://review.openstack.org/#/c/112923/ before it gets approved? | 01:03 |
| fungi | sure--it's probably still hours out from enqueuing anyway | 01:04 |
| fungi | lgtm, +1w | 01:05 |
| clarkb | thank you | 01:06 |
| clarkb | fungi: puppet return code checking worked \o/ | 01:06 |
| fungi | clarkb: awesome! one more major class of troubleshooting annoyance down! | 01:07 |
| fungi | i will definitely not miss the "puppet failed to set our image up, but we're going to use it anyway" issues | 01:07 |
| gus | Who does elastic-recheck reviews? https://review.openstack.org/#/c/110531/ has sat for a week and a half. Unsure when and whom I should prompt... | 01:07 |
| clarkb | gus: usually jogo mtreinish sdague and a few others. you can always check the gerrit group for membership to see who cores are | 01:08 |
| *** unicell has joined #openstack-infra | 01:10 | |
| *** packet has quit IRC | 01:10 | |
| gus | aha, thanks. | 01:10 |
| jogo | gus: usually I try to review them sooner then that | 01:10 |
| *** eharney has quit IRC | 01:10 | |
| gus | jogo: you commented earlier on https://review.openstack.org/#/c/110531/ - does my update address your concerns? | 01:11 |
| jogo | gus: just read your response | 01:12 |
| jogo | so the issue here is: we try to keep the queries fairly tight | 01:12 |
| gus | right, I understand I'm probably doing something unusual here. | 01:12 |
| jogo | so that we don't hide other issues | 01:12 |
| gus | The issue occurs at such a low level and across all projects, that it might surface anywhere. | 01:13 |
| jogo | so we should explicitly say which files (via tags) these line should be in | 01:13 |
| jogo | gus: do you really think its a single issue? | 01:13 |
| jogo | if so thats is a different story | 01:13 |
| *** aysyd has quit IRC | 01:13 | |
| gus | There is an issue with eventlet + mysqldb (the default mysql driver used by sqlalchemy) | 01:13 |
| jogo | but a prelinary look shows only neutro nhits this line (nova works around) | 01:13 |
| clarkb | mordred: pleia2: ok I am going to afk now before I die of heat stroke | 01:14 |
| clarkb | mordred: pleia2: I am curious to see what jeblair thinks of the all auth situation tomorrow | 01:14 |
| clarkb | pleia2: is that on the meeting agenda? | 01:14 |
| gus | yeah, neutron hits this often because they do lots of things inside transactions. | 01:14 |
| gus | but _anything_ that yields the eventlet inside a transaction may trigger this (eg: writing an entry to a logfile). | 01:14 |
| jogo | gus: so this query is a no go actually | 01:14 |
| jogo | gus: not becaause it is not a bug | 01:15 |
| jogo | gus: but because you hit passing jobs over half the time | 01:15 |
| gus | oh, yeah that sounds bad. | 01:15 |
| jogo | gus: in other words this query isn't a good fingerprint of a bug causing a job to fail | 01:15 |
| jogo | gus: its a real bug and oen we should fix, but not one that consistently causes jobs to fail | 01:15 |
| jogo | so its easy for this to hide another bug | 01:16 |
| gus | understood. | 01:16 |
| jogo | gus: I recomend adding this fingerprint to the bug itself in a comment | 01:16 |
| jogo | so you can track its progress and whatnot | 01:16 |
| jogo | but I don't think the query belongs in e-r unfortunately | 01:17 |
| gus | (and even in the nova catch-and-retry case, my understanding is that it would make the nova process freeze for ~30s) | 01:17 |
| krotscheck | mordred: I’m mostly in the “ignore it” camp as well, unless we want to lay the foundation for upstream task federation and build in origin/remoteId. | 01:17 |
| *** mkerrin has joined #openstack-infra | 01:17 | |
| jogo | gus: even if it does, if you use conductors its less of a big deal | 01:17 |
| jogo | gus: still bad | 01:17 |
| gus | jogo: ack. I'll drop the e-r query. | 01:17 |
| jogo | gus: as a single conductor may freeze for 30 seconds but you should have many | 01:17 |
| jogo | gus: thanks for digging into this bug, I would love to see it fixed | 01:18 |
| gus | It's so hard to get cross-project consensus on what is a simple change :( | 01:18 |
| jogo | sigh yeah | 01:18 |
| *** harlowja_at_home has quit IRC | 01:18 | |
| gus | I'm basically making zero progress because (I think) no-one realises it's a common underlying problem that occurs often. | 01:18 |
| jogo | gus: what is your current blocker? | 01:19 |
| gus | instead they debug and whack each specific instances of it. | 01:19 |
| gus | I need mysqlconnector installed on the jenkins environment so we can actually run tests with it. | 01:19 |
| jogo | gus: ahh | 01:20 |
| jogo | well this is the room to sort that out in | 01:20 |
| fungi | gus: what was the latest update on getting it published to pypi.python.org? i thought i heard the author agreed to start doing that? | 01:20 |
| gus | yep, I was told that by an oracle employee too, but then haven't heard anything since ... I | 01:21 |
| fungi | mordred: you talked to geert (or someobody) about that too right? | 01:21 |
| gus | 'll follow up with him. | 01:21 |
| mordred | sudo apt-get install python-mysql.connector | 01:21 |
| *** xianghuihui has quit IRC | 01:21 | |
| gus | mordred: yeah. it's so simple to fix :( | 01:21 |
| pleia2 | clarkb: I think jeblair will still be gone during our meeting tomorrow | 01:21 |
| mordred | this can all be done in devstack and doesn't need anything from infra | 01:21 |
| fungi | mordred: oh, for devstack-based jobs right. i think the other issues were raised about needing it in python 2.6 and 2.7 unit test jobs | 01:22 |
| mordred | are we actually running in to eventlet/mysql issues in unittests? | 01:22 |
| mordred | if it's needed in unittest jobs, then it'sa non-starter until it's on pypi. | 01:22 |
| mordred | if it's not needed in unittest jobs (which I do not believe it is) | 01:23 |
| gus | I _think_ we aren't. | 01:23 |
| gus | because the only unittests I'm aware of that (may) hit mysql are oslo.db's "opportunistic" tests. | 01:23 |
| fungi | it seems unlikely to me, but i don't recall why it was requested unless the proposal was to stop using python-mysqldb at all | 01:23 |
| mordred | then all we need to do is put ina path to devstack which installs python-mysql.connnector from apt and then changes the sqlalchemy string | 01:23 |
| gus | .. and oslo.db runs no tests with eventlet as well. | 01:23 |
| mordred | yip | 01:23 |
| mordred | gus: so, I suggest just making a patch to devstack | 01:24 |
| fungi | yeah, i'm with mordred on this... if the only place you need it is in devstack, then... make devstack bring it in from the distro | 01:24 |
| mordred | I, myself, see no problems with unittests using MySQLdb and devstack using python-mysql.connector | 01:24 |
| gus | I have an oslo.db test that reproduces the problem. That obviously can't go in until then. | 01:25 |
| gus | but that's a lower priority than getting devstack tests running. | 01:25 |
| openstackgerrit | A change was merged to openstack-infra/config: Update and normalize OpenStackID dev DB creds https://review.openstack.org/113345 | 01:26 |
| fungi | stable jobs up to icehouse run on precise and would get 0.3.2, while master jobs get 1.1.6 on trusty. presumably that won't pose a problem? | 01:26 |
| *** xianghui has joined #openstack-infra | 01:26 | |
| *** jyuso has joined #openstack-infra | 01:27 | |
| gus | Here's the devstack change: https://review.openstack.org/#/c/105209/ | 01:27 |
| gus | I think sdague's latest comments mean we won't actually be testing mysqlconnector in gate. | 01:28 |
| *** SumitNaiksatam has quit IRC | 01:28 | |
| mordred | gus: I read those differently | 01:28 |
| mordred | I think he's saying that this needs to be behind an option flag | 01:28 |
| mordred | that defaults to mysqldb | 01:28 |
| mordred | then, we can set up a job that runs with the flag turned to mysqlconnector | 01:28 |
| nibalizer | krotscheck: left one comment on your patch | 01:29 |
| mordred | and, after a while, we can decide to change the default | 01:29 |
| gus | ah ok. and then an additional test run that uses mysqlconnector. ack. | 01:29 |
| nibalizer | i'm +1 on it, just one question i'd like to see answered | 01:29 |
| *** gokrokve has joined #openstack-infra | 01:29 | |
| mordred | yah | 01:29 |
| *** SumitNaiksatam has joined #openstack-infra | 01:29 | |
| gus | until then we'll continue to have ~frequent neutron gate failures, just for full disclosure. | 01:29 |
| mordred | but make it so that devstack knows how to do both | 01:29 |
| mordred | sure | 01:29 |
| mordred | but it's a fairly large change to make, and even in the face of pain, we've learned that not going through stepwise with data can get us in worse trouble later | 01:30 |
| *** esker has joined #openstack-infra | 01:31 | |
| *** xianghui has quit IRC | 01:34 | |
| gus | mordred: yeah agreed. In this case however even if the unknown future locks up occasionally then it will still be no worse than the current situation :/ | 01:34 |
| *** xianghuihui has joined #openstack-infra | 01:34 | |
| gus | I'll poke Ihar to progress his devstack change. | 01:34 |
| gus | thanks all. | 01:34 |
| *** SumitNaiksatam has quit IRC | 01:36 | |
| *** yamahata has joined #openstack-infra | 01:37 | |
| *** mkerrin has quit IRC | 01:37 | |
| openstackgerrit | Aishwarya Thangappa proposed a change to openstack-infra/storyboard: Added a subscription_events table & subscription hander https://review.openstack.org/113016 | 01:38 |
| *** SumitNaiksatam has joined #openstack-infra | 01:38 | |
| *** marun has quit IRC | 01:38 | |
| *** marun has joined #openstack-infra | 01:40 | |
| *** SumitNaiksatam has quit IRC | 01:40 | |
| *** nosnos has joined #openstack-infra | 01:43 | |
| *** nosnos has quit IRC | 01:45 | |
| *** nosnos has joined #openstack-infra | 01:45 | |
| mrodden | so what happened with the keyston 2014.1.2 tag... | 01:46 |
| *** nosnos has quit IRC | 01:46 | |
| clarkb | why cant we use the other native python lib? | 01:46 |
| *** nosnos has joined #openstack-infra | 01:47 | |
| clarkb | mordred gus ^ | 01:48 |
| mordred | clarkb: what other native python lib? | 01:49 |
| fungi | clarkb: because we can't have nice^H^H^H^Hpy3k things | 01:49 |
| clarkb | pymysql | 01:49 |
| mrodden | (what is ^H) | 01:49 |
| clarkb | its native so can be monkey patched | 01:49 |
| clarkb | its MIT licensed iirc | 01:50 |
| clarkb | strives for mysql-python compat and has a cython implementation | 01:50 |
| mordred | clarkb: I have no clue | 01:50 |
| *** dims_ has quit IRC | 01:50 | |
| mordred | gus: ^^ ? | 01:50 |
| gus | so the issue is eventlet monkeypatching. | 01:51 |
| gus | C libraries obviously have a much harder time of that. | 01:51 |
| fungi | mrodden: it got initially pushed for (then cached as) the commit on the tip of master rather than stable/icehouse, so there's a 2014.1.2.1 now | 01:51 |
| mordred | yah. pymysql is pure python | 01:51 |
| gus | oh, sorry you meant another python library. | 01:51 |
| mordred | yah | 01:51 |
| gus | yeah we could. | 01:52 |
| gus | I went for mysqlconnector since its the "official" mysql python client. Presumably others would work too. | 01:52 |
| mrodden | fungi: a lot of my jenkins jobs internally are lodged because they re-tagged and its non-fast-forward :( | 01:52 |
| mrodden | but it sounds like i just need to clean the old tag out | 01:52 |
| fungi | mrodden: yeah, that was a spur of the moment attempt at fixing it, which for obvious reasons in hindsight was not a terribly great idea | 01:54 |
| mrodden | no biggie, as long as its just one project | 01:54 |
| mordred | gus: I like geert and all, but 'official' probably doesn't do as much for us as "released on pypi" and "developed somewhere that we can send patches" | 01:54 |
| fungi | gus: it was also suggested that pymysql already supports py3k | 01:55 |
| clarkb | and pypy | 01:55 |
| fungi | which is a nice step in the right direction there too | 01:55 |
| mordred | yup | 01:55 |
| gus | yeah sure. At the moment, the fear/blockers prevent us switching to pymsql equally. | 01:56 |
| mrodden | ooo pypy | 01:56 |
| gus | so it doesn't really matter exactly which alternative we moving to. | 01:56 |
| fungi | even if the places we use it today might not see immediate py3k/pypy compliance overall | 01:56 |
| mordred | gus: I actually think there are less blockers - infra being opposed to the other one because the files are not published is much harder to deal with than a few devs being afraid of change :) | 01:56 |
| gus | .. the speed tests in the blueprint will need to be repeated. | 01:57 |
| *** mkerrin has joined #openstack-infra | 01:57 | |
| gus | mordred: ack. | 01:57 |
| mordred | btw - I think at this point I'd happily trade a little raw speed for some overall stability | 01:57 |
| gus | I wasn't aware of the importance of pypi issues when I embarked on this particular path ;) | 01:58 |
| fungi | in infra, we get devs over their fear of jumping by pushing them out of the plane | 01:58 |
| gus | mordred: yeah, I'm all for correctness first too. It's been an interesting learning experience trying to address all the different objections raised to what is at it's core a 15 character change to a connection URL ;) | 02:00 |
| mordred | gus: welcome to my life :) | 02:00 |
| gus | haha | 02:00 |
| mordred | gus: the competing requirements up in here can kill a person :) | 02:00 |
| *** weshay has quit IRC | 02:01 | |
| gus | and *everyone* wants to be involved in each decision. | 02:01 |
| mrodden | is this a change for all projects connection strings or just one? | 02:01 |
| mordred | mrodden: all | 02:01 |
| mrodden | that'll be tough | 02:02 |
| gus | depends who you ask. we've had both versions of "-1 do the other thing" | 02:02 |
| *** dims_ has joined #openstack-infra | 02:02 | |
| mordred | hehe | 02:02 |
| mordred | mrodden: clearly do not listen to me | 02:02 |
| clarkb | no no it should be the same imo | 02:02 |
| clarkb | otherwise now our matrix is even bigger | 02:02 |
| clarkb | and it needs to be smaller | 02:03 |
| mrodden | mordred: i dont follow... | 02:03 |
| mordred | clarkb: I agree - but it does not surprise me that someone else disagrees | 02:03 |
| clarkb | if the intent is better gating we use one | 02:03 |
| mordred | mrodden: I was talking like I had an answer - I was clearly only a dude with an opinion | 02:03 |
| mordred | :) | 02:03 |
| mrodden | oh ok | 02:03 |
| mrodden | gotcha... | 02:03 |
| mordred | clarkb: I think the people who are advocating for one project at a time or whatever may find themselves crushed under the mighty power of the Gate | 02:04 |
| mrodden | i would think if using a pure mysql driver allows us to get rid of the _retry_on_deadlock crap then it would be a win | 02:09 |
| mrodden | pure python* | 02:09 |
| *** Ryan_Lane has quit IRC | 02:13 | |
| *** gokrokve has quit IRC | 02:14 | |
| *** flwang_ has joined #openstack-infra | 02:14 | |
| *** cody-somerville has joined #openstack-infra | 02:16 | |
| *** flwang_ has quit IRC | 02:18 | |
| *** bdpayne has joined #openstack-infra | 02:20 | |
| *** homeless_ has quit IRC | 02:22 | |
| *** bhuvan has quit IRC | 02:23 | |
| *** mfer has quit IRC | 02:30 | |
| *** adalbas has quit IRC | 02:33 | |
| *** dims_ has quit IRC | 02:39 | |
| *** dims_ has joined #openstack-infra | 02:40 | |
| *** cody-somerville has quit IRC | 02:41 | |
| *** david-lyle has joined #openstack-infra | 02:44 | |
| *** dims_ has quit IRC | 02:44 | |
| *** sdake_ has quit IRC | 02:47 | |
| *** pcrews has quit IRC | 02:49 | |
| chmouel | mordred: i think the swift functional tests run on a normal devstack (with others openstack services enabled) | 02:50 |
| openstackgerrit | Steve Martinelli proposed a change to openstack/requirements: Add a lower limit to oslosphinx https://review.openstack.org/113423 | 02:55 |
| *** yanghe has joined #openstack-infra | 03:00 | |
| *** dims_ has joined #openstack-infra | 03:00 | |
| *** yanghe has left #openstack-infra | 03:00 | |
| *** esker has quit IRC | 03:01 | |
| *** dims_ has quit IRC | 03:01 | |
| *** yanghe has joined #openstack-infra | 03:01 | |
| *** dims_ has joined #openstack-infra | 03:02 | |
| *** laxity_ has quit IRC | 03:02 | |
| *** laxity_ has joined #openstack-infra | 03:03 | |
| *** jgrimm has joined #openstack-infra | 03:05 | |
| *** lttrl has quit IRC | 03:06 | |
| *** dims_ has quit IRC | 03:06 | |
| *** mkerrin has quit IRC | 03:06 | |
| *** funzo has joined #openstack-infra | 03:10 | |
| *** daya_k has joined #openstack-infra | 03:10 | |
| *** ArxCruz has quit IRC | 03:12 | |
| *** jyuso has quit IRC | 03:12 | |
| *** jyuso has joined #openstack-infra | 03:12 | |
| openstackgerrit | A change was merged to openstack-infra/config: HPCloud has deprecated another 12.04 image https://review.openstack.org/112923 | 03:18 |
| *** nosnos has quit IRC | 03:22 | |
| *** yamamot__ has joined #openstack-infra | 03:26 | |
| *** AlexF_ has joined #openstack-infra | 03:27 | |
| *** yamamoto has quit IRC | 03:29 | |
| *** sdake_ has joined #openstack-infra | 03:29 | |
| *** dims_ has joined #openstack-infra | 03:32 | |
| *** rwsu has quit IRC | 03:38 | |
| *** xchu has quit IRC | 03:38 | |
| *** SumitNaiksatam has joined #openstack-infra | 03:38 | |
| *** dims_ has quit IRC | 03:39 | |
| mordred | chmouel: nod | 03:40 |
| openstackgerrit | Steve Kowalik proposed a change to openstack-infra/config: Add python3-jobs to os-refresh-config https://review.openstack.org/112493 | 03:41 |
| *** crc32 has quit IRC | 03:41 | |
| *** mkerrin has joined #openstack-infra | 03:42 | |
| *** KimJ has quit IRC | 03:42 | |
| mordred | StevenK: ^^ woot | 03:44 |
| mordred | StevenK: also, I'm running a launchpadlib program against your website and it's slow | 03:45 |
| *** otter768 has quit IRC | 03:45 | |
| mordred | StevenK: https://gist.github.com/emonty/4ca365d4bcd284ab9274 | 03:45 |
| *** sdake_ has quit IRC | 03:49 | |
| lifeless | mordred: so did you review my patch? | 03:49 |
| *** mkerrin has quit IRC | 03:49 | |
| *** mkerrin has joined #openstack-infra | 03:50 | |
| *** chandankumar has joined #openstack-infra | 03:55 | |
| *** rfolco has quit IRC | 03:58 | |
| StevenK | mordred: "My" website? :-) | 04:01 |
| openstackgerrit | A change was merged to openstack/requirements: Remove unneeded definitions of Python Source Code Encoding https://review.openstack.org/110912 | 04:02 |
| *** david-lyle has quit IRC | 04:08 | |
| *** david-lyle has joined #openstack-infra | 04:08 | |
| *** garyh has quit IRC | 04:09 | |
| *** david-lyle has quit IRC | 04:13 | |
| *** HenryG is now known as HenryG_afk | 04:14 | |
| *** flwang_ has joined #openstack-infra | 04:15 | |
| *** marun has quit IRC | 04:17 | |
| *** flwang_ has quit IRC | 04:19 | |
| *** chandankumar has quit IRC | 04:20 | |
| *** nosnos has joined #openstack-infra | 04:25 | |
| *** yfried has quit IRC | 04:29 | |
| *** AlexF_ has quit IRC | 04:30 | |
| StevenK | mordred: One suggestion. Set bug = task.bug and then use that, rather than task.bug.owner and so on | 04:33 |
| StevenK | mordred: But it is always going to slow, because lots of data, REST and LP | 04:34 |
| *** MarkAtwood has joined #openstack-infra | 04:38 | |
| *** garyh has joined #openstack-infra | 04:40 | |
| *** praneshp has quit IRC | 04:47 | |
| *** praneshp_ has joined #openstack-infra | 04:47 | |
| anteaya | yay done all the acl file change patchsets and new-project patchsets I could find, that weren't wip | 04:53 |
| anteaya | night | 04:53 |
| *** ArxCruz has joined #openstack-infra | 04:56 | |
| *** ildikov has quit IRC | 05:03 | |
| *** camunoz_ has joined #openstack-infra | 05:05 | |
| *** camunoz_ has quit IRC | 05:06 | |
| *** camunoz__ has joined #openstack-infra | 05:06 | |
| *** bcrochet has quit IRC | 05:06 | |
| *** jamielen- has joined #openstack-infra | 05:07 | |
| *** Longgeek has joined #openstack-infra | 05:07 | |
| *** ianw has quit IRC | 05:08 | |
| *** ianw has joined #openstack-infra | 05:08 | |
| *** jamielennox has quit IRC | 05:08 | |
| *** camunoz has quit IRC | 05:08 | |
| *** bcrochet has joined #openstack-infra | 05:08 | |
| *** maxbit_ has quit IRC | 05:08 | |
| *** camunoz has joined #openstack-infra | 05:09 | |
| *** shashankhegde has joined #openstack-infra | 05:09 | |
| *** camunoz__ has quit IRC | 05:10 | |
| *** enikanorov has quit IRC | 05:10 | |
| *** enikanorov has joined #openstack-infra | 05:11 | |
| *** gema has joined #openstack-infra | 05:12 | |
| *** asettle has quit IRC | 05:12 | |
| *** asettle has joined #openstack-infra | 05:13 | |
| *** jaypipes has joined #openstack-infra | 05:14 | |
| *** gema_ has quit IRC | 05:15 | |
| *** jamielen- is now known as jamielennox | 05:15 | |
| *** chandankumar has joined #openstack-infra | 05:16 | |
| openstackgerrit | A change was merged to openstack-infra/zuul: Bump Babel requirement to >=1.0 https://review.openstack.org/112574 | 05:16 |
| *** chandankumar has quit IRC | 05:17 | |
| *** chandankumar has joined #openstack-infra | 05:17 | |
| *** gema_ has joined #openstack-infra | 05:17 | |
| *** gema has quit IRC | 05:18 | |
| *** gema_ has quit IRC | 05:22 | |
| *** gema has joined #openstack-infra | 05:22 | |
| *** otherwiseguy has quit IRC | 05:22 | |
| *** dims_ has joined #openstack-infra | 05:23 | |
| *** otherwiseguy has joined #openstack-infra | 05:23 | |
| *** AlexF_ has joined #openstack-infra | 05:25 | |
| *** gema has quit IRC | 05:27 | |
| *** gema has joined #openstack-infra | 05:27 | |
| *** dims_ has quit IRC | 05:29 | |
| *** gema_ has joined #openstack-infra | 05:32 | |
| *** bdpayne has quit IRC | 05:33 | |
| *** gema has quit IRC | 05:35 | |
| *** bdpayne has joined #openstack-infra | 05:35 | |
| *** k4n0 has joined #openstack-infra | 05:37 | |
| *** gema has joined #openstack-infra | 05:37 | |
| *** gema_ has quit IRC | 05:38 | |
| openstackgerrit | A change was merged to openstack-infra/devstack-gate: address bashate style violations https://review.openstack.org/101268 | 05:40 |
| *** ianw has quit IRC | 05:42 | |
| *** ianw has joined #openstack-infra | 05:42 | |
| *** camunoz__ has joined #openstack-infra | 05:42 | |
| *** ianw has quit IRC | 05:43 | |
| *** ianw has joined #openstack-infra | 05:43 | |
| *** ianw has quit IRC | 05:43 | |
| *** jamielen| has joined #openstack-infra | 05:44 | |
| *** ianw has joined #openstack-infra | 05:44 | |
| *** ianw has quit IRC | 05:44 | |
| *** jamielen| has quit IRC | 05:44 | |
| *** yfried has joined #openstack-infra | 05:44 | |
| *** ianw has joined #openstack-infra | 05:44 | |
| *** bcrochet has quit IRC | 05:44 | |
| *** ianw has quit IRC | 05:45 | |
| *** ianw has joined #openstack-infra | 05:45 | |
| *** jamielen| has joined #openstack-infra | 05:45 | |
| *** jamielen| has quit IRC | 05:45 | |
| *** ianw has quit IRC | 05:46 | |
| *** jamielennox has quit IRC | 05:46 | |
| *** ianw has joined #openstack-infra | 05:46 | |
| *** camunoz has quit IRC | 05:46 | |
| *** camunoz has joined #openstack-infra | 05:46 | |
| *** jamielennox has joined #openstack-infra | 05:47 | |
| *** bcrochet has joined #openstack-infra | 05:48 | |
| *** ianw has quit IRC | 05:50 | |
| *** ianw has joined #openstack-infra | 05:51 | |
| openstackgerrit | François Magimel proposed a change to openstack-infra/config: Added python-cloudkittyclient to stackforge projects https://review.openstack.org/112507 | 05:52 |
| *** otherwiseguy has quit IRC | 05:53 | |
| *** dtantsur|afk is now known as dtantsur | 05:54 | |
| *** ildikov has joined #openstack-infra | 05:56 | |
| *** bdpayne has quit IRC | 05:58 | |
| *** e0ne has joined #openstack-infra | 05:58 | |
| openstackgerrit | Ilia Meerovich proposed a change to openstack-infra/jenkins-job-builder: Implements wrapper for ownership plugin. This wrapper sets "Inject ownership variables into environment" https://review.openstack.org/113348 | 05:59 |
| *** e0ne has quit IRC | 06:00 | |
| *** camunoz_ has joined #openstack-infra | 06:00 | |
| *** ianw has quit IRC | 06:01 | |
| *** abhishekk has joined #openstack-infra | 06:01 | |
| *** kashyap has joined #openstack-infra | 06:01 | |
| *** camunoz__ has joined #openstack-infra | 06:01 | |
| *** jamielen- has joined #openstack-infra | 06:01 | |
| *** reed has quit IRC | 06:01 | |
| *** reed has joined #openstack-infra | 06:02 | |
| *** loki184 has joined #openstack-infra | 06:02 | |
| *** e0ne has joined #openstack-infra | 06:02 | |
| *** bcrochet has quit IRC | 06:02 | |
| *** dkliban_afk has quit IRC | 06:03 | |
| *** portante has quit IRC | 06:03 | |
| *** markmc has joined #openstack-infra | 06:03 | |
| *** camunoz has quit IRC | 06:04 | |
| *** jamielennox has quit IRC | 06:05 | |
| *** camunoz_ has quit IRC | 06:05 | |
| *** portante has joined #openstack-infra | 06:06 | |
| *** bcrochet has joined #openstack-infra | 06:06 | |
| *** dkliban_afk has joined #openstack-infra | 06:06 | |
| *** SumitNaiksatam has quit IRC | 06:07 | |
| *** evgenyf has joined #openstack-infra | 06:07 | |
| *** shashankhegde has quit IRC | 06:08 | |
| *** SumitNaiksatam has joined #openstack-infra | 06:08 | |
| *** yamamoto has joined #openstack-infra | 06:09 | |
| *** yamamo___ has joined #openstack-infra | 06:10 | |
| *** _nadya_ has joined #openstack-infra | 06:10 | |
| *** e0ne has quit IRC | 06:11 | |
| *** yamamot__ has quit IRC | 06:12 | |
| *** _nadya_ has quit IRC | 06:14 | |
| *** yamamoto has quit IRC | 06:14 | |
| *** AlexF_ has quit IRC | 06:14 | |
| *** ArxCruz has quit IRC | 06:15 | |
| *** AlexF_ has joined #openstack-infra | 06:15 | |
| *** flwang_ has joined #openstack-infra | 06:16 | |
| *** _nadya_ has joined #openstack-infra | 06:16 | |
| *** e0ne has joined #openstack-infra | 06:16 | |
| *** oomichi has quit IRC | 06:16 | |
| *** e0ne has quit IRC | 06:17 | |
| *** Longgeek has quit IRC | 06:19 | |
| *** flwang_ has quit IRC | 06:20 | |
| *** Longgeek has joined #openstack-infra | 06:21 | |
| *** camunoz__ has quit IRC | 06:25 | |
| *** yanghe has quit IRC | 06:27 | |
| *** ildikov has quit IRC | 06:27 | |
| *** yamamo___ has quit IRC | 06:28 | |
| openstackgerrit | lokesh s proposed a change to openstack-infra/config: Adding windows disk-image builder project https://review.openstack.org/110562 | 06:29 |
| *** yanghe has joined #openstack-infra | 06:30 | |
| *** linkid has joined #openstack-infra | 06:30 | |
| *** flaper87|afk is now known as flaper87 | 06:33 | |
| *** p5ntangle has joined #openstack-infra | 06:33 | |
| *** stevemar has quit IRC | 06:34 | |
| *** e0ne has joined #openstack-infra | 06:35 | |
| *** camunoz__ has joined #openstack-infra | 06:38 | |
| *** e0ne has quit IRC | 06:40 | |
| *** ildikov has joined #openstack-infra | 06:40 | |
| *** e0ne has joined #openstack-infra | 06:44 | |
| *** e0ne has quit IRC | 06:46 | |
| *** denis_makogon_ has joined #openstack-infra | 06:50 | |
| *** e0ne has joined #openstack-infra | 06:51 | |
| *** linkid has left #openstack-infra | 06:51 | |
| *** andreykurilin_ has joined #openstack-infra | 06:51 | |
| *** AlexF_ has quit IRC | 06:51 | |
| *** unicell has quit IRC | 06:51 | |
| *** unicell has joined #openstack-infra | 06:52 | |
| *** abhishekk has quit IRC | 06:54 | |
| *** abhishekk has joined #openstack-infra | 06:54 | |
| *** e0ne has quit IRC | 06:56 | |
| *** yamamoto has joined #openstack-infra | 06:58 | |
| openstackgerrit | Kanagaraj Manickam proposed a change to openstack-infra/config: namos - openstack infra device and service manager https://review.openstack.org/109494 | 06:59 |
| *** MarkAtwood has quit IRC | 06:59 | |
| *** e0ne has joined #openstack-infra | 07:00 | |
| *** e0ne has quit IRC | 07:04 | |
| *** asettle has quit IRC | 07:05 | |
| *** _nadya_ has quit IRC | 07:06 | |
| *** trinaths has joined #openstack-infra | 07:07 | |
| *** camunoz__ has quit IRC | 07:07 | |
| *** e0ne has joined #openstack-infra | 07:08 | |
| *** yamamoto has quit IRC | 07:10 | |
| *** e0ne has quit IRC | 07:10 | |
| *** rdopieralski has joined #openstack-infra | 07:11 | |
| *** e0ne has joined #openstack-infra | 07:14 | |
| *** rcarrillocruz has joined #openstack-infra | 07:14 | |
| *** rcarrill` has quit IRC | 07:16 | |
| *** e0ne has quit IRC | 07:16 | |
| *** e0ne has joined #openstack-infra | 07:19 | |
| *** jcoufal has joined #openstack-infra | 07:20 | |
| *** jgallard has joined #openstack-infra | 07:20 | |
| *** e0ne has quit IRC | 07:22 | |
| *** jgallard has quit IRC | 07:23 | |
| *** jgallard has joined #openstack-infra | 07:23 | |
| *** e0ne has joined #openstack-infra | 07:26 | |
| *** sballe has joined #openstack-infra | 07:27 | |
| *** salv-orlando has quit IRC | 07:27 | |
| *** pblaho has joined #openstack-infra | 07:28 | |
| *** yamamoto has joined #openstack-infra | 07:28 | |
| *** e0ne has quit IRC | 07:29 | |
| *** afazekas has joined #openstack-infra | 07:31 | |
| *** yamamoto has quit IRC | 07:31 | |
| *** Guest35108 has joined #openstack-infra | 07:34 | |
| *** e0ne has joined #openstack-infra | 07:34 | |
| *** _nadya_ has joined #openstack-infra | 07:35 | |
| *** jistr has joined #openstack-infra | 07:36 | |
| *** e0ne has quit IRC | 07:36 | |
| *** andreykurilin_ has quit IRC | 07:36 | |
| *** linkid has joined #openstack-infra | 07:36 | |
| *** trinaths has quit IRC | 07:38 | |
| *** e0ne has joined #openstack-infra | 07:40 | |
| *** trinaths has joined #openstack-infra | 07:42 | |
| *** e0ne has quit IRC | 07:42 | |
| *** e0ne has joined #openstack-infra | 07:46 | |
| *** e0ne has quit IRC | 07:48 | |
| *** AlexF_ has joined #openstack-infra | 07:49 | |
| *** mrda is now known as mrda-away | 07:49 | |
| *** denis_makogon_ has quit IRC | 07:49 | |
| *** praneshp_ has quit IRC | 07:51 | |
| *** flwang_ has joined #openstack-infra | 07:51 | |
| *** e0ne has joined #openstack-infra | 07:52 | |
| *** praneshp has joined #openstack-infra | 07:52 | |
| *** jpich has joined #openstack-infra | 07:53 | |
| *** s0nea has quit IRC | 07:54 | |
| *** Krast has joined #openstack-infra | 07:55 | |
| *** e0ne has quit IRC | 07:58 | |
| *** fifieldt__ is now known as fifieldt | 07:59 | |
| *** Krast has quit IRC | 08:02 | |
| *** yamamoto has joined #openstack-infra | 08:02 | |
| *** e0ne has joined #openstack-infra | 08:02 | |
| *** yanghe has quit IRC | 08:03 | |
| *** gema has quit IRC | 08:04 | |
| *** AlexF_ has quit IRC | 08:04 | |
| *** gema has joined #openstack-infra | 08:04 | |
| *** dosaboy_ has joined #openstack-infra | 08:05 | |
| *** e0ne has quit IRC | 08:06 | |
| *** yamamoto has quit IRC | 08:06 | |
| *** Krast has joined #openstack-infra | 08:06 | |
| *** e0ne has joined #openstack-infra | 08:10 | |
| *** e0ne has quit IRC | 08:13 | |
| *** jlibosva has joined #openstack-infra | 08:14 | |
| *** derekh has joined #openstack-infra | 08:15 | |
| *** ihrachyshka has joined #openstack-infra | 08:16 | |
| *** e0ne has joined #openstack-infra | 08:17 | |
| *** zehicle_at_dell has joined #openstack-infra | 08:18 | |
| *** praneshp has quit IRC | 08:18 | |
| *** yaguang has quit IRC | 08:19 | |
| *** yaguang has joined #openstack-infra | 08:19 | |
| *** ihrachyshka has quit IRC | 08:20 | |
| *** ihrachyshka has joined #openstack-infra | 08:21 | |
| *** andreaf has quit IRC | 08:23 | |
| *** andreaf_ has joined #openstack-infra | 08:23 | |
| *** jamielen- is now known as jamielennox|away | 08:24 | |
| *** yanghe has joined #openstack-infra | 08:25 | |
| *** dims_ has joined #openstack-infra | 08:26 | |
| *** amuller has joined #openstack-infra | 08:27 | |
| *** yfried is now known as yfried_afk | 08:27 | |
| *** AlexF_ has joined #openstack-infra | 08:27 | |
| *** yamamoto has joined #openstack-infra | 08:28 | |
| *** p5ntangle has quit IRC | 08:29 | |
| *** p5ntangle has joined #openstack-infra | 08:29 | |
| *** yanghe has quit IRC | 08:29 | |
| *** dims_ has quit IRC | 08:31 | |
| *** gema has quit IRC | 08:33 | |
| *** yamamoto has quit IRC | 08:33 | |
| *** gema has joined #openstack-infra | 08:33 | |
| *** p5ntangle has quit IRC | 08:34 | |
| *** e0ne has quit IRC | 08:35 | |
| *** abhishekk has quit IRC | 08:36 | |
| *** loki184 has quit IRC | 08:36 | |
| *** dosaboy_ has quit IRC | 08:36 | |
| *** gema has quit IRC | 08:36 | |
| *** abhishekk has joined #openstack-infra | 08:37 | |
| *** andreaf has joined #openstack-infra | 08:37 | |
| *** e0ne has joined #openstack-infra | 08:39 | |
| *** Krast has quit IRC | 08:40 | |
| *** andreaf_ has quit IRC | 08:40 | |
| *** Krast has joined #openstack-infra | 08:40 | |
| *** AlexF_ has quit IRC | 08:41 | |
| *** kashyap has quit IRC | 08:41 | |
| *** andreaf_ has joined #openstack-infra | 08:43 | |
| *** gema has joined #openstack-infra | 08:45 | |
| *** andreaf has quit IRC | 08:45 | |
| *** dtantsur is now known as dtantsur|lunch | 08:45 | |
| *** dosaboy_ has joined #openstack-infra | 08:46 | |
| *** dosaboy_ is now known as dosaboy-alt | 08:46 | |
| *** penguinRaider has quit IRC | 08:46 | |
| *** abhishekk has quit IRC | 08:46 | |
| *** dosaboy-alt has quit IRC | 08:47 | |
| *** abhishekk has joined #openstack-infra | 08:47 | |
| *** zehicle_at_dell has quit IRC | 08:48 | |
| *** zehicle_at_dell has joined #openstack-infra | 08:48 | |
| *** penguinRaider has joined #openstack-infra | 08:49 | |
| *** e0ne has quit IRC | 08:52 | |
| openstackgerrit | A change was merged to openstack-infra/storyboard: Add task_id to Timeline Event info https://review.openstack.org/106759 | 08:55 |
| *** AlexF_ has joined #openstack-infra | 08:55 | |
| *** e0ne has joined #openstack-infra | 08:57 | |
| *** e0ne has quit IRC | 08:59 | |
| openstackgerrit | Timur Sufiev proposed a change to openstack-infra/config: Create merlin repo https://review.openstack.org/110325 | 09:03 |
| *** e0ne has joined #openstack-infra | 09:03 | |
| *** AlexF_ has quit IRC | 09:07 | |
| *** mkerrin has quit IRC | 09:07 | |
| *** e0ne has quit IRC | 09:07 | |
| *** AlexF_ has joined #openstack-infra | 09:08 | |
| *** jyuso has quit IRC | 09:08 | |
| *** e0ne has joined #openstack-infra | 09:13 | |
| *** laxity has joined #openstack-infra | 09:15 | |
| *** e0ne has quit IRC | 09:15 | |
| *** laxity_ has quit IRC | 09:15 | |
| *** zehicle_at_dell has quit IRC | 09:17 | |
| *** niedbalski has quit IRC | 09:18 | |
| openstackgerrit | A change was merged to openstack-infra/config: sahara-ptl group should be able to push 2014.1.* https://review.openstack.org/113274 | 09:19 |
| *** e0ne has joined #openstack-infra | 09:19 | |
| *** dosaboy has quit IRC | 09:20 | |
| *** wolsen has quit IRC | 09:20 | |
| openstackgerrit | Mike Heald proposed a change to openstack-infra/nodepool: Add stdout and stderr to exception when ready script fails https://review.openstack.org/113287 | 09:22 |
| *** dosaboy has joined #openstack-infra | 09:23 | |
| *** niedbalski has joined #openstack-infra | 09:24 | |
| *** e0ne has quit IRC | 09:24 | |
| *** wolsen has joined #openstack-infra | 09:25 | |
| *** e0ne has joined #openstack-infra | 09:26 | |
| *** evgenyf has quit IRC | 09:27 | |
| *** yamamoto has joined #openstack-infra | 09:29 | |
| *** e0ne has quit IRC | 09:30 | |
| *** AlexF_ has quit IRC | 09:32 | |
| *** evgenyf has joined #openstack-infra | 09:32 | |
| *** yamamoto has quit IRC | 09:34 | |
| openstackgerrit | Ilya Shakhat proposed a change to openstack-infra/config: Run stackalytics in infra https://review.openstack.org/98656 | 09:34 |
| *** e0ne has joined #openstack-infra | 09:34 | |
| *** loki184 has joined #openstack-infra | 09:36 | |
| openstackgerrit | afazekas proposed a change to openstack-infra/config: Do not fail the install_puppet.sh on 2th run https://review.openstack.org/105002 | 09:36 |
| *** AlexF_ has joined #openstack-infra | 09:36 | |
| *** pelix has joined #openstack-infra | 09:38 | |
| *** e0ne has quit IRC | 09:39 | |
| *** andreaf has joined #openstack-infra | 09:40 | |
| *** andreaf has quit IRC | 09:40 | |
| *** andreaf has joined #openstack-infra | 09:41 | |
| *** andreaf_ has quit IRC | 09:42 | |
| *** e0ne has joined #openstack-infra | 09:44 | |
| DinaBelova | good morning, infra folks! do you know if centos-6 image was updated on the gate? in ceilo we have failures with py26 jobs - like this one http://logs.openstack.org/71/112771/2/check/gate-ceilometer-python26/d6a7161/console.html - "Could not find mongod command"... | 09:44 |
| DinaBelova | SergeyLukjanov - may you take a look on this moment? ^^ | 09:45 |
| DinaBelova | we have blocked gate ATM | 09:45 |
| *** andreaf has quit IRC | 09:46 | |
| *** erw has quit IRC | 09:46 | |
| *** serverascode has quit IRC | 09:47 | |
| DinaBelova | jhesketh, may you looks as well sir? ^^ | 09:47 |
| *** zhiyan has quit IRC | 09:47 | |
| *** simonmcc_ has quit IRC | 09:48 | |
| *** serverascode has joined #openstack-infra | 09:48 | |
| *** Alec-A2 has quit IRC | 09:49 | |
| *** willroberts has quit IRC | 09:49 | |
| *** ctracey has quit IRC | 09:49 | |
| *** Ng has quit IRC | 09:49 | |
| *** skath has quit IRC | 09:50 | |
| *** sweston has quit IRC | 09:50 | |
| openstackgerrit | Mike Heald proposed a change to openstack-infra/nodepool: Add stdout and stderr to exception when ready script fails https://review.openstack.org/113287 | 09:50 |
| *** marcusvrn has quit IRC | 09:51 | |
| *** jraim has quit IRC | 09:51 | |
| *** Alec-A2 has joined #openstack-infra | 09:51 | |
| *** erw has joined #openstack-infra | 09:52 | |
| *** zhiyan has joined #openstack-infra | 09:52 | |
| *** nimrodsun has quit IRC | 09:52 | |
| *** willroberts has joined #openstack-infra | 09:52 | |
| *** simonmcc_ has joined #openstack-infra | 09:53 | |
| *** Alex_Gaynor has quit IRC | 09:53 | |
| *** serverascode has quit IRC | 09:53 | |
| *** ctracey has joined #openstack-infra | 09:53 | |
| *** erlon has quit IRC | 09:53 | |
| *** sweston has joined #openstack-infra | 09:54 | |
| *** skath has joined #openstack-infra | 09:54 | |
| *** jraim has joined #openstack-infra | 09:54 | |
| *** marcusvrn has joined #openstack-infra | 09:54 | |
| *** unicell has quit IRC | 09:55 | |
| *** flaviof_zzz is now known as flaviof | 09:55 | |
| *** zhiyan has quit IRC | 09:57 | |
| *** erw has quit IRC | 09:57 | |
| *** nimrodsun has joined #openstack-infra | 09:57 | |
| *** rlandy has joined #openstack-infra | 09:58 | |
| *** Alex_Gaynor has joined #openstack-infra | 09:59 | |
| *** Guest35108 has quit IRC | 09:59 | |
| *** AlexF_ has quit IRC | 09:59 | |
| *** erlon has joined #openstack-infra | 09:59 | |
| *** zhiyan has joined #openstack-infra | 09:59 | |
| *** serverascode has joined #openstack-infra | 09:59 | |
| *** Ng has joined #openstack-infra | 10:00 | |
| *** erw has joined #openstack-infra | 10:01 | |
| *** andreaf has joined #openstack-infra | 10:04 | |
| *** e0ne has quit IRC | 10:06 | |
| DinaBelova | dear infra folks, please pay attention to the problem I've listed above ^^ Ceilometer gate is actually blocked and we don't know the reason | 10:08 |
| *** yaguang has quit IRC | 10:10 | |
| *** mkerrin has joined #openstack-infra | 10:11 | |
| *** alexpilotti has joined #openstack-infra | 10:12 | |
| *** jgallard has quit IRC | 10:12 | |
| *** ianw has joined #openstack-infra | 10:14 | |
| openstackgerrit | Radomir Dopieralski proposed a change to openstack/requirements: Update xstatic-jquery-ui for Horizon https://review.openstack.org/113184 | 10:15 |
| *** dims_ has joined #openstack-infra | 10:15 | |
| d0ugal | We currently CI against sqlite for Tuskar, I'd like to include a job for mysql. Is this the right place to ask? Where do I start? Is there a doc I should read? | 10:19 |
| d0ugal | My Google fu is failing me thus far. | 10:19 |
| *** dims_ has quit IRC | 10:20 | |
| *** AlexF_ has joined #openstack-infra | 10:21 | |
| *** yaguang has joined #openstack-infra | 10:25 | |
| *** yamamoto has joined #openstack-infra | 10:29 | |
| *** p5ntangle has joined #openstack-infra | 10:30 | |
| *** AlexF_ has quit IRC | 10:31 | |
| *** yamamoto has quit IRC | 10:33 | |
| *** AlexF_ has joined #openstack-infra | 10:34 | |
| *** andreaf_ has joined #openstack-infra | 10:35 | |
| *** yfried_afk is now known as yfried | 10:35 | |
| *** andreaf has quit IRC | 10:37 | |
| *** rfolco has joined #openstack-infra | 10:37 | |
| *** daya_k_ has joined #openstack-infra | 10:38 | |
| *** daya_k has quit IRC | 10:38 | |
| *** daya_k_ is now known as daya_k | 10:39 | |
| *** daya_k has quit IRC | 10:43 | |
| *** andreaf_ has quit IRC | 10:43 | |
| *** sballe has quit IRC | 10:46 | |
| *** Krast has quit IRC | 10:50 | |
| *** ZZelle has joined #openstack-infra | 10:52 | |
| *** jedimike has quit IRC | 10:56 | |
| *** reed has quit IRC | 10:58 | |
| *** loki184 has quit IRC | 11:00 | |
| *** andreaf has joined #openstack-infra | 11:01 | |
| *** jedimike has joined #openstack-infra | 11:01 | |
| *** unicell has joined #openstack-infra | 11:04 | |
| *** andreaf_ has joined #openstack-infra | 11:04 | |
| *** andreaf has quit IRC | 11:08 | |
| *** yamamoto has joined #openstack-infra | 11:10 | |
| *** Longgeek has quit IRC | 11:10 | |
| *** Longgeek has joined #openstack-infra | 11:10 | |
| *** yamahata has quit IRC | 11:11 | |
| *** dims_ has joined #openstack-infra | 11:12 | |
| *** Longgeek has quit IRC | 11:14 | |
| *** yaguang has quit IRC | 11:15 | |
| *** zehicle_at_dell has joined #openstack-infra | 11:18 | |
| openstackgerrit | Mike Heald proposed a change to openstack-infra/nodepool: Add stdout and stderr to exception when ready script fails https://review.openstack.org/113287 | 11:18 |
| *** ArxCruz has joined #openstack-infra | 11:20 | |
| *** hashar has joined #openstack-infra | 11:20 | |
| *** zehicle_at_dell has quit IRC | 11:22 | |
| *** flaviof is now known as flaviof_zzz | 11:23 | |
| *** dims_ is now known as dims | 11:28 | |
| *** willroberts has quit IRC | 11:33 | |
| *** jamielennox|away has quit IRC | 11:34 | |
| *** Longgeek has joined #openstack-infra | 11:34 | |
| *** willroberts has joined #openstack-infra | 11:35 | |
| *** weshay has joined #openstack-infra | 11:36 | |
| *** jamielennox|away has joined #openstack-infra | 11:37 | |
| *** dims has quit IRC | 11:37 | |
| *** p5ntangle has quit IRC | 11:38 | |
| *** p5ntangle has joined #openstack-infra | 11:38 | |
| *** AlexF_ has quit IRC | 11:40 | |
| *** p5ntangle has quit IRC | 11:41 | |
| *** p5ntangle has joined #openstack-infra | 11:41 | |
| *** dtantsur|lunch is now known as dtantsur | 11:44 | |
| *** andreaf_ has quit IRC | 11:45 | |
| *** andreaf_ has joined #openstack-infra | 11:45 | |
| *** p5ntangle has quit IRC | 11:50 | |
| *** mbacchi has joined #openstack-infra | 11:50 | |
| *** p5ntangle has joined #openstack-infra | 11:50 | |
| *** flwang_ has quit IRC | 11:58 | |
| *** s0nea has joined #openstack-infra | 11:59 | |
| *** AlexF_ has joined #openstack-infra | 12:04 | |
| *** jgallard has joined #openstack-infra | 12:06 | |
| *** ominakov has quit IRC | 12:07 | |
| *** prad has joined #openstack-infra | 12:07 | |
| *** trinaths has quit IRC | 12:10 | |
| *** dprince has joined #openstack-infra | 12:10 | |
| *** salv-orlando has joined #openstack-infra | 12:10 | |
| *** p5ntangle has quit IRC | 12:14 | |
| *** adalbas has joined #openstack-infra | 12:16 | |
| *** AlexF_ has quit IRC | 12:17 | |
| SergeyLukjanov | fungi, clarkb, mordred, yeah, it was me (re non-commited change to hierra) | 12:21 |
| fungi | SergeyLukjanov: no worries--i forgot the first few times to | 12:22 |
| fungi | o | 12:22 |
| SergeyLukjanov | fungi, /me going to look how it should looks like | 12:23 |
| fungi | SergeyLukjanov: we basically all commit changes as root and then just sign one-liner commit messages | 12:23 |
| *** dkranz has joined #openstack-infra | 12:24 | |
| fungi | SergeyLukjanov: it's mainly so that we know who changed what roughly when, to make it easy to undo later if needed | 12:24 |
| *** ominakov has joined #openstack-infra | 12:24 | |
| SergeyLukjanov | fungi, yeah, I see git history | 12:24 |
| SergeyLukjanov | fungi, will do it next time ;) | 12:25 |
| fungi | awesome | 12:25 |
| *** ihrachyshka_ has joined #openstack-infra | 12:25 | |
| openstackgerrit | Peter Belanyi proposed a change to openstack-infra/config: Change selenium and jshint to voting for tuskar-ui https://review.openstack.org/113510 | 12:25 |
| *** aysyd has joined #openstack-infra | 12:28 | |
| *** ihrachyshka has quit IRC | 12:28 | |
| *** yamamoto has quit IRC | 12:31 | |
| openstackgerrit | Flavio Percoco proposed a change to openstack-infra/config: Rename Marconi to Zaqar https://review.openstack.org/111244 | 12:33 |
| *** maxbit has joined #openstack-infra | 12:41 | |
| *** miqui has joined #openstack-infra | 12:44 | |
| *** ominakov has quit IRC | 12:46 | |
| *** AlexF_ has joined #openstack-infra | 12:48 | |
| *** dims has joined #openstack-infra | 12:48 | |
| *** dims has quit IRC | 12:49 | |
| *** dims has joined #openstack-infra | 12:49 | |
| *** eglynn has joined #openstack-infra | 12:50 | |
| *** dims is now known as Guest75191 | 12:50 | |
| eglynn | greeting gentle-people! | 12:50 |
| eglynn | FYI a major snafu in Fedora-land has killed the ceilometer py26 gate for the last day | 12:53 |
| eglynn | mongodb packages were retired from EPEL6 by mistake (couchdb was the intended victim) | 12:53 |
| *** k4n0 has quit IRC | 12:53 | |
| *** vhoward has left #openstack-infra | 12:53 | |
| eglynn | for details see ... | 12:54 |
| eglynn | https://www.redhat.com/archives/pulp-list/2014-August/msg00002.html | 12:54 |
| eglynn | https://www.redhat.com/archives/pulp-list/2014-August/msg00016.html | 12:54 |
| eglynn | the bad commit has been reverted and the version re-tagged, pbrady is actively chasing +1s | 12:54 |
| eglynn | but it may take a while for the package restore to propogate thru' the suasage machine ... in the meantime, ceilo gating is blocked | 12:55 |
| * eglynn finally cuts to the chase ... | 12:55 | |
| *** jistr has quit IRC | 12:56 | |
| eglynn | as a stop-gap, would it be possible to install the RPM direct from the Fedora package build system (koji) onto the centos-6 jenkins slaves? | 12:56 |
| eglynn | e.g. yum install -y http://kojipkgs.fedoraproject.org/packages/mongodb/2.4.6/1.el6/x86_64/mongodb-server-2.4.6-1.el6.x86_64.rpm | 12:56 |
| *** jistr has joined #openstack-infra | 12:57 | |
| *** Guest35108 has joined #openstack-infra | 12:57 | |
| *** heyongli has joined #openstack-infra | 12:57 | |
| eglynn | fungi: ^^^ any thoughts on the above question about working around the breakage in EPEL6? | 12:58 |
| *** AlexF_ has quit IRC | 13:00 | |
| *** jcoufal has quit IRC | 13:00 | |
| eglynn | SergeyLukjanov: ^^^ you might know about the feasibility of the EPEL6 work-around proposed above? | 13:01 |
| *** e0ne has joined #openstack-infra | 13:01 | |
| *** HenryG_afk is now known as HenryG | 13:02 | |
| fungi | eglynn: i'd rather we didn't work around breakage if a fix is already percolating. by the time we get the change tested and merged to do that, it will probably no longer be necessary | 13:02 |
| *** gentlesaurus is now known as annegentle | 13:02 | |
| eglynn | fungi: OK, I'll try to getting a better estimate on when the fix will propogate through | 13:03 |
| *** lttrl has joined #openstack-infra | 13:03 | |
| eglynn | fungi: the first indication was "a few days" which is obviously waaay too long | 13:03 |
| *** jcoufal has joined #openstack-infra | 13:03 | |
| fungi | eglynn: keep in mind that everyone in the world using that is probably broken, which likely includes many/most of our downstream users, so having it impact our development process similarly seems minor by comparison | 13:04 |
| *** julim has joined #openstack-infra | 13:04 | |
| *** pblaho has quit IRC | 13:04 | |
| *** juzuluag has joined #openstack-infra | 13:05 | |
| *** pblaho has joined #openstack-infra | 13:05 | |
| *** dustins has joined #openstack-infra | 13:05 | |
| eglynn | fungi: always look on the bright side! ... as a wise Python once said/sang ;) | 13:06 |
| fungi | though i do have to say, in a perverse sort of way, it's hilarious that the epel maintainers would confuse mongodb with couchdb ;) | 13:06 |
| *** ominakov has joined #openstack-infra | 13:06 | |
| *** julim has quit IRC | 13:06 | |
| fungi | so, yeah, if they think it's going to be days before epel consumers will have access to mongodb again then we can try to shoehorn in a workaround. if they expect it to be back in sometime today on the other hand, that's likely to just be wasted effort on our part | 13:07 |
| eglynn | fungi: well, one DB is much the same as another, right! ... they all just store stuff :) | 13:07 |
| eglynn | fungi: cool, got it ... I'll try to get a better fix on a realistic ETA | 13:08 |
| *** bknudson has quit IRC | 13:09 | |
| *** AlexF_ has joined #openstack-infra | 13:10 | |
| *** flaper87 is now known as flaper87|afk | 13:10 | |
| *** dkranz has quit IRC | 13:11 | |
| eglynn | fungi: ildikov just had a eureka moment, how about we set ceilo/py26 as non-voting as the stop-gap? | 13:12 |
| eglynn | fungi: (... seeing as we still get the mongo coverage via py27 on Trusty) | 13:12 |
| *** nosnos has quit IRC | 13:13 | |
| *** shashankhegde has joined #openstack-infra | 13:13 | |
| ildikov | fungi: eglynn: also assuming that this issue will be fixed in more a shorter than longer amount of time, so most prolly we will not ruin the whole python 2.6 world of Ceilometer in that time frame if we choose this option | 13:14 |
| *** chat6062 has joined #openstack-infra | 13:15 | |
| chat6062 | hey all. an i ask here a question about Jenkins job builder? | 13:16 |
| eglynn | fungi: from the horse's mouth ... "I currently do not know when the next push will occur, but I guess within the next 24 hours, since everyone should be back from Flock now." | 13:16 |
| *** jgallard has quit IRC | 13:17 | |
| *** mrmartin has joined #openstack-infra | 13:20 | |
| *** achuprin has quit IRC | 13:20 | |
| fungi | eglynn: seems fine, i'll propose the config change for you real quick | 13:20 |
| eglynn | fungi: excellent thank you sir! | 13:21 |
| fungi | chat6062: you can ask... most of the developers who wrote it hang out here, as well as a lot of users | 13:21 |
| *** changbl has quit IRC | 13:21 | |
| *** AlexF_ has quit IRC | 13:21 | |
| fungi | chat6062: though depending on the specifics of your question, it may be a few hours before people with the right knowledge to answer definitively are awake and paying attention | 13:22 |
| chat6062 | cool. well, when i run Jenkins-jobs script with the test flag there is no output. what can cause this? | 13:23 |
| *** julim has joined #openstack-infra | 13:24 | |
| *** MarkAtwood has joined #openstack-infra | 13:25 | |
| *** pcrews has joined #openstack-infra | 13:26 | |
| *** ptoohill has quit IRC | 13:28 | |
| openstackgerrit | Jeremy Stanley proposed a change to openstack-infra/config: Temporarily ignore Ceilometer Python 2.6 job https://review.openstack.org/113521 | 13:28 |
| fungi | eglynn: so there | 13:28 |
| eglynn | fungi: give that man a chocolate fish! :) | 13:29 |
| eglynn | fungi: thank you sir! | 13:29 |
| *** mjturek has joined #openstack-infra | 13:29 | |
| *** evgenyf has quit IRC | 13:29 | |
| fungi | that will ignore the job result once it merges, but you probably want to consider disabling the mongodb-specific tests in your testsuite instead and having us revert the non-voting change instead if it persists | 13:29 |
| eglynn | cool, will do | 13:30 |
| * eglynn hopes for a quick resolution to the EPEL6 snafu | 13:30 | |
| *** jgallard has joined #openstack-infra | 13:30 | |
| fungi | eglynn: also, are mongodb tests running under trusty for you now? | 13:30 |
| *** bknudson has joined #openstack-infra | 13:30 | |
| *** AlexF_ has joined #openstack-infra | 13:31 | |
| eglynn | fungi: yes, on ceilo/py27 and also via an experimental Tempest job | 13:31 |
| eglynn | (we want to upgrade that experimental job to full voting status, see ... http://lists.openstack.org/pipermail/openstack-dev/2014-August/042505.html ) | 13:31 |
| fungi | eglynn: okay, so that probably wouldn't be the end of the world if you had to stop testing it on centos6 for a while, since you have mongodb coverage voting on trusty | 13:32 |
| *** yamamoto has joined #openstack-infra | 13:32 | |
| fungi | not ideal of course, but not terrible | 13:32 |
| eglynn | fungi: yeap, exactly, thanks! | 13:32 |
| *** esker has joined #openstack-infra | 13:32 | |
| mjturek | hey all, question. so posting 'recheck' as a comment (even as a substring in the comment) causes a Jenkins recheck. Is this intended? For example if I post 'recheck hyper-v' it will also recheck Jenkins | 13:33 |
| *** ptoohill has joined #openstack-infra | 13:33 | |
| alexpilotti | mjturek: we just repalced the Hyepr-V check w recheck-hyper-v because of this | 13:33 |
| mjturek | alexpilotti, I'll need to confirm but I think 'recheck-power-kvm' also triggered Jenkins for me. | 13:35 |
| *** wenlock has joined #openstack-infra | 13:35 | |
| *** rcarrill` has joined #openstack-infra | 13:35 | |
| *** yamamoto has quit IRC | 13:36 | |
| *** dkranz has joined #openstack-infra | 13:37 | |
| *** ociuhandu has joined #openstack-infra | 13:37 | |
| *** rcarrillocruz has quit IRC | 13:38 | |
| *** ptoohill has quit IRC | 13:39 | |
| *** maxbit_ has joined #openstack-infra | 13:40 | |
| alexpilotti | ociuhandu: ^^ | 13:40 |
| fungi | mjturek: yep, intended. we are phasing out bug tracking via recheck comments (because elastic-recheck patterns are far more reliable) and the first phases of that are to support "\nre(check|verify)( .*)?" to rerun jobs in upstream ci | 13:40 |
| *** reaper has joined #openstack-infra | 13:40 | |
| fungi | er, actually, the regex is currently "(?i)^(Patch Set [0-9]+:)?( [\w\\+-]*)*(\n\n)?\s*(recheck|reverify)" | 13:41 |
| *** reaper has quit IRC | 13:41 | |
| fungi | so yes, even "recheck-power-kvm" will do the same | 13:41 |
| ociuhandu | fungi: would it be ok if we update the trigger for Hyper-V CI to “hyper-v-recheck” for instance? | 13:41 |
| *** mriedem has joined #openstack-infra | 13:42 | |
| *** maxbit has quit IRC | 13:42 | |
| *** esker has quit IRC | 13:42 | |
| *** homeless has joined #openstack-infra | 13:42 | |
| fungi | ociuhandu: you could make it whatever you want. i'm not convinced that there's anything wrong with retriggering openstack project ci jobs when you also retrigger third-party jobs | 13:42 |
| *** mfer has joined #openstack-infra | 13:42 | |
| ociuhandu | fungi: I know there’s no established syntax for 3rd party CIs and it’s been under discussion if there should be any but I think it would be helpful for the community to find a way to trigger only a particular 3rd party CI | 13:43 |
| mjturek | fungi, thanks!! | 13:43 |
| fungi | see bug 1355480 and http://lists.openstack.org/pipermail/openstack-infra/2014-August/001681.html for discussion | 13:43 |
| uvirtbot | Launchpad bug 1355480 in openstack-ci "recheck/reverify comment_filter is too loose" [Undecided,Opinion] https://launchpad.net/bugs/1355480 | 13:43 |
| *** MarkAtwood has quit IRC | 13:43 | |
| ociuhandu | fungi: triggering Jenkins on any recheck will just increase the load on the CI, for patches that were already successful | 13:43 |
| fungi | ociuhandu: i personally think that's dangerous, but am open to other ideas | 13:44 |
| *** stevemar has joined #openstack-infra | 13:44 | |
| fungi | ociuhandu: and trust me, people rechecking because of a third-party ci failure are not going to make a measurable impact on our resource consumption | 13:44 |
| *** chat6062 has left #openstack-infra | 13:45 | |
| *** ptoohill has joined #openstack-infra | 13:45 | |
| *** e0ne has quit IRC | 13:48 | |
| *** jistr has quit IRC | 13:49 | |
| fungi | SergeyLukjanov: if you're around, care to review https://review.openstack.org/113521 (i think we can safely approve it if you're +2 on it) | 13:49 |
| *** jistr has joined #openstack-infra | 13:50 | |
| *** pblaho is now known as pblaho-meeting | 13:50 | |
| *** pblaho-meeting is now known as pblaho_ | 13:50 | |
| fungi | SergeyLukjanov: it's currently blocking ceilometer changes | 13:50 |
| SergeyLukjanov | fungi, /me looking | 13:50 |
| *** pblaho has joined #openstack-infra | 13:50 | |
| *** pblaho is now known as pblaho-meeting | 13:50 | |
| *** marun has joined #openstack-infra | 13:51 | |
| SergeyLukjanov | fungi, approved | 13:51 |
| SergeyLukjanov | eglynn, DinaBelova, ^^ | 13:52 |
| SergeyLukjanov | fungi, do you remember my attempt to grant myself permissions to push 2014.1.* tags to sahara? | 13:52 |
| SergeyLukjanov | fungi, it's not working | 13:52 |
| SergeyLukjanov | :( | 13:52 |
| *** maxbit_ has quit IRC | 13:53 | |
| *** flaviof_zzz is now known as flaviof | 13:53 | |
| *** evgenyf has joined #openstack-infra | 13:53 | |
| SergeyLukjanov | fungi, looks like th last rule isn't enough :( https://review.openstack.org/#/admin/projects/openstack/sahara,access | 13:54 |
| *** p5ntangle has joined #openstack-infra | 13:55 | |
| *** pdmars has joined #openstack-infra | 13:55 | |
| *** pdmars has quit IRC | 13:55 | |
| fungi | SergeyLukjanov: hrm, that's disappointing | 13:55 |
| *** pdmars has joined #openstack-infra | 13:56 | |
| fungi | SergeyLukjanov: i don't think we've ever tried to do that before (restricting/allowing particular tags by pattern) | 13:56 |
| fungi | i wonder if gerrit is merely incapable, or if it has to be implemented in a different manner | 13:56 |
| eglynn | SergeyLukjanov: thank you sir! | 13:57 |
| openstackgerrit | Doug Hellmann proposed a change to openstack-dev/pbr: add docs env to tox https://review.openstack.org/113529 | 13:57 |
| *** julim has quit IRC | 13:58 | |
| openstackgerrit | A change was merged to openstack-infra/config: Temporarily ignore Ceilometer Python 2.6 job https://review.openstack.org/113521 | 13:59 |
| *** heyongli has quit IRC | 13:59 | |
| *** flwang_ has joined #openstack-infra | 13:59 | |
| *** doug-fish has joined #openstack-infra | 13:59 | |
| *** harlowja_at_home has joined #openstack-infra | 13:59 | |
| SergeyLukjanov | fungi, yeah, I'm trying to find docs or it too | 14:00 |
| *** julim has joined #openstack-infra | 14:01 | |
| *** flwang_ has quit IRC | 14:03 | |
| openstackgerrit | Julien Danjou proposed a change to openstack-infra/config: Import pylockfile https://review.openstack.org/101911 | 14:04 |
| *** afazekas has quit IRC | 14:05 | |
| *** dkranz has quit IRC | 14:05 | |
| *** wenlock has quit IRC | 14:06 | |
| *** hashar has quit IRC | 14:08 | |
| *** gargola has joined #openstack-infra | 14:08 | |
| *** raildo has quit IRC | 14:09 | |
| *** flaper87|afk is now known as flaper87 | 14:09 | |
| *** armax has joined #openstack-infra | 14:10 | |
| *** zz_dimtruck is now known as dimtruck | 14:10 | |
| *** raildo has joined #openstack-infra | 14:11 | |
| stevemar | hey infra folks, looks like ceilometer builds are failing py26: http://lists.openstack.org/pipermail/openstack-dev/2014-August/042704.html and a bug has been opened: https://bugs.launchpad.net/ceilometer/+bug/1355596 | 14:12 |
| uvirtbot | Launchpad bug 1355596 in ceilometer "mongod command not found" [Undecided,New] | 14:12 |
| *** maxbit has joined #openstack-infra | 14:12 | |
| openstackgerrit | Christian Berendt proposed a change to openstack-infra/config: Check requirements of stackforge/gerrit-dash-creator https://review.openstack.org/112768 | 14:13 |
| fungi | stevemar: see https://review.openstack.org/113521 | 14:13 |
| fungi | stevemar: not a bug in ceilometer or our infrastructure. epel accidentally deleted mongodb | 14:14 |
| stevemar | fungi, excellent! | 14:14 |
| stevemar | fungi, d'oh | 14:14 |
| *** hdd has joined #openstack-infra | 14:14 | |
| *** jistr has quit IRC | 14:14 | |
| *** otherwiseguy has joined #openstack-infra | 14:15 | |
| fungi | stevemar: that job is no longer voting while epel gets a chance to restore those packages and/or ceilometer devs disable mongodb tests in the python 2.6 job | 14:15 |
| *** jistr has joined #openstack-infra | 14:15 | |
| *** p5ntangl_ has joined #openstack-infra | 14:16 | |
| stevemar | fungi, yeah, that makes sense, i hope mongodb comes back soon | 14:16 |
| *** p5ntangle has quit IRC | 14:19 | |
| openstackgerrit | Julien Danjou proposed a change to openstack-infra/config: Import pylockfile https://review.openstack.org/101911 | 14:19 |
| *** hdd has quit IRC | 14:20 | |
| *** afazekas has joined #openstack-infra | 14:20 | |
| yolanda | mordred, jeblair, did you have any chance to review nodepool changes for dib? | 14:20 |
| *** dkranz has joined #openstack-infra | 14:21 | |
| *** jistr has quit IRC | 14:21 | |
| *** zz_gondoi is now known as gondoi | 14:21 | |
| *** jistr has joined #openstack-infra | 14:21 | |
| alexpilotti | mjturek: confirmed, we now opted for “check hyper-v”, but it’d be great to have a standard wau to do this :-) | 14:21 |
| *** e0ne has joined #openstack-infra | 14:22 | |
| *** abhishekk has quit IRC | 14:23 | |
| *** pblaho-meeting is now known as pblaho | 14:24 | |
| *** nelsnelson has quit IRC | 14:24 | |
| *** jgallard has quit IRC | 14:25 | |
| *** pblaho has quit IRC | 14:26 | |
| *** david-lyle has joined #openstack-infra | 14:26 | |
| *** pblaho has joined #openstack-infra | 14:27 | |
| *** wenlock has joined #openstack-infra | 14:28 | |
| mjturek | alexpilotti, cool! it would be nice | 14:29 |
| *** nelsnelson has joined #openstack-infra | 14:30 | |
| *** ildikov has quit IRC | 14:30 | |
| *** p5ntangl_ has quit IRC | 14:31 | |
| *** p5ntangle has joined #openstack-infra | 14:31 | |
| *** yamamoto has joined #openstack-infra | 14:32 | |
| *** salv-orlando has quit IRC | 14:33 | |
| SergeyLukjanov | fungi, could you please grant me god permissions on review-dev? | 14:33 |
| SergeyLukjanov | fungi, is it possible to create projects on it? | 14:34 |
| *** dangers_away is now known as dangers | 14:35 | |
| fungi | SergeyLukjanov: yeah, you can do it through the webui or via review_dev.projects.yaml i think | 14:35 |
| fungi | SergeyLukjanov: what's your account on review-dev? i can't find you to add to the group | 14:35 |
| SergeyLukjanov | fungi, slukjanov | 14:35 |
| fungi | SergeyLukjanov: you're in the administrators group now | 14:36 |
| SergeyLukjanov | fungi, thank you! | 14:36 |
| fungi | you're welcome, and good luck! | 14:36 |
| *** yamamoto has quit IRC | 14:37 | |
| SergeyLukjanov | fungi, heh, gerrit said "Invalid Name: refs/tags/2014.2.*" :) | 14:38 |
| SergeyLukjanov | fungi, refs/tags/* is ok, but refs/tags/2014.1* isn't working | 14:39 |
| SergeyLukjanov | fungi, that's funny | 14:39 |
| fungi | yeah, not sure what the limits on gerrit acl ref patterns are | 14:40 |
| *** datsun180b has joined #openstack-infra | 14:43 | |
| openstackgerrit | Devananda van der Veen proposed a change to openstack/requirements: Set a minimum version for python-ironicclient https://review.openstack.org/113542 | 14:43 |
| *** markmcclain has joined #openstack-infra | 14:44 | |
| *** funzo has quit IRC | 14:47 | |
| *** funzo has joined #openstack-infra | 14:48 | |
| *** esker has joined #openstack-infra | 14:49 | |
| *** gokrokve has joined #openstack-infra | 14:50 | |
| openstackgerrit | Devananda van der Veen proposed a change to openstack/requirements: Set a minimum version for python-ironicclient https://review.openstack.org/113542 | 14:52 |
| *** AlexF_ has quit IRC | 14:52 | |
| *** Guest75191 is now known as dims | 14:54 | |
| *** ihrachyshka_ has quit IRC | 14:55 | |
| *** dmsimard_away is now known as dmsimard | 14:55 | |
| fungi | jraim: ttx: should https://review.openstack.org/84811 have included python-kiteclient too? | 14:57 |
| *** AlexF_ has joined #openstack-infra | 14:57 | |
| fungi | trying to figure out the scope of the stackforge->openstack transition for the kite projects | 14:57 |
| david-lyle | I'd like to actually use https://launchpad.net/~horizon-bugs to manage Horizon bugs, can someone in Openstack Infra add me as an adminstrator? | 14:59 |
| david-lyle | I know jeblair has been able to make such changes in the past | 15:00 |
| *** markmcclain1 has joined #openstack-infra | 15:00 | |
| ttx | fungi: if there is such thing, certainly | 15:00 |
| fungi | david-lyle: i can | 15:00 |
| *** markmcclain has quit IRC | 15:01 | |
| *** jlibosva has quit IRC | 15:01 | |
| david-lyle | fungi: thanks! | 15:01 |
| fungi | david-lyle: are you also david-lyle on lp? | 15:02 |
| david-lyle | yes | 15:02 |
| david-lyle | I'm uncomplicated | 15:02 |
| *** james_li has joined #openstack-infra | 15:02 | |
| fungi | david-lyle: done | 15:02 |
| david-lyle | fungi: thank you | 15:02 |
| *** maxbit has quit IRC | 15:03 | |
| *** rcarrillocruz has joined #openstack-infra | 15:03 | |
| fungi | you're welcome | 15:03 |
| *** rcarrill` has quit IRC | 15:04 | |
| *** radez_g0n3 is now known as radez | 15:04 | |
| SergeyLukjanov | fungi, I found the solution - such regex should be prefixed with ^ | 15:05 |
| SergeyLukjanov | fungi, it works on review-dev | 15:05 |
| SergeyLukjanov | fungi, /me creating CR to fix | 15:05 |
| *** maxbit has joined #openstack-infra | 15:07 | |
| *** UtahDave has joined #openstack-infra | 15:07 | |
| *** UtahDave has left #openstack-infra | 15:07 | |
| fungi | excellent | 15:08 |
| *** Ajaeger has joined #openstack-infra | 15:08 | |
| *** dkranz has quit IRC | 15:08 | |
| *** afazekas has quit IRC | 15:08 | |
| openstackgerrit | Sergey Lukjanov proposed a change to openstack-infra/config: Fix sahara tag acl to make it working https://review.openstack.org/113552 | 15:09 |
| SergeyLukjanov | fungi, ^^ | 15:09 |
| SergeyLukjanov | fungi, I think gerrit is trying to parse refs string and not always could understand that it's a real regexp | 15:10 |
| SergeyLukjanov | fungi, and supports only one simple wildcards otherwise | 15:11 |
| SergeyLukjanov | fungi, could you please fast approve it when have some time to unblock sahara release? | 15:11 |
| fungi | SergeyLukjanov: done. amusing that adding a (seemingly unimportant) ^ anchor makes all the difference | 15:12 |
| SergeyLukjanov | fungi, I think it's some kind of parsing magic :( | 15:13 |
| SergeyLukjanov | fungi, thank you | 15:13 |
| *** amuller_ has joined #openstack-infra | 15:13 | |
| *** todd_dsm has joined #openstack-infra | 15:13 | |
| Ajaeger | fungi: Is my change to merge the propose-requirements scripts fine now or do you have further suggestions for improvement? Could you have a look, please? https://review.openstack.org/111344 | 15:14 |
| *** amuller has quit IRC | 15:15 | |
| *** MarkAtwood has joined #openstack-infra | 15:16 | |
| fungi | Ajaeger: i have it up in a tab hoping to get to it after the current meeting i'm in | 15:17 |
| openstackgerrit | Steve Martinelli proposed a change to openstack-dev/pbr: cleanup tox.ini https://review.openstack.org/113553 | 15:18 |
| YorikSar | fungi, mordred, jhesketh: Hi. Have you happen to see my thread on ML about local branches support for git-review? I'd like to know your overall opinion on this as you're those who might -2 this change if I do it within git-review. In such case I'd start working on a tool for this outside git-review. | 15:18 |
| Ajaeger | Thanks! Just have a few minutes while waiting for my plane and wanted to chat with you if the time is fine. But let's followup the usual process and if I have questions, I'll come back... | 15:18 |
| YorikSar | fungi, mordred, jhesketh: http://lists.openstack.org/pipermail/openstack-dev/2014-August/041872.html | 15:19 |
| fungi | YorikSar: it's not a feature i would use, and my main concerns over it would be adding complexity to the tool and possibly encouraging confusing/dangerous workflow choices for new contributors, but i've mostly just been watching the thread to see what other users of the tool think | 15:20 |
| *** dkranz has joined #openstack-infra | 15:21 | |
| YorikSar | fungi: Yes, the concern that such feature can be used as "autosquash" we'd like to avoid for newcommers has been raised on the thread... | 15:21 |
| *** afazekas has joined #openstack-infra | 15:21 | |
| fungi | in my opinion git-review is, first and foremost, a means of simplifying the contribution process for developers who are new to git and gerrit, far more than encoding arbitrary workflows for the benefit of seasoned devs | 15:22 |
| YorikSar | fungi: Doesn't hiding it behind config option solve that? | 15:22 |
| YorikSar | fungi: Ok, I see... | 15:22 |
| phschwartz | YorikSar: No it doesn't as the commit still has to be pushed to a repo other then what was cloned for it to appear in gerrit, so git-review makes it easier, by handling that for you. | 15:23 |
| fungi | possibly, but i want to see the conversation play out between people who are developers and new contributor mentors, since their opinions are what i'm more interested in | 15:23 |
| YorikSar | fungi: I thought it'd a right place to simplify life of seasoned dev as well :) | 15:23 |
| fungi | my opinion is, to a great extent, secondary. i will support consensus among the community if it makes sense | 15:24 |
| *** changbl has joined #openstack-infra | 15:24 | |
| YorikSar | fungi: Unfortunatelly that thread had died off after strong opposition provided by Ben... | 15:25 |
| phschwartz | fungi: I have talked about this exact thing with a group of other devs before, and their thoughts were the same as yours. | 15:25 |
| phschwartz | It would be interesting to see the discussion to a greater extent | 15:26 |
| YorikSar | phschwartz: I'm not sure I understand what repos you're talking about... | 15:26 |
| *** skolekonov has quit IRC | 15:26 | |
| *** eharney has joined #openstack-infra | 15:26 | |
| Ajaeger | SergeyLukjanov: could you recheck https://review.openstack.org/113080, please? Do you propose a different solution? | 15:26 |
| Ajaeger | right now the requirements-proposal job fails completely. | 15:26 |
| Ajaeger | (I mean the post job) | 15:26 |
| openstackgerrit | afazekas proposed a change to openstack-infra/elastic-recheck: Add query for fixed ip shortage with n-net https://review.openstack.org/112558 | 15:26 |
| phschwartz | repos is a generic term. I was just reiterating why git-review is used. in the repos It refered to the base clone off of git.openstack.org or github.com and the gerrit internal repo that you push to when you do a git-review | 15:27 |
| *** amotoki has joined #openstack-infra | 15:27 | |
| YorikSar | phschwartz: My proposal is to support workflow a bit different from 'git review' - fix issues - amend/rebase - 'git review' | 15:28 |
| *** Ajaeger has quit IRC | 15:28 | |
| *** alkari has joined #openstack-infra | 15:29 | |
| YorikSar | phschwartz: And it will still work for repos cloned from elsewhere. | 15:29 |
| phschwartz | YorikSar: I have read your proposal, and don't agree that it solves that as to how gerrit uses patchsets. I personally feel it will cause more greif then help for new developers. | 15:30 |
| fungi | YorikSar: it was also unclear to me how you use your local branch to distill/refactor your various historical edits into a series of dependent changes on a topic branch before pushing to gerrit (rather than the trivial use case of squashing them to one commit before pushing) | 15:31 |
| YorikSar | phschwartz: By 'solve that' I meant that new devs will have to explicitely turn new behavior on to use that. And they'll have to accept that they understand that their local view of the repo will differ a bit from what Gerrit sees. | 15:31 |
| phschwartz | And to note, if that is how you like to work, you can do it. gerrit only requires that you send on patch with a change-id to gerrit. Nothing is stopping you from doing it locally that way. | 15:31 |
| *** todd_dsm has quit IRC | 15:32 | |
| *** yamamoto has joined #openstack-infra | 15:32 | |
| fungi | YorikSar: do the changes with change-ids in their commit messages act as sequential checkpoints in your local history? | 15:32 |
| phschwartz | YorikSar: Which is the exact reason why I would personally be agaisnt it being in there. If there is an issue, it will make it harder for the dev using it to troubleshoot it with help from other users as they will see different things in their repo (makes rebase/merge conflicts harder to help a developer fix) | 15:33 |
| *** todd_dsm has joined #openstack-infra | 15:33 | |
| YorikSar | fungi: For every change request in upstream I'd like to have a separate local branch. Tip point of that branch will be uploaded as a new patchset when you run git-review. If your branch has merge commits in it, that patchset will depend either on master (last commit merged from master to local branch) or another change request (last local branch merged into this branch). | 15:34 |
| *** afazekas has quit IRC | 15:35 | |
| *** rwsu has joined #openstack-infra | 15:35 | |
| YorikSar | phschwartz: Well... For seasoned Git user, handling merge conflicts is not a huge issue. And those conflicts will be the same in uploaded patchset as in your local branch since they will reference the same tree object. | 15:37 |
| *** yamamoto has quit IRC | 15:37 | |
| openstackgerrit | Spencer Krum proposed a change to openstack-infra/config: Bump postgres module version https://review.openstack.org/113561 | 15:38 |
| phschwartz | YorikSar: Actually I am not refereing to local merge conflicts, but when the zuul merge has a conflict. The conflict tree from zuul merge will be diferent as it will see the mainline source tree and not the localbranch tree for the developer using this. | 15:38 |
| openstackgerrit | Salvatore Orlando proposed a change to openstack-infra/config: Add neutron full job to integrated gate https://review.openstack.org/113562 | 15:38 |
| *** cdub has joined #openstack-infra | 15:38 | |
| fungi | YorikSar: also, git rerere may be able to solve some of your "rebase repetition" pain. give it a try if you haven't... | 15:39 |
| YorikSar | phschwartz: If Zuul fails with merge conflict, you can merge master into your local branch, fix conflicts and run git-review once again. | 15:39 |
| *** dtantsur is now known as dtantsur|afk | 15:40 | |
| YorikSar | fungi: I don't find rebase painful... I just don't like that it doesn't keep history of what I changed during rebase... | 15:41 |
| fungi | depends on what you mean by "keep history" | 15:41 |
| fungi | git rerere will record your resolutions so they can be reapplied | 15:42 |
| YorikSar | fungi: But with Gerrit interactive rebase becomes the only way to fix any commit in patch series but the last one. | 15:43 |
| fungi | at least for me, that's the only history i care about from my rebases... "how did i solve this conflict so i can repeat it?" | 15:43 |
| YorikSar | fungi: So it's not merge conflicts I want to record. It's just usual edits. | 15:43 |
| fungi | i also don't see what's wrong with interactive rebase. i use it for tons of things besides just rebasing a change on another branch. you can use it to edit changes in your series, reorder them, insert or delete them, et cetera | 15:44 |
| fungi | i suppose it depends on how you view git. it was designed as a machine for developing patches (for eventual posting on the lkml) and as such it's well adapted to building a series ("history") of changes for proposing to the kernel | 15:45 |
| fungi | it's less well adapted to keeping tons of revision cruft around | 15:46 |
| *** hdd has joined #openstack-infra | 15:46 | |
| fungi | though it has grown features for dealing with it | 15:46 |
| *** shashankhegde has quit IRC | 15:48 | |
| YorikSar | fungi: But for Linux, once you've published pull request to ML, you start to carefully keep published commits instead of rebasing them around which is the case when you use Gertit. | 15:49 |
| YorikSar | *Gerrit | 15:49 |
| YorikSar | fungi: So instead of "fixing that issue in 3rd commit of 10" you fix the issue and upload new commit. | 15:50 |
| YorikSar | fungi: But for Gerrit you do 'rebase -i' and your change gets lost in the pile of changes rebase does automatically... | 15:51 |
| fungi | true, though the lkml also wasn't well adapted to code review, and thus stacking new edits on top of your public tree made it easier for reviewers to catch up on your edits. gerrit on the other hand provides different mechanisms to accomplish that | 15:52 |
| fungi | however, i also don't keep any local copy of my commits. i only use my local clone to prepare one or more patches to push to gerrit, then delete the local topic branch and forget about it, pulling back from gerrit again if it needs to be modified | 15:53 |
| mordred | YorikSar: also, you said the key "For seasoned Git user" | 15:53 |
| mordred | we do _not_ have seasoned git users | 15:53 |
| *** jgallard has joined #openstack-infra | 15:53 | |
| *** ociuhandu has quit IRC | 15:53 | |
| YorikSar | fungi: I mean how would you undo change between N-10th patchset and N-9th in the change request that's in the middle of your series of change requests. | 15:53 |
| *** ramashri has joined #openstack-infra | 15:53 | |
| fungi | YorikSar: git review can check both out from gerrit quite easily | 15:54 |
| mordred | YorikSar: I pull the tip of the series in gerrit, do a git rebase -i - edit the patch in the middle, and then re-push | 15:54 |
| phschwartz | mordred: I think that is the exact flow most of us use | 15:54 |
| *** reed has joined #openstack-infra | 15:54 | |
| *** emagana has joined #openstack-infra | 15:55 | |
| fungi | i think the question there was about undoing earlier edits to a particular patchset in a programmatic fashion, rather than altering a change in the missle of a proposed series | 15:55 |
| *** ihrachyshka has joined #openstack-infra | 15:55 | |
| fungi | s/missle/middle/ | 15:55 |
| *** annegent_ has joined #openstack-infra | 15:55 | |
| openstackgerrit | Matt Riedemann proposed a change to openstack-infra/elastic-recheck: Add query for py26/centos6 node setup bug 1355907 https://review.openstack.org/113567 | 15:55 |
| uvirtbot | Launchpad bug 1355907 in openstack-ci ""sudo: yum: command not found" in py26 jobs (bare-centos6-rax* nodes)" [Undecided,New] https://launchpad.net/bugs/1355907 | 15:56 |
| YorikSar | mordred: I just repeated (almost) what fungi said :) But from some replies to ML, it seems that we have people in OpenStack that do almost the same workflow. So I doubt we don't have people that know Git well in our community... | 15:56 |
| fungi | but regardless, git-review lets you fetch specific patchsets from a proposed change | 15:56 |
| phschwartz | fungi: correct and with some bash pipes you can generate it out as a patch file that you can roll back from a later patch using git am | 15:57 |
| YorikSar | mordred: Yes, you can _change_ that middle commit, but you can't _undo_ that change later, for example. | 15:57 |
| YorikSar | I mean, you can, but you can't use simple git revert for that because that change is not in a separate commit. | 15:57 |
| *** hdd has quit IRC | 15:58 | |
| *** hdd has joined #openstack-infra | 15:58 | |
| YorikSar | phschwartz: Reverting to explicit patch storage is what I want to prevent using Git to its potential. | 15:58 |
| stevemar | bnemec, ping | 15:58 |
| bnemec | stevemar: pong | 15:59 |
| stevemar | bnemec, patch https://review.openstack.org/#/c/113553/1/tox.ini , just to be sure, you wanted line 4 to be: envlist = py33,py26,py27,pypy,pep8 ? | 15:59 |
| *** Guest35108 has quit IRC | 16:00 | |
| *** flwang_ has joined #openstack-infra | 16:00 | |
| *** jcoufal has quit IRC | 16:00 | |
| phschwartz | YorikSar: I am not sure that most developers that are experienced with gerrit workflows would agree | 16:00 |
| stevemar | bnemec, also, thoughts on adding docs to that list? | 16:00 |
| *** arnaud has joined #openstack-infra | 16:00 | |
| fungi | YorikSar: it sounds like what would be actually useful is a git branch in gerrit representing the patchsets of each change over the course of its development. chances are that would simplify their interdiff and other things as well | 16:00 |
| phschwartz | YorikSar: yes, outside of using gerrit I would use the exact flow your talking about, but gerrit makes it so you don't have to manage history to that point locally. | 16:01 |
| bnemec | stevemar: Oh, yeah we should probably do that. | 16:01 |
| phschwartz | fungi: if it was on the gerrit side, I agree, but YorikSar had reiterated that he wants it as a local branch on the developers system | 16:01 |
| stevemar | bnemec, so it should read: envlist = py33,py26,py27,pypy,pep8,docs | 16:01 |
| bnemec | stevemar: Yep | 16:02 |
| * clarkb dives into the discussion. nothing prevents that workflow with gerrit today. are we purely arguing if it should be codified by git review? | 16:02 | |
| stevemar | i don't see the point in having py33 first, but probably my env. | 16:02 |
| marcusvrn | Hi all, I'm trying to install the CI system and after I setup the Jenkins master (using the script provided by http://www.joinfu.com/) the zuul/debug.log returns "SSHException: not a valid DSA private key file". Does anyone know what is happening? | 16:03 |
| *** jgallard has quit IRC | 16:03 | |
| YorikSar | fungi: Unfortunately storing these dependencies on Gerrit side is not really possible. You'll end up having different parents for different patchsets and that won't be the full history in the end anyway. | 16:03 |
| clarkb | I personally think git review should contain the simplest of tools to do the simplest of workflows and was opposed to git review -x | 16:03 |
| clarkb | which I have never used | 16:03 |
| openstackgerrit | Steve Martinelli proposed a change to openstack-dev/pbr: cleanup tox.ini https://review.openstack.org/113553 | 16:03 |
| bnemec | stevemar: There's some sort of weird thing with one of our test libs where it creates a db, and if the py2 version runs before the py3 version the db is incompatible. | 16:04 |
| *** juzuluag has quit IRC | 16:04 | |
| clarkb | marcusvrn: I would check your ssh private key file for correctness and do sone manuak ssh testing | 16:04 |
| *** flwang_ has quit IRC | 16:04 | |
| bnemec | stevemar: Basically you only see it if you run tox on a system with both py2 and py3 and the py2 version runs first. | 16:04 |
| stevemar | bnemec, i trust ya, tossed it up ^ | 16:04 |
| fungi | YorikSar: well, it will be the full history of the patchsets which were proposed, as long as you do it for all changes consistently | 16:05 |
| YorikSar | fungi: I've described the process of reverting from almost what Gerrit has (with intermediate commits that don't get uploaded) to branch-based history in one of emails... | 16:05 |
| bnemec | stevemar: Thanks! | 16:05 |
| stevemar | np at all bnemec | 16:05 |
| * bnemec exits stage left before he gets pulled into the git-review discussion again | 16:05 | |
| *** ihrachyshka has quit IRC | 16:06 | |
| YorikSar | fungi: http://lists.openstack.org/pipermail/openstack-dev/2014-August/042168.html | 16:06 |
| dims | ack. i've seen py2 then py3 causing issues | 16:07 |
| fungi | YorikSar: yep, i read it when you posted it | 16:07 |
| *** hdd has quit IRC | 16:07 | |
| YorikSar | fungi: So you don't get all history uploaded to Gerrit. Some intermediate commits get lost and they can still be valuable for bisecting... | 16:08 |
| fungi | you get all the history of what *was* uploaded to gerrit, is what i was trying to say | 16:08 |
| fungi | i don't personally care about multiple adjustments i made which i never pushed to gerrit, nor can i imagine wanting to bisect between them (particularly not if they're muddied by constantly pulling from master into my topic branch) | 16:09 |
| openstackgerrit | Matt Riedemann proposed a change to openstack-infra/elastic-recheck: Add query for infra setup bug 1354829 https://review.openstack.org/113575 | 16:09 |
| uvirtbot | Launchpad bug 1354829 in tempest ""sudo: 3 incorrect password attempts" in host setup" [Undecided,Invalid] https://launchpad.net/bugs/1354829 | 16:09 |
| *** kashyap has joined #openstack-infra | 16:10 | |
| phschwartz | Is there something that would purposefully remove glance.hacking from the installed code when you do a setup.py install? | 16:10 |
| openstackgerrit | Matt Riedemann proposed a change to openstack-infra/elastic-recheck: Add query for infra setup bug 1354829 https://review.openstack.org/113575 | 16:10 |
| uvirtbot | Launchpad bug 1354829 in tempest ""sudo: 3 incorrect password attempts" in host setup" [Undecided,Invalid] https://launchpad.net/bugs/1354829 | 16:10 |
| *** vhoward has joined #openstack-infra | 16:10 | |
| fungi | mriedem: is 1354829 actually a bug? | 16:11 |
| mriedem | fungi: by what definition? | 16:11 |
| *** vermarah_ has joined #openstack-infra | 16:11 | |
| YorikSar | fungi: Well... Have you ever had to maintain long change request chains (> 4) for a long time (> 3 months)? | 16:11 |
| mriedem | fungi: it's a thing that failed jobs | 16:11 |
| mriedem | fungi: 193 hits in 7 days, check and gate | 16:11 |
| clarkb | phschwartz what is glance.hacking? | 16:11 |
| fungi | mriedem: as opposed to correct behavior, refusing to run sudo'd commands for users who weren't configured to be allowed to do so | 16:12 |
| YorikSar | fungi: I guess that experience is what left me wanting to have some smarter tool to keep track of all intermediate changes... | 16:12 |
| mriedem | fungi: it's a failure that's not related to the actual change being tested, | 16:12 |
| mriedem | fungi: so not sure what you're asking | 16:12 |
| clarkb | phschwartz but yes it may be excluded by manifest.in or similar if not needed to be installed | 16:13 |
| phschwartz | clarkb: source path in the glance code base. It is loading by hacking from the tox.ini for glance. I installed a venv and can't run the hacking tests on it as it fails to import it. | 16:13 |
| *** yfried_ has joined #openstack-infra | 16:13 | |
| fungi | YorikSar: i have in fact, but having a history of how i dealt with earlier rebases would not have helped me keep up with changes on master | 16:13 |
| phschwartz | clarkb: doesn't look to be excluded in manifest.in | 16:13 |
| *** mpaolino has joined #openstack-infra | 16:14 | |
| *** yfried has quit IRC | 16:14 | |
| YorikSar | fungi: Well, even merges from master would have been recorded (they essentially replace rebases). | 16:14 |
| fungi | mriedem: just not sure whether it's suited to an e-r query since it's a very common failure mode which is usually triggered by people trying to add sudo calls into jobs where they're not allowed, or forgetting to adjust the jobs to allow them | 16:14 |
| clarkb | mriedem we dont allow sudo in many cases.I think fungi is saying sudo failing is probably intentional | 16:14 |
| phschwartz | wait, no graft for it in manifest.in so it is not installed as part of it | 16:14 |
| clarkb | mriedem the actual bug is the test that needs sudo when it cant sudo | 16:14 |
| fungi | mriedem: so they could in fact end up in the gate as well given that changes to teh config repo (where job configuration resides) is not co-gated nor self-testing | 16:15 |
| mriedem | fungi: clarkb: but this isn't in a test... | 16:15 |
| *** evgenyf has quit IRC | 16:15 | |
| *** amuller__ has joined #openstack-infra | 16:15 | |
| clarkb | you said it was in the gate | 16:15 |
| clarkb | I use test in the large sense here | 16:15 |
| mriedem | yes, check and gate, multiple jobs | 16:15 |
| mriedem | it's in the devstack setup | 16:16 |
| *** yfried__ has joined #openstack-infra | 16:16 | |
| mriedem | sudo yum install -y libffi-devel | 16:16 |
| mriedem | sudo: no tty present and no askpass program specified | 16:16 |
| clarkb | oh that. can we just stop using fedora? | 16:16 |
| mriedem | i would not be the person to ask | 16:17 |
| mriedem | but given we're talking about testing latest libvirt, possibly with a fedora job, i'd think not | 16:17 |
| clarkb | fedora has decided you must be at a tty to root | 16:17 |
| clarkb | which is wrong (tm) | 16:17 |
| fungi | mriedem: is it possible these were non-voting jobs in the gate pipeline? | 16:18 |
| mriedem | fungi: yeah, i didn't recognize some of the names | 16:18 |
| mriedem | however, https://bugs.launchpad.net/neutron/+bug/1354829 | 16:18 |
| uvirtbot | Launchpad bug 1354829 in tempest ""sudo: 3 incorrect password attempts" in host setup" [Undecided,Invalid] | 16:18 |
| mriedem | the guy reported the failure originally against a py26 job | 16:18 |
| *** arnaud has quit IRC | 16:18 | |
| clarkb | py26 should fail | 16:18 |
| *** loki184 has joined #openstack-infra | 16:19 | |
| clarkb | it not failing would be a bug | 16:19 |
| fungi | right, if it was a sudo failure reported against a unit test job, then that was the fault of the change being proposed | 16:19 |
| *** amuller_ has quit IRC | 16:19 | |
| mriedem | https://review.openstack.org/#/c/110186/1/neutron/plugins/ofagent/agent/ofa_neutron_agent.py | 16:20 |
| mriedem | how so? | 16:20 |
| clarkb | mriedem unittest slaves do not allow jobs to sudo | 16:20 |
| fungi | but as far as "hits in the gate," we probably want to confirm whether e-r is actually counting non-voting changes in the gate pipeline, and possible ways to make it not | 16:20 |
| *** yfried_ has quit IRC | 16:20 | |
| clarkb | because running `tox` is unpriveleged and should never require root | 16:20 |
| mriedem | fungi: we don't have anything in logstash to filter on voting/non-voting as far as i know | 16:20 |
| mriedem | clarkb: fungi: i guess i'm at a loss to how joe developer understands that a unit test host setup failures is the fault of this change https://review.openstack.org/#/c/110186/1/neutron/plugins/ofagent/agent/ofa_neutron_agent.py | 16:21 |
| mriedem | which is a docstring change | 16:21 |
| fungi | mriedem: yeah, we're going to end up with a number of false gate failure hits in that case... we try to limit the number of non-voting jobs run in the gate pipeline, but there are certainly some from time to time | 16:21 |
| clarkb | mriedem its neutrons fault | 16:21 |
| clarkb | they dont properly mock in their tests | 16:21 |
| clarkb | the tests then run sudo. the tests fail | 16:22 |
| *** yfried__ has quit IRC | 16:22 | |
| clarkb | neutron is basically the reason we explicitly enforce sudo = failure in unittests | 16:22 |
| *** shashankhegde has joined #openstack-infra | 16:22 | |
| clarkb | there should be a note in the test log that says use of sudo is not allowed and explains this | 16:23 |
| mriedem | so http://goo.gl/mRtcVn | 16:23 |
| fungi | i suspect that the particular failure for that change was actually because someone approved a poorly-thought-out alteration to gate-neutron-python26 | 16:23 |
| mriedem | there are also devstack/tempest/grenade jobs in there | 16:23 |
| fungi | it looks like there was briefly a "sudo yum install -y libffi-devel" in the job builders after revoke-sudo | 16:24 |
| fungi | checking config history now to confirm | 16:24 |
| clarkb | fungi: mordreds fix for that wasnt correct? that is possible | 16:25 |
| fungi | yep, caused by https://review.openstack.org/112972 which was only that way briefly i think | 16:25 |
| *** jistr has quit IRC | 16:25 | |
| flaper87 | hey guys, any idea when this will be merged? https://review.openstack.org/#/c/111244/ NOt sure if you already discussed it or are planning to. :) | 16:26 |
| asselin__ | clarkb, fungi, would it be safe to +3 this change soon? https://review.openstack.org/#/c/96551/ | 16:26 |
| flaper87 | (marconi->zaqar) | 16:26 |
| fungi | flaper87: as part of the (not yet scheduled) gerrit outage to do the renaming | 16:26 |
| fungi | flaper87: it's on our agenda to discuss in the infra team meeting today | 16:26 |
| flaper87 | fungi: awesome. thanks for the heads up. I'll read the logs :) | 16:27 |
| *** rcarrillocruz has quit IRC | 16:27 | |
| *** ildikov has joined #openstack-infra | 16:27 | |
| fungi | flaper87: if you're not goung to be around for the meeting, do you have any particular suggestions for dates/times which would be (in)convenient for your developer community? | 16:27 |
| fungi | s/goung/going/ | 16:28 |
| clarkb | asselin__ ya I can do it now | 16:28 |
| *** pballand has joined #openstack-infra | 16:28 | |
| fungi | flaper87: noting that all their git remotes will break in their local clones and need updating to the new url | 16:28 |
| clarkb | actually because I am paranoid let me noop apply somewhere first | 16:29 |
| asselin__ | clarkb, thanks I'll take out my workaround and test | 16:29 |
| *** rcarrillocruz has joined #openstack-infra | 16:29 | |
| *** hogepodge has quit IRC | 16:29 | |
| *** pblaho has quit IRC | 16:31 | |
| *** yamamoto has joined #openstack-infra | 16:32 | |
| flaper87 | fungi: the sooner we do it, the better. We've already renamed the project internally (imports, files and docs) | 16:32 |
| clarkb | mriedem: note the yum -y is done there because yum returns success even when it fails when 1) -y is used and 2) multiple packages are being installed | 16:32 |
| fungi | flaper87: noted. thanks! | 16:33 |
| flaper87 | fungi: np, thank you guys. | 16:33 |
| mordred | krotscheck, ttx: what do we want to do about assignees and bug creators who do not yet have storyboard users | 16:33 |
| clarkb | mriedem so our puppet stuff succeeds but really failed. to "fix" this the jobs double tap that package install | 16:33 |
| mriedem | clarkb: as long as one package is installed it returns success right? i heard about that 'feature' recently | 16:33 |
| clarkb | mriedem yup | 16:33 |
| mordred | krotscheck: if I create a User object without an openid string associated with it, how many things break? | 16:33 |
| krotscheck | mordred: Let me check. | 16:34 |
| clarkb | mordred: could you prepopulate users? | 16:34 |
| *** shashankhegde has quit IRC | 16:35 | |
| clarkb | mordred lp gives you enough info to make it not terrible right? | 16:35 |
| mordred | not the openid | 16:35 |
| mordred | because they STILL wrongly think that they are write about that | 16:35 |
| mordred | right | 16:35 |
| clarkb | you can look that up like account sync did right? | 16:35 |
| krotscheck | mordred: Lots of things would break. | 16:35 |
| mordred | krotscheck: kk | 16:35 |
| mordred | clarkb: I REALLY don't want to do that | 16:35 |
| clarkb | thats understandable :) | 16:36 |
| krotscheck | mordred: We have a uniqueness constraint on email address in the users table, and when resolving an openID auth request we query by openid, not by email. | 16:36 |
| krotscheck | So you can create a user fine, but without an openID we’ll end up trying to create duplicate users. | 16:36 |
| *** chandankumar has quit IRC | 16:36 | |
| krotscheck | And that will fail because of the uniqueness constraint. | 16:36 |
| *** rpodolyaka is now known as rpodolyaka_afk | 16:36 | |
| *** AlexF_ has quit IRC | 16:37 | |
| *** yamamoto has quit IRC | 16:37 | |
| fungi | clarkb: mriedem: !!! | 16:37 |
| openstackgerrit | Matt Riedemann proposed a change to openstack-infra/elastic-recheck: Add query for test_postgresql_opportunistically bug 1355929 https://review.openstack.org/113581 | 16:37 |
| uvirtbot | Launchpad bug 1355929 in nova "test_postgresql_opportunistically fails in stable/havana due to: "ERROR: source database "template1" is being accessed by other users"" [Undecided,New] https://launchpad.net/bugs/1355929 | 16:37 |
| fungi | clarkb: mriedem: actually, look at the build_node value for all those failures | 16:37 |
| mriedem | trusty | 16:37 |
| mriedem | yeah, no yum | 16:38 |
| fungi | clarkb: mriedem: strangely, this may be related to the run-pep8.sh failures | 16:38 |
| clarkb | fungi dfw only too? | 16:38 |
| fungi | mriedem: the relevant part is that they're *all* in rax-dfw | 16:38 |
| fungi | yeah | 16:38 |
| openstackgerrit | A change was merged to openstack-infra/config: Fix sahara tag acl to make it working https://review.openstack.org/113552 | 16:38 |
| mriedem | what is dfw? | 16:38 |
| fungi | mriedem: the airport code for dallas/fort worth | 16:38 |
| *** _nadya_ has quit IRC | 16:38 | |
| fungi | mriedem: a region in rackspace cloud | 16:39 |
| mriedem | then i blame tony romo | 16:39 |
| fungi | i think we have something seriously odd with instances booted in rax-dfw | 16:39 |
| mriedem | i could restrict the query to those nodes | 16:39 |
| mriedem | but i'd have to use wildcards | 16:40 |
| fungi | i'm not sure we can wildcard in that field | 16:40 |
| fungi | clarkb: ^ ? | 16:40 |
| mriedem | i think we can | 16:40 |
| clarkb | fungi you can | 16:40 |
| clarkb | so *-dfw-* or whatever | 16:40 |
| mriedem | clarkb: but should i? | 16:40 |
| mriedem | i.e. perf? | 16:40 |
| mriedem | or doesn't matter in this case | 16:40 |
| *** derekh has quit IRC | 16:41 | |
| clarkb | it matters. I think we are ok doing it if necessary | 16:41 |
| *** bdpayne has joined #openstack-infra | 16:41 | |
| mriedem | ok, i'll add it | 16:41 |
| clarkb | problem before was a lot of queries didnt need it | 16:41 |
| mriedem | clarkb: fungi: ok this is also yum not found on rax-dfw jobs https://bugs.launchpad.net/openstack-ci/+bug/1355907 | 16:42 |
| uvirtbot | Launchpad bug 1355907 in openstack-ci ""sudo: yum: command not found" in py26 jobs (bare-centos6-rax* nodes)" [Undecided,New] | 16:42 |
| *** markmcclain1 has quit IRC | 16:42 | |
| mriedem | those are bare-centos6-rax-dfw nodes so i thought yum was more appropriate | 16:42 |
| *** hogepodge has joined #openstack-infra | 16:43 | |
| fungi | mriedem: so this may be the same underlying cause as bug 1355167 | 16:44 |
| uvirtbot | Launchpad bug 1355167 in openstack-ci "run-pep8.sh usage now being output" [Critical,In progress] https://launchpad.net/bugs/1355167 | 16:44 |
| openstackgerrit | A change was merged to openstack-infra/elastic-recheck: Add query for fixed ip shortage with n-net https://review.openstack.org/112558 | 16:44 |
| fungi | phschwartz: are you aware of any snapshot booting weirdness specifically in dfw? | 16:45 |
| clarkb | does nova show show us the image booted from? | 16:45 |
| clarkb | (trying to sort out how we might even debug this) | 16:45 |
| fungi | clarkb: not once the node is deleted i don't think... nodepool's debug log might mention it | 16:45 |
| *** mpaolino has quit IRC | 16:45 | |
| *** dims has quit IRC | 16:46 | |
| *** dims has joined #openstack-infra | 16:47 | |
| *** dims is now known as Guest27104 | 16:47 | |
| *** harlowja_at_home has quit IRC | 16:47 | |
| clarkb | we could probably just nova show all dfw nodes every few minutes | 16:47 |
| clarkb | of spscific types and look for deltas | 16:48 |
| *** e0ne has quit IRC | 16:48 | |
| openstackgerrit | Michael Krotscheck proposed a change to openstack-infra/storyboard: Added subscription API https://review.openstack.org/108516 | 16:49 |
| *** harlowja_at_home has joined #openstack-infra | 16:49 | |
| *** penguinRaider has quit IRC | 16:50 | |
| openstackgerrit | Matt Riedemann proposed a change to openstack-infra/elastic-recheck: Add query for infra setup bug 1354829 https://review.openstack.org/113575 | 16:50 |
| uvirtbot | Launchpad bug 1354829 in tempest ""sudo: 3 incorrect password attempts" in host setup" [Undecided,Invalid] https://launchpad.net/bugs/1354829 | 16:50 |
| *** Guest27104 has quit IRC | 16:51 | |
| *** penguinRaider has joined #openstack-infra | 16:51 | |
| mordred | krotscheck, clarkb: I'm a bit stumped at to how to handle this ... | 16:54 |
| mordred | krotscheck: stories and tasks don't allow not having a creator | 16:55 |
| *** sandywalsh_ has quit IRC | 16:56 | |
| *** SumitNaiksatam has quit IRC | 16:56 | |
| *** aviau has joined #openstack-infra | 16:56 | |
| aviau | Hello! Can someone help me with my rtfd hook? it doesn't seem to work. | 16:56 |
| clarkb | aviau it uses the project name so make sure it matched on the rtfd side | 16:57 |
| aviau | I have it setup in : modules/openstack_project/files/jenkins_job_builder/config/projects.yaml (Project surveil) | 16:57 |
| *** rdopieralski has quit IRC | 16:57 | |
| aviau | clarkb, And my project is here: https://readthedocs.org/projects/surveil/ | 16:57 |
| clarkb | mordred: Can we create as a system user then allow individuals to adopt? | 16:58 |
| *** ihrachyshka has joined #openstack-infra | 16:58 | |
| clarkb | mordred: I really think we may need user syncage though :( | 16:58 |
| *** amuller__ has quit IRC | 16:58 | |
| *** amuller__ has joined #openstack-infra | 16:59 | |
| vishy | mordred: happy birthday! Stop working and go get drunk | 16:59 |
| aviau | Clabbe, The project names look like they match but nothing updated. | 17:00 |
| *** jpich has quit IRC | 17:00 | |
| mordred | vishy: done! | 17:00 |
| mordred | vishy: and thanks | 17:01 |
| marcusvrn | clarkb: The problem was with my private key...thx! But now...the CI returns this error : "This change was unable to be automatically merged with the current state of the repository. Please rebase your change and upload a new patchset." | 17:01 |
| openstackgerrit | Ben Nemec proposed a change to openstack/requirements: Add oslo.serialization to projects.txt https://review.openstack.org/112993 | 17:01 |
| clarkb | aviau: next step is to look at jobs logs. http://logs.openstack.org/firsttwoofsha1/sha1 | 17:01 |
| marcusvrn | clarkb: I'm using the sandbox testing environment | 17:01 |
| clarkb | marcusvrn make sure the zuul merger is running | 17:01 |
| *** arnaud has joined #openstack-infra | 17:01 | |
| *** praneshp has joined #openstack-infra | 17:01 | |
| clarkb | vishy mordred wait. we like it when mordred works on stuff | 17:02 |
| clarkb | I will accept drunk mordred work though :) | 17:02 |
| mordred | krotscheck: ok. what about if there is no email address, but we _do_ have the openid? | 17:02 |
| *** mestery has quit IRC | 17:02 | |
| aviau | clarkb, How can I find the sha? | 17:02 |
| *** mestery has joined #openstack-infra | 17:03 | |
| *** shashankhegde has joined #openstack-infra | 17:04 | |
| aviau | clarkb, I found it. | 17:04 |
| phschwartz | fungi: I haven't heard of one. I will check | 17:05 |
| aviau | clarkb, I had forgotten to add the job in modules/openstack_project/files/zuul/layout.yaml | 17:06 |
| *** sandywalsh has joined #openstack-infra | 17:06 | |
| *** mrmartin has quit IRC | 17:06 | |
| *** ianw has quit IRC | 17:08 | |
| krotscheck | mordred: Does Gerrit have the openid? | 17:08 |
| openstackgerrit | Nicholas Randon proposed a change to openstack-infra/config: Add firewall traffic on port 27410 to 27510. https://review.openstack.org/108737 | 17:08 |
| krotscheck | mordred: i.e. can we do a batch user import from Gerrit first? | 17:08 |
| mordred | krotscheck: hrm | 17:09 |
| openstackgerrit | A change was merged to openstack-infra/elastic-recheck: Add query for py26/centos6 node setup bug 1355907 https://review.openstack.org/113567 | 17:09 |
| uvirtbot | Launchpad bug 1355907 in openstack-ci ""sudo: yum: command not found" in py26 jobs (bare-centos6-rax* nodes)" [Undecided,New] https://launchpad.net/bugs/1355907 | 17:09 |
| *** praneshp has quit IRC | 17:10 | |
| *** james_li_ has joined #openstack-infra | 17:10 | |
| openstackgerrit | Alexandre Viau proposed a change to openstack-infra/config: Added rtfd template to Surveil https://review.openstack.org/113587 | 17:10 |
| krotscheck | mordred: I was just thinking that the authoritative source of “who’s involved in openstack” is the CLA, which is managed through Gerrit, yes? | 17:10 |
| fungi | krotscheck: well, not necessarily (not all projects require a cla, also not all users of gerrit are submitting changes which is the only activity a cla is enforced on) | 17:12 |
| krotscheck | fungi: Well, doh. | 17:12 |
| *** eglynn has quit IRC | 17:12 | |
| *** james_li has quit IRC | 17:12 | |
| *** james_li_ is now known as james_li | 17:12 | |
| *** SumitNaiksatam has joined #openstack-infra | 17:12 | |
| fungi | krotscheck: oh, though yes if you mean more broadly all gerrit users rather than just those who have agreed to a cla, probably closer | 17:12 |
| mordred | however, it probably gets us darned close | 17:13 |
| fungi | yeah | 17:13 |
| krotscheck | What does launchpad use to link openID and their user accounts? | 17:13 |
| marcusvrn | clarkb: the zuul merger is running... :( | 17:14 |
| mordred | well, launchpad consumes the ubuntu SSO system | 17:14 |
| mordred | krotscheck: the problem is, we don't necessarily have API access to all of the data we need | 17:14 |
| mordred | to get a person's openid identifier, we actually have to go make a fake openid consumer request to the login page and pull the identifier | 17:15 |
| *** krtaylor has quit IRC | 17:15 | |
| mordred | but then that can be wrong, because the ubuntu sso people havea very strange idea of how openid works | 17:15 |
| mordred | so if a person has ever merged an account, it's possible that requesting the ID from the IDP for a user will give you a _different_ openid string from what the finalization of the login process will return | 17:16 |
| mordred | and then also, users are allowed to hide their email address | 17:16 |
| fungi | mordred: well, there *is* the lp api's get_user_by_openid method | 17:16 |
| mordred | and there is NO way for us to scrape that information without their involved | 17:16 |
| krotscheck | mordred: because that makes perfect sense. | 17:16 |
| mordred | fungi: right. but it's the wrong way | 17:16 |
| *** praneshp has joined #openstack-infra | 17:16 | |
| fungi | mordred: agreed | 17:16 |
| mordred | so, more and more I'm leaning towards krotscheck suggestion of just slurping from teh gerrit db | 17:17 |
| *** andreaf_ has quit IRC | 17:17 | |
| mordred | I'm sure there will be bugs somewhere with a user who does not have a gerrit account ... but maybe we handle those as exceptions | 17:17 |
| krotscheck | mordred: I don’t suppose we can just ask canonical for a DB dump? | 17:17 |
| fungi | yeah, lp doesn't have a get_openid_by_user method | 17:17 |
| mordred | well, from their pov, email is privileged, so they can't give it to us if the user objects | 17:18 |
| fungi | also, most of the people who could have gotten it for us have been hired away to work on openstack now ;) | 17:18 |
| krotscheck | I don’t think we need that though. | 17:18 |
| *** harlowja_at_home has quit IRC | 17:18 | |
| krotscheck | We need the userId that’s associated as a task author, and the openid associated with that userid, no? | 17:18 |
| fungi | so they're a bit strapped just keeping things running there | 17:18 |
| mordred | krotscheck: but to make a User object, I need an email | 17:19 |
| mordred | which is a unique key | 17:19 |
| mordred | krotscheck: unless we wanted to make email nullable | 17:19 |
| *** Ryan_Lane has joined #openstack-infra | 17:20 | |
| krotscheck | mordred: fff… well, yeah. That, or we dump in an autogenerated email address. That’s overwritten the first time the user logs in anyway. | 17:20 |
| *** andreaf has joined #openstack-infra | 17:20 | |
| krotscheck | (User name and email address is auto-updated from openid on every login) | 17:20 |
| fungi | i think smarcet and mrmartin might have some similar needs for bootstrapping the openstackid.org backend | 17:21 |
| *** rcarrill` has joined #openstack-infra | 17:23 | |
| * anteaya looks at the agenda for today's meeting: https://wiki.openstack.org/wiki/Meetings/InfraTeamMeeting | 17:23 | |
| anteaya | jesusaurus nibalizer do you still need Puppet module Split out on today's infra agenda? | 17:24 |
| fungi | i don't know if jeblair planned to be around today for the meeting, but if not i'm happy to chair it | 17:24 |
| nibalizer | uh we can talk about it | 17:24 |
| * anteaya nods | 17:24 | |
| anteaya | nibalizer: great it can stay | 17:24 |
| *** rcarrillocruz has quit IRC | 17:24 | |
| nibalizer | im lacking a 'next step' short of 'get more consensus' so yea lets put it on the agenda | 17:25 |
| * anteaya removes the git-upstream item, it has been discussed | 17:25 | |
| clarkb | marcusvrn: check the merger logs then | 17:25 |
| clarkb | marcusvrn: should be in /var/log/zuul/merger*.log | 17:25 |
| anteaya | nibalizer: very good | 17:25 |
| clarkb | nibalizer: anteaya wait I thought we had that last meeting? | 17:26 |
| anteaya | I'll remove the third party item from the agenda, I am following up with folks in the third party space and they are aware that coordinating with nibalizer and jesusaurus is a good idea | 17:26 |
| *** bhuvan has joined #openstack-infra | 17:26 | |
| clarkb | basically it sounded like we needed a guinea pig project to do it on | 17:26 |
| anteaya | clarkb: we did, I am removing the item | 17:26 |
| clarkb | do it, figure out how to make it go better then do it for all the things | 17:26 |
| anteaya | so we don't have to discuss it again today | 17:26 |
| anteaya | clarkb: right | 17:26 |
| *** mmaglana has joined #openstack-infra | 17:26 | |
| nibalizer | oh we're gonna go ahead with one? | 17:27 |
| nibalizer | krotscheck: lets do storyboard! | 17:27 |
| *** james_li has quit IRC | 17:27 | |
| clarkb | nibalizer: I thought that is what we agreed on | 17:27 |
| fungi | clearly it was a popular choice | 17:27 |
| clarkb | pleia2: I am going to try forward proting jeblair's pootle SSO change to 2.6.0-rc1 now | 17:27 |
| krotscheck | I’m confused, what are we doing with storyboard? | 17:27 |
| clarkb | pleia2: hopefully that keeps us moving | 17:27 |
| nibalizer | krotscheck: split it out into its own module, that then is consumed by infra/config | 17:28 |
| anteaya | nibalizer: your agenda item stays, the third party agenda item goes | 17:28 |
| pleia2 | clarkb: thanks | 17:28 |
| anteaya | nibalizer: I'm directing third party folks to track you and your work | 17:28 |
| anteaya | nibalizer: and review and help | 17:28 |
| krotscheck | nibalizer: Oh, neat! | 17:28 |
| clarkb | asselin__: but first I am going to apply that change somewhere as a noop then hopefully merge it | 17:28 |
| pleia2 | clarkb: learned that zanata does support openid too (their live version has it), fiddling with it on my own demo server now, I think they have out of date instructions again :\ | 17:28 |
| krotscheck | nibalizer: Does that mean I have to move my patches to other places? | 17:29 |
| clarkb | pleia2: :/ | 17:29 |
| asselin__ | clarkb, sure | 17:29 |
| anteaya | clarkb: the trusty and tox agenda items don't need air time this week, do they? | 17:29 |
| *** kmartin has quit IRC | 17:29 | |
| nibalizer | krotscheck: so what pending patches do you have? | 17:29 |
| nibalizer | just the rabbit one or are there more? | 17:29 |
| krotscheck | There’s one more, sec... | 17:29 |
| nibalizer | i thought your larger refactor landed | 17:30 |
| krotscheck | nibalizer: https://review.openstack.org/#/c/112998/ | 17:30 |
| *** kmartin has joined #openstack-infra | 17:30 | |
| krotscheck | But I can move those as long as we can get it done this week. | 17:30 |
| * krotscheck is doing honeymoon things next week. | 17:30 | |
| anteaya | krotscheck: congratulations | 17:30 |
| krotscheck | anteaya: Thanks! | 17:30 |
| nibalizer | krotscheck++ awesome! | 17:30 |
| *** gyee has joined #openstack-infra | 17:31 | |
| *** MarkAtwood has quit IRC | 17:32 | |
| *** yamamoto has joined #openstack-infra | 17:32 | |
| clarkb | anteaya: nope, got that settled last time. thanks | 17:32 |
| clarkb | anteaya: doing it on the 20th | 17:32 |
| *** stevemar has quit IRC | 17:34 | |
| *** gyee has quit IRC | 17:34 | |
| clarkb | mordred: fungi: reviews on https://review.openstack.org/#/c/111094/ https://review.openstack.org/#/c/111853/ and https://review.openstack.org/#/c/112431/ would be great this week | 17:35 |
| anteaya | clarkb: kk | 17:35 |
| clarkb | mordred: fungi: if we can get that stuff merged before sdague gets back from vacation that will allow him to make ^ work | 17:35 |
| clarkb | along with adam_g and devananda | 17:35 |
| *** yfried__ has joined #openstack-infra | 17:36 | |
| *** emagana has quit IRC | 17:36 | |
| *** whoops has joined #openstack-infra | 17:36 | |
| *** yamamoto has quit IRC | 17:37 | |
| *** emagana has joined #openstack-infra | 17:37 | |
| *** esker has quit IRC | 17:38 | |
| mordred | krotscheck: ok. if I can do autogenerated email and it gets updated, then I'll just do that | 17:39 |
| mordred | krotscheck: that makes it easy | 17:40 |
| *** rcarrillocruz has joined #openstack-infra | 17:40 | |
| *** salv-orlando has joined #openstack-infra | 17:40 | |
| *** e0ne has joined #openstack-infra | 17:40 | |
| *** rcarrill` has quit IRC | 17:41 | |
| clarkb | fungi: mordred ok https://review.openstack.org/#/c/96551/ appears to apply cleaning to es03 and review.o.o | 17:41 |
| clarkb | any opposition to the method of handling unbounnd there before I approve | 17:41 |
| *** mriedem has quit IRC | 17:41 | |
| mordred | clarkb: fine by me | 17:42 |
| *** eglynn has joined #openstack-infra | 17:44 | |
| fungi | clarkb: sorry, was stuffing my face | 17:45 |
| fungi | clarkb: are you planning to rebase it or something? | 17:46 |
| fungi | oh, nevermind. i misread one of the needed-by patches as depends-on | 17:46 |
| clarkb | fungi: ya no rebase, just wanted to make sure people are onboard | 17:47 |
| *** melwitt has joined #openstack-infra | 17:47 | |
| openstackgerrit | Oscar Romero proposed a change to openstack-infra/jeepyb: Fix close_pull_requests from jeepyb https://review.openstack.org/113598 | 17:47 |
| clarkb | also a thunderstorm is rolling through so this could be fun | 17:47 |
| *** dims has joined #openstack-infra | 17:47 | |
| clarkb | you know what github needs? git review -d | 17:48 |
| *** dims is now known as Guest61254 | 17:48 | |
| *** flwang_ has joined #openstack-infra | 17:48 | |
| jeblair | hello all | 17:52 |
| dprince | clarkb: hi sir, so I created the ACL here https://review.openstack.org/#/c/112331/3. Will that automatically create the group in gerrit, or is that a manual step? | 17:52 |
| anteaya | hello jeblair | 17:52 |
| *** flwang_ has quit IRC | 17:52 | |
| clarkb | jeblair: o/ we have new pootle stories | 17:52 |
| clarkb | dprince: it will create the group | 17:52 |
| jroll | clarkb: github kind of has -d https://gist.github.com/piscisaureus/3342247 | 17:53 |
| *** dmsimard is now known as dmsimard_away | 17:53 | |
| dprince | clarkb: cool, thanks | 17:53 |
| clarkb | jroll: oh I fetched the pr I awnted just fine | 17:53 |
| jeblair | clarkb: i caught the gist that their implementation of openid has an impedence mismatch with mine... | 17:53 |
| clarkb | jroll: it just requires jumping through hoops | 17:53 |
| clarkb | jeblair: a couple of them yes | 17:53 |
| jroll | clarkb: yeah, a little annoying | 17:54 |
| clarkb | jeblair: I am working on forward porting your patch to 2.6.0-rc1 but there are a few conflicts I am not sure how to fix yet | 17:54 |
| marcusvrn | clarkb: looking at merger-logs, I found that the problem is when try to execute this command: "git clone -v ssh://hds-ci@review.openstack.org:29418/openstack-dev/sandbox /var/lib/zuul/git/openstack-dev/sandbox"....but looking at /var/lib/zuul/ssh/ the private key is there... | 17:54 |
| jroll | did any devstack cores show back up today? :) | 17:54 |
| jeblair | clarkb: let me know if i can help. i remember little of pootle, but the auth stuff is mostly basic django; i have a bit of openid knowledge | 17:54 |
| marcusvrn | clarkb: the returning error is : "Cloning into '/var/lib/zuul/git/openstack-dev/sandbox'... | 17:55 |
| marcusvrn | Permission denied (publickey). | 17:55 |
| marcusvrn | fatal: The remote end hung up unexpectedly | 17:55 |
| marcusvrn | " | 17:55 |
| clarkb | jeblair: in this case they appear to have removed enough of the auth mechanism and handed it off to allauth that pootle doesn't provide its own login page? | 17:55 |
| clarkb | marcusvrn: the merger is using ssh and public key auth to clone that repo | 17:55 |
| clarkb | marcusvrn: but that is failing. make sure you are using the correct key | 17:55 |
| jeblair | clarkb: neat. does allauth not have an openid single sign on mode? | 17:56 |
| *** mriedem has joined #openstack-infra | 17:56 | |
| clarkb | jeblair: nope, and it doesn't work with pootle as implemented | 17:56 |
| *** gyee has joined #openstack-infra | 17:56 | |
| jeblair | that's kind of confusing | 17:56 |
| clarkb | jeblair: so basically they rejected your thing, implemented allauth, closed the bug and never tested that it actualy fixed the bug | 17:57 |
| *** pdmars has quit IRC | 17:57 | |
| jeblair | clarkb: neat. to be fair, i did zero followup on that, mostly because we couldn't decide if we actually wanted to use pootle. | 17:57 |
| clarkb | jeblair: https://github.com/translate/pootle/issues/3244 and https://github.com/translate/pootle/issues/3243 | 17:57 |
| marcusvrn | clarkb: the private key that need to be there must be the pair of public key sent to the openstack-infra team? | 17:58 |
| *** vhoward has left #openstack-infra | 17:58 | |
| clarkb | jeblair: maybe you can fill in the sso use case on that first issue | 17:58 |
| clarkb | marcusvrn: yes | 17:58 |
| clarkb | marcusvrn: also make sure you are using teh correct username for the account | 17:58 |
| clarkb | marcusvrn: you will get failures like that if usernames don't match | 17:58 |
| *** dkliban_afk is now known as dkliban | 17:59 | |
| clarkb | jeblair: in particular with allauth pootle/apps/pootle_profile/urls.py is deleted in pootle | 17:59 |
| clarkb | jeblair: because allauth deals with it | 17:59 |
| clarkb | jeblair: pootle/templates/user_profile_menu.html is also gone for similar reasons | 17:59 |
| *** mestery has quit IRC | 18:00 | |
| *** sabeen3 has joined #openstack-infra | 18:01 | |
| jeblair | clarkb: http://bazaar.launchpad.net/~django-openid-auth/django-openid-auth/trunk/view/head:/README.txt | 18:01 |
| jeblair | clarkb: that's basically the process i followed in creating that patch | 18:02 |
| *** sabeen1 has quit IRC | 18:02 | |
| mordred | jeblair: did you see that we're going to need to do openid scraping again for lp->storyboard import? | 18:02 |
| jeblair | mordred: nope; that's a one-time thing though, right? | 18:02 |
| *** amuller__ has quit IRC | 18:03 | |
| jeblair | clarkb: i think issue https://github.com/translate/pootle/issues/3244 probably wants the allauth issue taken care of first, yeah? because otherwise, it's hard to say what things pootle will need to pass through? | 18:03 |
| clarkb | jeblair: no I think pootle can implement the only allow openid bit concurently | 18:04 |
| *** andreykurilin_ has joined #openstack-infra | 18:04 | |
| clarkb | jeblair: further restricting which openid providers are allowed is what is needed in allauth | 18:04 |
| *** pdmars has joined #openstack-infra | 18:04 | |
| *** abhishekk has joined #openstack-infra | 18:04 | |
| jeblair | clarkb: okay, then i think you ought to reply with that because i don't know what settings to say need to be implemented :) | 18:04 |
| *** e0ne has quit IRC | 18:04 | |
| clarkb | jeblair: right now pootle always allows local auth | 18:05 |
| clarkb | at least that was my experience with it | 18:05 |
| jeblair | clarkb: however, i think 3243 might warrant a mention of http://openid.net/specs/openid-simple-registration-extension-1_1-01.html to provide usernames | 18:05 |
| clarkb | jeblair: can you update 3243 with that? I will respond to the other one | 18:06 |
| zaro | clarkb, jeblair, fungi, mordred : is an infra core able to approve puppetize installation of 3rd party plugins? (https://review.openstack.org/#/c/91193). i'm able to babysit now. | 18:06 |
| *** jamielennox|away is now known as jamielennox | 18:06 | |
| openstackgerrit | A change was merged to openstack-infra/storyboard: Updated storyboard documentation https://review.openstack.org/107490 | 18:06 |
| marcusvrn | clarkb: where can I check the username information? | 18:07 |
| clarkb | marcusvrn: it should have been given to you in the email for the third party account | 18:08 |
| *** otherwiseguy has quit IRC | 18:08 | |
| clarkb | marcusvrn: and your /etc/zuul/zuul.conf should have an entry for it under the gerrit section | 18:08 |
| clarkb | asselin__: optional unbound is approved | 18:09 |
| *** linkid has left #openstack-infra | 18:09 | |
| asselin__ | thanks! :) | 18:09 |
| krotscheck | nibalizer: I recall there being a nodeps option in the install_modules script so as to avoid version problems. That seems to have dissapeared… is it ok for me to just assume module dependencies will be run? | 18:09 |
| jeblair | zaro: done | 18:09 |
| nibalizer | krotscheck: so if you want to have nodeps use the source install situation | 18:10 |
| nibalizer | er | 18:10 |
| nibalizer | tool | 18:10 |
| nibalizer | well there is a way to install from source now | 18:10 |
| nibalizer | which gets nodeps for free | 18:10 |
| krotscheck | Alright, let me consider that. I _think_ deps resolution for upgradeing ot rabbitmq 4.0.0 is ok though... | 18:11 |
| jogo | did the e-r bot die? I don't see any IRC messages of late | 18:14 |
| jogo | and I am gettint reports of it not posting comments | 18:14 |
| jeblair | jogo: checking | 18:14 |
| jogo | jeblair: thanks | 18:15 |
| openstackgerrit | A change was merged to openstack-infra/config: puppetize installation of gerrit third party plugins https://review.openstack.org/91193 | 18:15 |
| *** loki184 has quit IRC | 18:15 | |
| jeblair | it's still running; it may be netsplit | 18:15 |
| jeblair | 2014-08-12 18:15:38,349 DEBUG recheckwatchbot: Console logs not ready for gate-trove-python26 109824,6,ad24724 | 18:16 |
| jogo | jeblair: thanks, no stacktraces it the logs I presume | 18:16 |
| jogo | jeblair: ahh | 18:16 |
| jogo | that explains it | 18:16 |
| *** dmsimard_away is now known as dmsimard | 18:16 | |
| jogo | jeblair: its not commenting because its taking too long to get the logs from ES | 18:16 |
| jeblair | nope, es requests are returning quickly | 18:16 |
| *** zehicle_at_dell has joined #openstack-infra | 18:17 | |
| jeblair | jogo: or yes, depending on exactly what your question was :) | 18:17 |
| jogo | jeblair: hmm no how long it takes for ES to respond but the logs not getting added in the 15 minute window | 18:17 |
| jeblair | yeah | 18:17 |
| jeblair | jogo: however, the graph does not suggest a backlog | 18:18 |
| clarkb | its possible that particular log is backed up but there is no consistent backlog | 18:18 |
| jogo | clarkb jeblair: hmm yeah I see recent logs in ES | 18:18 |
| *** bhuvan_ has joined #openstack-infra | 18:19 | |
| *** krtaylor has joined #openstack-infra | 18:19 | |
| *** marun has quit IRC | 18:21 | |
| *** stevemar has joined #openstack-infra | 18:21 | |
| clarkb | jeblair: https://github.com/cboylan/pootle/compare/translate:stable/2.6.0...openid going to see if something as simple as that will work | 18:21 |
| *** marun has joined #openstack-infra | 18:21 | |
| *** markmcclain has joined #openstack-infra | 18:21 | |
| *** bhuvan has quit IRC | 18:22 | |
| jogo | clarkb jeblair: hmm ER doesn't appear to be commenting | 18:23 |
| jogo | I found https://review.openstack.org/#/c/111837/ | 18:23 |
| jogo | which matched | 18:23 |
| jogo | message:"Failed to commit reservations" AND tags:"screen-n-cond.txt" AND loglevel:"ERROR" | 18:23 |
| jogo | but has no ER comment | 18:23 |
| jeblair | jogo: yeah, it's still waiting for that trove job | 18:23 |
| jeblair | jogo: oh, you expected it to comment because that job would have completed before the trove one? | 18:24 |
| *** rcarrill` has joined #openstack-infra | 18:25 | |
| *** pelix has quit IRC | 18:25 | |
| jeblair | jogo: http://paste.openstack.org/show/93983/ | 18:26 |
| *** hdd has joined #openstack-infra | 18:26 | |
| jeblair | jogo: when i run that now, i get a response, but at the time, it did not. | 18:26 |
| *** rcarrillocruz has quit IRC | 18:26 | |
| clarkb | jeblair: 'django_openid_auth.useropenid: 'user' defines a relation with the model 'auth.User', which has been swapped out. Update the relation to point at settings.AUTH_USER_MODEL.' so we will need to patch django_openid_auth too | 18:28 |
| jeblair | clarkb: what's the source of that? | 18:31 |
| clarkb | jeblair: running pootle setup which does db migrations | 18:31 |
| jogo | jeblair: hmm | 18:31 |
| jogo | jeblair: can you share the last few hours of e-r logs? | 18:31 |
| *** Longgeek has quit IRC | 18:32 | |
| clarkb | jeblair: maybe that is a django version thing? | 18:32 |
| jeblair | clarkb: what version are you using? | 18:32 |
| *** yamamoto has joined #openstack-infra | 18:32 | |
| *** crc32 has joined #openstack-infra | 18:32 | |
| clarkb | 1.6.5 | 18:32 |
| jeblair | clarkb: https://docs.djangoproject.com/en/1.5/topics/auth/customizing/#auth-custom-user | 18:33 |
| *** crc32 has quit IRC | 18:33 | |
| jeblair | clarkb: trunk django-openid-auth supports django 1.6; i think they have just not implemented support for this optional new setting | 18:33 |
| *** crc32 has joined #openstack-infra | 18:34 | |
| *** e0ne has joined #openstack-infra | 18:35 | |
| mordred | jeblair: yes - the sync is one time - although it'll effectively be _everytime_ we run a sync | 18:36 |
| *** hdd has quit IRC | 18:36 | |
| jeblair | mordred: but this is just for importing existing bugs, which we will do once for infra, and once for openstack, right? | 18:36 |
| *** yamamoto has quit IRC | 18:36 | |
| jeblair | mordred: it's not an ongoing operational concern? | 18:37 |
| mordred | yes | 18:37 |
| mordred | so it shouldn't be _terrible_ | 18:37 |
| jeblair | mordred: what's the bit of data we need? | 18:37 |
| mordred | jeblair: well, we need a user account to deal with created_by and friends | 18:37 |
| mordred | jeblair: and user accoutns need openids, otherwise there's no way to match them when someone logs in | 18:38 |
| mordred | so we mainly just need the openid url thing | 18:38 |
| *** gokrokve_ has joined #openstack-infra | 18:38 | |
| *** alexpilotti has quit IRC | 18:39 | |
| jeblair | mordred: what was the new api call they added? openid -> username? | 18:40 |
| mordred | jeblair: yah. exactly the opposite of what we needed | 18:40 |
| mordred | jeblair: it's ok - I've got the 'fetch openid' code working | 18:40 |
| jeblair | jogo: sanitizing the logs | 18:41 |
| jeblair | jogo: (it logs passwords) | 18:41 |
| *** gokrokve has quit IRC | 18:41 | |
| *** eglynn has quit IRC | 18:41 | |
| *** homeless has quit IRC | 18:42 | |
| *** gokrokve_ has quit IRC | 18:43 | |
| openstackgerrit | A change was merged to openstack-infra/elastic-recheck: Add query for infra setup bug 1354829 https://review.openstack.org/113575 | 18:43 |
| uvirtbot | Launchpad bug 1354829 in tempest ""sudo: 3 incorrect password attempts" in host setup" [Undecided,Invalid] https://launchpad.net/bugs/1354829 | 18:43 |
| jogo | jeblair: thanks | 18:45 |
| jeblair | jogo: http://status.openstack.org/debug.log.txt | 18:45 |
| jeblair | jogo: let me know when you have a local copy of that so i can rm it | 18:46 |
| abhishekk | dstanek:hi, you there? | 18:46 |
| jogo | jeblair: got it | 18:46 |
| jogo | thanks | 18:46 |
| *** hdd has joined #openstack-infra | 18:46 | |
| openstackgerrit | Matt Riedemann proposed a change to openstack-infra/elastic-recheck: Add query for ceilometer py26 bug 1355983 https://review.openstack.org/113612 | 18:47 |
| uvirtbot | Launchpad bug 1355983 in ceilometer "gate-ceilometer-python26 failing with "Could not find mongod command"" [Undecided,New] https://launchpad.net/bugs/1355983 | 18:47 |
| jogo | aha Traceback (most recent call last): | 18:48 |
| jogo | File "/usr/local/lib/python2.7/dist-packages/elastic_recheck/bot.py", line 211, in run | 18:49 |
| jogo | debug=not self.commenting) | 18:49 |
| jogo | File "/usr/local/lib/python2.7/dist-packages/elastic_recheck/elasticRecheck.py", line 333, in leave_comment | 18:49 |
| jogo | msg = msgs['found_bug'] % {'bugs': event.bug_list()} | 18:49 |
| jogo | found the bug | 18:49 |
| jogo | jeblair: thank you | 18:49 |
| jeblair | jogo: thank you" | 18:49 |
| jeblair | ! | 18:49 |
| *** Ajaeger has joined #openstack-infra | 18:49 | |
| *** abhishekk has quit IRC | 18:52 | |
| clarkb | django.db.utils.OperationalError: (1170, "BLOB/TEXT column 'claimed_id' used in key specification without a key length") anyone know how to set the foreignkey legnth restriction with django orm? | 18:52 |
| *** gokrokve has joined #openstack-infra | 18:52 | |
| clarkb | also I think that is an allauth bug... | 18:52 |
| *** hdd has quit IRC | 18:52 | |
| clarkb | basically allauth isn't quite mysql safe | 18:52 |
| *** MarkAtwood has joined #openstack-infra | 18:54 | |
| *** ianw has joined #openstack-infra | 18:58 | |
| pleia2 | yeesh, openid is hard | 18:59 |
| clarkb | meeting time | 19:00 |
| *** zehicle_at_dell has quit IRC | 19:00 | |
| *** dimtruck is now known as zz_dimtruck | 19:00 | |
| fungi | the openid spec is a bit more like retroactive codification of the original implementation, from what i gather, iterated over with some incremental improvements | 19:00 |
| pleia2 | fungi: that seems like the best way to do things ;) | 19:01 |
| openstackgerrit | Andreas Jaeger proposed a change to openstack-infra/config: Move openstack-security-notes to attic https://review.openstack.org/113614 | 19:01 |
| fungi | well, it was developed as an auth api-ish for lj | 19:01 |
| *** zz_dimtruck is now known as dimtruck | 19:01 | |
| clarkb | in this cause its just django plugins only working in the initial lego set they came in I think | 19:01 |
| fungi | if my memory is correct | 19:01 |
| pleia2 | huh, I didn't know that | 19:01 |
| clarkb | there is clearly very little testing of ow they fit together | 19:02 |
| *** hashar has joined #openstack-infra | 19:02 | |
| fungi | worth noting, though, that most early ietf standards fall into the same category ("this is how we implemented protocol in tool foo") | 19:03 |
| *** reed has quit IRC | 19:03 | |
| stevemar | just a heads up - possible new infra bug: 1355998 | 19:04 |
| *** pdmars has quit IRC | 19:04 | |
| fungi | bug 1355998 | 19:05 |
| uvirtbot | Launchpad bug 1355998 in devstack "PostgreSQL's request for a shared memory segment exceeded your kernel's SHMMAX parameter" [Undecided,New] https://launchpad.net/bugs/1355998 | 19:05 |
| *** SergeyLu_ has joined #openstack-infra | 19:05 | |
| clarkb | sounds like a postgres bug >_> | 19:05 |
| *** rcarrillocruz has joined #openstack-infra | 19:05 | |
| *** SergeyLu_ is now known as SergeyLukjanov2 | 19:05 | |
| openstackgerrit | Michael Krotscheck proposed a change to openstack-infra/config: Create storyboard configuration directory https://review.openstack.org/113616 | 19:06 |
| *** rcarrill` has quit IRC | 19:07 | |
| zaro | jeblair: review & review-dev are not automatically updated on merges to config. I don't remember if that's on purpose or not but i have manually updated review-dev and have verified that the change to automate 3rd party plugin installation did work as expected. | 19:08 |
| *** rcarrill` has joined #openstack-infra | 19:08 | |
| fungi | stevemar: search on logstash. looks like it started around 1500-1530 when the nodepool images would have been finishing their daily update. the build_node column indicates that it's only impacting devstack-trusty nodes on hpcloud-b1 | 19:09 |
| fungi | stevemar: so i'll delete that image | 19:09 |
| fungi | clarkb: ^ worth digging into the image update log for that when our meeting concludes | 19:10 |
| clarkb | fungi: well if it is an issue with our kernel settings it will pop up tomorrow too | 19:10 |
| fungi | clarkb: why only on one image in one provider though? | 19:10 |
| *** annegent_ has quit IRC | 19:10 | |
| fungi | (e.g. not impacting hpcloud-b2 through b5) | 19:10 |
| *** pdmars has joined #openstack-infra | 19:11 | |
| *** dimtruck is now known as zz_dimtruck | 19:11 | |
| clarkb | did the image update there? | 19:11 |
| *** rcarrillocruz has quit IRC | 19:11 | |
| clarkb | or maybe it was a race around kernel updates and when image updates ran | 19:11 |
| clarkb | (I don't have a good answer, but the error message indicates kernel settings) | 19:12 |
| *** gema_ has joined #openstack-infra | 19:12 | |
| openstackgerrit | Aishwarya Thangappa proposed a change to openstack-infra/storyboard: Added a subscription_events table & subscription hander https://review.openstack.org/113016 | 19:14 |
| aviau | Can someone help me with this error? http://logs.openstack.org/87/113587/1/check/gate-config-layout/7dfb8ee/console.html | 19:15 |
| *** gema has quit IRC | 19:15 | |
| aviau | here is the changeset: https://review.openstack.org/#/c/113587/ | 19:15 |
| aviau | It looks like the check script is failing? | 19:16 |
| openstackgerrit | Michael Krotscheck proposed a change to openstack-infra/storyboard: Added subscription API https://review.openstack.org/108516 | 19:16 |
| *** otherwiseguy has joined #openstack-infra | 19:16 | |
| clarkb | aviau: that isn't quite how you specify a template iirc. grep for template in that file and you should see the diff | 19:17 |
| openstackgerrit | Joe Gordon proposed a change to openstack-infra/elastic-recheck: Fix MessageConfig https://review.openstack.org/113620 | 19:18 |
| aviau | clarkb, My bad! =) | 19:18 |
| aviau | thanks | 19:18 |
| *** imcsk8 has joined #openstack-infra | 19:18 | |
| *** mrmartin has joined #openstack-infra | 19:18 | |
| jogo | jeblair: the fix ^ | 19:19 |
| Ajaeger | aviau: use "- name: docs-on-rtfd" | 19:19 |
| *** andreykurilin_ has quit IRC | 19:19 | |
| imcsk8 | hello, is there any problem with the jenkins jobs of review.openstack.org? | 19:19 |
| pleia2 | imcsk8: what are you having trouble with? | 19:20 |
| Ajaeger | imcsk8: jenkins has quite some work in the queue do - see status.openstack.org/zuul | 19:20 |
| *** ihrachyshka has quit IRC | 19:21 | |
| openstackgerrit | Alexandre Viau proposed a change to openstack-infra/config: Added rtfd template to Surveil https://review.openstack.org/113587 | 19:21 |
| *** stevemar has quit IRC | 19:22 | |
| aviau | Ajaeger, Done! Thanks | 19:22 |
| imcsk8 | pleia2, Ajaeger some of my reviews are taking too long to merge and get checked, i guessed that jenkins was really busy, i just needed to confirm. thanks! | 19:22 |
| Ajaeger | aviau: lgtm | 19:23 |
| *** rlandy has quit IRC | 19:23 | |
| Ajaeger | imcsk8: check on the status page for your review and see where it is in the queue... | 19:24 |
| pleia2 | jogo: you in town at all these next couple weeks? I could use some jogo brain picking time :) | 19:24 |
| *** Ryan_Lane has quit IRC | 19:24 | |
| *** e0ne has quit IRC | 19:24 | |
| *** gokrokve has quit IRC | 19:26 | |
| *** mestery has joined #openstack-infra | 19:27 | |
| openstackgerrit | Monty Taylor proposed a change to openstack-infra/storyboard: Add loader subcommand to sync from launchpad https://review.openstack.org/113624 | 19:27 |
| mordred | krotscheck: ^^ | 19:27 |
| mordred | krotscheck: UNTESTED WITH ACTUALY RUNNING AGAINST A DATABASE | 19:27 |
| openstackgerrit | A change was merged to openstack-infra/config: Cleanup elasticsearch related config https://review.openstack.org/112448 | 19:31 |
| *** annegent_ has joined #openstack-infra | 19:32 | |
| *** yamamoto has joined #openstack-infra | 19:32 | |
| *** ramashri has quit IRC | 19:32 | |
| *** stevemar has joined #openstack-infra | 19:34 | |
| krotscheck | mordred: So, I’m not entirely certain that that script should live in storyboard. | 19:35 |
| krotscheck | For now, maybe…? It feels like too-specific a tool though. | 19:35 |
| mordred | yah | 19:35 |
| mordred | I couldn't come up with a GREAT home for it | 19:35 |
| mordred | but I tend to agree with you | 19:35 |
| *** SergeyLukjanov2 has quit IRC | 19:36 | |
| nibalizer | its kindof a oneshot | 19:36 |
| *** todd_dsm has quit IRC | 19:36 | |
| nibalizer | also i think i was supposed to write that once, sorry about that | 19:36 |
| * mordred stabs nibalizer in face | 19:36 | |
| * nibalizer dies | 19:36 | |
| * nibalizer regenrates | 19:36 | |
| fungi | well, it might not be oneshot for other projects who want to run their own storyboard instance and migrate off lp, but the audience is probably pretty limited | 19:36 |
| nibalizer | but yea after we do the migration we're gonna throw the script away and never test to see if it works ever again right? | 19:37 |
| openstackgerrit | Monty Taylor proposed a change to openstack-infra/storyboard: Add loader subcommand to sync from launchpad https://review.openstack.org/113624 | 19:37 |
| krotscheck | nibalizer fungi mordred: Well, wait a sec. | 19:37 |
| *** yamamoto has quit IRC | 19:37 | |
| openstackgerrit | A change was merged to openstack-infra/config: Make use of unbound optional https://review.openstack.org/96551 | 19:37 |
| krotscheck | One of the long-term visions is the ability to set up remote federated sources for stories. I.e. if you have a dependency on a project that’s still tracked in launchpad, you can slurp those in and keep track of them. This _could_ just be the first step. | 19:38 |
| mordred | indeed | 19:38 |
| mordred | I mean, it would need work and data model additions and stuff - but one could imagine a world in which the script could be slightly more generally useful | 19:38 |
| krotscheck | So it’d evolve into being a sync mechanism with pluggable upstream providers. | 19:39 |
| *** _nadya_ has joined #openstack-infra | 19:39 | |
| krotscheck | With, perhaps a “one time only” or a “continuous” flag. | 19:39 |
| mordred | yah | 19:40 |
| *** ArxCruz has quit IRC | 19:40 | |
| openstackgerrit | A change was merged to openstack-infra/jenkins-job-builder: Allow `test` command to use conf files https://review.openstack.org/105620 | 19:41 |
| *** shashankhegde has quit IRC | 19:41 | |
| *** Ryan_Lane has joined #openstack-infra | 19:41 | |
| mordred | ttx: btw ... please see https://review.openstack.org/113624 | 19:43 |
| *** annegent_ has quit IRC | 19:43 | |
| *** raildo has left #openstack-infra | 19:43 | |
| *** _nadya_ has quit IRC | 19:44 | |
| fungi | krotscheck: oh, interesting (and potentially useful for our phased migration plans) | 19:44 |
| mordred | ttx: I split it into two pieces, so you can run the first and get it to spit out a json file with the lp data for verification | 19:44 |
| mordred | ttx: since you are the kind of this area - I'd love verification from you that you think it does what you think it should do | 19:44 |
| krotscheck | mordred: I’ll look at your thing as soon as I figure out why this rabbit patch is being stupid | 19:45 |
| ttx | mordred: will.. try to look at it. I'm supposedly on vacation starting today until the end of the week | 19:45 |
| mordred | krotscheck: I blame your barolo | 19:46 |
| ttx | I'm supposed to fix my roof. | 19:46 |
| mordred | ttx: sure. I mean, I can also just soldier forward - I think you did the important thing which was look at the data mapping | 19:46 |
| fungi | ttx: they have professionals who do those sorts of things for you now ;) | 19:46 |
| krotscheck | mordred: Dude. I pulled two bottles of that before the wedding just in case people loved it. And now those are the only two bottles left. | 19:46 |
| ttx | fungi: they just replace my old tiles though. I like my old tiles. | 19:46 |
| *** e0ne has joined #openstack-infra | 19:46 | |
| krotscheck | mordred: I have a whole case of the pinot though. | 19:46 |
| fungi | ttx: good point | 19:46 |
| ttx | I'll post pictures. | 19:47 |
| * fungi assumes terracotta tile roof | 19:47 | |
| clarkb | fungi: stevemar http://www.postgresql.org/docs/9.1/static/kernel-resources.html postgres says linux defaults are bad. devstack should probably update them | 19:48 |
| *** david-lyle has quit IRC | 19:49 | |
| *** flwang_ has joined #openstack-infra | 19:49 | |
| *** david-lyle has joined #openstack-infra | 19:49 | |
| *** marun has quit IRC | 19:50 | |
| *** marun has joined #openstack-infra | 19:50 | |
| fungi | clarkb: good to know. did they *just* change something wrt that and for some reason it only impacted one image build in one provider? | 19:51 |
| mordred | krotscheck: both were delicious | 19:51 |
| *** vhoward has joined #openstack-infra | 19:51 | |
| clarkb | fungi: I don't think they did. maybe it has to do with other services using some of the shared memory? | 19:51 |
| openstackgerrit | Spencer Krum proposed a change to openstack-infra/config: Send the puppet-dashboard module to the attic https://review.openstack.org/113627 | 19:51 |
| *** david-ly_ has joined #openstack-infra | 19:52 | |
| fungi | clarkb: maybe. i'm hoping when i have time to look at the image update log for that one it'll be some sort of obvious breakage which our prep scripts ignored | 19:52 |
| clarkb | dtroyer: ^ basically the postgres devstack configurator may want to increate the shared memory space on linux to a larger value | 19:52 |
| *** flwang_ has quit IRC | 19:53 | |
| rcarrill` | guys, i got drowned with some stuff at work and haven't had time to push a spec i had discussed at mid-cycle about issue tracker agnostic jeepyb, aka plugin based jeepyb | 19:53 |
| *** david-lyle has quit IRC | 19:53 | |
| rcarrill` | am I on time? | 19:53 |
| rcarrill` | not familiar on the spec procedure, i have time till the feature freeze to push? | 19:54 |
| *** dkliban is now known as dkliban_brb | 19:54 | |
| clarkb | rcarrill`: infra doesn't follow normal feature freeze | 19:55 |
| clarkb | rcarrill`: you are fine | 19:55 |
| fungi | rcarrill`: we don't so much have an infra feature freexe | 19:55 |
| fungi | we do sort of slush up a bit around releases to avoid screwing up our change throughput for the other projects | 19:55 |
| *** unicell has quit IRC | 19:55 | |
| *** SumitNaiksatam has quit IRC | 19:55 | |
| *** esker has joined #openstack-infra | 19:55 | |
| clarkb | ya | 19:56 |
| *** resker has joined #openstack-infra | 19:57 | |
| *** gokrokve has joined #openstack-infra | 19:57 | |
| *** AlexF has joined #openstack-infra | 19:57 | |
| *** esker has quit IRC | 19:57 | |
| *** shashankhegde has joined #openstack-infra | 19:57 | |
| *** esker has joined #openstack-infra | 19:57 | |
| stevemar | clarkb, fungi cool. thanks for looking into it | 19:58 |
| jogo | pleia2: I'm around the first half the week | 19:59 |
| jogo | then off to chicago | 19:59 |
| dtroyer | clarkb: is the shmmax value in the gate images left at the default? | 19:59 |
| clarkb | dtroyer: we don't touch it, it should be the default as set by ubuntu cloud images | 20:00 |
| pleia2 | jogo: lunch on monday? | 20:00 |
| jogo | pleia2: sounds good | 20:00 |
| pleia2 | great | 20:00 |
| pleia2 | speaking of which, lunchtime :) | 20:00 |
| *** e0ne has quit IRC | 20:00 | |
| openstackgerrit | Andreas Jaeger proposed a change to openstack-infra/config: Move openstack-security-notes to attic https://review.openstack.org/113614 | 20:01 |
| *** resker has quit IRC | 20:01 | |
| tchaypo | #info Last week's meeting notes got misfiled under http://eavesdrop.openstack.org/meetings/tripleo_weekly/ - I'll ping infra to see if we can get them moved | 20:01 |
| rcarrill` | fungi: ok, thx, i 'hope' to be able to push the spec next week | 20:01 |
| clarkb | tchaypo: ww | 20:01 |
| tchaypo | Who could I talk to about getting ^^ moved back with the rest of the tripleo meetings? | 20:01 |
| clarkb | or did you copy pasta? | 20:02 |
| tchaypo | I deliberately pasted, it was in my buffer from being ready to mention in our meeting | 20:02 |
| *** amotoki has quit IRC | 20:02 | |
| clarkb | I see, I can copy pasta the logs. where should they go? | 20:02 |
| tchaypo | under /meetings/tripleo/ | 20:02 |
| *** jedimike has quit IRC | 20:03 | |
| fungi | tchaypo: i assume someone did #startmeeting tripleo_weekly last week? | 20:03 |
| clarkb | pleia2: jeblair: I wanted to ask in the meeting if we need to chagne tactics for pootle to be ready by the 20th? | 20:03 |
| tchaypo | yep. I'm updating the wiki to clarify that.. | 20:03 |
| *** stevemar has quit IRC | 20:03 | |
| *** MarkAtwood has quit IRC | 20:04 | |
| anteaya | jeblair: so what is your decision on ci renames? | 20:05 |
| *** zz_dimtruck is now known as dimtruck | 20:05 | |
| anteaya | jeblair: I'll support whatever you decide | 20:05 |
| anteaya | I'm just asking by way of a heads up for myself | 20:05 |
| clarkb | tchaypo: done | 20:06 |
| * fungi grumbles at lack of any obvious error in image update log | 20:06 | |
| tchaypo | clarkb: <3 | 20:06 |
| clarkb | fungi: I really don't think this is an error | 20:06 |
| fungi | clarkb: have a theory on why it would impact only one nodepool provider out of 8? | 20:07 |
| clarkb | fungi: packages | 20:07 |
| openstackgerrit | Davanum Srinivas (dims) proposed a change to openstack-infra/elastic-recheck: Add query for Postgres startup failure https://review.openstack.org/113634 | 20:07 |
| *** pdmars has quit IRC | 20:08 | |
| *** homeless has joined #openstack-infra | 20:08 | |
| clarkb | postgres asked for >32MB of shared memory | 20:08 |
| clarkb | default max is 32MB | 20:08 |
| fungi | clarkb: different package versions on hpcloud-b1 than the others you think? got balanced to a different mirror? | 20:08 |
| clarkb | either new postgres or new kernel package affected the numbers? | 20:08 |
| openstackgerrit | Monty Taylor proposed a change to openstack-infra/storyboard: Add loader subcommand to sync from launchpad https://review.openstack.org/113624 | 20:08 |
| clarkb | fungi: ya, or the image builds raced mirror updates | 20:08 |
| fungi | i'll run a comparison | 20:08 |
| openstackgerrit | Monty Taylor proposed a change to openstack-infra/storyboard: Assocate StoryTag with Story via mapping https://review.openstack.org/113635 | 20:08 |
| fungi | see whether this is a precursor to everywhere breaking tomorrow | 20:09 |
| mordred | krotscheck: ^^ there's the bug I found in StoryTag | 20:09 |
| mordred | krotscheck: basically, it was not possible to actually have the same tag on more than one story | 20:09 |
| *** andreaf has quit IRC | 20:10 | |
| *** bhuvan_ has quit IRC | 20:10 | |
| *** andreaf has joined #openstack-infra | 20:12 | |
| *** rcarrillocruz has joined #openstack-infra | 20:12 | |
| clarkb | there is a lot of purple on our graph but we don't appear to have leaked floating ips like crazy | 20:14 |
| clarkb | and the hpcloud error rate is much lower than yesterday | 20:14 |
| clarkb | I am guessing that is related to nova on rax side cleanup then? | 20:14 |
| *** rcarrill` has quit IRC | 20:14 | |
| clarkb | looks like a small chunk of nodes in delete state for a day or so | 20:15 |
| *** pdmars has joined #openstack-infra | 20:16 | |
| *** MarkAtwood has joined #openstack-infra | 20:17 | |
| mordred | clarkb: so why were you opposed to re-using floating-ips again/ | 20:18 |
| clarkb | fungi: did you delete the image that failed with postgres shmmax? | 20:18 |
| *** gema has joined #openstack-infra | 20:18 | |
| clarkb | mordred: beacuse the cleanup aspect of it is hard | 20:18 |
| krotscheck | Anyone here familiar with rabbitmq’s permissions structure? I cannot for the sake of me figure out why I can’t connect with a non-admin user. | 20:18 |
| *** gema_ has quit IRC | 20:19 | |
| clarkb | mordred: right now with a 1:1 relationship cleanup is easy | 20:19 |
| mordred | clarkb: can you give me _slightly_ more to go on than that? I'd like to think about whether or not it's hard or impossible | 20:19 |
| clarkb | mordred: sure. So right now we use all 600 ips in our quota, but on the weekend we only need a max of 100 | 20:19 |
| clarkb | mordred: nodepool needs to free the 500 it isn't using | 20:19 |
| mordred | clarkb: why? | 20:20 |
| mordred | (just for sake of argument) | 20:20 |
| clarkb | mordred: because they cost money | 20:20 |
| mordred | ok | 20:20 |
| mordred | fair enough | 20:20 |
| clarkb | we have users of nodepool that explicitly want to use it to use only the resources they need | 20:20 |
| *** pdmars_ has joined #openstack-infra | 20:20 | |
| ianw | fungi: a lot of check jobs seem to be waiting for dsvm-f20 ATM | 20:21 |
| mordred | clarkb: so, what would you think about a brief logic of "if there is a floating ip that is unattached, I should try to use it, and node deletion should delete floating ips" | 20:21 |
| mordred | clarkb: so that if we happen to leak the occasional ip, it gets re-used | 20:21 |
| mordred | clarkb: but we still reap as usual | 20:21 |
| clarkb | mordred: that races the allocation of the ip | 20:21 |
| clarkb | mordred: unfortunately neutron/nova don't really indicate this well to their users | 20:21 |
| ianw | several solum jobs | 20:21 |
| anteaya | cinder sprint question, if core abandons a patch can the owner unabandon the patch? | 20:22 |
| *** pdmars has quit IRC | 20:22 | |
| clarkb | anteaya: yes I think so, you cn always hve them test it | 20:22 |
| anteaya | kk | 20:22 |
| clarkb | ianw: we are behind on node alloactions | 20:22 |
| clarkb | ianw: and your patch was reverted | 20:22 |
| homeless | hi guys, I was having problems with jeepyb and the close pull request behavior for github, then I found that the root cause was an if statement in the close_pull_requests.py, if its possible, could someone review this one, https://review.openstack.org/#/c/113598/ | 20:22 |
| *** andreaf_ has joined #openstack-infra | 20:23 | |
| *** flaper87 is now known as flaper87|afk | 20:23 | |
| fungi | clarkb: i did delete it... we should be running off yesterday's image there at this point | 20:23 |
| ianw | clarkb: yeah, just more f20 use than i was aware of | 20:23 |
| mordred | jogo, mikal: see above conversation about floating ips and nodepool for a real-world example of why I think nova should grow MORE smarts around proxying to neutron | 20:23 |
| mordred | jogo, mikal: specifically, nova boot --gimme-floating-ip should be a thing I should be able to ask my cloud to do | 20:24 |
| mordred | beause it's REALLY hard to get right as a user | 20:24 |
| mordred | also, you should see the floating-ip related patches I recently landed upstream in ansible | 20:24 |
| mordred | they'll make you cry a gbit | 20:24 |
| jogo | mordred: ++ good idea | 20:25 |
| *** lbragstad has joined #openstack-infra | 20:25 | |
| jogo | salv-orlando marun: ^ | 20:25 |
| *** andreaf has quit IRC | 20:26 | |
| clarkb | fungi: ok, I was going to say we could boot a node off of it and check the shmmax | 20:26 |
| fungi | clarkb: i may still be able to hold one. checking | 20:26 |
| *** dprince has quit IRC | 20:26 | |
| mordred | marun, salv-orlando, jogo: https://github.com/ansible/ansible/blob/devel/library/cloud/nova_compute#L370 | 20:28 |
| jogo | mordred: hehe, I am talking to arosen about this in #openstack-neutron | 20:28 |
| *** amcrn has joined #openstack-infra | 20:28 | |
| fungi | clarkb: 15.126.233.190 was booted over an hour ago and has 33554432 | 20:29 |
| pleia2 | clarkb: re pootle: we could just set it up with boring old logins for now, and if we actually decide to use it we'll keep pushing on openid stuff | 20:29 |
| clarkb | fungi: that looks correct to me | 20:29 |
| fungi | checking now to see if that's booted from the image i deleted or the other | 20:29 |
| clarkb | pleia2: that is an option | 20:29 |
| jogo | mordred: so you should join the room and join the fun | 20:29 |
| pleia2 | my zanata demo just has plain logins, and it looks like SSO for us on that will take work too, so we have work there | 20:30 |
| fungi | clarkb: 2014-08-12 19:10:41,532 INFO nodepool.NodeLauncher: Creating server with hostname devstack-trusty-hpcloud-b1-1429306.slave.openstack.org in hpcloud-b1 from image devstack-trusty for node id: 1429306 | 20:30 |
| clarkb | fungi: is that the old image? | 20:31 |
| fungi | clarkb: auth.log says i deleted the image at 19:30:24 | 20:31 |
| marcusvrn | clarkb: now it's working!! the problem was that zuul didn't find the private key on /home/zuul/.ssh/ so it couldn't clone git repo | 20:31 |
| anteaya | clarkb: yes you are correct we just tested: 92233 | 20:31 |
| fungi | clarkb: so it should be booted from the old image | 20:31 |
| *** emagana has quit IRC | 20:32 | |
| clarkb | fungi: ok so I think the kernel is doing the correct thing there. it looks a lot like my shmmax and the documented default | 20:32 |
| *** yamamoto has joined #openstack-infra | 20:32 | |
| clarkb | fungi: so signs would point at postgres asking for more all of a sudden? | 20:32 |
| *** emagana has joined #openstack-infra | 20:32 | |
| * salv-orlando reading scrollback | 20:32 | |
| clarkb | fungi: oh wait, devstack isntalls the packages. we cache them but it should udpate the cache if necessary right? | 20:33 |
| fungi | clarkb: the last job to report with that error and get indexed into logstash was at 19:47:00 but i'll wait and see if we grow more of them | 20:33 |
| *** ramashri has joined #openstack-infra | 20:33 | |
| openstackgerrit | Michael Krotscheck proposed a change to openstack-infra/config: Added RabbitMQ to StoryBoard https://review.openstack.org/98474 | 20:33 |
| clarkb | fungi: so probably not purely a package issue | 20:33 |
| krotscheck | Anyone want to review this? I promise it’s quick. https://review.openstack.org/#/c/113616/1 | 20:33 |
| fungi | krotscheck: 0400 for a dir in /etc? full of super secret stuff? | 20:34 |
| krotscheck | fungi: SUPER secret. | 20:34 |
| krotscheck | fungi: It’s my default “hey there’s passwords in here go away” perms | 20:34 |
| openstackgerrit | A change was merged to openstack-infra/elastic-recheck: Fix MessageConfig https://review.openstack.org/113620 | 20:35 |
| clarkb | krotscheck: done | 20:35 |
| salv-orlando | mordred, jogo: a sort of macro then? boot a vm and allocate also a floating IP (for all interfaces by default) | 20:35 |
| fungi | krotscheck: makes sense for the config file, but the config dir? | 20:35 |
| clarkb | and I -1'd for that raisin | 20:35 |
| clarkb | fungi: see my omment on the change | 20:35 |
| *** Ajaeger has quit IRC | 20:35 | |
| fungi | the "security guy" is unparanoid enough to vote for 0755 on the dir | 20:36 |
| clarkb | it should be 0500 or 0700 | 20:36 |
| openstackgerrit | Michael Krotscheck proposed a change to openstack-infra/config: Create storyboard configuration directory https://review.openstack.org/113616 | 20:36 |
| *** julim has quit IRC | 20:36 | |
| clarkb | 0400 will only work by way of puppet weirdness | 20:36 |
| *** emagana has quit IRC | 20:36 | |
| fungi | 0600 or 0400 on the file is expected, but the dir itself doesn't need that treatment | 20:36 |
| *** yamamoto has quit IRC | 20:37 | |
| jogo | salv-orlando: we are talking about this with arosen in -neutron | 20:37 |
| krotscheck | Done | 20:37 |
| krotscheck | Wanna look at it again? | 20:37 |
| clarkb | krotscheck: yup +2 | 20:37 |
| krotscheck | yay! | 20:37 |
| *** crc32 has quit IRC | 20:38 | |
| openstackgerrit | Sean Roberts proposed a change to openstack-infra/config: fixup congress-specs infra configuration https://review.openstack.org/112423 | 20:38 |
| openstackgerrit | Sergey Lukjanov proposed a change to openstack/requirements: Remove d-g from projects.txt https://review.openstack.org/113643 | 20:38 |
| clarkb | pleia2: I would be fine using the plan login for now. I will go ahead and set up my server that way and give you an admin account | 20:39 |
| clarkb | pleia2: if nothing else it gets us through the demo phase | 20:39 |
| openstackgerrit | Ian Wienand proposed a change to openstack-infra/nodepool: Log error on over-allocation https://review.openstack.org/113644 | 20:39 |
| pleia2 | clarkb: sounds good | 20:39 |
| *** bhuvan has joined #openstack-infra | 20:40 | |
| *** Guest61254 has quit IRC | 20:41 | |
| krotscheck | Ok, nibalizer - that stability patch I asked for just got approved, once it gets merged we can land all the things. | 20:42 |
| *** homeless has quit IRC | 20:44 | |
| *** stevemar has joined #openstack-infra | 20:44 | |
| openstackgerrit | Michael Krotscheck proposed a change to openstack-infra/config: Added RabbitMQ to StoryBoard https://review.openstack.org/98474 | 20:45 |
| clarkb | pleia2: ok thats done | 20:45 |
| clarkb | pleia2: but it looks like profile still break | 20:45 |
| clarkb | I think that bug may go deeper than just openid setting username to none | 20:45 |
| *** homeless has joined #openstack-infra | 20:46 | |
| *** rfolco has quit IRC | 20:46 | |
| * pleia2 nods | 20:46 | |
| clarkb | pleia2: if you register I think I can make you an admin | 20:46 |
| *** dims has joined #openstack-infra | 20:46 | |
| pleia2 | clarkb: ip again? | 20:46 |
| *** dims is now known as Guest60092 | 20:46 | |
| clarkb | pleia2: http://23.253.231.198 | 20:46 |
| pleia2 | thanks | 20:47 |
| openstackgerrit | Oscar Romero (homeless) proposed a change to openstack-infra/jeepyb: Fix close_pull_requests from jeepyb https://review.openstack.org/113598 | 20:47 |
| marun | mordred, jogo: So the suggestion is that nova and/or neutron should know how to allocate floating ips rather than having to implement the same behavior in every single orchestration mechanism under the sun? | 20:47 |
| pleia2 | clarkb: and can you also add how you set up django to the install notes? | 20:47 |
| mordred | marun: yes | 20:47 |
| krotscheck | mordred: So about tags - is that going to be required for your data import? Because we’re still arguing about the spec. | 20:47 |
| mordred | marun: we're talking about it in #openstack-neutron, btw | 20:47 |
| *** dkliban_brb is now known as dkliban | 20:47 | |
| pleia2 | clarkb: registered as lyz | 20:47 |
| mordred | krotscheck: oh! well, if we're still arguing about the spec, I can revert to just commenting out actually using them | 20:47 |
| clarkb | pleia2: I didn't have to do anything to setup django, pip installed it | 20:48 |
| clarkb | pleia2: I haven't gotten to using wsgi | 20:48 |
| pleia2 | oh, neat :) | 20:48 |
| ianw | clarkb / fungi : is it possible to get nodepool logs into logstash? totally new to logstash-agent, etc, digging around to see how it works now | 20:48 |
| clarkb | ianw: I want to say no :) I have tried to not put system logs nd test logs in the same cluster | 20:48 |
| clarkb | ianw: mostly because system logs can be much more sensitive and shouldn't be 100% public imo | 20:49 |
| jogo | marun: converation moved to #openstack-neutron | 20:49 |
| clarkb | ianw: but sorting that out with logstash/elasticsearch is weird because they have rudimentary support for ACLs | 20:49 |
| krotscheck | mordred: Actually, nevermind. https://review.openstack.org/#/c/97211/ | 20:49 |
| marun | jogo: got it, thanks. | 20:49 |
| krotscheck | mordred: No code has landed for that yet though, so the data might end up having to be migrated. | 20:50 |
| ianw | clarkb: yeah, that is a good policy. just wondering how i can monitor for https://review.openstack.org/#/c/113644/ | 20:50 |
| mordred | krotscheck: so, I _think_ the patch I did is still in line with that spec | 20:51 |
| mordred | krotscheck: the tags table probably needs more metadata to deal with system/protected or something ... but the many/many relationship seems like it's still needed? | 20:51 |
| clarkb | ianw: maybe graphite and have nodepool bump a counter? that is really not enough info tough :/ | 20:52 |
| clarkb | pleia2: arg I may have to update the db to give us superuser powers, this is so weird | 20:54 |
| clarkb | clearly there is a reason I haven't really used django before | 20:54 |
| pleia2 | clarkb: hm, also having trouble going to my account page | 20:55 |
| clarkb | pleia2: ya I think that is just generally broken | 20:55 |
| clarkb | pleia2: regression from the allauth move | 20:55 |
| ianw | clarkb: hmm, yeah, knowing it happened would be enough for now | 20:55 |
| * pleia2 nods | 20:55 | |
| *** rfolco has joined #openstack-infra | 20:56 | |
| clarkb | pleia2: I updated the db directly you should be a super user now. can you log out then back in and see if you have an admin tab? | 20:57 |
| pleia2 | clarkb: yep, thanks | 20:57 |
| clarkb | woot | 20:57 |
| *** emagana has joined #openstack-infra | 20:59 | |
| *** markmc has quit IRC | 21:02 | |
| *** miqui has quit IRC | 21:02 | |
| *** praneshp has quit IRC | 21:03 | |
| *** praneshp has joined #openstack-infra | 21:04 | |
| *** bhuvan_ has joined #openstack-infra | 21:04 | |
| *** MarkAtwood has quit IRC | 21:05 | |
| *** bhuvan has quit IRC | 21:07 | |
| homeless | Hi clarkb, regarding https://review.openstack.org/#/c/113598/ (close pull requests) I think we cannot leave the if statement into one line because if the option seciont is not here, the it fails. look at this, http://paste.forj.io/show/305/ | 21:09 |
| *** pdmars_ has quit IRC | 21:09 | |
| homeless | seciont=section | 21:09 |
| clarkb | homeless: if you say 'if 'options' not in section or use options:' that will garuntee that options is there via short circuiting right? | 21:11 |
| clarkb | if it is not there you short circuit. if it is ther you go to the next thing | 21:11 |
| homeless | is what I thought, but as is an "or" its validate both conditions | 21:11 |
| *** gondoi is now known as zz_gondoi | 21:12 | |
| mordred | homeless: or only touches the second if the first option is false | 21:12 |
| mordred | homeless: if the first is true, it will not bother with the second side | 21:13 |
| *** aysyd has quit IRC | 21:13 | |
| *** pblaho has joined #openstack-infra | 21:13 | |
| pleia2 | clarkb: can you: apt-get install python3-aeidon unzip iso-codes | 21:13 |
| clarkb | pleia2: on the translate host? | 21:14 |
| pleia2 | clarkb: yep | 21:14 |
| clarkb | pleia2: we don't python3 there unless I am mistaken | 21:14 |
| *** weshay has quit IRC | 21:14 | |
| *** ociuhandu has joined #openstack-infra | 21:14 | |
| clarkb | pleia2: or is that just a tool? | 21:14 |
| pleia2 | clarkb: ah, right, I'll see if something else provides aeidon | 21:15 |
| mordred | homeless: http://paste.openstack.org/show/94046 | 21:15 |
| pleia2 | actually, no need for that really, I don't think we do "subtitles" | 21:15 |
| pleia2 | unzip and iso-codes we need though | 21:15 |
| clarkb | pleia2: done | 21:16 |
| pleia2 | thanks | 21:16 |
| *** radez is now known as radez_g0n3 | 21:17 | |
| openstackgerrit | Anita Kuno proposed a change to openstack-infra/config: Added a missing single quote https://review.openstack.org/113655 | 21:17 |
| *** yamamoto has joined #openstack-infra | 21:17 | |
| *** mrda-away is now known as mrda | 21:18 | |
| nibalizer | krotscheck: wooooooot | 21:19 |
| *** sandywalsh has quit IRC | 21:19 | |
| *** emagana has quit IRC | 21:20 | |
| openstackgerrit | A change was merged to openstack/requirements: Updated an explanation to the sqla requirement https://review.openstack.org/111531 | 21:20 |
| krotscheck | mordred: Yeah, I think your changes are still in line. | 21:20 |
| mordred | krotscheck: ossum | 21:20 |
| *** emagana has joined #openstack-infra | 21:21 | |
| homeless | clarkb, mordred, this is my close_pull_requests.py file http://paste.forj.io/show/306/, and this is what I got http://paste.forj.io/show/307/ | 21:22 |
| homeless | probably im missing something but for me that doesnt work | 21:22 |
| openstackgerrit | Michael Krotscheck proposed a change to openstack-infra/storyboard: Added subscriber and publisher modules https://review.openstack.org/102842 | 21:24 |
| openstackgerrit | Michael Krotscheck proposed a change to openstack-infra/storyboard: Added subscription API https://review.openstack.org/108516 | 21:24 |
| *** yamamoto has quit IRC | 21:24 | |
| openstackgerrit | Michael Krotscheck proposed a change to openstack-infra/storyboard: Added a subscription_events table & subscription hander https://review.openstack.org/113016 | 21:24 |
| *** yfried__ has quit IRC | 21:25 | |
| *** dkranz has quit IRC | 21:25 | |
| *** ominakov has quit IRC | 21:25 | |
| *** mfer has quit IRC | 21:27 | |
| *** Guest60092 is now known as dims_ | 21:27 | |
| anteaya | so making a patch that makes a change to http://git.openstack.org/cgit/openstack-infra/config/tree/modules/openstack_project/files/jenkins_job_builder/config/projects.yaml | 21:28 |
| anteaya | I don't see any format for how to decide where to make my addition | 21:28 |
| mordred | homeless: you're missing quotes | 21:28 |
| anteaya | so anywhere I feel like it? | 21:28 |
| mordred | homeless: if 'options' not in section or 'has-pull-requests' in section['options']: | 21:29 |
| nibalizer | hrm how did i do this the first time | 21:31 |
| homeless | omg.. that is right, also on the has-pull-requests, Im sorry mordred | 21:31 |
| homeless | I just copy/paste the proposal from the comment on gerrit | 21:31 |
| mordred | homeless: no worries! that's why we all talk all the time :) | 21:31 |
| homeless | yup, I will change it right now | 21:32 |
| *** mrmartin has quit IRC | 21:32 | |
| *** lyxus has quit IRC | 21:33 | |
| *** marun has quit IRC | 21:33 | |
| *** whoops has quit IRC | 21:34 | |
| *** bknudson has quit IRC | 21:35 | |
| *** crc32 has joined #openstack-infra | 21:36 | |
| mordred | jeblair: oh - btw - while you were out, my all of my fixes to the nova module to ansible landed | 21:36 |
| YorikSar | anteaya: Can you please lift -1 from https://review.openstack.org/112666 ? Doug has put his +1 there, repo is reviewed and considered good. | 21:37 |
| openstackgerrit | Aishwarya Thangappa proposed a change to openstack-infra/storyboard: Added a subscription_events table & subscription hander https://review.openstack.org/113016 | 21:37 |
| openstackgerrit | Oscar Romero (homeless) proposed a change to openstack-infra/jeepyb: Fix close_pull_requests from jeepyb https://review.openstack.org/113598 | 21:37 |
| jeblair | mordred: neat, that even took less than 6 months | 21:37 |
| anteaya | YorikSar: +1, thanks | 21:38 |
| mordred | jeblair: yah! | 21:38 |
| mordred | jeblair: so, once 1.8 comes out, there are several things I can drop from that ansible patch to config I put up the other day | 21:38 |
| YorikSar | anteaya: Thanks :) | 21:38 |
| jeblair | mordred: so ansible set low expectations, and then exceeded them and you were happy | 21:38 |
| *** ominakov has joined #openstack-infra | 21:39 | |
| jeblair | (as far as project workflow goes) | 21:39 |
| mordred | yup | 21:39 |
| mordred | althought to be fair, I did ping the project lead out of band | 21:39 |
| mordred | after having had a beer with him | 21:39 |
| jeblair | this is all useful information! | 21:40 |
| mordred | so it's entirely possible that the key in any situation is social engineering | 21:40 |
| jeblair | mordred: i have a list of people for you to have beers with | 21:40 |
| mordred | jeblair: consider it done | 21:40 |
| krotscheck | nibalizer: Need any help? I have a vested interest in getting a new feature into storyboard-puppet :) | 21:42 |
| openstackgerrit | Joe Gordon proposed a change to openstack-infra/devstack-gate: Collect javelin2 log https://review.openstack.org/113658 | 21:42 |
| openstackgerrit | Aaron Rosen proposed a change to openstack-infra/nodepool: Remove unneeded call to removeFloatingIP in cleanupServer https://review.openstack.org/113659 | 21:42 |
| anteaya | so any comment on whether is is a viable job? http://git.openstack.org/cgit/openstack-infra/config/tree/modules/openstack_project/files/jenkins_job_builder/config/hooks.yaml | 21:43 |
| anteaya | hook-{name}-rtfd | 21:44 |
| nibalizer | krotscheck: i dont think what im doing now is easily parallelizable | 21:44 |
| anteaya | I'm making a patch for a stackforge repo, looking for the right job for publishing docs | 21:44 |
| krotscheck | nibalizer: You got it. I’ll be standing in the wings if you nee dme. | 21:45 |
| *** dims_ has quit IRC | 21:45 | |
| *** dims_ has joined #openstack-infra | 21:45 | |
| mordred | anteaya: the rtfd hooks are the right way for stackforge stuff | 21:46 |
| *** hashar has quit IRC | 21:47 | |
| anteaya | mordred: thanks | 21:48 |
| openstackgerrit | A change was merged to openstack-infra/config: Create storyboard configuration directory https://review.openstack.org/113616 | 21:49 |
| *** flwang_ has joined #openstack-infra | 21:50 | |
| openstackgerrit | Joe Gordon proposed a change to openstack-infra/config: Add javelin.log to logstash https://review.openstack.org/113663 | 21:51 |
| jogo | mtreinish: patch is up in advance | 21:52 |
| mtreinish | cool | 21:53 |
| *** armax has quit IRC | 21:53 | |
| *** flwang_ has quit IRC | 21:54 | |
| *** homeless_ has joined #openstack-infra | 21:55 | |
| mtreinish | jogo: I gave reviewed the other 2 | 21:56 |
| mtreinish | wow, I can't type today... | 21:56 |
| anteaya | SergeyLukjanov: are you about? | 21:57 |
| anteaya | SergeyLukjanov: cinder has a question about a hadoop dependency | 21:57 |
| * anteaya is at the cinder sprint | 21:57 | |
| *** homeless has quit IRC | 21:58 | |
| *** doug-fish has left #openstack-infra | 21:59 | |
| *** homeless_ has quit IRC | 21:59 | |
| *** rcarrill` has joined #openstack-infra | 21:59 | |
| *** yamamoto has joined #openstack-infra | 21:59 | |
| *** homeless has joined #openstack-infra | 21:59 | |
| *** rcarrillocruz has quit IRC | 22:01 | |
| jogo | mtreinish: thanks | 22:02 |
| dansmith | um | 22:02 |
| lifeless | mordred: https://review.openstack.org/#/c/94108/ - I'm losing hope | 22:02 |
| *** bknudson1 has joined #openstack-infra | 22:02 | |
| dansmith | the check queue is growing crazy right now | 22:02 |
| dansmith | like someone just rechecked everything | 22:02 |
| clarkb | dansmith: its node issues | 22:03 |
| *** amotoki has joined #openstack-infra | 22:03 | |
| dansmith | wow, okay | 22:03 |
| clarkb | dansmith: those upstream bugs in downstream clouds are killing us | 22:03 |
| dansmith | so is this just refilling the queue after a restart or something? | 22:03 |
| clarkb | dansmith: no, it is rax leaking nodes, hpcloud leaking floating ips, + a couple other bugs we know about | 22:04 |
| dansmith | so just to be clear, | 22:04 |
| clarkb | dansmith: that decreases our effective quota and results in overall less throughput | 22:04 |
| dansmith | ten minutes ago, the check queue was under 100 and in the last three minutes it's gotten to about 300 | 22:05 |
| mordred | lifeless: don't lose hope | 22:05 |
| clarkb | dansmith: oh I see, so the 100 is because of what I describe | 22:05 |
| clarkb | the 300 is interesting | 22:05 |
| dansmith | yeah, the 300 is the "um..." above :) | 22:05 |
| mordred | jesusaurus: this failure confuses me: http://logs.openstack.org/01/113401/1/check/gate-config-puppet-apply-precise/bfa0abb/console.html | 22:06 |
| mordred | nibalizer: ^^ | 22:06 |
| clarkb | dansmith: vmware minesweeper is to blame | 22:06 |
| mordred | lifeless: did you see https://review.openstack.org/#/c/113401/ | 22:06 |
| dansmith | clarkb: ah, did it just report on 300 things that triggered stale rechecks? | 22:06 |
| mordred | lifeless: ah, you did | 22:06 |
| jesusaurus | mordred: huh, im not sure what's timing out there | 22:07 |
| mordred | jesusaurus: maybe just bad node that ran REALLY slow? | 22:07 |
| jesusaurus | also, wow, those logs are UGLY | 22:07 |
| clarkb | dansmith: minesweeper just rechecked a million things | 22:07 |
| mordred | yah | 22:07 |
| mordred | they make me want to die | 22:07 |
| *** amotoki has quit IRC | 22:08 | |
| mordred | dansmith: why did it do that? | 22:08 |
| dansmith | mordred: why are you asking me? | 22:08 |
| clarkb | dansmith: it is tempting to disable their account | 22:08 |
| clarkb | but we should probably find them and ask why | 22:08 |
| *** scottda__ has joined #openstack-infra | 22:08 | |
| mordred | dansmith: I meant to say... | 22:08 |
| scottda__ | hi | 22:08 |
| mordred | clarkb: why did it do that? | 22:08 |
| anteaya | dansmith isn't vmware | 22:08 |
| * mordred knows | 22:08 | |
| dansmith | hehe, okay :) | 22:08 |
| anteaya | arosen and salv-orlando are | 22:08 |
| dansmith | I was about to be really insulted :P | 22:08 |
| anteaya | so this is scottda__ | 22:08 |
| jogo | can we turn off stale recehcks on 3rd party CI system votes | 22:08 |
| ianw | clarkb: http://graphite.openstack.org/render/?target=stats.nodepool.launch.devstack-f20.*.*&rawData=True <- i'm assuming this is likely due to a bug? | 22:08 |
| anteaya | he is at the cinder sprint with me | 22:08 |
| openstackgerrit | lifeless proposed a change to openstack-dev/pbr: Teach pbr about post versioned dev versions. https://review.openstack.org/80449 | 22:08 |
| anteaya | and also works on hp public cloud | 22:08 |
| openstackgerrit | lifeless proposed a change to openstack-dev/pbr: Raise an error if preversion versions are too low https://review.openstack.org/108269 | 22:08 |
| openstackgerrit | lifeless proposed a change to openstack-dev/pbr: Look for and process sem-ver pseudo headers in git https://review.openstack.org/108270 | 22:08 |
| openstackgerrit | lifeless proposed a change to openstack-dev/pbr: Introduce a SemanticVersion object https://review.openstack.org/94108 | 22:09 |
| mordred | scottda__: hey there | 22:09 |
| dansmith | ...429 | 22:09 |
| scottda__ | I can be a proxy for pinging the HP Public cloud .NOC room... | 22:09 |
| salv-orlando | jogo, dansmith: I was told the stale recheck issue on vmw minesweeper was sorted? | 22:09 |
| nibalizer | mordred: o.O | 22:09 |
| anteaya | clarkb mordred what is useful for scottda__ to pass along to help about hp public cloud? | 22:09 |
| nibalizer | rastu/win 16 | 22:09 |
| lifeless | mordred: 94108 doesn't need the haveged fix - it needs your and dhellmann's +2's. | 22:09 |
| scottda__ | or someone with a proper hipchat account can also do this... | 22:09 |
| dansmith | salv-orlando: apparently not | 22:09 |
| lifeless | mordred: the one *above* it needs haveged | 22:09 |
| salv-orlando | dansmith: still the same patch? | 22:09 |
| scottda__ | but they are very interested if large numbers of NOva instances are stuck in delete. | 22:09 |
| * anteaya has forgotten her hipchat password :( | 22:09 | |
| dansmith | salv-orlando: no, every patch (talk to clarkb, I don't know anything) | 22:09 |
| jogo | salv-orlando: why is is minesweeper calling its own recheck | 22:10 |
| jogo | that is silly | 22:10 |
| salv-orlando | jogo: I am afraid I do not understand what do you mean by “its own”… | 22:10 |
| clarkb | scottda__: yesterday we were leaking floating ips. still trying to figure out if that is our side of hpcloud side | 22:10 |
| dansmith | we're almost at 500 and there are 100 things in the backlog | 22:10 |
| jogo | salv-orlando: see https://review.openstack.org/#/c/65452/ | 22:11 |
| dansmith | I'm thinking disabling the account is a good plan right now :) | 22:11 |
| jogo | dansmith: ++ | 22:11 |
| dansmith | 513+102 | 22:11 |
| dansmith | woo | 22:11 |
| clarkb | salv-orlando: see https://review.openstack.org/#/c/65474/ | 22:11 |
| clarkb | salv-orlando: it just did that for hundreds of changes | 22:11 |
| jogo | this is funny | 22:11 |
| jogo | and bad | 22:11 |
| dansmith | jogo: funny in a "can't look away" sort of way | 22:12 |
| scottda__ | anteaya: https://www.hipchat.com/sign_in has a forgot password feature | 22:12 |
| anteaya | scottda__: thanks | 22:12 |
| jogo | dansmith: yup, like a crash | 22:12 |
| anteaya | clarkb mordred anything specific for scottda__ ? | 22:12 |
| jogo | where no one gets hurt | 22:12 |
| clarkb | anteaya: yesterday and day before we were leaking floating ips | 22:12 |
| dansmith | 555+111 | 22:12 |
| clarkb | anteaya: thats all i have right now | 22:12 |
| * jogo decides to not review any more patches for the day | 22:13 | |
| jogo | as it will take hours for this to flush out | 22:13 |
| anteaya | scottda__: is the fact we were leaking floating ips enough to go on? | 22:13 |
| dansmith | at least hours, methinks | 22:13 |
| clarkb | anteaya: scottda__ looking at the list of nodes to see if we have a lot in delete states | 22:13 |
| anteaya | clarkb: thanks | 22:13 |
| *** cdub has quit IRC | 22:13 | |
| *** rcarrillocruz has joined #openstack-infra | 22:14 | |
| clarkb | anteaya: scottda__ nothing is more than ~15 minutes | 22:14 |
| scottda__ | I just asked the NOC if they are seeing issues. | 22:14 |
| clarkb | er nothing more than 15 minutes old in a delete state | 22:14 |
| salv-orlando | clarkb: just like you I’m playing on several tables… is vmware recheck triggering a jenkins build upstream? | 22:14 |
| clarkb | salv-orlando: yes | 22:14 |
| salv-orlando | crap. I’ve spawn a recheck now on the jenkins for neutron patches. | 22:14 |
| clarkb | salv-orlando: are you all using zuul? if yes you can recheck en mass directly via the zuul client | 22:15 |
| clarkb | salv-orlando: which is proably a much better way to recheck 500 changes | 22:15 |
| dansmith | backlog is dropping, | 22:15 |
| dansmith | so I think the storm is ending | 22:15 |
| russellb | salv-orlando: see the check queue on http://status.openstack.org/zuul/ | 22:15 |
| *** rcarrill` has quit IRC | 22:16 | |
| russellb | and the zuul job queue graph at the bottom | 22:16 |
| russellb | that's what it just caused :) | 22:16 |
| clarkb | scottda__: anteaya: there is also a specific upstream bug that we ran into, let me find the bugs | 22:16 |
| anteaya | clarkb: thanks | 22:17 |
| clarkb | scottda__: anteaya https://bugs.launchpad.net/python-neutronclient/+bug/1297309 and https://bugs.launchpad.net/neutron/+bug/1211915 I don't remember which one was hpcloud and which was rax | 22:17 |
| uvirtbot | Launchpad bug 1297309 in python-neutronclient "Unauthorized: Unknown auth strategy" [Medium,Fix committed] | 22:17 |
| clarkb | but we hit them one each in each cloud | 22:17 |
| *** emagana has quit IRC | 22:17 | |
| anteaya | clarkb: was that yesterday? | 22:18 |
| salv-orlando | clarkb, russellb: ok I’ve disabled the job for the neutron part. The owner of the jenkins instances for the nova part have been warned as well and should disable recheck soon | 22:18 |
| clarkb | anteaya: no this was a week ago? I sent mail to the list on the 5th so I think it was the 5th | 22:18 |
| *** emagana has joined #openstack-infra | 22:18 | |
| anteaya | kk thanks | 22:18 |
| anteaya | which list? | 22:18 |
| clarkb | dev list | 22:18 |
| salv-orlando | clarkb: I’m not the only one with control over the CI system, so I need to check what happened to the recheck monitor | 22:18 |
| clarkb | anteaya: Gate bugs in the real world | 22:18 |
| anteaya | kk | 22:18 |
| anteaya | salv-orlando: thank you | 22:18 |
| anteaya | clarkb: thanks | 22:18 |
| *** lemongrabber has joined #openstack-infra | 22:20 | |
| *** changbl has quit IRC | 22:21 | |
| *** emagana has quit IRC | 22:22 | |
| *** armax has joined #openstack-infra | 22:22 | |
| *** dustins has quit IRC | 22:23 | |
| fungi | catching up on scrollback, it *might* make sense to not trigger stale rechecks on comments which carry a verify vote | 22:23 |
| *** pblaho has quit IRC | 22:23 | |
| *** dmsimard is now known as dmsimard_away | 22:23 | |
| clarkb | fungi: I think CIs should trigger their side if the intent is 500 change recheck | 22:23 |
| clarkb | fungi: zuul enqueue --change foo --pipeline bar | 22:23 |
| dansmith | clarkb: you think they scripted a recheck-vmware comment on every change to trigger minesweeper to re-run everything? | 22:24 |
| clarkb | dansmith: ye | 22:24 |
| dansmith | clarkb: ah, okay.. also: yikes | 22:24 |
| dansmith | that's a bad idea for more than just this reason | 22:24 |
| fungi | clarkb: oh, gotcha, yes did they comment on those changes with no verify to trigger their own ci runs? i didn't look at an example | 22:25 |
| clarkb | fungi: yup https://review.openstack.org/#/c/65474/ | 22:26 |
| fungi | ugh | 22:28 |
| anteaya | https://review.openstack.org/#/q/reviewer:%22VMware+Mine+Sweeper%22+status:open,p,002f10630001a6da | 22:28 |
| jeblair | so | 22:30 |
| jeblair | how about we disable minesweeper | 22:30 |
| jeblair | and restart zuul | 22:30 |
| jeblair | fungi, clarkb, mordred: ? | 22:31 |
| anteaya | +1 | 22:31 |
| mordred | jeblair: ++ | 22:31 |
| *** p5ntangl_ has joined #openstack-infra | 22:32 | |
| *** rhsu has joined #openstack-infra | 22:32 | |
| anteaya | salv-orlando: we have to disable you | 22:32 |
| fungi | jeblair: yes | 22:32 |
| anteaya | salv-orlando: please don't take it personally | 22:32 |
| jeblair | fungi: i will do it | 22:32 |
| clarkb | wfm | 22:32 |
| fungi | thanks. i'm approaching time for eveningish activities which will take me away from computers a while | 22:33 |
| *** emagana has joined #openstack-infra | 22:33 | |
| *** dkranz has joined #openstack-infra | 22:33 | |
| *** cody-somerville has joined #openstack-infra | 22:33 | |
| *** emagana has quit IRC | 22:33 | |
| salv-orlando | anteaya: if you are disabling me I’d take it personally ;) but I think you mean minesweeper? | 22:34 |
| *** emagana has joined #openstack-infra | 22:34 | |
| fungi | salv-orlando: yeah, having it leave recheck comments on 300+ changes to trigger itself is sort of impactful | 22:34 |
| *** weshay has joined #openstack-infra | 22:34 | |
| fungi | and noisy | 22:35 |
| *** weshay is now known as weshay_afk | 22:35 | |
| anteaya | salv-orlando: minesweeper yes, not you | 22:35 |
| *** p5ntangle has quit IRC | 22:35 | |
| salv-orlando | fungi: I’d do the same. I have stopped neutron rechecks. I have no credentials for stopping rechecks from the jenkins instance managing the nova vc driver | 22:36 |
| salv-orlando | I can only alert the relevant people and phone them to their desks | 22:36 |
| salv-orlando | more than that I cant I’m afraid | 22:36 |
| *** jaypipes has quit IRC | 22:36 | |
| fungi | salv-orlando: your efforts are appreciated! | 22:36 |
| jeblair | AttributeError: 'NoneType' object has no attribute 'db' | 22:37 |
| jeblair | nodepool is throwing errors after a restart ^ | 22:37 |
| fungi | ugh, did it lose the sql socket object? | 22:37 |
| salv-orlando | clarkb fungi anteaya: can you defer disablement of minesweeper by 10 minutes only? I’m talking with the rest of people managing CI. we should disable recheck soon. | 22:39 |
| fungi | i don't see any recent changes to the repo which should have caused that as a regression | 22:39 |
| *** AlexF has quit IRC | 22:40 | |
| jeblair | ssh review gerrit set-account --inactive 9008 | 22:40 |
| jeblair | i did that | 22:40 |
| jeblair | it should be reversible with '--active' | 22:40 |
| *** david-ly_ is now known as david-lyle | 22:41 | |
| fungi | yep, used --inactive on another account over the weekend | 22:41 |
| fungi | seems to work great | 22:41 |
| *** shashankhegde has quit IRC | 22:42 | |
| clarkb | jeblair: do we need to debug nodepool? | 22:42 |
| fungi | it's thinking self.config is none... | 22:42 |
| salv-orlando | fungi: I think there is no way to go back active? | 22:42 |
| fungi | salv-orlando: there is, it's a simple api call from our (admin) end | 22:43 |
| jeblair | clarkb, fungi: yeah, i'm not seeing the problem yet | 22:43 |
| salv-orlando | fungi: ha! I actually meant there is no way I can convince you I’ve fixed the issue or at least I’ve disabled the recheck altogether? | 22:43 |
| jeblair | it's supposed to set up the db before the managers, which it has logged that it has done | 22:44 |
| *** prad has quit IRC | 22:44 | |
| fungi | salv-orlando: probably after current latest fire is extinguished | 22:44 |
| jeblair | just did a thread dump | 22:45 |
| jeblair | oh | 22:45 |
| salv-orlando | fungi: cool we’ll talk about this tomorrow them | 22:45 |
| jeblair | it may be stuck waiting to connect to geard | 22:45 |
| jeblair | for zuul | 22:45 |
| jeblair | yes it seems much happier now | 22:46 |
| jeblair | so apparently the problem is that it does not completely start if it can not connect to gearman | 22:46 |
| fungi | ohhh | 22:46 |
| clarkb | jeblair: fun, but it recovered once gearman showed up? | 22:46 |
| jeblair | yep | 22:47 |
| fungi | without needing any additional help? | 22:47 |
| jeblair | fungi: correct | 22:47 |
| *** arosen has joined #openstack-infra | 22:47 | |
| fungi | can't complain then | 22:47 |
| jeblair | btw, the nodepool restart was to facilitate a mass-delete of all workers | 22:47 |
| fungi | right, i figured as much | 22:48 |
| fungi | salv-orlando: is everything nailed down on the minesweeper end? issues identified? | 22:48 |
| rhsu | hi guys, salv-orlando just pinged me about the recheck chaos | 22:48 |
| fungi | rhsu: was that an accident, or...? | 22:48 |
| salv-orlando | fungi, rhsu is my counterpart for the nova vc driver | 22:48 |
| rhsu | fungi: we had our build queue cleared and we did the rechecks to get all the reviews back onto the queue | 22:49 |
| anteaya | rhsu: ah maybe don't do that in future? | 22:49 |
| rhsu | fungi: we've done it twice before, but for some reason this time all the CIs got triggered somehow | 22:49 |
| anteaya | rhsu: did you manually trigger the rechecks? | 22:50 |
| anteaya | this was a conscious behaviour? | 22:50 |
| fungi | rhsu: right. so we really don't want system accounts leaving what basically amount to "noise" comments on reviews. a lot of reviewers subscribe to e-mail from project change comments, so that was effectively a 300+ e-mail blast out to each and every one of them | 22:50 |
| rhsu | anteaya: we had a script go through and post rechecks on the reviews we had lost | 22:50 |
| anteaya | rhsu: how did you lose those reviews? | 22:51 |
| jeblair | rhsu: if you are using zuul, there is a command to enqueue changes in to zuul without comments | 22:51 |
| jeblair | rhsu: and you should know that "recheck" followed by anything will now trigger openstack's ci system | 22:51 |
| morganfainberg | before i recheck anything is zuul stable/loading in checks? | 22:51 |
| jeblair | morganfainberg: yep, gtg. | 22:51 |
| * morganfainberg noticed you guys were working on it and doesn't want to add fuel to a fire | 22:51 | |
| morganfainberg | jeblair, cool | 22:51 |
| fungi | as to why it also caused us to rerun checks on all of them, that's because we've loosened our recheck comment syntax | 22:52 |
| rhsu | anteaya: we have an issue with our jenkins acting up.. | 22:52 |
| *** shashankhegde has joined #openstack-infra | 22:52 | |
| anteaya | rhsu: care to elaborate? | 22:52 |
| *** gokrokve_ has joined #openstack-infra | 22:52 | |
| rhsu | jeblair: we're not using zuul yet, but good to know | 22:52 |
| rhsu | anteaya: our jenkins is used for many other projects, not just the CI. we have some internal plugins that messed with the build queue. that resulted in us having to clear it. | 22:53 |
| *** emagana has quit IRC | 22:53 | |
| fungi | also worth noting, since our zuul deleted the jenkins user vote on those changes (which were then cancelled in the zuul restart) i think they're not not going to get refreshed on stale recheck comments or change approvals since they have no existing jenkins user vote | 22:54 |
| anteaya | rhsu: is it a separate jenkins from the jenkins running the neutron tests? | 22:54 |
| anteaya | or one jenkins for both nova and neutron plus other things? | 22:54 |
| anteaya | fungi: /me nods | 22:55 |
| rhsu | anteaya: yes, a different team manages the neutron one | 22:55 |
| fungi | so they'll probably all have to get rechecked manually at some point | 22:55 |
| *** gokrokve has quit IRC | 22:56 | |
| anteaya | rhsu: would it make sense to have one jenkins for both nova and neutron tests and a separate jenkins for internal testing? | 22:56 |
| *** julim has joined #openstack-infra | 22:56 | |
| anteaya | rhsu: then when you use zuul you probably will have a happier time | 22:56 |
| clarkb | mordred: fungi: jeblair: arosen and I seem to have a good handle on why we can leak floating ips | 22:57 |
| mordred | clarkb: ooh! excellent | 22:57 |
| anteaya | clarkb: yay, what channel? | 22:57 |
| mordred | arosen: thanks! | 22:57 |
| clarkb | anteaya: -neutron | 22:57 |
| * anteaya switches | 22:57 | |
| *** bdpayne has quit IRC | 22:57 | |
| clarkb | basically our reliance on https://git.openstack.org/cgit/openstack-infra/nodepool/tree/nodepool/provider_manager.py#n504 to get the ips is killing us | 22:57 |
| clarkb | novaclient delete ip does a disassocaite then a delete ip. This means the disassociate can succeed but the delete fail. We then short circuit and pop out of that code | 22:58 |
| clarkb | later we try again to delete the node, no ips are listed because we disassociated, and we delete the node (note we never deleted the ip) | 22:58 |
| *** bdpayne has joined #openstack-infra | 22:59 | |
| *** shashankhegde has quit IRC | 22:59 | |
| fungi | that would do it | 23:00 |
| rhsu | anteaya: yes, we could do that but it will take time (the CIs were started/maintained by different teams so all the scripts are different) | 23:00 |
| jeblair | clarkb: can we delete without disassociating? | 23:00 |
| *** julim has quit IRC | 23:00 | |
| clarkb | jeblair: yes but only with neutron client | 23:00 |
| clarkb | so we may have to add a new task manager for neutron client and use it instead | 23:00 |
| fungi | otherwise we end up needing to track the ips separately to retry deletion after disassociation | 23:01 |
| *** bdpayne_ has joined #openstack-infra | 23:01 | |
| *** datsun180b has quit IRC | 23:03 | |
| anteaya | rhsu: understood | 23:03 |
| anteaya | rhsu: also you are welcome to join the third party meeting anytime: https://wiki.openstack.org/wiki/Meetings/ThirdParty | 23:04 |
| *** bdpayne has quit IRC | 23:04 | |
| anteaya | rhsu: I think if you work towards steering in that direction you will be eventually happier with the results than if you keep the structure you have | 23:04 |
| anteaya | especially for being able to use zuul | 23:04 |
| anteaya | I'm off to dinner and dancing | 23:05 |
| salv-orlando | anteaya: enjoy both things | 23:05 |
| *** eharney has quit IRC | 23:06 | |
| *** gargola has quit IRC | 23:06 | |
| *** ZZelle has quit IRC | 23:08 | |
| rhsu | anteaya: thanks for your comments, we are definitely heading in that direction | 23:09 |
| rhsu | anteaya: see you in the 3rd party meeting | 23:09 |
| *** scottda__ has quit IRC | 23:11 | |
| fungi | ohcrappity | 23:15 |
| *** emagana has joined #openstack-infra | 23:15 | |
| fungi | clarkb: i think we booted precise nodes as trusty in b1 | 23:15 |
| clarkb | huh? thats cool | 23:16 |
| clarkb | fungi: this related to postgres? | 23:16 |
| fungi | or rather built today's devstack-trusty image in b1 using a precise base image | 23:16 |
| clarkb | oh was this jesusaurus change? did it chagne a base image it shouldn't have? | 23:16 |
| fungi | yeah | 23:16 |
| * fungi checks the config | 23:16 | |
| *** homeless has quit IRC | 23:17 | |
| clarkb | yup thats it | 23:17 |
| clarkb | fungi: I can push fix or you can then the other can approve it | 23:17 |
| jesusaurus | whoops | 23:17 |
| fungi | jesusaurus: you want to find/submit the fix? then clarkb and i can both review/approve it ;) | 23:18 |
| fungi | ahh, yep, thar she blows | 23:19 |
| jesusaurus | yep, just found it | 23:19 |
| jesusaurus | i'll push up a quick patch | 23:19 |
| fungi | just what and where i expected to see it | 23:19 |
| *** emagana has quit IRC | 23:20 | |
| fungi | jesusaurus: closes-bug: #1355998 | 23:20 |
| nibalizer | commit bomb incoming | 23:21 |
| nibalizer | everyone take cover | 23:21 |
| openstackgerrit | K Jonathan Harker proposed a change to openstack-infra/config: Accidentally turned a trusty image into precise https://review.openstack.org/113678 | 23:21 |
| openstackgerrit | K Jonathan Harker proposed a change to openstack-infra/config: Accidentally turned a trusty image into precise https://review.openstack.org/113678 | 23:21 |
| clarkb | jesusaurus: can you double check that was the only one? | 23:21 |
| clarkb | fungi: also weird that it would crop up like this on precise but not trusty | 23:22 |
| *** sdake_ has joined #openstack-infra | 23:22 | |
| clarkb | fungi: might be a config difference for postgres I guess | 23:22 |
| fungi | clarkb: perhaps | 23:22 |
| jogo | clarkb: random question - what type of testing process do we have around moving to a newer distro? | 23:22 |
| fungi | jogo: mostly manual, and some hopeful prayer | 23:22 |
| jogo | clarkb: this is related to the libvirt version_cap flag. | 23:22 |
| clarkb | jogo: we add image to nodepool, then can manually test on those images or do experimental with them | 23:22 |
| clarkb | jogo: the libvirt verson_cap thing has nothing to do with newer distro testing aiui | 23:23 |
| jesusaurus | clarkb: it appears to be the only s/14.04/12.04/ | 23:23 |
| jogo | clarkb: well it *may*. "Yes, that's the idea. Rather than having to scramble when the new | 23:23 |
| clarkb | jesusaurus: great | 23:23 |
| jogo | devstack-gate image shows up, we'd be able to work on any issues in the | 23:23 |
| jogo | context of a patch series to bump the version_cap." | 23:23 |
| clarkb | jogo: I don't understand that sentence without more context | 23:23 |
| fungi | jesusaurus: i double-checked it too, and lgtm | 23:23 |
| jogo | clarkb: as in the version_cap thing decouples moving to a newer version of libvirt and enabling new libvirt dependant features | 23:24 |
| clarkb | jogo: I still don't understand :/ | 23:24 |
| clarkb | jogo: the libvirt cap change is self testing in the gate | 23:24 |
| jogo | clarkb: so for the record I am trying to understand myself | 23:24 |
| mattoliverau | Morning | 23:25 |
| fungi | clarkb: nova has libvirt-version-dependent features apparently so testing with a newer version of libvirt means we suddenly start exercising new code paths | 23:25 |
| jogo | clarkb: so we hit the live snapshot issue when we upgraded to the new libvirt. | 23:25 |
| *** dangers is now known as dangers_away | 23:25 | |
| clarkb | oh that | 23:25 |
| jogo | fungi: correct, and the version_cap flag is an attempt at controlling when those new code paths are excercised independant of the libvirt version underneath | 23:25 |
| fungi | clarkb: the idea (from what i gathered in the ml thread) would be to put turning that on in the hands of the nova devs | 23:26 |
| jogo | clarkb fungi: so it makes a single transition into a two phase | 23:26 |
| clarkb | jogo: right the version cap would make it self testing | 23:26 |
| fungi | yep | 23:26 |
| mordred | yah. install the new version of libvirt | 23:26 |
| mordred | then propose a patch exercising new code paths | 23:26 |
| mordred | profit | 23:26 |
| jogo | clarkb fungi: right, so doesn't this go against our model for all other dependencies | 23:26 |
| mordred | except without profit | 23:26 |
| mordred | nope. this is actually exactly how _ALL_ of them should work | 23:27 |
| clarkb | jogo: sort of | 23:27 |
| clarkb | jogo: but we acn pin our python reqs | 23:27 |
| jogo | I thought we have exclicitly left upper bounds open to force us to detect and fix issues ASAP | 23:27 |
| clarkb | jogo: we cannot pin libvirt easily | 23:27 |
| clarkb | jogo: we do, but we also have the option of pinning | 23:27 |
| mordred | jogo: to force us to detect and fix issues that peopel would see | 23:27 |
| clarkb | without version capping we have no way of pinnin | 23:27 |
| mordred | if you have the dependency version unbound | 23:27 |
| mordred | but you have in-code feature flags | 23:27 |
| mordred | then a user upgrading their version of a library will not affect the features in their system | 23:28 |
| fungi | i guess part of the struggle here, which is alluded to in the ml thread, is that users who run nova with a newer libvirt end up using features we're not yet testing | 23:28 |
| mordred | it's a much more scalable thing | 23:28 |
| *** emagana has joined #openstack-infra | 23:28 | |
| mordred | fungi: only if they raise the version_cap flag though, right? | 23:28 |
| fungi | mordred: i believe so, yes. warranty voided and all that ;) | 23:28 |
| jogo | mordred: the version_cap is back in hypothetica lland | 23:28 |
| dansmith | the other option, of course, is for us to not add code to nova that we know we can't test :) | 23:29 |
| mordred | oh. this isn't a realy thing? | 23:29 |
| jogo | hypothetical_land | 23:29 |
| jogo | mordred: it was | 23:29 |
| jogo | for a week or two | 23:29 |
| mordred | jogo: gotcha | 23:29 |
| fungi | dansmith: novel concept! | 23:29 |
| mordred | dansmith: except that this problem is a real problem in the world | 23:29 |
| mordred | dansmith: not just our gate | 23:29 |
| mordred | people need to do rolling updates of things | 23:29 |
| dansmith | mordred: because we put code into our tree that we don't test | 23:29 |
| *** dims_ has quit IRC | 23:30 | |
| fungi | mordred: right, this is that nova landed features supporting newer libvirt functionality but which we had no platform on which to validate | 23:30 |
| jogo | so I see this version_cap trying to address two different yet related issues: 1) stop us from upgrading to a new version of libvirt and everything breaks in gate | 23:30 |
| *** dims_ has joined #openstack-infra | 23:30 | |
| jogo | 2) reduce risk for folks doing upgrades in production by making the libvirt upgrade a two phase | 23:30 |
| *** dimtruck is now known as zz_dimtruck | 23:30 | |
| mordred | right. I'm on board with waiting until we can test a new feature before landing it | 23:30 |
| dansmith | jogo: both of which go away if we test things we land :) | 23:31 |
| jogo | dansmith: yup | 23:31 |
| mordred | but I think we need jogo's (1) case sorted for us to be able to do that | 23:31 |
| mordred | no | 23:31 |
| mordred | 1 does not go away | 23:31 |
| mordred | we have to solve 1 to be able to land the changes to test this | 23:31 |
| dansmith | why not? | 23:31 |
| jogo | dansmith: and TBH the libvirt break the gate for two days isn't bad | 23:31 |
| mordred | because howelse do we test an upgrade to libvirt? | 23:31 |
| jogo | mordred: we have a good system developed to detect and triage gate bugs | 23:31 |
| dansmith | mordred: I think you're confusing two things | 23:31 |
| fungi | mordred: well, the specific case of gate breakage because we're testing new features of nova we didn't test before does go away, i think was the point | 23:31 |
| mordred | right | 23:32 |
| mordred | but libvirt _itself_ could behave differently | 23:32 |
| dansmith | mordred: one is that there are bugs in a new libvirt, and one is that there is a new libvirt that suddenly enables code in nova we've not been exercising | 23:32 |
| jogo | mordred: for gate upgrades, things besides libvirt cause us huge pains | 23:32 |
| dansmith | mordred: yeah, but just capping some of our code on libvirt versions doesn't really help that | 23:32 |
| clarkb | dansmith: mordred is right | 23:32 |
| mordred | dansmith: yes. the second part is poopy and I agree with your solution to that | 23:32 |
| clarkb | dansmith: it does, we can update libvirt, then push a change to increase the cap | 23:32 |
| clarkb | dansmith: that change becomes self gating with the newer libvirt | 23:32 |
| mordred | but the sane path to getting a new libvirt available so that we can test thigns that use its new features | 23:32 |
| clarkb | but without the cap you immediately jump into using new libvirt | 23:32 |
| jogo | clarkb: self gating isn't enough IMHO | 23:32 |
| jogo | if new failure is 1% | 23:33 |
| clarkb | jogo: I agree | 23:33 |
| mordred | ++ | 23:33 |
| dansmith | clarkb: I don't get what you're saying | 23:33 |
| jogo | clarkb: and this is a libvirt specific issue | 23:33 |
| clarkb | jogo: but nothing but running the tests 100000 times is goingto catch that | 23:33 |
| clarkb | jogo: so self gating is just another check | 23:33 |
| *** reed has joined #openstack-infra | 23:33 | |
| clarkb | it won't catch everything | 23:33 |
| fungi | clarkb: i believe with the cap in nova (which switches features on/off) you're still running new libvirt at that point, just not new nova features dependent on it | 23:33 |
| jogo | when really libvirt isn't the only thing we hit on upgrades | 23:33 |
| clarkb | fungi: correct | 23:33 |
| clarkb | fungi: which is what we need to fix the chicken and egg | 23:33 |
| fungi | clarkb: so "things broke because new libvirt has bugs" is still an issue there | 23:34 |
| dansmith | fungi: right | 23:34 |
| clarkb | fungi: yes to a degree, but not the nova broke because untested code paths | 23:34 |
| clarkb | which is the actual problem we have run into | 23:34 |
| jogo | clarkb: so I am bringing this up here because I don't think the version_cap really helps with gaet issues in a meaningful way | 23:34 |
| clarkb | jogo: it does for the reason above | 23:34 |
| dansmith | clarkb: the chicken and egg is solved by always testing on a libvirt that supports all the features we have in tree, no? | 23:34 |
| *** dims_ has quit IRC | 23:34 | |
| openstackgerrit | A change was merged to openstack-infra/config: Accidentally turned a trusty image into precise https://review.openstack.org/113678 | 23:34 |
| clarkb | dansmith: yes that solves it too but I don't ever expect that to happen | 23:34 |
| fungi | clarkb: yes. i was trying to separate the two types of breakage there, to narrow what we're talking about ;) | 23:34 |
| mordred | dansmith: we need a sane path to propose an update to the version of libvirt we test with | 23:35 |
| clarkb | dansmith: we can say noav should do that but it won't happen | 23:35 |
| dansmith | clarkb: we need to stop landing code we don't test | 23:35 |
| mordred | dansmith: yes | 23:35 |
| mordred | but that's only half the picture | 23:35 |
| jogo | mordred: /libvirt/any dependency/ | 23:35 |
| mordred | we have to enable testing new code | 23:35 |
| mordred | jogo: yes | 23:35 |
| mordred | jogo: but libvirt is special right now | 23:35 |
| jogo | which is why i don't like a libvirt specific solution | 23:35 |
| mordred | libvirt is a snowflake | 23:35 |
| dansmith | mordred: I'm confused about the other half I guess | 23:35 |
| jogo | mordred: not really, sqlalchemy is too then | 23:35 |
| clarkb | jogo: sqlalchemy isn't | 23:36 |
| mordred | jogo: sqlalchemy has pip pins | 23:36 |
| clarkb | jogo: sqlalchemy is already self gating | 23:36 |
| mordred | libvirt does not yet for us | 23:36 |
| jogo | mordred: oh when did that happen? | 23:36 |
| mordred | in _THIS_CASE_ | 23:36 |
| mordred | libvirt is different from every other python library we use | 23:36 |
| mordred | once we get to modern libvirt, this will change | 23:36 |
| mordred | but we have to get there | 23:36 |
| clarkb | jogo: its always been the case | 23:36 |
| jogo | clarkb: oh never mind | 23:36 |
| clarkb | jogo: since 0.7 broke us | 23:36 |
| clarkb | mordred: well we are on trusty libvirt now | 23:37 |
| dansmith | clarkb: I like that we've codified a grudge :) | 23:37 |
| mordred | clarkb: are we installing that from pip? | 23:37 |
| clarkb | mordred: ya | 23:37 |
| clarkb | mordred: but not the C lib | 23:37 |
| clarkb | mordred: just the python lib | 23:37 |
| clarkb | mordred: in theory libvirt-python could bundle teh C lib too | 23:37 |
| clarkb | mordred: but it doesn't | 23:37 |
| mordred | please no | 23:37 |
| mordred | please please now | 23:37 |
| mordred | now | 23:37 |
| mordred | no | 23:38 |
| clarkb | :P | 23:38 |
| mordred | python is terrible at compiling c | 23:38 |
| clarkb | mordred: I agree, just pointing out the real issue is the C side | 23:38 |
| clarkb | mordred: because that is what enables the features that aren't being tested | 23:38 |
| clarkb | mordred: but yes we made nova stop using site packages | 23:38 |
| mordred | ok. so we're back to the thing we've been saying | 23:38 |
| clarkb | mordred: its glorious | 23:38 |
| mordred | which is that we need a path to deal with getting to new versions of things | 23:39 |
| mordred | I thought it was more specific because of libvirt, I'm sorry | 23:39 |
| mordred | it's related to ALL c-based libraries | 23:39 |
| clarkb | mordred: it is actually | 23:39 |
| *** emagana has quit IRC | 23:39 | |
| mordred | libxml could release a new version | 23:39 |
| mordred | how do we deal with that? | 23:39 |
| clarkb | well except that the other C based libs haven't been a problem | 23:39 |
| mordred | yes. this is true :) | 23:39 |
| clarkb | libvirt is a perpetual problem | 23:39 |
| *** emagana has joined #openstack-infra | 23:39 | |
| jogo | so to wrap around, yes we need a path t odeal with etting new versions of things. do you think the version_cap for libvirt makes sense with that in mind? | 23:39 |
| clarkb | lxml is not | 23:39 |
| mordred | jogo: I think to clarkb's point, because libvirt has show itself to be a problem child, we may have to treat it as such | 23:40 |
| clarkb | mordred: I think that if we want to solve this "correctly" we use gentoo | 23:40 |
| dansmith | I wonder if we're talking about different things, | 23:40 |
| jogo | assuming that nova won't land any new features that don't have at least some testing | 23:40 |
| mordred | dansmith: I'm certain that we are | 23:40 |
| dansmith | because the version cap jogo and I are talking about is purely to allow us to land things we're not testing | 23:40 |
| jogo | dansmith: well there was some other pretext there but yeah | 23:40 |
| clarkb | dansmith: there are two things here. 1) test nova features that are nwe and require new libvirt 2) test that new libvirt works with existing nova features | 23:41 |
| clarkb | dansmith: you are correct that your proposal fixes 1 | 23:41 |
| clarkb | dansmith: if we can get buy in I think your propsal is the way to solve 1 | 23:41 |
| dansmith | clarkb: what is "my proposal" ? | 23:41 |
| jogo | clarkb: if this flag doesn't exist then 2 isn't a possible code path | 23:41 |
| clarkb | dansmith: no untested features | 23:41 |
| dansmith | clarkb: okay, then yes | 23:41 |
| mordred | ++ | 23:41 |
| *** rfolco has quit IRC | 23:41 | |
| mordred | but that leads us to 2 ... | 23:41 |
| dansmith | clarkb: the version cap doesn't address #2 tho right? | 23:41 |
| clarkb | dansmith: not completely at least not for the gate | 23:42 |
| clarkb | dansmith: but it does help fix it for our "customers" | 23:42 |
| mordred | because I dont think waiting for trusty+next-lts is the way to get us moving with more modern libvirt on a regular basis | 23:42 |
| *** emagana has quit IRC | 23:42 | |
| dansmith | clarkb: I don't think it does | 23:42 |
| clarkb | dansmith: it is a way of saying "look we ran libvirt version X.Y 5.6millions times the last 6 months and it is good" | 23:42 |
| clarkb | dansmith: we test libvirt and qemu more than anyone else I bet | 23:42 |
| jogo | clarkb: so I am confused about 2) | 23:43 |
| *** emagana has joined #openstack-infra | 23:43 | |
| mordred | so I wat to know a) how we as infra get newer libvirt onto systems so that nova devs who want to land new features that depend on newer libvirt have a way to test those | 23:43 |
| dansmith | clarkb: right, so what does disabling some piece of code that worked in one version of libvirt when we're running another one? | 23:43 |
| dansmith | because things that used to work can break in the future, | 23:43 |
| *** wenlock has quit IRC | 23:43 | |
| mordred | b) how we do that without screwing nova devs | 23:43 |
| clarkb | dansmith: in our experience most versions of libvirt just don't work | 23:43 |
| clarkb | dansmith: regardless of nova feature | 23:43 |
| dansmith | mordred: requiring testing is not screwing nova devs in any way, IMHO :) | 23:43 |
| clarkb | dansmith: like flat out broken | 23:43 |
| jogo | testing that new libvirt works with old features. the case we are looking at here is if libvirt > X do a, else do b | 23:43 |
| dansmith | clarkb: right, so I don't see how disabling random pieces of nova based on libvirt version helps with that | 23:44 |
| clarkb | dansmith: it doesn't but nova saying this is the version I work on does | 23:44 |
| dansmith | clarkb: you mean just like a ReleaseNotes thing? | 23:44 |
| jogo | dansmith: 'break the future save the present' | 23:44 |
| mordred | dansmith: I mean that how can infra add new libvirt to test machines without screwing nova devs | 23:44 |
| mordred | dansmith: :) | 23:44 |
| dansmith | jogo: so we have to go back to 1955 to stop biff?! | 23:44 |
| clarkb | dansmith: that is one way of doing it | 23:44 |
| dansmith | mordred: okay :) | 23:44 |
| clarkb | mordred: it can't | 23:45 |
| krotscheck | wha... | 23:45 |
| clarkb | mordred: I mean we went to trusty and that screwed people | 23:45 |
| clarkb | mordred: and that was the easy case | 23:45 |
| dansmith | clarkb: because such a communication would be helpful indeed, but I don't think random knobs to disable parts of nova helps in that area at all :) | 23:45 |
| mordred | clarkb: right. I'd love to keep treating that as a problem to try to solve | 23:45 |
| jogo | dansmith: ++ I think we should document which libvirt we test in release notes | 23:45 |
| mordred | clarkb: I don't have a solution right now, but that doesn't mean there is no solution | 23:45 |
| jogo | but not a random knob | 23:45 |
| *** sdake_ has quit IRC | 23:45 | |
| dansmith | mordred: I'd like to be testing a newer libvirt than is coming in the next LTS before the LTS is released | 23:45 |
| dansmith | mordred: that doesn't mean the LTS will work of course, | 23:46 |
| clarkb | mordred: sure, I think it is something we make better but not necessarily "solve" | 23:46 |
| dansmith | mordred: but it helps everyone I think | 23:46 |
| clarkb | mordred: we could write libvirt in python | 23:46 |
| clarkb | mordred: or bundle libvirt in libvirt-python | 23:46 |
| dansmith | jogo: sure that's a very helpful thing to consumers | 23:46 |
| clarkb | dansmith: right so we should use gentoo if that is what we are going to do or arch (and I am not trolling that is how we solve that problem) | 23:46 |
| clarkb | if you want to test a rolling release pipeline then us a rolling release | 23:47 |
| mordred | clarkb: please stop saying bundle libvirt in libvirt-python | 23:47 |
| clarkb | mordred: :P | 23:47 |
| *** emagana has quit IRC | 23:47 | |
| dansmith | clarkb: well, I dunno about that, but if you tell me that's the easiest way, I'm not likely to argue :) | 23:47 |
| mordred | clarkb: I also don't think we need to go to gentoo | 23:47 |
| mordred | or arch | 23:47 |
| clarkb | dansmith: its not the easiest way, but that is what you want to test | 23:47 |
| mordred | I think there are plenty of ways to skin this cat that do not involve arch or gentoo | 23:47 |
| clarkb | dansmith: you are stating you want to get ahead of the slower release based distros and check things in a pipelien as they arrive | 23:47 |
| dansmith | mordred: basically "oh god, hell no" ? :P | 23:48 |
| clarkb | mordred: there are, but they all involve essentially running a rolling release distro | 23:48 |
| jogo | so just to clarify before I say the wrong thing on the ML about this: libvirt version_cap doesn't really solve a lot. it may help a little | 23:48 |
| mordred | clarkb: I disagree | 23:48 |
| clarkb | mordred: then how do you test newer libvirt? | 23:48 |
| dansmith | jogo: I totally disagree that it helps this problem *at*all* | 23:48 |
| mordred | clarkb: I think that we can take important elements into our own hands without having to jump 100% to a rolling release distro | 23:48 |
| mordred | there are like, maybe 5 pieces of the puzzle tht we need more explicit control over | 23:48 |
| jogo | dansmith: I am with you, but I want to make sure I get infras thoughts | 23:49 |
| dansmith | mordred: I agree, I think it's not that hard | 23:49 |
| clarkb | mordred: libvirt has a lot of deps | 23:49 |
| clarkb | mordred: for the way we use it at least | 23:49 |
| clarkb | those dpes have dpes | 23:49 |
| clarkb | and you start getting in kernel land | 23:49 |
| mordred | clarkb: they have a very fixed amount of deps | 23:49 |
| openstackgerrit | David Pursehouse proposed a change to openstack-infra/jenkins-job-builder: SCM: rename `self` arguments to `parser` https://review.openstack.org/106942 | 23:49 |
| fungi | fwiw russellb already has some patches up for a start at sucking newer libvirt upstream packages into the distros on which we test (though starting with fedora first i think, as an experiment) | 23:49 |
| mordred | ++ | 23:50 |
| dansmith | fungi: clarkb: yes, but that will break at some point as upstream libvirt pulls a new dep that the base distro doesn't have | 23:50 |
| dansmith | I know it will happen, | 23:50 |
| dansmith | because the dep tree is very deep | 23:50 |
| nibalizer | hrm i having an issue with storyborad | 23:50 |
| clarkb | mordred: my concern isn't the small point reelase. my concern is I need to have a new kernel and new systemd and new iptables and new openvswitch | 23:50 |
| clarkb | mordred: and at that point I am not a rolling release distro | 23:50 |
| nibalizer | remote: (W) 32af657: no files changed, was rebased | 23:50 |
| nibalizer | remote: (W) e85550e: no files changed, was rebased | 23:50 |
| nibalizer | remote: (W) No changes between prior commit c6b132f and new commit de8daa | 23:50 |
| nibalizer | what does that even mean | 23:50 |
| clarkb | s/not/now/ | 23:51 |
| fungi | dansmith: yes, probably. they have to target specific releases and add further packages which we can pin, and there's a risk that if we do that for too many unrelated subsystems we want to test against they will at some point conflict | 23:51 |
| mordred | clarkb: no, you're a rolling release overlay distro | 23:51 |
| nibalizer | oh are we having the systemd argument in here | 23:51 |
| *** flwang_ has joined #openstack-infra | 23:51 | |
| clarkb | nibalizer: no | 23:51 |
| * nibalizer gets soapbox | 23:51 | |
| fungi | s/which we can pin/which we can't pin/ | 23:51 |
| mordred | clarkb: I don't need to rolling release the entire thing | 23:51 |
| dansmith | fungi: however, if we're pulling from upstream releases, I think that maybe we can pin the libvirt version when we hit that problem, and work it out before we unpin, just like we do for python packages | 23:51 |
| clarkb | mordred: but you are doing the hard bits | 23:51 |
| mordred | clarkb: I need to rolling release the parts that are important to openstack | 23:51 |
| clarkb | mordred: those same bits are the ones that take up all the time | 23:52 |
| jhesketh | Morning | 23:52 |
| openstackgerrit | A change was merged to openstack-infra/storyboard-webclient: Formalized Resource Criteria API https://review.openstack.org/110160 | 23:52 |
| nibalizer | they explictly say they wont test or even consider it a bug when you dont lockstep upgrade systemd and the kernel | 23:52 |
| clarkb | mordred: this is why I think using a rolling release distro is fine | 23:52 |
| nibalizer | which to me is grump | 23:52 |
| fungi | dansmith: yeah, though we'll need better mechanisms for spotting when our image builds cease producing new images | 23:52 |
| clarkb | mordred: and a better approach | 23:52 |
| clarkb | mordred: because someone else can deal with teh kernel and systemd and openvswitch | 23:52 |
| dansmith | fungi: yeah, but it's manageable I think | 23:52 |
| mordred | clarkb: except that the rolling release distros are all managed like horseshit | 23:52 |
| mordred | and they will break CONSTANTLY | 23:52 |
| clarkb | mordred: they really aren't | 23:52 |
| mordred | so we'll STILL do the work | 23:52 |
| * fungi sees foreshadowing in this comment | 23:52 | |
| clarkb | mordred: especailly with us rebuulding them every day | 23:52 |
| clarkb | mordred: if we tried to keep gerrit alive for 2 years on arch that would be creazy | 23:53 |
| mordred | clarkb: remember back when I used to run the drizzle build farm and we did care about things like c compilation across multiple platforms? | 23:53 |
| *** dims_ has joined #openstack-infra | 23:53 | |
| dansmith | clarkb: I think mordred just challenged you to a year of gentoo-on-your-laptop | 23:53 |
| clarkb | but building a new image for devstack every day on arch is fine | 23:53 |
| mordred | clarkb: I could not keep the gentoo slaves going for more than a week at any point in time | 23:53 |
| mordred | because deps would break | 23:53 |
| *** andreaf has joined #openstack-infra | 23:53 | |
| mordred | in very strange ways | 23:53 |
| clarkb | mordred: right they were not single use right? | 23:53 |
| mordred | it didn't matter | 23:53 |
| clarkb | mordred: our single use mechanism is the thing that makes it work | 23:53 |
| mordred | their behavior changed | 23:53 |
| mordred | no | 23:53 |
| mordred | the libraries | 23:53 |
| mordred | the header files | 23:53 |
| mordred | gcc | 23:53 |
| mordred | BEHAVIOR changed | 23:54 |
| mordred | I'm not saing the machiens couldn't operate | 23:54 |
| *** otherwiseguy has quit IRC | 23:54 | |
| mordred | I'm saying they have no consistent interface | 23:54 |
| openstackgerrit | Michael Krotscheck proposed a change to openstack-infra/storyboard-webclient: Updated story list interface. https://review.openstack.org/110165 | 23:54 |
| *** rfolco has joined #openstack-infra | 23:54 | |
| clarkb | mordred: if that is the worry you won't fixthat with your plan | 23:54 |
| mordred | clarkb: no, it's exactly the same amount of trouble on our side | 23:54 |
| clarkb | mordred: grabbing new kernel which needs new gcc to support new systemd to support new libvirt has the same problem | 23:54 |
| *** bdpayne_ has quit IRC | 23:54 | |
| mordred | except in my plan | 23:54 |
| mordred | the ONLY people making cowboy changes to library versions are US | 23:54 |
| openstackgerrit | Michael Krotscheck proposed a change to openstack-infra/storyboard-webclient: Updated project list interface. https://review.openstack.org/110169 | 23:54 |
| mordred | not some random ass hipster doing thigns in gentoo outside of our gate | 23:55 |
| clarkb | mordred: right that is the problem :) | 23:55 |
| clarkb | I have zero interest in doing that because it is a ton of busy work | 23:55 |
| mordred | we will have to do all of that work ourselves in either scenario | 23:55 |
| mordred | all of it | 23:55 |
| mordred | we will not be able to count on gentoo for any of it | 23:55 |
| mordred | we'll have to check every single thing they do | 23:55 |
| mordred | so we might as well cut them out, since they will be providing no value | 23:55 |
| dansmith | so, | 23:56 |
| *** flwang_ has quit IRC | 23:56 | |
| mordred | and only pain | 23:56 |
| dansmith | just to point out,m | 23:56 |
| dansmith | if we use fedora/rawhide as our rolling release, | 23:56 |
| *** andreaf_ has quit IRC | 23:56 | |
| dansmith | if we point out that fedora is broken with a libvirt, | 23:56 |
| clarkb | dansmith: I want to rip out fedora completely rsn | 23:56 |
| clarkb | dansmith: fedora has the gentoo problem | 23:56 |
| dansmith | I think we'll be able to get the attention of the fedora and libvirt owners | 23:56 |
| clarkb | dansmith: and noen of the benefits of a stable release apparently | 23:56 |
| dansmith | well, I think you'll be able to get the attention of paid folks, is my point :) | 23:56 |
| *** bdpayne has joined #openstack-infra | 23:56 | |
| *** andreaf has quit IRC | 23:56 | |
| dansmith | I agree that fedora is no more stable | 23:56 |
| mordred | I think the problem with both things is that it's churn from outside of ourselves | 23:57 |
| *** andreaf has joined #openstack-infra | 23:57 | |
| clarkb | and with centos7 we can test the things that people interested in fedora want to see | 23:57 |
| jogo | maybe we should change nova's default virt driver to xen ;) | 23:57 |
| mordred | jogo: hehe | 23:57 |
| mordred | jogo: ++ | 23:57 |
| jogo | I bet rax would love that | 23:57 |
| clarkb | jogo: have you read the nodepool scripts for xen? | 23:57 |
| clarkb | jogo: its even worse than this situation | 23:58 |
| dansmith | heh | 23:58 |
| jogo | clarkb: I have not, link? | 23:58 |
| clarkb | jogo: tehre are heredoc'd bash scripts nested recursively a few layers deep that boot things in various places with restarts and all kinds of crazy | 23:58 |
| openstackgerrit | Michael Krotscheck proposed a change to openstack-infra/storyboard-webclient: Point search requests to search endpoints https://review.openstack.org/102536 | 23:58 |
| mordred | clarkb: so, I think what I'm getting at is taht I'd like to give _our_ devs the ability to take the reins of their destiny with some of these things | 23:59 |
| openstackgerrit | Michael Krotscheck proposed a change to openstack-infra/storyboard-webclient: Sort stories and projects. https://review.openstack.org/112426 | 23:59 |
| mordred | it's not just that I want to go faster or slower - it's that I think that external distro release decisions about which version of libvirt should exist for us give us no value | 23:59 |
| openstackgerrit | K Jonathan Harker proposed a change to openstack-infra/config: Move some jenkinsuser bits to openstack_project https://review.openstack.org/106884 | 23:59 |
| clarkb | mordred: I agree | 23:59 |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!