| sdague | mordred: that I don't know, deflate is pretty late in the request processing | 00:01 |
|---|---|---|
| sdague | so it might actually be gzipping after the cache | 00:01 |
| openstackgerrit | Khai Do proposed a change to openstack-infra/config: upgrade jenkins build-timeout plugin due to a performance bug https://review.openstack.org/97651 | 00:01 |
| openstackgerrit | Clark Boylan proposed a change to openstack-infra/zuul: Cache zuul status.json for one second. https://review.openstack.org/97652 | 00:02 |
| sdague | In random web world where I've lived you'd often put varnish on top of all of that to cache the output | 00:02 |
| clarkb | jhesketh: mordred ^ that should do caching. I looked to see if paste.httpserver has it built in but didn't see anything so meh | 00:02 |
| *** trinaths has joined #openstack-infra | 00:02 | |
| mordred | sdague: that seems excessive for one file... | 00:03 |
| *** yamahata has joined #openstack-infra | 00:03 | |
| sdague | mordred: agreed | 00:04 |
| sdague | apache should be pretty fast here regardless, faster than doing this in python | 00:05 |
| mordred | clarkb: I agree with you | 00:05 |
| *** shivharis has joined #openstack-infra | 00:07 | |
| jhesketh | mordred: why did you get rid of the check for the deflate apache module? | 00:07 |
| mordred | jhesketh: sdague said it wasn't needed | 00:07 |
| jhesketh | right, I just figured it can't hurt | 00:08 |
| sdague | jhesketh: we don't need it on the other box where we use this | 00:08 |
| clarkb | jhesketh: mordred I am actually wondering if I should keep the Vary: Accept-Encoding in the zuul change | 00:08 |
| clarkb | or should we set that in the apache change. apache is the one doing the deflate so should deal with that instead | 00:09 |
| sdague | clarkb: I think apache will just do it with magic | 00:09 |
| harlowja | magci! | 00:09 |
| clarkb | oh good | 00:09 |
| sdague | it's pretty easily testable once it's in place | 00:09 |
| clarkb | because zuul doesn't need to set it as it only ever has one encoding with my change | 00:09 |
| clarkb | and zuul shouldn't assume it is proxied so apache should magic | 00:10 |
| sdague | yeh | 00:10 |
| mordred | yah | 00:10 |
| *** praneshp has joined #openstack-infra | 00:11 | |
| clarkb | ok stuff is approved | 00:14 |
| *** ArxCruz has joined #openstack-infra | 00:14 | |
| *** ArxCruz has quit IRC | 00:15 | |
| clarkb | mordred: oh crap there is even a test in zuul to make sure that the data is gzipped... | 00:15 |
| * clarkb rips this out | 00:16 | |
| *** sarob has quit IRC | 00:16 | |
| *** sarob_ has joined #openstack-infra | 00:16 | |
| *** homeless has quit IRC | 00:17 | |
| mordred | clarkb: haha | 00:18 |
| *** mriedem has joined #openstack-infra | 00:19 | |
| *** mmaglana has joined #openstack-infra | 00:21 | |
| *** radez` is now known as radez_g0n3 | 00:21 | |
| *** lcheng_ has quit IRC | 00:22 | |
| openstackgerrit | Clark Boylan proposed a change to openstack-infra/zuul: Stop gzipping zuul status json in zuul https://review.openstack.org/97649 | 00:22 |
| openstackgerrit | Clark Boylan proposed a change to openstack-infra/zuul: Cache zuul status.json for one second. https://review.openstack.org/97652 | 00:22 |
| clarkb | mordred: jhesketh ^ that should do it | 00:22 |
| *** matsuhashi has joined #openstack-infra | 00:23 | |
| clarkb | and sorry for not actually checking. of course there would be a test to test this problem :) | 00:23 |
| morganfainberg | clarkb, it's the test you expect to not exist that does (and break things) ... always... | 00:23 |
| clarkb | morganfainberg: indeed | 00:24 |
| harlowja | qq, are reviews that should be running through the gate actively working/processing? | 00:24 |
| clarkb | harlowja: yes | 00:24 |
| harlowja | hmmm | 00:24 |
| harlowja | k | 00:24 |
| clarkb | harlowja: slowly but yes | 00:24 |
| harlowja | gotcha | 00:24 |
| clarkb | harlowja: do you have an example of something that isn't right? | 00:24 |
| mordred | clarkb: you haz reviews from jhesketh | 00:25 |
| *** swes has quit IRC | 00:25 | |
| *** _nadya_ has quit IRC | 00:25 | |
| *** unicell has joined #openstack-infra | 00:26 | |
| *** olaph has quit IRC | 00:26 | |
| harlowja | hmmm, clarkb probably a mess up in my czuul tool, shows up on the zuul site, odd | 00:26 |
| *** arnaud has joined #openstack-infra | 00:26 | |
| harlowja | clarkb ya, its something broke in my tool, hmmm | 00:27 |
| harlowja | nm then :-P | 00:28 |
| harlowja | not showing ones that are full queued for some reason | 00:28 |
| *** zhiyan_ is now known as zhiyan | 00:28 | |
| sdague | oh mordred while you seem to be active, this is what you were talking about at summit right - https://review.openstack.org/#/c/97442/ | 00:32 |
| *** nati_uen_ has quit IRC | 00:32 | |
| *** nati_ueno has joined #openstack-infra | 00:33 | |
| *** sarob_ has quit IRC | 00:33 | |
| openstackgerrit | K Jonathan Harker proposed a change to openstack-infra/config: Move variables in manifests/site.pp to top of file https://review.openstack.org/97659 | 00:33 |
| mordred | sdague: yup | 00:33 |
| sdague | I think the neutron fail is a normal race, and not an actual issue | 00:34 |
| sdague | but I'll wait for things to calm down a little before rechecking that | 00:34 |
| *** bcrochet is now known as bcrochet|g0ne | 00:34 | |
| *** lcostantino has quit IRC | 00:35 | |
| *** lcheng_ has joined #openstack-infra | 00:35 | |
| sdague | hey, ianw so the fact that Red Hat CI is running check experimental automatically is putting some additional load back on all the things. Can we have it stop that? | 00:35 |
| sdague | that kicks off another 6 dsvm jobs for every devstack change | 00:36 |
| ianw | sdague: yes | 00:36 |
| sdague | ianw: cool | 00:37 |
| *** otter768 has joined #openstack-infra | 00:37 | |
| sdague | just thinking about ways to save load until we recover | 00:37 |
| ianw | sdague: no worries; just restarted it and it won't post any more experimental requests for now | 00:38 |
| *** nati_ueno has quit IRC | 00:38 | |
| sdague | awesome, thanks | 00:39 |
| *** nati_ueno has joined #openstack-infra | 00:39 | |
| *** ramashri has quit IRC | 00:40 | |
| sdague | ianw: this was mostly because of the f20 job? | 00:40 |
| sdague | we could probably move that to devstack non-voting | 00:40 |
| sdague | it looks close | 00:40 |
| ianw | sdague: yes, i was only interested in the stability of that | 00:40 |
| sdague | you want to propose moving that out of experimental to main non-voting? | 00:40 |
| sdague | I'd be ok with that | 00:41 |
| ianw | sdague: per infra meeting this morning, i'm just validating the f20 partner image on hpcloud to give it some redundancy | 00:41 |
| openstackgerrit | Clark Boylan proposed a change to openstack-infra/zuul: Cache zuul status.json for one second. https://review.openstack.org/97652 | 00:41 |
| clarkb | jhesketh: mordred ^ | 00:41 |
| ianw | sdague: so i'd say let's get that up, and then we can move to non-voting | 00:41 |
| sdague | cool | 00:41 |
| ianw | s/morning/time of day for you/ :) | 00:42 |
| sdague | ianw: yeh, what TZ are you in? I realize I have no grounding on that | 00:42 |
| clarkb | mordred: jhesketh I think I am going to relocate to home now | 00:43 |
| ianw | sdague: AEST (UTC+10) | 00:43 |
| clarkb | mordred: jhesketh but afterwards maybe we do the scary thing and restart zuul | 00:43 |
| clarkb | and apply those apache changes | 00:43 |
| jhesketh | sure | 00:43 |
| sdague | oh, cool. I thought UTC+1 for some reason. | 00:44 |
| *** lcheng_ has quit IRC | 00:45 | |
| trinaths | anteaya: What is the criteria for CI to be good for voting. | 00:45 |
| *** kevinbenton has joined #openstack-infra | 00:45 | |
| mordred | clarkb: ++ | 00:46 |
| *** nati_ueno has quit IRC | 00:47 | |
| sdague | when did zuul get a favicon? | 00:47 |
| sdague | or is that a side effect of the apache front end | 00:47 |
| * krotscheck looks askance at the current puppet module versions in config.... | 00:47 | |
| *** nati_ueno has joined #openstack-infra | 00:48 | |
| sdague | or a side effect of me not noticing things | 00:48 |
| clarkb | ok suspending now back in a bit ish | 00:48 |
| clarkb | actually wait. going to make a change ot install yappi on zuul nodes | 00:49 |
| *** mmaglana has quit IRC | 00:51 | |
| openstackgerrit | Clark Boylan proposed a change to openstack-infra/config: Install yappi on zuul nodes. https://review.openstack.org/97664 | 00:51 |
| clarkb | jhesketh: mordred ^ | 00:51 |
| *** mmaglana has joined #openstack-infra | 00:51 | |
| *** SumitNaiksatam has quit IRC | 00:53 | |
| *** yamahata has quit IRC | 00:53 | |
| *** yamahata has joined #openstack-infra | 00:54 | |
| *** mmaglana has quit IRC | 00:55 | |
| *** trinath_s has joined #openstack-infra | 00:57 | |
| fungi | clarkb: 97652 lgtm. any reason not to approve yet? | 00:59 |
| clarkb | fungi: probably not | 01:00 |
| * fungi is caught back up on all the exciting | 01:00 | |
| clarkb | it passes tests locally | 01:00 |
| clarkb | fungi: I still need to move myself to home | 01:00 |
| *** maurosr has quit IRC | 01:00 | |
| clarkb | but next steps are to merge things and restart zuul and apply new apache config | 01:00 |
| fungi | right. i'm here for the restart. have a good stroll | 01:01 |
| clarkb | cool | 01:01 |
| clarkb | ok really suspending now | 01:01 |
| clarkb | back in ~20 minutes | 01:01 |
| *** maurosr has joined #openstack-infra | 01:02 | |
| *** marcoemorais has quit IRC | 01:04 | |
| *** _nadya_ has joined #openstack-infra | 01:04 | |
| *** mmaglana has joined #openstack-infra | 01:08 | |
| *** arnaud has quit IRC | 01:08 | |
| openstackgerrit | Ian Wienand proposed a change to openstack-infra/config: Make install_puppet.sh more robust against failure https://review.openstack.org/97665 | 01:10 |
| *** trinaths has quit IRC | 01:11 | |
| *** camunoz has quit IRC | 01:16 | |
| openstackgerrit | A change was merged to openstack-infra/nodepool: Check the returned image status https://review.openstack.org/97564 | 01:17 |
| openstackgerrit | A change was merged to openstack-infra/nodepool: Prevent listserver tasks from piling up https://review.openstack.org/97570 | 01:18 |
| openstackgerrit | A change was merged to openstack-infra/nodepool: Log task manager queue length https://review.openstack.org/97574 | 01:18 |
| *** nati_uen_ has joined #openstack-infra | 01:18 | |
| openstackgerrit | A change was merged to openstack-infra/nodepool: Log task durations https://review.openstack.org/97575 | 01:18 |
| *** shivharis has quit IRC | 01:21 | |
| openstackgerrit | Ian Wienand proposed a change to openstack-infra/config: Make install_puppet.sh more robust against failure https://review.openstack.org/97665 | 01:21 |
| *** nati_ueno has quit IRC | 01:22 | |
| *** otherwiseguy has joined #openstack-infra | 01:24 | |
| mordred | fungi: in the 30G nodes - are we making use of all 8 cpus? | 01:25 |
| fungi | mordred: good question. tempest bases its parallelism on cpu count i think | 01:27 |
| mordred | also - | 01:27 |
| fungi | not sure whether straight testr runs do | 01:27 |
| clarkb | we are | 01:28 |
| lifeless | testr --parallel works off of cpu count by default | 01:28 |
| fungi | that answers that, then | 01:28 |
| clarkb | ttempest uses 4 | 01:29 |
| clarkb | services use the others | 01:29 |
| *** camunoz has joined #openstack-infra | 01:29 | |
| *** gokrokve has joined #openstack-infra | 01:29 | |
| clarkb | and for unittests we use all 8 | 01:29 |
| bodepd | my latest build blew up: http://logs.openstack.org/57/97357/3/check/gate-config-layout/26774e2/console.html | 01:29 |
| bodepd | I assume I just wait patiently ;) | 01:29 |
| *** jhesketh has quit IRC | 01:30 | |
| *** praneshp has quit IRC | 01:30 | |
| bodepd | this has weird puppet failures: http://logs.openstack.org/57/97357/3/check/gate-config-puppet-apply-precise/87c0a49/console.html | 01:31 |
| jesusaurus | bodepd: yeah, puppet-apply-precise is a pet project of mine. hopefully soon it will start passing as i work out the issues | 01:32 |
| jesusaurus | there's also puppet-apply-centos6 as an experimental job | 01:33 |
| ianw | bodepd: i'm seeing that on hpcloud too | 01:33 |
| ianw | i'm manually running some of the setup scripts, and getting various "error: [Errno 104] Connection reset by peer" | 01:33 |
| ianw | from curl get's of scripts, pip installs | 01:33 |
| jesusaurus | is pypi.python having a sad? | 01:33 |
| ianw | http://logs.openstack.org/57/97357/3/check/gate-config-layout/26774e2/console.html <- you're seeing it on pip | 01:34 |
| ianw | jesusaurus: i had issues wget'ing from openstack git, so not pypi specific | 01:34 |
| jesusaurus | ive noticed it a few times, always with docutils... | 01:34 |
| ianw | i wonder if it's more flaky networking on the hpcloud side | 01:34 |
| harlowja | clarkb alright fixed up gerrit-view, was wondering why i wasn't seeing any queued up jobs :-P | 01:36 |
| fungi | https://community.hpcloud.com/status/incident/2619 | 01:36 |
| fungi | bodepd: jesusaurus: ianw: ^ | 01:36 |
| *** bknudson has quit IRC | 01:37 | |
| jesusaurus | fungi: i don't think that's causing the error in http://logs.openstack.org/57/97357/3/check/gate-config-layout/26774e2/console.html | 01:37 |
| ianw | fungi: yeah, yum updates did seem *slow* | 01:37 |
| bodepd | jesusaurus: wrong like, I meant to post this one: http://logs.openstack.org/57/97357/1/check/check-projects-yaml-alphabetized/fa31884/console.html | 01:38 |
| *** nosnos has joined #openstack-infra | 01:38 | |
| *** jp_at_hp has quit IRC | 01:39 | |
| *** nati_uen_ has quit IRC | 01:39 | |
| jesusaurus | oh, that's strange | 01:39 |
| bodepd | I am totally confused about how the spell check treats - vs. _ vs. [a-z] | 01:39 |
| fungi | jesusaurus: looks like a struggling network device somewhere. pip got a stray tcp/rst during download | 01:39 |
| *** nati_ueno has joined #openstack-infra | 01:39 | |
| *** amotoki has quit IRC | 01:40 | |
| fungi | jesusaurus: it's sorting with LC_ALL=C (remember that sort order is locale-dependent) | 01:40 |
| fungi | er, bodepd: ^ | 01:41 |
| jesusaurus | fungi: could be, but that brief incident description makes it sound like the issue causes nova boot to hang, not random network outages | 01:41 |
| fungi | jesusaurus: ahh, yes agreed | 01:42 |
| *** jhesketh_ has joined #openstack-infra | 01:42 | |
| *** jhesketh_ is now known as jhesketh | 01:43 | |
| ianw | fungi, jesusaurs: http://paste.openstack.org/show/82705/ | 01:44 |
| fungi | ianw: nice random fail there | 01:45 |
| ianw | fungi: yeah, it will keep failing just enough that you think it's working, then it isn't :) (that's how i noticed https://review.openstack.org/97665) | 01:47 |
| *** amcrn has quit IRC | 01:48 | |
| *** _nadya__ has joined #openstack-infra | 01:49 | |
| ianw | that's on a vm in az2, which i guess is not us-east from https://community.hpcloud.com/status/incident/2619 | 01:49 |
| *** wenlock has joined #openstack-infra | 01:49 | |
| *** _nadya_ has quit IRC | 01:50 | |
| *** otherwiseguy has quit IRC | 01:50 | |
| fungi | well, us east has an az2 as well | 01:52 |
| *** otherwiseguy has joined #openstack-infra | 01:52 | |
| *** Ryan_Lane has quit IRC | 01:52 | |
| fungi | but it's an actual os-ext-az availability zone, not a region called an az | 01:52 |
| *** _nadya__ has quit IRC | 01:53 | |
| ianw | oh, i see, yeah it's in us east | 01:54 |
| ianw | silly me thought az might mean the state, and that seems like it would be "west" :) | 01:55 |
| fungi | bwahahaha | 01:55 |
| Kiall | LOL | 01:55 |
| fungi | that would be awesome though | 01:55 |
| Kiall | Also - https://community.hpcloud.com/status/incident/2619 should have zero effect on networking for running instances | 01:57 |
| openstackgerrit | A change was merged to openstack-infra/config: Use Apache to compress status.json https://review.openstack.org/97648 | 01:57 |
| openstackgerrit | A change was merged to openstack-infra/config: Cache status.json for 5 seconds in Apache https://review.openstack.org/97650 | 01:59 |
| *** gokrokve has quit IRC | 01:59 | |
| *** hs634 has left #openstack-infra | 02:00 | |
| *** mmaglana has quit IRC | 02:05 | |
| *** mmaglana has joined #openstack-infra | 02:06 | |
| *** mmaglana has quit IRC | 02:10 | |
| clarkb | fungi: mordred should we do the zuul stuff soonish? | 02:15 |
| clarkb | looks like the zuul and config changes are about to finish check testing | 02:16 |
| fungi | they'll end up flying through the gate once they do | 02:16 |
| clarkb | cool | 02:16 |
| clarkb | may as well let them go through that then | 02:16 |
| *** _nadya_ has joined #openstack-infra | 02:17 | |
| clarkb | my dinner is just about ready though | 02:17 |
| mordred | clarkb: yes | 02:18 |
| *** crc32 has quit IRC | 02:20 | |
| openstackgerrit | A change was merged to openstack-infra/zuul: Stop gzipping zuul status json in zuul https://review.openstack.org/97649 | 02:22 |
| openstackgerrit | A change was merged to openstack-infra/config: Install yappi on zuul nodes. https://review.openstack.org/97664 | 02:23 |
| *** zz_gondoi is now known as gondoi | 02:23 | |
| openstackgerrit | A change was merged to openstack-infra/zuul: Cache zuul status.json for one second. https://review.openstack.org/97652 | 02:23 |
| *** gyee has quit IRC | 02:27 | |
| *** mriedem has quit IRC | 02:27 | |
| *** mbacchi has quit IRC | 02:27 | |
| clarkb | fungi: mordred it looks like config changes have merged | 02:30 |
| clarkb | now we are waiting on zuul? | 02:30 |
| clarkb | fungi: https://review.openstack.org/#/c/97641/ though that didn't end up in the queues like the others so maybe we restart for that another day? | 02:31 |
| clarkb | the info is potentially valuable for debugging though | 02:31 |
| clarkb | apache change is in place though | 02:32 |
| mordred | woot | 02:33 |
| clarkb | most important is the zuul gzipping | 02:33 |
| clarkb | I am ok with doing that without the profiler change | 02:33 |
| clarkb | let me know what you guys think | 02:33 |
| mordred | clarkb: I think we can do it without the profiler change | 02:35 |
| *** camunoz has quit IRC | 02:36 | |
| *** gokrokve has joined #openstack-infra | 02:37 | |
| *** gondoi is now known as zz_gondoi | 02:39 | |
| fungi | clarkb: for 97641, we would kill -USR2 twice (once to start the profiler and again to stop)? | 02:39 |
| *** otter768 has quit IRC | 02:41 | |
| clarkb | fungi: yup | 02:42 |
| clarkb | fungi: mostly want a way to stop it so that an performance impact can be bypassed once we have data | 02:42 |
| *** harlowja is now known as harlowja_away | 02:42 | |
| clarkb | it is upposed to be light weight though | 02:43 |
| *** melwitt has quit IRC | 02:44 | |
| clarkb | I'm eating quickly now but can be ready to restart zuul whenever | 02:44 |
| clarkb | fungi: mordred just let me know when you are ready and we can coordinate this | 02:44 |
| *** dims has quit IRC | 02:45 | |
| clarkb | the code in /opt/zuul is up to date on zuul.o.o | 02:45 |
| fungi | well, the profiler change has been approved and ought to merge momentarily since it already had a +1 check | 02:45 |
| mordred | neat | 02:45 |
| clarkb | fungi: perfect I will eat more then | 02:46 |
| fungi | because gate priority | 02:46 |
| fungi | supposed eta 2 minutes | 02:46 |
| *** mmaglana has joined #openstack-infra | 02:46 | |
| *** asettle is now known as asettle-luncheon | 02:47 | |
| openstackgerrit | A change was merged to openstack-infra/zuul: Add toggleable yappi profiling to zuul https://review.openstack.org/97641 | 02:48 |
| fungi | mergola | 02:48 |
| clarkb | now to get that in /opt and installed | 02:48 |
| *** camunoz has joined #openstack-infra | 02:49 | |
| *** trinath_s has quit IRC | 02:50 | |
| clarkb | fungi: mordred can one of you kick off a puppet run on zuul.o.o? that should get us to the latest version of all e things | 02:51 |
| clarkb | I am refreshing on zuul killing process | 02:53 |
| clarkb | looks like grab zuul changes with script in root's home dir then init script stop start | 02:54 |
| clarkb | then run script to reenqueue change | 02:54 |
| clarkb | and looks like zuul yappi is on zuul host | 02:55 |
| clarkb | everyone ready? | 02:55 |
| ianw | jesusaurus / fungi: i raised an issue with hpcloud about the network issues; the support guy could replicate it so it's been escalated, apparently | 02:55 |
| clarkb | I can start pipeline capture as soon as I get a node from someone not me :) | 02:56 |
| clarkb | s/node/nod/ | 02:58 |
| clarkb | mordred: fungi ^ | 03:00 |
| fungi | nod | 03:02 |
| fungi | yes | 03:02 |
| clarkb | ok capturing pipelines now | 03:02 |
| fungi | you lost me at node ;) | 03:02 |
| clarkb | will ping here before I actually stop zuul | 03:02 |
| clarkb | ok check gate and check-tripleo captured | 03:04 |
| clarkb | ready to stop zuul? | 03:04 |
| * clarkb does it | 03:04 | |
| fungi | yeah | 03:04 |
| clarkb | got to requeue gate now | 03:05 |
| fungi | looks like it restarted | 03:05 |
| fungi | i guess we lost anything in post | 03:06 |
| clarkb | fungi: yeah I checked it and no docs so meh | 03:06 |
| clarkb | and nothing in release | 03:06 |
| fungi | probably not critical at this point... i manually triggered the remaining branch tarballs for apevec's call for 2014.1.1 testing | 03:06 |
| fungi | and those all appeared a while before the restart | 03:06 |
| clarkb | check is queueing now | 03:07 |
| fungi | and stable/icehouse is basically in freeze now, so shouldn't have been any stragglers | 03:07 |
| clarkb | and now tripleo | 03:08 |
| *** changbl has joined #openstack-infra | 03:08 | |
| clarkb | fungi: can you do some curling against the status.json endpoint and see if it is really caching and gzipping | 03:09 |
| *** gokrokve has quit IRC | 03:09 | |
| fungi | yep, on it | 03:10 |
| clarkb | danke | 03:10 |
| *** rwsu has quit IRC | 03:10 | |
| fungi | Length: 1493566 (1.4M) [application/json] | 03:11 |
| clarkb | fungi: from my testing it grabs the gzip if you do curl -H 'Accept-Encoding: gzip' | 03:12 |
| clarkb | but I get differnt zip files each time | 03:13 |
| clarkb | going to inflate and check contents now | 03:13 |
| openstackgerrit | Marc Abramowitz proposed a change to stackforge/python-jenkins: Fix "H404 multi line docstring should start with a summary" hacking warnings https://review.openstack.org/97676 | 03:14 |
| clarkb | mordred: ^ I am not sure your five second thing is working | 03:15 |
| openstackgerrit | Marc Abramowitz proposed a change to stackforge/python-jenkins: Fix hacking docstring warnings https://review.openstack.org/97676 | 03:15 |
| mordred | clarkb: no?. damm | 03:15 |
| fungi | unless it's caching the uncompressed content and then compressing on each request | 03:16 |
| fungi | oh, or you get different uncompressed content | 03:17 |
| clarkb | fungi: well I uncompressed too and get differnet file sizes there | 03:17 |
| fungi | fun | 03:17 |
| clarkb | nothign in the error log indicating apache is unhappy | 03:18 |
| clarkb | let me format this json and look at it in a readable way | 03:18 |
| clarkb | fungi: the diffs appear to be in elapsed and remaining time | 03:19 |
| clarkb | so definitely different | 03:19 |
| clarkb | but zuul should cache it for at least a second now. going to see if I can do two subsecond requests and get the same one that way | 03:19 |
| *** unicell has quit IRC | 03:19 | |
| mordred | and apache is gzipping it, yeah? | 03:21 |
| mordred | that should also be nice | 03:21 |
| clarkb | mordred: yes apache is definitely gzipping if you set Accept-Encoding | 03:21 |
| clarkb | so zuul should be much happier even without the 5 second cache | 03:21 |
| *** asettle-luncheon is now known as asettle | 03:22 | |
| clarkb | yup really quickly curling I get the same value back each time | 03:22 |
| clarkb | so the 1 second cache is working I think | 03:23 |
| fungi | just not at the apache layer | 03:23 |
| clarkb | ya | 03:23 |
| fungi | and that we can tweak without a zuul restart | 03:23 |
| clarkb | yup | 03:23 |
| * clarkb goes to debug apache now | 03:23 | |
| fungi | i'm not finding a smoking gun in the apache logs yet | 03:23 |
| *** W00dy has joined #openstack-infra | 03:25 | |
| *** W00dy is now known as Guest87350 | 03:25 | |
| clarkb | I think Ifound it | 03:25 |
| clarkb | MCacheMaxObjectSize 2048 | 03:26 |
| clarkb | is way to conservative (that value is bytes according to apache docs) | 03:26 |
| clarkb | we need to tweak those numbers | 03:26 |
| lifeless | clarkb: you could put a real cache (squid/varnish) in there | 03:26 |
| clarkb | lifeless: as mordred said earlier that is a bit heavyweight for one file | 03:27 |
| openstackgerrit | Lance Bragstad proposed a change to openstack/requirements: Add bash8 to global-requirements https://review.openstack.org/97625 | 03:27 |
| lifeless | clarkb: mmm I guess | 03:27 |
| clarkb | currently status.json is 1.7MB. I think jeblair said it was in the 3MB range before. What if we set the MCacheSize to ~12MB and max object size to ~6MB | 03:27 |
| mordred | clarkb: ++ | 03:28 |
| clarkb | or we can be even less conservative. That box isn't memory constrained | 03:28 |
| clarkb | also the streaming buffer may be important as we are proxying this file | 03:29 |
| *** nosnos has quit IRC | 03:29 | |
| mordred | clarkb: we could make zuul return protobuf, and then have an apache module that transforms that into json ... | 03:30 |
| * mordred ducks | 03:30 | |
| *** matsuhashi has quit IRC | 03:31 | |
| *** unicell has joined #openstack-infra | 03:31 | |
| clarkb | arg apache uses kilobytes in some places and bytes in others | 03:32 |
| clarkb | this isn't confusing | 03:32 |
| *** Guest87350 has quit IRC | 03:33 | |
| *** Longgeek has joined #openstack-infra | 03:33 | |
| mordred | clarkb: I blame perl | 03:34 |
| clarkb | mordred: fungi: you ok with me ninja applying apache config really quicly to see if it works? | 03:36 |
| fungi | yeah, should be safe | 03:36 |
| *** talluri has joined #openstack-infra | 03:36 | |
| fungi | and 2k is definitely not enough to cover this file ;) | 03:37 |
| clarkb | gah puppet overwrote my changes very quicky; | 03:38 |
| * clarkb figures out how to make it not do that | 03:38 | |
| clarkb | puppet disabled on zuul.o.o | 03:39 |
| clarkb | nope that still seems to not work | 03:41 |
| clarkb | I am doing curl ; sleep 2 ; curl ; sleep 2; curl | 03:41 |
| clarkb | at least 2 of the 3 curls should be identical | 03:41 |
| *** otherwiseguy has quit IRC | 03:42 | |
| mattoliverau | clarkb: what return value are you getting from curl? | 03:42 |
| clarkb | oh wait | 03:42 |
| clarkb | mod mem cache is per process | 03:42 |
| clarkb | derp | 03:42 |
| clarkb | so this may be working with my changes just harder to confirm | 03:42 |
| *** JoshNang has quit IRC | 03:43 | |
| * clarkb sets some LogLevel debug | 03:46 | |
| mattoliverau | clarkb: you could turn the cacheheader directiive on and look at the returned x-headers in curl. | 03:47 |
| mordred | clarkb: should we switch to cache_disk? | 03:47 |
| clarkb | mordred: oh good point | 03:47 |
| clarkb | mordred: maybe? | 03:47 |
| *** sballe has joined #openstack-infra | 03:47 | |
| mordred | I read in another place that it does weird things with user agent combos | 03:48 |
| mordred | maybe we _Should_ just stick a varnish there | 03:48 |
| clarkb | I am not against doing that :) | 03:48 |
| clarkb | mod_cache.c(635): cache: /status.json not cached. Reason: No Last-Modified, Etag, Expires, Cache-Control:max-age or Cache-Control:s-maxage headers | 03:50 |
| clarkb | well that solves it | 03:50 |
| clarkb | I will propose my apache changes as they will be needed anyways and reenable puppet | 03:50 |
| clarkb | then tomorrow we can fix the zuul wsgi app | 03:50 |
| clarkb | and scheduel another restart | 03:50 |
| mordred | clarkb: ++ | 03:52 |
| openstackgerrit | Clark Boylan proposed a change to openstack-infra/config: Better apache mem cache tuneable values https://review.openstack.org/97678 | 03:52 |
| *** otherwiseguy has joined #openstack-infra | 03:52 | |
| clarkb | that ^ is what I had running while testing | 03:52 |
| *** trinath_s has joined #openstack-infra | 03:53 | |
| *** trinaths has joined #openstack-infra | 03:55 | |
| clarkb | ok puppet ran and put us back to normal | 03:57 |
| clarkb | I am going to shift back to more normal evening things | 03:57 |
| *** trinath_s has quit IRC | 03:57 | |
| *** amotoki has joined #openstack-infra | 03:58 | |
| mordred | clarkb: like beer? | 03:59 |
| clarkb | and television | 04:00 |
| clarkb | also paste.httpserver doesn't support http/1.1 out of the box | 04:00 |
| clarkb | might be worth investing in a different server so that we can get things like Last-Modified for free | 04:00 |
| fungi | or we could just format the additional headers we want in the response | 04:02 |
| clarkb | yeah I am working on that real qucik right now | 04:03 |
| clarkb | probably the quickest path forward | 04:03 |
| fungi | oh, though if it reports http/1.0 then apache might ignore them. eh, worth testing | 04:03 |
| *** lcheng_ has joined #openstack-infra | 04:08 | |
| clarkb | < HTTP/1.1 200 OK from apache though that may just be apache being nice | 04:09 |
| clarkb | but webob response objects have a last_modified attribute we can set | 04:12 |
| clarkb | so thats simple | 04:12 |
| *** mburned is now known as mburned_out | 04:12 | |
| *** zhiyan is now known as zhiyan_ | 04:15 | |
| *** zhiyan_ is now known as zhiyan | 04:17 | |
| *** mmaglana has quit IRC | 04:18 | |
| mordred | clarkb: does pecan/wsme? | 04:19 |
| ryanpetrello | pecan.response is just a webob.Response instance | 04:20 |
| mordred | ah | 04:20 |
| ryanpetrello | (so yes) | 04:20 |
| mordred | but if we're already using webob, in this case, it's probably not going to get us much? | 04:21 |
| mordred | ryanpetrello: http://git.openstack.org/cgit/openstack-infra/zuul/tree/zuul/webapp.py | 04:21 |
| ryanpetrello | yea, if you’re already just using webob, pecan’s not going to do anything special for you w/ Last-Modified | 04:22 |
| openstackgerrit | Clark Boylan proposed a change to openstack-infra/zuul: Set Last-Modified header when serving status.json https://review.openstack.org/97682 | 04:23 |
| clarkb | mordred: fungi ryanpetrello ^ pretty simple | 04:24 |
| ryanpetrello | yep | 04:24 |
| ryanpetrello | that’d do it | 04:24 |
| clarkb | or should I do it in the object creation? | 04:24 |
| clarkb | meh | 04:24 |
| mordred | clarkb: lgtm | 04:25 |
| *** matsuhashi has joined #openstack-infra | 04:25 | |
| *** nosnos has joined #openstack-infra | 04:26 | |
| *** oomichi has quit IRC | 04:26 | |
| *** lcheng_ has quit IRC | 04:27 | |
| *** rcarrill` has joined #openstack-infra | 04:29 | |
| *** rcarrillocruz has quit IRC | 04:32 | |
| *** lcheng_ has joined #openstack-infra | 04:36 | |
| *** amotoki has quit IRC | 04:39 | |
| *** pcrews has quit IRC | 04:44 | |
| openstackgerrit | A change was merged to openstack-infra/config: Better apache mem cache tuneable values https://review.openstack.org/97678 | 04:46 |
| *** lcheng_ has quit IRC | 04:52 | |
| boris-42 | mordred hi there | 04:53 |
| boris-42 | mordred can we make a bit simpler life for gates | 04:53 |
| boris-42 | mordred and remove py33 job in rally | 04:53 |
| boris-42 | mordred https://review.openstack.org/#/c/97448/ | 04:53 |
| clarkb | boris-42: done | 04:55 |
| *** nati_ueno has quit IRC | 04:55 | |
| boris-42 | clarkb thank you | 04:55 |
| boris-42 | clarkb it's sad but we are not able to add support for py33 until all python clients won't have py33 voting job .. | 04:56 |
| boris-42 | clarkb =( | 04:56 |
| *** sarob has joined #openstack-infra | 04:56 | |
| *** Longgeek has quit IRC | 04:56 | |
| *** nati_uen_ has joined #openstack-infra | 04:57 | |
| *** nati_uen_ has quit IRC | 04:57 | |
| boris-42 | clarkb btw could you point me where can I found code that organizes chain in gates? | 04:57 |
| boris-42 | find* | 04:57 |
| clarkb | boris-42: the zuul dependent pipeline manager does that | 05:00 |
| openstackgerrit | A change was merged to openstack-infra/config: Remove rally-python33 job https://review.openstack.org/97448 | 05:01 |
| mordred | boris-42: http://git.openstack.org/cgit/openstack-infra/zuul/tree/zuul/scheduler.py#n1628 | 05:01 |
| boris-42 | mordred thanks | 05:01 |
| *** Longgeek has joined #openstack-infra | 05:01 | |
| *** yaguang has joined #openstack-infra | 05:02 | |
| *** otherwiseguy has quit IRC | 05:02 | |
| boris-42 | mordred so what I am thinking about is that limiting size of changes that are in gates queueu | 05:03 |
| boris-42 | mordred may improve performance of gates a lot | 05:03 |
| *** camunoz has quit IRC | 05:03 | |
| boris-42 | mordred I made some math to calculate overhead depending on possibility of fail + queue length + time that is required for testing patch | 05:04 |
| *** ildikov has quit IRC | 05:04 | |
| *** yaguang has quit IRC | 05:04 | |
| boris-42 | mordred need to make one more how depends duration of merging queue on size of queue | 05:04 |
| StevenK | boris-42: We already limit the number of jobs that the gate runs | 05:04 |
| StevenK | But not the length of the queue itself | 05:05 |
| *** lcheng_ has joined #openstack-infra | 05:05 | |
| boris-42 | StevenK you mean jobs in gate chain? | 05:05 |
| clarkb | boris-42: we already do that | 05:05 |
| boris-42 | StevenK or overall? | 05:05 |
| clarkb | if you look at the zuul status page now you will see it happening | 05:05 |
| clarkb | the windowing method could probably be improved though | 05:06 |
| clarkb | right now it is similar to tcp slow start | 05:06 |
| boris-42 | clarkb nice | 05:07 |
| clarkb | but the gate pipelines are windowed. the window grows as changes merge and shrinks as they fail to merge | 05:07 |
| clarkb | anything with a gray blob next to it in the gate pipeline is currently outside of the window | 05:07 |
| boris-42 | clarkb ah nice | 05:08 |
| boris-42 | clarkb so how did you chose the number of jobs? | 05:08 |
| boris-42 | clarkb experimental? | 05:08 |
| clarkb | boris-42: its actually number of changes instead of jobs. and yes we started at 10 iirc but found that was too small | 05:08 |
| clarkb | so we have 20 as the base window size and it grows by 2 ever merge iirc | 05:09 |
| boris-42 | clarkb actually if we collect info about success possibility for tasks and avg duration of all jobs we can make it dynamic | 05:10 |
| boris-42 | clarkb but not sure that it will help a lot | 05:10 |
| clarkb | yeah we can definitely make it more dynamic | 05:10 |
| clarkb | the trouble is over doing it | 05:10 |
| boris-42 | clarkb so should i continue work on formulas? | 05:10 |
| clarkb | the nice thing this dumb implementation has going for it is that it has a good worst case | 05:10 |
| clarkb | boris-42: its probably worth looking into | 05:11 |
| clarkb | boris-42: a pid loop may be appropriate | 05:11 |
| boris-42 | clarkb I mean sometimes it's better to limit by 10 (when we have a lot of races in gates, which sometimes happens) | 05:11 |
| clarkb | yeah and I think a pid loop can learn that | 05:12 |
| boris-42 | clarkb sometimes we can make 20 or even more if there is no super big races | 05:12 |
| *** yaguang has joined #openstack-infra | 05:12 | |
| boris-42 | clarkb I think it's nice as a first task=) | 05:13 |
| StevenK | boris-42: The 20 that clarkb mentioned isn't static -- it changes depending on if things merge or fail | 05:13 |
| clarkb | StevenK: well 20 is a static floor | 05:13 |
| clarkb | but it is dynamic above that | 05:13 |
| StevenK | Ah, right | 05:13 |
| *** praneshp has joined #openstack-infra | 05:13 | |
| clarkb | sdague found that if we let it be very dynamic we drive to 0 | 05:13 |
| StevenK | Hehe | 05:13 |
| boris-42 | clarkb heh=) | 05:13 |
| StevenK | Sounds like an excellent plan to have stuff merge | 05:13 |
| *** camunoz has joined #openstack-infra | 05:14 | |
| *** Kiall has quit IRC | 05:15 | |
| boris-42 | clarkb btw one more thing that I was thinking, for example I am doing change in heat | 05:15 |
| boris-42 | clarkb why we are testing that glance, nova, keystone, cinder and other projects still work | 05:15 |
| boris-42 | clarkb I think it's enough to run just heat benchmarks | 05:16 |
| clarkb | because it is an integrated cloud? | 05:16 |
| boris-42 | clarkb no I mean we can do this a bit accurate | 05:16 |
| clarkb | boris-42: so this comes up constantly and the issue is no one actually understands how all of the compoenents talk to each other | 05:16 |
| boris-42 | clarkb e.g. changing nova requires testing heat | 05:16 |
| clarkb | and whenever you remove something you wedge the gate | 05:16 |
| boris-42 | clarkb but not vise versa | 05:16 |
| clarkb | boris-42: but it does. what if heat does something to break their compatibility with nova? | 05:16 |
| boris-42 | clarkb for example?) | 05:17 |
| clarkb | if they typo a rest call | 05:17 |
| boris-42 | clarkb they will break heat | 05:17 |
| boris-42 | clarkb not nova | 05:17 |
| boris-42 | clarkb and it should be catches by heat tempest tests | 05:17 |
| clarkb | boris-42: yes but you won't know that its broken unless you test it against nova | 05:17 |
| clarkb | boris-42: I guess I don't follow then | 05:17 |
| boris-42 | clarkb yep a bit=) | 05:17 |
| boris-42 | clarkb lemme explain more carefully | 05:18 |
| clarkb | oh why do the nova tempest tests run? | 05:18 |
| clarkb | so that goes back to my former statement | 05:18 |
| boris-42 | clarkb yep | 05:18 |
| clarkb | there are tons of implicit dependencies and people always get them wrong | 05:18 |
| clarkb | and the gate breaks | 05:18 |
| boris-42 | clarkb so you already try to make such experiment? | 05:18 |
| clarkb | boris-42: no, but we have inadverdently done it and discovered it when things break | 05:19 |
| clarkb | the other hard problem is that there isn't one openstack | 05:19 |
| *** Kiall has joined #openstack-infra | 05:19 | |
| boris-42 | clarkb you mean nova + neutron or nova + nova network? | 05:19 |
| clarkb | if you start counting all the combos you start crying | 05:20 |
| boris-42 | clarkb yep=) | 05:20 |
| clarkb | boris-42: yes nova network or neutron (neutron with openvswitch, other drivers and so on). kvm vs lxc vs docker vs xen and so on. rabbit, 0mq, qpid. mysql, postgres, mongodb. and on and on | 05:20 |
| boris-42 | clarkb hm but did you think about next thing (works for everything except ceilometer) | 05:21 |
| openstackgerrit | A change was merged to openstack-infra/zuul: Set Last-Modified header when serving status.json https://review.openstack.org/97682 | 05:21 |
| clarkb | just from my example that is 2 * 4 * 3 * 3 openstacks | 05:21 |
| boris-42 | clarkb if project has in requirements python clients of other project | 05:21 |
| clarkb | and we didn't even get into architecture, just components | 05:21 |
| boris-42 | clarkb yep sure but any way, lemme try to explain non-human approach to collect dependencies | 05:22 |
| clarkb | boris-42: that is inaccurate because for example swiftclient doesnt depend on keystone | 05:22 |
| boris-42 | clarkb hm | 05:22 |
| clarkb | you can automate this. you just can't do it today | 05:22 |
| boris-42 | clarkb but if we automate this stuff | 05:22 |
| boris-42 | clarkb we can safe a lot of time | 05:23 |
| clarkb | yes but someone has to do the work to untangle the mess first | 05:23 |
| clarkb | btw 72 is the number from above | 05:23 |
| clarkb | and I am sure I can make that number much bigger if you actually take the time to count | 05:23 |
| boris-42 | clarkb yep it's huge | 05:23 |
| *** mmaglana has joined #openstack-infra | 05:23 | |
| boris-42 | clarkb but this number makes work on carefully choosing what to test even more important | 05:25 |
| *** yaguang has quit IRC | 05:25 | |
| *** yaguang has joined #openstack-infra | 05:25 | |
| clarkb | I agree | 05:25 |
| clarkb | but we can't jump ahead or everything will fall apar | 05:25 |
| boris-42 | clarkb I will try to get some resources from Mirnatis to work on this task | 05:25 |
| boris-42 | clarkb as well I think if we change order of running tempest scenarios we can increate concurrency | 05:26 |
| clarkb | for tempest specific stuff you will need to talk to the qa team | 05:27 |
| clarkb | I also volunteered to start documenting what we test today so that we can see that from a high level | 05:27 |
| clarkb | because really very few people know what is tested | 05:27 |
| boris-42 | clarkb +1 | 05:27 |
| *** praneshp_ has joined #openstack-infra | 05:27 | |
| clarkb | which is another part of untangling | 05:27 |
| clarkb | then we map that onto the inter project dependencies and figure it out | 05:28 |
| boris-42 | clarkb so I will try to make some document | 05:29 |
| *** praneshp has quit IRC | 05:29 | |
| *** praneshp_ is now known as praneshp | 05:29 | |
| boris-42 | 1) testing different projects simultaneously | 05:30 |
| boris-42 | 2) dependencies between projects | 05:30 |
| clarkb | boris-42: and if we look at it from another perspective one nice thing of overtesting is we get really good statistically sound data that we can look at | 05:30 |
| boris-42 | clarkb actually tempest testing is quite syntetic | 05:31 |
| clarkb | it shakes out a lot of bugs | 05:31 |
| boris-42 | clarkb e.g. if you try to run simultaneously bunch of scenarios you may be schoked=) | 05:31 |
| boris-42 | shocked* | 05:31 |
| clarkb | boris-42: as someone that runs thousnads of nodes every hour probably not :) | 05:31 |
| boris-42 | clarkb hehe=) | 05:31 |
| *** Ivo has quit IRC | 05:32 | |
| boris-42 | clarkb I was shocked when I saw this http://pavlovic.me/rally/glance_list.html | 05:32 |
| boris-42 | clarkb 1k images list will take 20 seconds=) | 05:32 |
| boris-42 | clarkb and it's exponential growth gg | 05:33 |
| boris-42 | clarkb in nova same situation =) | 05:33 |
| boris-42 | clarkb with vms=) | 05:33 |
| clarkb | is it exponential? that looks linear | 05:34 |
| clarkb | still not good | 05:34 |
| boris-42 | clarkb on the second graph | 05:34 |
| boris-42 | clarkb you can click on create_image | 05:34 |
| boris-42 | clarkb and turn it off | 05:34 |
| clarkb | 200 iamges is 5 seconds. 1000 images is ~23 seconds | 05:34 |
| *** oomichi has joined #openstack-infra | 05:34 | |
| boris-42 | clarkb it's with create | 05:34 |
| boris-42 | clarkb on the graph on bottom (stacked are) click on glance.create_image to hide it | 05:35 |
| boris-42 | clarkb so 200 = 2.5 sec 1000 ~20sec | 05:35 |
| *** ildikov has joined #openstack-infra | 05:35 | |
| boris-42 | 40 takes 0.5 | 05:36 |
| clarkb | right so 40 to 200 is linear | 05:36 |
| openstackgerrit | Isaku Yamahata proposed a change to openstack-infra/config: Add tacker project on StackForge https://review.openstack.org/97435 | 05:37 |
| boris-42 | but from 200 to 1000 it raises faster | 05:37 |
| clarkb | 40 * 5 = 200. .5 * 5 = 2.5 | 05:37 |
| clarkb | yes the factor seems to change | 05:37 |
| clarkb | anyways we should fix that :) | 05:38 |
| boris-42 | clarkb yep I need to get merged profiler | 05:38 |
| boris-42 | clarkb going to move it on stackforge | 05:38 |
| boris-42 | clarkb otherwise it's unclear what the hell is happaning=) | 05:38 |
| Alex_Gaynor | regardless of the scaling characferistic, 2.5 seconds to return 200 items is bonkers | 05:38 |
| clarkb | Alex_Gaynor: sure. just poitning out it doesn't look exponential :) | 05:38 |
| clarkb | still terrible | 05:39 |
| boris-42 | clarkb btw neutorn list ports | 05:39 |
| *** alexpilotti has joined #openstack-infra | 05:39 | |
| boris-42 | clarkb will win golden poo awarad | 05:39 |
| boris-42 | clarkb need to get data for it=) | 05:39 |
| clarkb | boris-42: this is a special profiler? | 05:39 |
| boris-42 | clarkb nope this is just plain rally | 05:40 |
| boris-42 | clarkb profiler will show what the hell is happening inside | 05:40 |
| boris-42 | clarkb http://pavlovic.me/rally/profiler/ | 05:40 |
| boris-42 | clarkb boxes are clickable | 05:40 |
| boris-42 | clarkb and there is a typo osapvi_volume.cinder.db not nova.db | 05:40 |
| clarkb | oh I see it renders the call tree | 05:41 |
| boris-42 | clarkb yep via all services of all projects | 05:41 |
| boris-42 | clarkb and collect out of box db calls with sql statements, wsgi calls, and rpc | 05:41 |
| boris-42 | clarkb but we can add more points | 05:41 |
| boris-42 | clarkb so this one iteration of rally with profiling enabled in cinder/nova | 05:41 |
| boris-42 | clarkb it creates volume from image and boot vm from this volume | 05:42 |
| boris-42 | clarkb so for example we can see ultimate number of db request | 05:43 |
| boris-42 | clarkb uncountable* for booting single vm | 05:43 |
| *** pcrews has joined #openstack-infra | 05:44 | |
| *** basha has joined #openstack-infra | 05:44 | |
| boris-42 | clarkb this will organize huge load on DB if we try for example to run 500 vm simultaneosuly | 05:45 |
| boris-42 | clarkb as well regarding to nova conductor there will be super huge load on rabbit as well | 05:45 |
| *** sarob has quit IRC | 05:46 | |
| *** sarob has joined #openstack-infra | 05:46 | |
| *** yaguang has quit IRC | 05:47 | |
| boris-42 | clarkb so when I get profiler in upstream I will start working on fixing this stuff=) | 05:47 |
| *** yaguang has joined #openstack-infra | 05:47 | |
| clarkb | cool. does the profiler use a standard(ish) output format like callgrind or pstat? | 05:48 |
| *** jamespage_ has joined #openstack-infra | 05:48 | |
| clarkb | might be interesting to feed data generated by other tools into ti too | 05:48 |
| boris-42 | clarkb actually we are using ceilometer to collect all data | 05:48 |
| clarkb | oh well that will put a huge laod on your cloud :) | 05:49 |
| boris-42 | clarkb yep I know=) | 05:49 |
| clarkb | I was playing with yappi today which is why I am curious | 05:49 |
| boris-42 | clarkb but swift guys said that probably we should print in logs | 05:49 |
| clarkb | it has learned to spit out callgrind and pstat files | 05:49 |
| boris-42 | clarkb all messages and then collected | 05:49 |
| boris-42 | collect | 05:49 |
| clarkb | which you can use other tools to view | 05:49 |
| *** _nadya_ has quit IRC | 05:49 | |
| boris-42 | clarkb we can transform json into that format | 05:50 |
| boris-42 | clarkb I don't see any issues here=) | 05:50 |
| boris-42 | clarkb I have limitation in rally that I am using only API of OpenStack | 05:50 |
| *** sarob has quit IRC | 05:50 | |
| boris-42 | clarkb so as we don't have in moment logging-as-a-service with python client | 05:50 |
| boris-42 | clarkb i have to use ceilometer for collecting all resources | 05:51 |
| clarkb | could you use a python profiler directly? (I am not sure how they play with eventlet) | 05:51 |
| *** jlibosva has joined #openstack-infra | 05:52 | |
| boris-42 | clarkb the main goal of my profiler was to collect all traces from different services | 05:52 |
| boris-42 | clarkb in different projects | 05:52 |
| boris-42 | clarkb not to get full traces of all calls of everything in code | 05:52 |
| boris-42 | in single process | 05:53 |
| boris-42 | cause if something is broken in keystone and you are seeing issues in nova | 05:53 |
| boris-42 | nova traces won't help you to detect what the hell is happaning | 05:53 |
| boris-42 | =) | 05:53 |
| *** jamespage_ has quit IRC | 05:54 | |
| clarkb | well they will help you figure out what you need to do to make nova list less bad | 05:54 |
| clarkb | its one thing to know nova list is bad | 05:54 |
| *** alexpilotti has quit IRC | 05:54 | |
| morganfainberg | clarkb, oh interesting yappi spitsout callgrind? /me is going to have to check this out | 05:54 |
| clarkb | morganfainberg: yup and pstat. and it works fine with threads. I am really happy with it | 05:54 |
| boris-42 | clarkb but with profiler that I am doing you'll get trace of everything interesting | 05:54 |
| morganfainberg | clarkb, thats awesome! | 05:54 |
| clarkb | morganfainberg: I shoved it into zuul as an optional thing today | 05:54 |
| morganfainberg | clarkb, ++ nice! | 05:55 |
| boris-42 | clarkb that nova list works bad I already now even without profiler | 05:55 |
| boris-42 | clarkb it is collected by rally only using api calls | 05:55 |
| kashyap | There's no such thing as NEEDINFO flag on Launchpad I guess? | 05:55 |
| clarkb | kashyap: incomplete | 05:55 |
| kashyap | clarkb, Okay I thought so. but a fix is posted, and it's languishing since 8 months in review | 05:56 |
| *** pcrews has quit IRC | 05:56 | |
| kashyap | soren, I can still mark it incomplete I guess? | 05:56 |
| kashyap | (Err, didn't mean to prompt him :-( sorry) | 05:56 |
| clarkb | kashyap: if there is a fix do you need more info? | 05:56 |
| kashyap | clarkb, The fix has two NACKs | 05:57 |
| kashyap | I should have been more clear | 05:57 |
| clarkb | just because the code has been -1'd doesnt mean the bug is incomplete (or needs info) | 05:57 |
| kashyap | I see your point. It was just meant as a gentle ping to raise the attention | 05:58 |
| *** srenatus has quit IRC | 05:58 | |
| clarkb | I would just leave a comment in that case | 05:58 |
| *** srenatus has joined #openstack-infra | 05:58 | |
| clarkb | either on the review or bug depending on where you need people to look | 05:59 |
| *** alexpilotti has joined #openstack-infra | 06:01 | |
| *** andreaf has joined #openstack-infra | 06:03 | |
| *** sarob has joined #openstack-infra | 06:08 | |
| *** nati_ueno has joined #openstack-infra | 06:10 | |
| *** sarob has quit IRC | 06:12 | |
| *** mmaglana has quit IRC | 06:14 | |
| *** andreykurilin has joined #openstack-infra | 06:16 | |
| *** _nadya_ has joined #openstack-infra | 06:16 | |
| *** sarob has joined #openstack-infra | 06:17 | |
| *** sarob_ has joined #openstack-infra | 06:19 | |
| *** Ivo has joined #openstack-infra | 06:19 | |
| *** mkoderer has joined #openstack-infra | 06:20 | |
| *** pcrews has joined #openstack-infra | 06:21 | |
| kashyap | Yep, done that already. | 06:21 |
| *** sarob has quit IRC | 06:21 | |
| *** sarob_ has quit IRC | 06:24 | |
| *** rfolco has quit IRC | 06:24 | |
| *** nati_ueno has quit IRC | 06:28 | |
| *** afazekas is now known as _afazekas | 06:29 | |
| *** rgerganov has joined #openstack-infra | 06:34 | |
| *** ccit has joined #openstack-infra | 06:39 | |
| *** ihrachyshka has joined #openstack-infra | 06:41 | |
| *** ominakov has joined #openstack-infra | 06:42 | |
| *** bauzas has joined #openstack-infra | 06:43 | |
| *** wenlock has quit IRC | 06:43 | |
| *** Guest22618 has joined #openstack-infra | 06:45 | |
| *** andreykurilin has quit IRC | 06:46 | |
| *** rgerganov has quit IRC | 06:54 | |
| *** fbo_away is now known as fbo | 06:56 | |
| *** alexpilotti has quit IRC | 06:59 | |
| *** ihrachyshka has quit IRC | 06:59 | |
| *** tkelsey has joined #openstack-infra | 07:01 | |
| *** _nadya_ has quit IRC | 07:04 | |
| *** freyes has joined #openstack-infra | 07:04 | |
| *** rgerganov has joined #openstack-infra | 07:06 | |
| *** ihrachyshka has joined #openstack-infra | 07:07 | |
| *** denis_makogon has joined #openstack-infra | 07:09 | |
| *** fbo is now known as fbo_away | 07:13 | |
| *** ihrachyshka has quit IRC | 07:14 | |
| *** Ivo has quit IRC | 07:15 | |
| *** skolekonov has joined #openstack-infra | 07:16 | |
| *** matsuhashi has quit IRC | 07:17 | |
| openstackgerrit | James Polley proposed a change to openstack-dev/hacking: Clarify where the "following template" is https://review.openstack.org/97698 | 07:17 |
| *** matsuhashi has joined #openstack-infra | 07:18 | |
| *** sarob has joined #openstack-infra | 07:19 | |
| soren | SpamapS: Done. | 07:22 |
| *** sarob has quit IRC | 07:23 | |
| SpamapS | soren: cool thanks! | 07:24 |
| *** ccit has quit IRC | 07:26 | |
| *** fifieldt has joined #openstack-infra | 07:29 | |
| *** _nadya_ has joined #openstack-infra | 07:30 | |
| *** hashar has joined #openstack-infra | 07:32 | |
| *** Alexei_987 has quit IRC | 07:33 | |
| *** zhiyan is now known as zhiyan_ | 07:34 | |
| openstackgerrit | Prashanth Raghu proposed a change to openstack/requirements: Adding hiredis as a dependancy https://review.openstack.org/97701 | 07:36 |
| *** srenatus has quit IRC | 07:37 | |
| *** srenatus has joined #openstack-infra | 07:38 | |
| *** oomichi has quit IRC | 07:38 | |
| *** lcheng_ has quit IRC | 07:45 | |
| *** pblaho has joined #openstack-infra | 07:45 | |
| *** matrohon has joined #openstack-infra | 07:46 | |
| *** renlt has joined #openstack-infra | 07:46 | |
| *** mrda is now known as mrda-away | 07:51 | |
| *** lcheng_ has joined #openstack-infra | 07:53 | |
| openstackgerrit | Antoine Musso proposed a change to openstack-infra/zuul: Document usage of yappi profiler https://review.openstack.org/97708 | 07:55 |
| *** jpich has joined #openstack-infra | 07:55 | |
| *** andreaf has quit IRC | 07:55 | |
| *** yaguang has quit IRC | 07:59 | |
| *** derekh_ has joined #openstack-infra | 07:59 | |
| *** yaguang has joined #openstack-infra | 07:59 | |
| *** doude has joined #openstack-infra | 08:00 | |
| *** afazekas has joined #openstack-infra | 08:03 | |
| *** lcheng_ has quit IRC | 08:05 | |
| *** praneshp has quit IRC | 08:06 | |
| *** habib has joined #openstack-infra | 08:10 | |
| *** jistr has joined #openstack-infra | 08:10 | |
| *** viktors has quit IRC | 08:14 | |
| *** sarob has joined #openstack-infra | 08:17 | |
| *** srenatus has quit IRC | 08:18 | |
| *** srenatus has joined #openstack-infra | 08:18 | |
| *** oomichi has joined #openstack-infra | 08:19 | |
| *** sarob has quit IRC | 08:21 | |
| *** dizquierdo has joined #openstack-infra | 08:22 | |
| *** ihrachyshka has joined #openstack-infra | 08:25 | |
| *** Longgeek has quit IRC | 08:27 | |
| *** pelix has joined #openstack-infra | 08:27 | |
| *** Longgeek has joined #openstack-infra | 08:28 | |
| *** Longgeek has quit IRC | 08:29 | |
| *** markmc has joined #openstack-infra | 08:30 | |
| *** Longgeek has joined #openstack-infra | 08:30 | |
| *** flaper87|afk is now known as flaper87 | 08:35 | |
| *** IvanBerezovskiy1 has joined #openstack-infra | 08:35 | |
| *** _nadya_ has quit IRC | 08:35 | |
| *** _nadya_ has joined #openstack-infra | 08:35 | |
| *** fbo_away is now known as fbo | 08:35 | |
| *** e0ne has joined #openstack-infra | 08:38 | |
| *** markmcclain has joined #openstack-infra | 08:40 | |
| *** jooools has joined #openstack-infra | 08:42 | |
| *** matsuhashi has quit IRC | 08:42 | |
| *** dizquierdo has quit IRC | 08:44 | |
| *** pblaho has quit IRC | 08:48 | |
| *** rgerganov has quit IRC | 08:48 | |
| *** mkoderer has quit IRC | 08:48 | |
| *** jlibosva has quit IRC | 08:48 | |
| *** ildikov has quit IRC | 08:48 | |
| *** sballe has quit IRC | 08:48 | |
| *** changbl has quit IRC | 08:48 | |
| *** openstackgerrit has quit IRC | 08:48 | |
| *** IvanBerezovskiy1 has quit IRC | 08:48 | |
| *** rickerc has quit IRC | 08:48 | |
| *** ityaptin has quit IRC | 08:48 | |
| *** pblaho has joined #openstack-infra | 08:48 | |
| *** rgerganov has joined #openstack-infra | 08:48 | |
| *** ityaptin has joined #openstack-infra | 08:48 | |
| *** mkoderer has joined #openstack-infra | 08:48 | |
| *** IvanBerezovskiy has joined #openstack-infra | 08:48 | |
| *** changbl has joined #openstack-infra | 08:49 | |
| *** ildikov has joined #openstack-infra | 08:49 | |
| *** IvanBerezovskiy has left #openstack-infra | 08:49 | |
| *** jlibosva has joined #openstack-infra | 08:49 | |
| *** rickerc has joined #openstack-infra | 08:49 | |
| *** sballe has joined #openstack-infra | 08:49 | |
| *** IvanBerezovskiy has joined #openstack-infra | 08:49 | |
| *** matsuhashi has joined #openstack-infra | 08:50 | |
| *** openstackgerrit has joined #openstack-infra | 08:50 | |
| *** e0ne has quit IRC | 08:53 | |
| *** liam_ has joined #openstack-infra | 08:54 | |
| isviridov | hello infra. Could anybody give me a hint where can I check status or release job? | 08:54 |
| *** liam_ has quit IRC | 08:54 | |
| *** jgallard has joined #openstack-infra | 08:54 | |
| isviridov | I've pushed a tag and tar.gz is available at http://tarballs.openstack.org/magnetodb/, but not at pypi https://pypi.python.org/pypi/magnetodb | 08:55 |
| *** e0ne has joined #openstack-infra | 08:57 | |
| *** jp_at_hp has joined #openstack-infra | 08:58 | |
| *** fifieldt has quit IRC | 09:00 | |
| *** timrc is now known as timrc-afk | 09:01 | |
| openstackgerrit | Monty Taylor proposed a change to openstack-infra/nodepool: Build images using diskimage-builder https://review.openstack.org/46482 | 09:01 |
| *** freyes has quit IRC | 09:02 | |
| *** andreaf has joined #openstack-infra | 09:05 | |
| *** freyes has joined #openstack-infra | 09:05 | |
| openstackgerrit | Le Tian Ren proposed a change to openstack-infra/config: Add acl to create branches on powervc-driver https://review.openstack.org/97726 | 09:06 |
| *** SpamapS has quit IRC | 09:08 | |
| *** pblaho has quit IRC | 09:09 | |
| *** rcarrillocruz has joined #openstack-infra | 09:11 | |
| *** rcarrill` has quit IRC | 09:11 | |
| *** SpamapS has joined #openstack-infra | 09:11 | |
| *** ociuhandu has quit IRC | 09:11 | |
| *** ihrachyshka has quit IRC | 09:12 | |
| openstackgerrit | Antoine Musso proposed a change to openstack-infra/config: Bring docs jobs for openstack-infra/gerritlib https://review.openstack.org/95006 | 09:16 |
| openstackgerrit | Antoine Musso proposed a change to openstack-infra/config: Bring docs jobs for openstack-infra/gerritlib https://review.openstack.org/95006 | 09:20 |
| *** ihrachyshka has joined #openstack-infra | 09:40 | |
| *** e0ne has quit IRC | 09:43 | |
| *** e0ne has joined #openstack-infra | 09:43 | |
| *** markmcclain has quit IRC | 09:44 | |
| *** freyes has quit IRC | 09:44 | |
| *** e0ne_ has joined #openstack-infra | 09:44 | |
| *** markmcclain has joined #openstack-infra | 09:45 | |
| *** e0ne has quit IRC | 09:48 | |
| openstackgerrit | A change was merged to openstack-infra/jenkins-job-builder: update URL references to project. https://review.openstack.org/94430 | 09:50 |
| *** Sam-I-Am has quit IRC | 09:52 | |
| *** Sam-I-Am has joined #openstack-infra | 09:52 | |
| *** StevenK has quit IRC | 09:53 | |
| *** monester has quit IRC | 09:53 | |
| *** StevenK has joined #openstack-infra | 09:53 | |
| *** dizquierdo has joined #openstack-infra | 09:54 | |
| *** monester has joined #openstack-infra | 09:58 | |
| *** renlt has quit IRC | 09:59 | |
| *** apevec has joined #openstack-infra | 10:01 | |
| *** jcoufal has joined #openstack-infra | 10:01 | |
| *** jamespage_ has joined #openstack-infra | 10:05 | |
| *** denis_makogon has quit IRC | 10:06 | |
| *** nosnos has quit IRC | 10:08 | |
| *** hashar has quit IRC | 10:10 | |
| *** markmcclain has quit IRC | 10:11 | |
| *** jgallard has quit IRC | 10:13 | |
| *** sarob has joined #openstack-infra | 10:17 | |
| *** salv-orlando has joined #openstack-infra | 10:17 | |
| *** salv-orlando has quit IRC | 10:18 | |
| *** salv-orlando has joined #openstack-infra | 10:18 | |
| sdague | boris-42: 1k glance images is also something that I'd be surprised if you find anywhere. Rax image list is 38 entries | 10:19 |
| *** e0ne_ has quit IRC | 10:22 | |
| *** sarob has quit IRC | 10:22 | |
| *** jamespage_ has quit IRC | 10:23 | |
| *** matsuhashi has quit IRC | 10:24 | |
| *** ociuhandu has joined #openstack-infra | 10:26 | |
| boris-42 | sdague in any case=) | 10:26 |
| boris-42 | sdague it doesn't work => it should be fixed | 10:26 |
| sdague | boris-42: there are a ton of things that don't work, that's a really low priority one | 10:27 |
| boris-42 | sdague what about snapshots? | 10:27 |
| boris-42 | sdague actually most of all functionality of openstack don't work properly | 10:27 |
| boris-42 | sdague and they all should be just fixed | 10:27 |
| boris-42 | sdague profiler + benchmarks will allow to find these issues fast | 10:28 |
| boris-42 | sdague and just fix them | 10:28 |
| boris-42 | sdague without too much discussion=) | 10:28 |
| sdague | boris-42: sure, but you randomly picking some benchmark out of a hat which is bad, that almost no one will actually hit, seems like a pretty terrible way to prioritize | 10:28 |
| boris-42 | sdague actually we hit the same perofrmance | 10:28 |
| boris-42 | sdague in nova vm list , and cinder volume list | 10:28 |
| boris-42 | sdague and in nova there could be much more then 1k vms | 10:29 |
| boris-42 | sdague this bug just came from real production cloud | 10:29 |
| boris-42 | sdague in nova | 10:29 |
| boris-42 | sdague so we are adding benchmarks for all list commands | 10:29 |
| sdague | so is it if 1k vms exist? or are returned | 10:29 |
| boris-42 | sdague returned | 10:29 |
| sdague | right, returning 1k vms is non sensible | 10:30 |
| boris-42 | sdague it depends | 10:30 |
| sdague | there was a whole conversation about that in the paging sessions | 10:30 |
| sdague | I get that we'd like these calls to be faster, however they are many orders of magnitude less important than actually getting a compute to reliably boot and connect to the network :) | 10:31 |
| sdague | like the top bugs here - http://status.openstack.org/elastic-recheck/ | 10:31 |
| boris-42 | sdague and for example nova conductor that is bottlneck | 10:31 |
| boris-42 | sdague in real deployments | 10:32 |
| sdague | boris-42: sure, I'd believe that | 10:32 |
| boris-42 | sdague btw I made some proposal | 10:32 |
| boris-42 | sdague don't want to publish it in mailing list | 10:32 |
| sdague | why not? | 10:32 |
| boris-42 | sdague but a better approach for real zero downtime upgraeds | 10:32 |
| boris-42 | sdague nova team will hate me=) | 10:32 |
| *** sulochan has joined #openstack-infra | 10:32 | |
| sdague | we do everything on the mailing lists | 10:32 |
| boris-42 | sdague it's not yet the proper time for that | 10:33 |
| boris-42 | sdague but if you are interested take a look at https://docs.google.com/a/mirantis.com/document/d/15SVB2VrO8ZczVjk9C1rQ2uXgNTuNQ9DddZbFU1UR8UM/edit | 10:33 |
| *** bcrochet|g0ne is now known as bcrochet | 10:33 | |
| sdague | honestly, right now I'd be more interested in help fixing the fact that we can't reliably boot a compute and connect it to the network | 10:33 |
| boris-42 | sdague no need in unified objects and conductor to resolve zero downtime upgrades and security issues=) | 10:34 |
| sdague | I think until that's a 100% guaruntee | 10:34 |
| boris-42 | sdague sure I understand | 10:34 |
| *** Longgeek has quit IRC | 10:34 | |
| *** Longgeek has joined #openstack-infra | 10:35 | |
| *** sballe has quit IRC | 10:35 | |
| *** matrohon has quit IRC | 10:36 | |
| *** sballe has joined #openstack-infra | 10:40 | |
| *** lttrl has quit IRC | 10:42 | |
| *** sballe_ has joined #openstack-infra | 10:42 | |
| *** sballe has quit IRC | 10:45 | |
| openstackgerrit | Valeriy Ponomaryov proposed a change to openstack-infra/config: Set env vars explicitly for manila jobs https://review.openstack.org/97747 | 10:46 |
| sdague | SergeyLukjanov: you up yet? | 10:47 |
| *** yamahata has quit IRC | 10:47 | |
| *** matsuhashi has joined #openstack-infra | 10:47 | |
| *** sballe_ has quit IRC | 10:48 | |
| *** torgomatic has quit IRC | 10:48 | |
| *** dims_ has joined #openstack-infra | 10:50 | |
| *** SpamapS has quit IRC | 10:50 | |
| *** _nadya_ has quit IRC | 10:51 | |
| *** SpamapS has joined #openstack-infra | 10:52 | |
| sdague | I'm staring at zuul, and the giant rush that's going through the gate now has issues. The clean check hole that SpamapS found where something can jump to the gate even if a dependant patch is -1ed | 10:52 |
| *** freyes has joined #openstack-infra | 10:53 | |
| openstackgerrit | A change was merged to openstack-infra/config: Fix zuul's app.js name https://review.openstack.org/91314 | 10:53 |
| *** torgomatic has joined #openstack-infra | 10:55 | |
| openstackgerrit | Mark McLoughlin proposed a change to stackforge/gertty: Handle multiple child revisions of same parent https://review.openstack.org/97749 | 10:57 |
| openstackgerrit | A change was merged to openstack-infra/config: Fix scenario name in rally-neutron job https://review.openstack.org/97469 | 10:57 |
| openstackgerrit | A change was merged to openstack-infra/config: Make murano-dsvm jobs voting https://review.openstack.org/97462 | 10:57 |
| openstackgerrit | Darragh Bailey proposed a change to openstack-infra/git-review: Provide nicer user message for missing remote ref https://review.openstack.org/97751 | 10:58 |
| pelix | would appreciate feedback on https://review.openstack.org/#/c/48783/ as well if anyone would like to review a custom yaml loader for JJB | 10:59 |
| openstackgerrit | A change was merged to openstack-infra/config: Fix jquery-visibility.min.js name https://review.openstack.org/96968 | 11:02 |
| *** ociuhandu has quit IRC | 11:04 | |
| openstackgerrit | Mark McLoughlin proposed a change to stackforge/gertty: Add ctrl-o to help dialog https://review.openstack.org/97753 | 11:05 |
| *** _nadya_ has joined #openstack-infra | 11:05 | |
| openstackgerrit | Mark McLoughlin proposed a change to stackforge/gertty: Set log mode to append https://review.openstack.org/97754 | 11:05 |
| *** ociuhandu has joined #openstack-infra | 11:07 | |
| *** amotoki has joined #openstack-infra | 11:11 | |
| *** zzelle has joined #openstack-infra | 11:12 | |
| SergeyLukjanov | sdague, I'm here, just wfh tody, so, not stable online | 11:12 |
| sdague | ok | 11:13 |
| sdague | yeh, so at the current burn rate, we probably won't get through the gate queue until some time next week | 11:13 |
| *** dims_ has quit IRC | 11:14 | |
| sdague | definitely not until a weekend shows up | 11:15 |
| sdague | especially given there are things like a huge heat stack of patches that are in there, but have some early parts -1ed | 11:16 |
| SergeyLukjanov | sdague, yeah... | 11:17 |
| sdague | my instinct is to dump the gate queue. Because the problem is that when check got to 500 a ton of stuff was +Aed, and the rush like that is something our gate can't really handle | 11:17 |
| SergeyLukjanov | sdague, you mean drop? | 11:17 |
| *** sarob has joined #openstack-infra | 11:17 | |
| sdague | yeh, basically drop everything out of it | 11:17 |
| SergeyLukjanov | sdague, I'm ok with it, I'd like to have 5 patches in into the stable/icehouse today ;( | 11:18 |
| sdague | are those the ones currently in the gate? | 11:20 |
| *** dims_ has joined #openstack-infra | 11:20 | |
| *** sarob has quit IRC | 11:21 | |
| sdague | I would be surprised if they landed before tomorrow | 11:23 |
| *** srenatus has quit IRC | 11:23 | |
| *** srenatus has joined #openstack-infra | 11:23 | |
| openstackgerrit | Radomir Dopieralski proposed a change to openstack/requirements: Require django-pyscss 1.0.1 for django-compressor 1.4 support https://review.openstack.org/97760 | 11:24 |
| *** matsuhashi has quit IRC | 11:25 | |
| *** flaper87 is now known as flaper87|afk | 11:25 | |
| *** matsuhashi has joined #openstack-infra | 11:26 | |
| *** mburned_out is now known as mburned | 11:27 | |
| *** matsuhashi has quit IRC | 11:29 | |
| *** matsuhashi has joined #openstack-infra | 11:29 | |
| *** eharney has joined #openstack-infra | 11:34 | |
| *** yamahata has joined #openstack-infra | 11:41 | |
| zzelle | hi everyone | 11:47 |
| zzelle | it seems opendaylight CI is broken | 11:47 |
| *** yamahata has quit IRC | 11:50 | |
| *** yamahata has joined #openstack-infra | 11:50 | |
| *** hashar has joined #openstack-infra | 11:52 | |
| *** alexpilotti has joined #openstack-infra | 11:53 | |
| *** alaski_ is now known as alaski | 11:53 | |
| *** e0ne has joined #openstack-infra | 11:56 | |
| *** adalbas has joined #openstack-infra | 11:57 | |
| *** weshay has joined #openstack-infra | 12:00 | |
| rgerganov | sdague, hi, do you have a minute to ask you something about https://review.openstack.org/#/c/95743 | 12:02 |
| *** e0ne_ has joined #openstack-infra | 12:03 | |
| *** e0ne has quit IRC | 12:05 | |
| *** achuprin has quit IRC | 12:06 | |
| sdague | SergeyLukjanov: https://review.openstack.org/#/c/97768/ - any idea who is even the approvers on that? | 12:07 |
| *** alexpilotti has quit IRC | 12:07 | |
| sdague | rgerganov: sure | 12:07 |
| rgerganov | sdague, do I need to make any puppet changes if I want to move the CI names in external file? | 12:08 |
| rgerganov | sdague, and how to test this | 12:08 |
| *** trinaths has quit IRC | 12:08 | |
| SergeyLukjanov | sdague, yup, they are in gate | 12:09 |
| sdague | rgerganov: I don't think so | 12:09 |
| *** jgallard has joined #openstack-infra | 12:09 | |
| rgerganov | sdague, so we deploy all files in the gerrit directory and I just need to create a new one for the CI names | 12:10 |
| sdague | rgerganov: so I'd just say make that a separate .js file | 12:10 |
| sdague | then have the header just include it by url | 12:10 |
| SergeyLukjanov | sdague, I'm looking on your patch | 12:10 |
| rgerganov | sdague, thanks | 12:10 |
| SergeyLukjanov | sdague, I think sahara missed | 12:11 |
| sdague | SergeyLukjanov: yeh, my concern remains that no one is awake yet, and we're basically merging about 2 changes an hour | 12:11 |
| SergeyLukjanov | sdague, yeah | 12:12 |
| *** rfolco has joined #openstack-infra | 12:12 | |
| sdague | though determining run rate is kind of hard | 12:14 |
| *** esker has joined #openstack-infra | 12:16 | |
| *** lcostantino has joined #openstack-infra | 12:16 | |
| SergeyLukjanov | sdague, I've added comments to https://review.openstack.org/#/c/97768/ | 12:16 |
| *** sarob has joined #openstack-infra | 12:17 | |
| *** basha has quit IRC | 12:17 | |
| *** akrivoka has joined #openstack-infra | 12:18 | |
| *** jistr is now known as jistr|eng | 12:18 | |
| *** ArxCruz has joined #openstack-infra | 12:19 | |
| *** esker has quit IRC | 12:21 | |
| *** sarob has quit IRC | 12:21 | |
| *** alexpilotti has joined #openstack-infra | 12:22 | |
| *** lcostantino has quit IRC | 12:23 | |
| *** radez_g0n3 is now known as radez | 12:23 | |
| *** lcostantino has joined #openstack-infra | 12:23 | |
| *** achuprin has joined #openstack-infra | 12:24 | |
| openstackgerrit | Radoslav Gerganov proposed a change to openstack-infra/config: Add button that shows/hides CI comments in Gerrit https://review.openstack.org/95743 | 12:24 |
| *** jamespage_ has joined #openstack-infra | 12:28 | |
| *** ominakov_ has joined #openstack-infra | 12:28 | |
| *** ociuhandu has quit IRC | 12:28 | |
| *** dims_ has quit IRC | 12:29 | |
| *** aysyd has joined #openstack-infra | 12:29 | |
| *** jamespage_ has quit IRC | 12:29 | |
| *** dims_ has joined #openstack-infra | 12:29 | |
| sdague | SergeyLukjanov: ok, cool | 12:30 |
| *** mbacchi has joined #openstack-infra | 12:30 | |
| sdague | I think I addressed those, though clearly I don't understand submodules, as this doesn't seem to be doing quite the right thing | 12:30 |
| *** ominakov has quit IRC | 12:31 | |
| *** talluri has quit IRC | 12:31 | |
| *** talluri has joined #openstack-infra | 12:31 | |
| sdague | maybe when mordred gets up he can explain what I did terribly wrong :) | 12:32 |
| *** Ivo has joined #openstack-infra | 12:32 | |
| mordred_phone | submodules are evil | 12:33 |
| *** ominakov_ has quit IRC | 12:34 | |
| *** ociuhandu has joined #openstack-infra | 12:35 | |
| sdague | I entirely agree | 12:36 |
| sdague | however openstack/openstack is really out of date in whats included | 12:36 |
| *** talluri has quit IRC | 12:36 | |
| sdague | here is my attempt to fix that - https://review.openstack.org/#/c/97768/ - but clearly I'm missing something | 12:37 |
| sdague | mordred_phone or SergeyLukjanov - can someone promote this - https://review.openstack.org/#/c/96824/ | 12:38 |
| sdague | it will fix a bunch of failures where we are breaking because of n-cond not having clean logs | 12:38 |
| SergeyLukjanov | sdague, /melooking | 12:39 |
| *** lttrl has joined #openstack-infra | 12:39 | |
| SergeyLukjanov | sdague, oh, it's approved in nova, yup, I can promote it | 12:39 |
| SergeyLukjanov | sdague, please, ack that it should be promoted | 12:41 |
| *** ominakov has joined #openstack-infra | 12:41 | |
| sdague | yes, it should be | 12:41 |
| *** viktors has joined #openstack-infra | 12:43 | |
| SergeyLukjanov | sdague, anything else to promote with this change? | 12:43 |
| * SergeyLukjanov will run "zuul promote --pipeline gate --changes 96824,2" | 12:43 | |
| mordred | SergeyLukjanov: ++ | 12:43 |
| *** viktors has quit IRC | 12:44 | |
| mordred | sdague: ok - what thing that you did did you want me to look at? | 12:44 |
| mordred | sdague: got it | 12:44 |
| mordred | I need to work on figuring out tags and o/o too | 12:45 |
| SergeyLukjanov | mordred, publishing to pypi doesn't work right no, isn't it? | 12:46 |
| *** dkranz has joined #openstack-infra | 12:46 | |
| mordred | SergeyLukjanov: aroo? it should ... | 12:46 |
| SergeyLukjanov | mordred, magnetodb folks pushed the tag and there is no new package at pypi | 12:47 |
| mordred | sdague: why not read the integrated projects list from the governance repo? | 12:47 |
| mordred | SergeyLukjanov: how long ago? | 12:47 |
| SergeyLukjanov | mordred, could you please take a look too? I've not find any issues | 12:47 |
| SergeyLukjanov | mordred, today | 12:47 |
| mordred | I mean - the gate is very backed up | 12:47 |
| sdague | mordred: will that include pbr, tempest, devstack? | 12:47 |
| SergeyLukjanov | mordred, https://github.com/stackforge/magnetodb/releases/tag/2.0.5 | 12:47 |
| mordred | hrm. maybe not | 12:47 |
| mordred | SergeyLukjanov: yah - but the post job may not have run yet | 12:47 |
| mordred | it responds to the tag landing | 12:48 |
| sdague | I know that jogo and I use openstack/openstack to figure out merge velocity on the integrated gate | 12:48 |
| sdague | which is what I was trying to heuristic out | 12:48 |
| SergeyLukjanov | mordred, yeah... it could be, thx | 12:48 |
| SergeyLukjanov | sdague, promoted | 12:48 |
| sdague | SergeyLukjanov: thanks | 12:48 |
| sdague | I also just kicked all the ironic changes out of the gate pipeline | 12:49 |
| SergeyLukjanov | sdague, I've used os/os several times for reports :) | 12:49 |
| sdague | they seem to all be failing unit tests | 12:49 |
| *** dprince has joined #openstack-infra | 12:49 | |
| sdague | which means I think in the giant flush something merged there which broke everything else | 12:49 |
| mordred | SergeyLukjanov: looking | 12:49 |
| mordred | SergeyLukjanov: I see the tarball here: http://tarballs.openstack.org/magnetodb/ | 12:51 |
| SergeyLukjanov | mordred, yup, and wheel too | 12:51 |
| SergeyLukjanov | mordred, but there are no runs in https://jenkins.openstack.org/job/magnetodb-pypi-both-upload/ | 12:52 |
| mordred | SergeyLukjanov: but I don't see the upload one having run in jenkins.o.o | 12:52 |
| mordred | yeah | 12:52 |
| mordred | hrm | 12:52 |
| *** yaguang has quit IRC | 12:52 | |
| sdague | there is also a 17 deep patch queue in heat that has a jenkins -1 on patch #4 | 12:53 |
| sdague | but they all are in the gate | 12:53 |
| *** mfer has joined #openstack-infra | 12:53 | |
| SergeyLukjanov | sdague, :( | 12:54 |
| sdague | mordred: so my knee jerk reaction to the gate status right now is to dump everything out of it | 12:54 |
| sdague | with the following rationale | 12:54 |
| sdague | yesterday the check queue got to 500 and people just kept approving things | 12:55 |
| *** ominakov has quit IRC | 12:55 | |
| sdague | so once we could finally process a ton of stuff all came through at once | 12:55 |
| sdague | however 2 things happen in such a stampede | 12:55 |
| sdague | the first, is the ironic situation | 12:55 |
| fungi | mordred: pypi.slave.openstack.org seems to have not been recreated after jeblair accidentally deleted all the long-running slaves on monday | 12:55 |
| sdague | where code merged which breaks the unit tests for others | 12:56 |
| sdague | but our assumptions is that it will all be fine | 12:56 |
| sdague | so we keep grinding | 12:56 |
| sdague | the second is the heat case | 12:56 |
| *** ominakov has joined #openstack-infra | 12:56 | |
| sdague | where because of a bug in clean check it is possible to land items in the gate with dependencies that have -1 from jenkins | 12:57 |
| mordred | fungi: aha. that would explain it | 12:57 |
| sdague | which heat mostly finds a way to trip because they have very long patch series | 12:57 |
| fungi | i can start spinning up a replacement while i find some coffee | 12:57 |
| sdague | and it has to do with timing of when things enter | 12:57 |
| mordred | hrm | 12:57 |
| mordred | I don't understand the ironic bug | 12:58 |
| sdague | there are 20 ironic changes in check | 12:58 |
| sdague | they all get approved simultaneously | 12:58 |
| fungi | sdague: at the moment, because of jgit problems, gerrit tends to fail at merging everything in a series of dependent changes past about the first four | 12:58 |
| sdague | change #2 in gate changes an interface | 12:58 |
| sdague | that's not enough to create a merge conflict | 12:59 |
| sdague | #3 - #20 now all fail unit tests | 12:59 |
| *** markmcclain has joined #openstack-infra | 12:59 | |
| mordred | but should not land | 12:59 |
| mordred | oh | 12:59 |
| mordred | Oh. ok. I follow | 12:59 |
| mordred | I thought you were saying code had landed that broke unittest in trunk | 12:59 |
| mordred | which is weird | 12:59 |
| mordred | but I get it now | 13:00 |
| *** markmcclain1 has joined #openstack-infra | 13:00 | |
| mordred | sdague: I'm fine with it - fungi? | 13:00 |
| openstackgerrit | Boris Pavlovic proposed a change to openstack-infra/config: Finally add osprofiler to stackforge https://review.openstack.org/97778 | 13:00 |
| fungi | i'd need to ponder the messaging a bit | 13:00 |
| *** gokrokve has joined #openstack-infra | 13:00 | |
| sdague | we actually also see it with pep8 | 13:00 |
| sdague | which is the clearly explanation, the automatic merge resolution creates a pep8 violation | 13:01 |
| fungi | yeah, a ton of nova changes got approved and now violate pep8 tests | 13:01 |
| fungi | and keep causing gate resets because of it | 13:01 |
| *** esker has joined #openstack-infra | 13:01 | |
| sdague | basically we have a race in the fact that we have a pipeline | 13:01 |
| fungi | saw a bunch of those in the gate yesterday | 13:01 |
| *** fanhe has joined #openstack-infra | 13:02 | |
| sdague | fungi: well, there is also the n-cond clean log failure | 13:02 |
| sdague | which is what 96824,2 is about | 13:02 |
| sdague | that SergeyLukjanov just promoted | 13:02 |
| mordred | well, there was another thing clark wanted to land that requires a zuul restart - we could kill 2 birds withi one stone | 13:02 |
| sdague | mordred: a zuul restart will kill the queues? | 13:02 |
| sdague | I thought they were often reloaded | 13:02 |
| *** HenryG has quit IRC | 13:02 | |
| sdague | we should land 96824,2 before we do anything, as that will reduce failure rate | 13:03 |
| *** markmcclain has quit IRC | 13:03 | |
| mordred | sdague: we usually save the queues and reload them | 13:05 |
| * fungi will brb | 13:07 | |
| *** mriedem has joined #openstack-infra | 13:07 | |
| fungi | pypi.slave replacement is building now | 13:07 |
| *** signed8bit has joined #openstack-infra | 13:07 | |
| *** ArxCruz has quit IRC | 13:07 | |
| *** markmc has quit IRC | 13:09 | |
| *** ArxCruz has joined #openstack-infra | 13:11 | |
| sdague | mordred: https://review.openstack.org/#/c/97768/ | 13:12 |
| sdague | updated | 13:12 |
| mordred | sdague: python-ganttclient is in there | 13:15 |
| *** masayukig has quit IRC | 13:15 | |
| sdague | it sure is | 13:16 |
| Shrews | sdague: such scrollback. an ironic change has broken the gate? or is that just another symptom? | 13:16 |
| sdague | it's also in the integrated gate :) | 13:16 |
| *** pdmars has joined #openstack-infra | 13:16 | |
| mordred | sdague: I think that's the thing I was pointing out | 13:16 |
| mordred | :) | 13:16 |
| sdague | yeh, but I mean if you pushed a change to it, it would run 4 tempest jobs and grenade | 13:16 |
| sdague | also, ceilometer isn't in the integrated gate definition | 13:17 |
| sdague | as discovered by this | 13:17 |
| mordred | sdague: did we not move gantt and python-ganttclient to the attic? | 13:17 |
| sdague | Shrews: there were enough ironic changes in flight that something that was approved and landed merged in a way that all ironic changes now fail their unit tests | 13:17 |
| *** sarob has joined #openstack-infra | 13:17 | |
| sdague | mordred: apparently not the client | 13:17 |
| *** julim has joined #openstack-infra | 13:18 | |
| *** Longgeek has quit IRC | 13:18 | |
| Shrews | lovely | 13:18 |
| mordred | sdague: not either | 13:18 |
| *** bknudson has joined #openstack-infra | 13:18 | |
| mordred | I'm so confused | 13:18 |
| *** Longgeek has joined #openstack-infra | 13:18 | |
| sdague | mordred: yeh, me too, because I actually thought I'd fixed that ceilometer issue before as well | 13:18 |
| *** Longgeek has quit IRC | 13:19 | |
| sdague | now I understand why all of facebook is 1 git tree :) | 13:19 |
| mordred | sdague: :) | 13:19 |
| *** Longgeek has joined #openstack-infra | 13:19 | |
| *** Longgeek has quit IRC | 13:19 | |
| *** Longgeek_ has joined #openstack-infra | 13:19 | |
| *** jistr|eng is now known as jistr | 13:20 | |
| *** Longgeek_ has quit IRC | 13:21 | |
| *** masayukig has joined #openstack-infra | 13:22 | |
| *** unicell has quit IRC | 13:22 | |
| *** sarob has quit IRC | 13:22 | |
| mordred | sdague: Release Managers has +2 on o/o | 13:23 |
| mordred | ttx: ^^ | 13:23 |
| *** Longgeek has joined #openstack-infra | 13:24 | |
| *** otherwiseguy has joined #openstack-infra | 13:24 | |
| openstackgerrit | Mark McLoughlin proposed a change to stackforge/gertty: Refactor duplicated code in dependencies handling https://review.openstack.org/97786 | 13:26 |
| *** radez is now known as radez_g0n3 | 13:26 | |
| *** habib has quit IRC | 13:27 | |
| *** palar has joined #openstack-infra | 13:27 | |
| *** habib has joined #openstack-infra | 13:28 | |
| openstackgerrit | Longgeek proposed a change to openstack-infra/config: Modify jenkins home directory https://review.openstack.org/97240 | 13:28 |
| sdague | heh, so I guess I have +2 on it then | 13:29 |
| openstackgerrit | Bob Ball proposed a change to openstack-infra/nodepool: Support install phase with nodepool https://review.openstack.org/97787 | 13:29 |
| openstackgerrit | Peter Belanyi proposed a change to openstack/requirements: Add nodeenv for horizon https://review.openstack.org/97788 | 13:30 |
| *** alexpilotti has quit IRC | 13:30 | |
| Longgeek | sdague: My patch? | 13:30 |
| SergeyLukjanov | sorry, I've been on unexpected meeting, reading scrollback | 13:31 |
| *** markmcclain1 has quit IRC | 13:31 | |
| SergeyLukjanov | something awful happens? | 13:31 |
| *** habib has quit IRC | 13:33 | |
| anteaya | Longgeek: read scrollback or check the channel logs | 13:34 |
| *** habib has joined #openstack-infra | 13:34 | |
| anteaya | Longgeek: sdague is in the middle of another conversation and was not referring to your patch | 13:34 |
| Longgeek | anteaya: ok | 13:34 |
| *** jamespage_ has joined #openstack-infra | 13:35 | |
| SergeyLukjanov | mordred, fungi, how to enqueue the release pipeline job? (there is no change id for them) | 13:35 |
| SergeyLukjanov | I mean to restart job | 13:35 |
| openstackgerrit | Sean Dague proposed a change to openstack-infra/elastic-recheck: fix bug 1324793 to determine what the issue actually is https://review.openstack.org/97790 | 13:36 |
| uvirtbot | Launchpad bug 1324793 in nova "Nova conductor throws ERROR messages, which is forbidden in our contract" [Critical,In progress] https://launchpad.net/bugs/1324793 | 13:36 |
| openstackgerrit | Marc Abramowitz proposed a change to stackforge/python-jenkins: Fix hacking docstring warnings https://review.openstack.org/97676 | 13:36 |
| *** freyes has quit IRC | 13:36 | |
| *** habib has quit IRC | 13:37 | |
| *** habib has joined #openstack-infra | 13:37 | |
| *** matrohon has joined #openstack-infra | 13:38 | |
| *** _nadya_ has quit IRC | 13:38 | |
| *** ArxCruz_ has joined #openstack-infra | 13:39 | |
| *** ArxCruz has quit IRC | 13:39 | |
| SergeyLukjanov | and mordred, fungi, I can help with 3rd party testing accounts creation if you point me to the process :) | 13:40 |
| *** otherwiseguy has quit IRC | 13:41 | |
| fungi | SergeyLukjanov: once i get pypi.slave.openstack.org rebuilt and readded to jenkins.openstack.org, we use the trigger script in the tools directory of zuul's git repo | 13:42 |
| *** otherwiseguy has joined #openstack-infra | 13:43 | |
| fungi | SergeyLukjanov: as for third-party service account creation, it's documented in http://ci.openstack.org/gerrit.html#third-party-testing-access | 13:44 |
| *** doug-fish has joined #openstack-infra | 13:44 | |
| SergeyLukjanov | fungi, oh, I'm always forget to try searching using third-party vs. 3rd party :( | 13:44 |
| SergeyLukjanov | fungi, thank you | 13:45 |
| SergeyLukjanov | fungi, I see the 3rd party account request, will try to create one ;) | 13:45 |
| fungi | SergeyLukjanov: there are actually several pending requests that i haven't gotten to yet... let me get you links | 13:45 |
| srenatus | ok, last resort: http://memecrunch.com/meme/3P8WU/merge-97196/image.jpg (https://review.openstack.org/#/c/97196/) | 13:47 |
| openstackgerrit | Radomir Dopieralski proposed a change to openstack/requirements: Add xstatic-packaged JavaScript libraries for Horizon https://review.openstack.org/97793 | 13:48 |
| ttx | mordred: missing context -- what would I use that +2 on o/o power for ? | 13:48 |
| *** timrc-afk is now known as timrc | 13:48 | |
| sdague | ttx: https://review.openstack.org/#/c/97768/ | 13:48 |
| fungi | SergeyLukjanov: http://lists.openstack.org/pipermail/openstack-infra/2014-May/001261.html http://lists.openstack.org/pipermail/openstack-infra/2014-May/001269.html https://bugs.launchpad.net/openstack-ci/+bug/1326150 | 13:49 |
| uvirtbot | Launchpad bug 1326150 in openstack-ci "Request 3rd Party CI Account for HP Cinder" [Undecided,New] | 13:49 |
| SergeyLukjanov | fungi, will do | 13:49 |
| anteaya | SergeyLukjanov: do you mind if we get some agreement on naming of third party accounts first please? | 13:49 |
| anteaya | SergeyLukjanov: I would like to go over that list on the etherpad and ensure we all agree how we want to handle multiple accounts from the same company | 13:49 |
| anteaya | as well as username and gerrit account name agreement | 13:50 |
| anteaya | it will save chasing down people and renaming in future | 13:50 |
| SergeyLukjanov | anteaya, do we have an estimate for this agreement? | 13:50 |
| anteaya | as well, we had agreed at summit that they needed a wikipage before account creation and we are almost there | 13:50 |
| *** zz_gondoi is now known as gondoi | 13:51 | |
| fungi | anteaya: well, some of those requests are already a couple weeks old, i've just been too busy to pick them up yet | 13:51 |
| anteaya | well for the wikipage creation if we get agreement at next week's meeting that we like the seed accounts, we can create wikipages | 13:51 |
| SergeyLukjanov | anteaya, it's not documented now, so, people don't know what should be done | 13:51 |
| fungi | yeah, the current requests are following the currently documented process | 13:51 |
| anteaya | I know, but they can't vote yet anyway and if we do this according to the new rules, it saves me a heap of time | 13:51 |
| SergeyLukjanov | fungi, + | 13:52 |
| anteaya | this just creates so much more work for me | 13:52 |
| sdague | ok, so we seem to have at least a triumvirant of race fails that are liking to land on each other | 13:52 |
| anteaya | since we had agreed to it at summit, I had thought there would be a little support for it | 13:52 |
| fungi | i'm just considering that these vendors have been instructed (primary by the block storage program now) that they need to be doing testing, so i feel sort of bad for holding up their requests and thus their progress toward better integration with cinder | 13:53 |
| anteaya | I can't stop you, but this just digs the hole deeper | 13:53 |
| *** _nadya_ has joined #openstack-infra | 13:53 | |
| *** wenlock has joined #openstack-infra | 13:53 | |
| ttx | sdague: I suppose the new .gitmodules is the result of the script, not something you cobbled together ? | 13:53 |
| anteaya | jgriffith: ping | 13:53 |
| srenatus | thanks jeremy | 13:53 |
| fungi | the hole is already pretty deep, so i'm not sure that a few more on top while we work out the remaining details make much difference | 13:53 |
| *** mwagner_lap has joined #openstack-infra | 13:53 | |
| SergeyLukjanov | I think that while we have now documented process we couldn't point folks to the new process | 13:53 |
| sdague | ttx: yep | 13:53 |
| anteaya | fungi: the difference is for me | 13:54 |
| anteaya | do I feel supported or alone | 13:54 |
| ttx | sdague: ok, +2. Will wait for another release manager to +2 or a bit more infra-core to +A before I APRV it | 13:54 |
| anteaya | right now I don't feel supported | 13:54 |
| anteaya | I feel alone | 13:54 |
| *** unicell has joined #openstack-infra | 13:55 | |
| *** dkliban_afk is now known as dkliban | 13:55 | |
| ttx | mordred: don't forget you're supposed to submit a scoring proposal as a governance resolution at some point | 13:56 |
| fungi | just to confirm, your suggestion is to announce a freeze on any new gerrit systems accounts until we determine a more consistent way to add them, and that programs should adjust their driver approval deadlines accordingly | 13:56 |
| *** jgrimm has joined #openstack-infra | 13:56 | |
| * ttx just got home | 13:56 | |
| *** fanhe has quit IRC | 13:56 | |
| *** e0ne_ has quit IRC | 13:56 | |
| anteaya | fungi: no | 13:56 |
| *** heyongli has joined #openstack-infra | 13:56 | |
| fungi | i'm not sure i understand then | 13:57 |
| *** e0ne has joined #openstack-infra | 13:57 | |
| SergeyLukjanov | me too | 13:57 |
| fungi | we should or shouldn't stop adding new system accounts right now? | 13:57 |
| *** fanhe has joined #openstack-infra | 13:58 | |
| *** dizquierdo has quit IRC | 13:58 | |
| anteaya | I am checking timing with cinder | 13:58 |
| fungi | okay | 13:59 |
| *** ilyashakhat has joined #openstack-infra | 13:59 | |
| SergeyLukjanov | ack, I;m on hold for creating acc | 13:59 |
| fungi | if there is a good synergy of schedules for them, then i can understand telling the current requestors we're changing the process, with an eta on when the new process will be unveiled for them to follow instead | 14:00 |
| sdague | SergeyLukjanov / fungi: https://etherpad.openstack.org/p/gatetriage-june2014 | 14:00 |
| fungi | sdague: looking those over now | 14:00 |
| openstackgerrit | Alexandre Levine proposed a change to openstack-infra/config: Initial commit for the AWS EC2 and VPC API standalone service project https://review.openstack.org/97795 | 14:00 |
| SergeyLukjanov | fungi, ++ otherwise we should keep the old process unfortunately while the new one isn't ready | 14:00 |
| openstackgerrit | Boris Pavlovic proposed a change to openstack-infra/config: Finally add osprofiler to stackforge https://review.openstack.org/97778 | 14:01 |
| sdague | fungi: the tempest one isn't approved yet, just discussing with mtreinish | 14:01 |
| *** habib has quit IRC | 14:01 | |
| sdague | the conductor one was just at top of gate, and killed by the tempest one | 14:01 |
| SergeyLukjanov | sdague, candidates for promoting? | 14:01 |
| sdague | SergeyLukjanov: yep | 14:01 |
| fungi | those all look like sound measures to me | 14:01 |
| SergeyLukjanov | sdague, so, I think we should wait for tempest change to be approved and the promote all three changes | 14:02 |
| sdague | SergeyLukjanov: I agree | 14:02 |
| fungi | yeah, presumably in the order listed in that pad | 14:02 |
| *** pdmars has quit IRC | 14:02 | |
| openstackgerrit | Kiall Mac Innes proposed a change to openstack-infra/config: Add Designate DevStack/Requirements/Docs Jobs https://review.openstack.org/97348 | 14:02 |
| fungi | --changes 97755,2 96824,2 92893,5 | 14:02 |
| *** pdmars has joined #openstack-infra | 14:02 | |
| Kiall | Would love reviews on ^ TC people holding off on incubation vote until it's running :( | 14:03 |
| anteaya | fungi SergeyLukjanov right now cinder has no timeline that I can see: https://etherpad.openstack.org/p/juno-cinder-3rd-party-cert-and-verification | 14:03 |
| anteaya | I am looking to find jgriffith to get additional information | 14:03 |
| fungi | anteaya: okay, there's no rush if he's busy (he almost certainly is). this was merely a good opportunity for SergeyLukjanov to get some additional experience with the basic account creation api calls | 14:04 |
| sdague | fungi: ok, we have a +A on the tempest change | 14:04 |
| *** jamespage_ has quit IRC | 14:05 | |
| sdague | so I think the promotes should be good to go | 14:05 |
| fungi | anteaya: and also to help with the fact that we've not gotten back on a couple of account requests for a week or two | 14:05 |
| SergeyLukjanov | sdague, I'll do it | 14:05 |
| fungi | SergeyLukjanov: did you want to do that promote command? | 14:05 |
| fungi | go for it | 14:05 |
| sdague | SergeyLukjanov: thanks | 14:05 |
| SergeyLukjanov | fungi, yup | 14:05 |
| fungi | Kiall: adding it near the top of my pile. if my pile stops growing for the day and starts shrinking, i'll have a look | 14:06 |
| Kiall | Thanks fungi :) | 14:06 |
| SergeyLukjanov | zuul promote --pipeline gate --changes 97755,2 96824,2 92893,5 | 14:06 |
| SergeyLukjanov | fungi, sdague ^^ | 14:07 |
| anteaya | fungi: I understand that | 14:07 |
| anteaya | and thanks for letting me get some additional information on this | 14:07 |
| fungi | SergeyLukjanov: that looks right to me | 14:07 |
| SergeyLukjanov | fungi, hm, Exception: Unable to find shared change queue for 97755,2 | 14:08 |
| *** basha has joined #openstack-infra | 14:08 | |
| *** markmcclain has joined #openstack-infra | 14:09 | |
| fungi | SergeyLukjanov: that's odd... | 14:09 |
| SergeyLukjanov | fungi, yup... | 14:10 |
| *** trinaths has joined #openstack-infra | 14:10 | |
| sdague | it's in check | 14:11 |
| sdague | does that matter | 14:11 |
| sdague | I guess we need to move it over to gate | 14:11 |
| fungi | sdague: shouldn't | 14:11 |
| *** [1]trinaths has joined #openstack-infra | 14:11 | |
| fungi | SergeyLukjanov: i wonder if you tried to run it before zuul saw the approve vote. try a second time now? | 14:11 |
| fungi | oh! | 14:12 |
| fungi | sdague: you are correct | 14:12 |
| fungi | SergeyLukjanov: any of them which aren't in the gate pipeline yet need a zuul enqueue command first | 14:12 |
| SergeyLukjanov | fungi, okay | 14:12 |
| SergeyLukjanov | fungi, got it | 14:12 |
| fungi | sudo zuul enqueue --trigger gerrit --project openstack/tempest --pipeline gate --change 97755,2 | 14:13 |
| fungi | like that | 14:13 |
| * fungi clearly didn't get enough coffee | 14:13 | |
| fungi | that will pop them onto the end of the gate, and then you can promote them all in one shot. you'll also need to delete any jenkins -1 or -2 votes | 14:14 |
| fungi | before trying to enqueue | 14:14 |
| *** trinaths has quit IRC | 14:14 | |
| *** [1]trinaths is now known as trinaths | 14:14 | |
| fungi | possibly | 14:15 |
| fungi | though it looks like that's not necessary now that zuul clears the old jenkins vote when enqueuing anyway | 14:15 |
| SergeyLukjanov | done | 14:15 |
| SergeyLukjanov | fungi, sdague ^^ | 14:15 |
| sdague | cool | 14:15 |
| fungi | yep, now they're on top and in order. awesome SergeyLukjanov | 14:16 |
| sdague | thanks | 14:16 |
| SergeyLukjanov | fungi, it works after enqueuing them to the gate pipeline and only after that promoting | 14:16 |
| fungi | SergeyLukjanov: yep | 14:16 |
| SergeyLukjanov | fungi, adding it to my infra how to notes ;) | 14:16 |
| *** pblaho has joined #openstack-infra | 14:16 | |
| openstackgerrit | Bob Ball proposed a change to openstack-infra/nodepool: Support nodes with launch condition https://review.openstack.org/97798 | 14:17 |
| *** sarob has joined #openstack-infra | 14:17 | |
| *** skolekonov has quit IRC | 14:18 | |
| *** wenlock has quit IRC | 14:20 | |
| anteaya | SergeyLukjanov: who might I talk to about fuel ci and fuel watcher? | 14:20 |
| SergeyLukjanov | anteaya, let me check | 14:20 |
| anteaya | thanks | 14:21 |
| anteaya | if there is a channel I should join, do tell me what it is | 14:21 |
| anteaya | BobBall: any idea who maintains CitrixJenkins? | 14:21 |
| *** sarob has quit IRC | 14:21 | |
| anteaya | BobBall: it has to be renamed and I need to talk to someone to find out what it actually does | 14:22 |
| BobBall | I saw that the other day on the mailing list | 14:22 |
| BobBall | I think it might have been from the good old days of Citrix and OpenStack | 14:22 |
| BobBall | i.e. probably Euan Mellor's automated account that is highly deprecated now | 14:23 |
| *** Alexei_987 has joined #openstack-infra | 14:23 | |
| *** mrodden has quit IRC | 14:24 | |
| fungi | anteaya: yeah, worth considering some of these accounts may be defunct (and if so should just be removed from the associated groups). figuring that out might require some interesting api queries looking for the age of their most recent comments though | 14:24 |
| anteaya | BobBall: do you feel it is safe to mark CitrixJenkins as deprecated then? | 14:24 |
| BobBall | yes | 14:24 |
| openstackgerrit | A change was merged to openstack-infra/elastic-recheck: fix bug 1324793 to determine what the issue actually is https://review.openstack.org/97790 | 14:24 |
| uvirtbot | Launchpad bug 1324793 in nova "Nova conductor throws ERROR messages, which is forbidden in our contract" [Critical,In progress] https://launchpad.net/bugs/1324793 | 14:24 |
| anteaya | BobBall: thanks I will make a note of that | 14:25 |
| *** dizquierdo has joined #openstack-infra | 14:25 | |
| anteaya | fungi: yes, I will compile a list of suspected deprecated accounts so they can be confirmed | 14:25 |
| anteaya | and then we can discuss what to do with them | 14:25 |
| openstackgerrit | Sergey Skripnick proposed a change to openstack-infra/config: Add rally-neutron-neutron job https://review.openstack.org/97805 | 14:27 |
| *** Longgeek has quit IRC | 14:27 | |
| e0ne | Hi Infra Team. I'm trying to setup local environment for debug using manifests from https://github.com/openstack-infra/config. Looks like it isn't flexible enough: a lot of hardcoded git repos urls inside. Does it make sense to create bug and fix it? | 14:29 |
| *** heyongli has quit IRC | 14:29 | |
| e0ne | e.g.: https://github.com/openstack-infra/config/blob/master/modules/jenkins/manifests/job_builder.pp#L27 | 14:30 |
| e0ne | I propose to make git repos urls configurable | 14:30 |
| tteggel | heads up- hpcloud 1.1 is experiencing scheduling delay again | 14:35 |
| tteggel | https://community.hpcloud.com/status/incident/2622 | 14:35 |
| ruhe | anteaya: there is #fuel-dev channel, i hope you'll find people you're looking at that channel | 14:36 |
| SergeyLukjanov | anteaya, Alexandra Fedorova <afedorova@mirantis.com> | 14:36 |
| rcarrillocruz | tteggel: interesting, i created an instance in east two hours ago and it was quite slow | 14:36 |
| rcarrillocruz | thx for sharing | 14:36 |
| anteaya | ruhe, SergeyLukjanov thank you | 14:36 |
| *** mrodden has joined #openstack-infra | 14:37 | |
| openstackgerrit | Bob Ball proposed a change to openstack-infra/nodepool: Support install phase with nodepool https://review.openstack.org/97787 | 14:37 |
| openstackgerrit | Bob Ball proposed a change to openstack-infra/nodepool: Support nodes with launch condition https://review.openstack.org/97798 | 14:37 |
| fungi | e0ne: i believe there are already some changes proposed to do just that. asselin_ had at least one up for review in that theme | 14:38 |
| *** doude has quit IRC | 14:39 | |
| *** rwsu has joined #openstack-infra | 14:40 | |
| *** andreaf has quit IRC | 14:41 | |
| e0ne | fungi, thanks, but i found only https://review.openstack.org/#/c/96539 | 14:42 |
| *** basha has quit IRC | 14:42 | |
| *** reed has joined #openstack-infra | 14:43 | |
| e0ne | it doesn't fix all issues with hardcoded git repos | 14:43 |
| anteaya | fungi SergeyLukjanov so DuncanT in -cinder said that if we can get some agreement on naming in 24 hours, he is okay with that | 14:44 |
| *** thedodd has joined #openstack-infra | 14:44 | |
| anteaya | I am working on this etherpad: https://etherpad.openstack.org/p/automated-gerrit-account-naming-format | 14:44 |
| *** gondoi is now known as zz_gondoi | 14:44 | |
| anteaya | and have my suggestions for new names as #comments after the current name | 14:44 |
| anteaya | if we can at least agree that the new accounts will follow the format of the suggestions, I am fine with creating them | 14:45 |
| anteaya | if we agree my suggestions for new names are good suggestions | 14:45 |
| *** markmcclain has quit IRC | 14:45 | |
| fungi | i'll have a look once i'm done wrangling the bitrot in our server launch script and get release uploads back online again | 14:46 |
| anteaya | fungi: *nod | 14:46 |
| anteaya | yeah I know this is not top priority for you | 14:46 |
| anteaya | just trying to keep you up to date | 14:46 |
| *** wenlock has joined #openstack-infra | 14:46 | |
| trinaths | hi. In Jenkins testing, I got this error for my patchset which doesn't relate me http://paste.openstack.org/show/82784/ . Can I go for recheck no bug | 14:47 |
| *** achuprin has quit IRC | 14:48 | |
| * anteaya is afk for a bit | 14:48 | |
| *** matsuhashi has quit IRC | 14:49 | |
| openstackgerrit | A change was merged to openstack-infra/config: Fix Chef cookbook-related strainer issue https://review.openstack.org/97196 | 14:49 |
| *** jpich has quit IRC | 14:50 | |
| *** sballe has joined #openstack-infra | 14:50 | |
| *** sballe_ has joined #openstack-infra | 14:52 | |
| *** klindgren_ has left #openstack-infra | 14:53 | |
| *** sballe has quit IRC | 14:56 | |
| openstackgerrit | A change was merged to stackforge/gertty: Re-add alembic to requirements https://review.openstack.org/96807 | 14:56 |
| *** sballe__ has joined #openstack-infra | 14:57 | |
| *** jpich has joined #openstack-infra | 14:57 | |
| openstackgerrit | Kiall Mac Innes proposed a change to openstack-infra/config: Add Designate DevStack/Requirements/Docs Jobs https://review.openstack.org/97348 | 14:58 |
| *** basha has joined #openstack-infra | 14:58 | |
| openstackgerrit | A change was merged to stackforge/gertty: Add hyperlinks https://review.openstack.org/95766 | 14:59 |
| openstackgerrit | A change was merged to stackforge/gertty: Correct a problem with tables at very small widths https://review.openstack.org/95767 | 14:59 |
| *** sballe_ has quit IRC | 15:00 | |
| *** achuprin has joined #openstack-infra | 15:01 | |
| fungi | trinaths: that looks like it might be bug 1310852 but it's very hard to tell from the limited amount of detail you provided | 15:01 |
| uvirtbot | Launchpad bug 1310852 in neutron "tempest.scenario.test_security_groups_basic_ops.TestSecurityGroupsBasicOps.test_in_tenant_traffic Timeout" [High,Confirmed] https://launchpad.net/bugs/1310852 | 15:01 |
| *** atiwari has joined #openstack-infra | 15:01 | |
| *** dangers_away is now known as dangers | 15:02 | |
| openstackgerrit | A change was merged to stackforge/gertty: Add patchset selection in diff https://review.openstack.org/95768 | 15:03 |
| openstackgerrit | A change was merged to stackforge/gertty: Handle (ignore) no-diff renames https://review.openstack.org/95769 | 15:03 |
| *** basha has quit IRC | 15:04 | |
| *** mriedem has quit IRC | 15:04 | |
| fungi | sdague: this got hit on a cinder change and i'm having a hard time fathoming that it could be a random failure... known issue? http://logs.openstack.org/52/97452/1/check/check-tempest-dsvm-full/8e9b941/console.html | 15:05 |
| russellb | to request a new tag on openstack mailing lists, do i file an openstack-ci bug? or ping someone/somehwere else? | 15:05 |
| fungi | sdague: "Job timeout set to: -128 minutes ... timeout: invalid option -- '1'" | 15:05 |
| sdague | fungi: that one I don't know | 15:05 |
| openstackgerrit | A change was merged to stackforge/gertty: Handle missing commits https://review.openstack.org/96801 | 15:05 |
| *** mriedem has joined #openstack-infra | 15:05 | |
| openstackgerrit | A change was merged to stackforge/gertty: Handle file-level comments https://review.openstack.org/96985 | 15:05 |
| openstackgerrit | A change was merged to stackforge/gertty: Add project updated column https://review.openstack.org/97031 | 15:06 |
| openstackgerrit | A change was merged to stackforge/gertty: Support background sync of missing refs https://review.openstack.org/97066 | 15:06 |
| fungi | russellb: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev suggests reed or ttx can help | 15:06 |
| openstackgerrit | A change was merged to stackforge/gertty: Add dependency navigation https://review.openstack.org/97067 | 15:06 |
| *** sballe__ has quit IRC | 15:06 | |
| reed | russellb, whazzup? | 15:06 |
| fungi | russellb: "OpenStack-dev list run by thierry at openstack.org, stefano at openstack.org" | 15:06 |
| openstackgerrit | A change was merged to stackforge/gertty: Remove stray debug line https://review.openstack.org/97068 | 15:07 |
| russellb | reed: was looking to get the "NFV" tag added to both openstack@ and openstack-dev@ | 15:07 |
| russellb | reed: NFV team would like to have that | 15:07 |
| reed | do people use tags? :) | 15:07 |
| reed | sure, glad to hear that people use them | 15:07 |
| russellb | heh, i don't know, but it came up in our meeting just now | 15:07 |
| russellb | people certainly use them when they post ... not sure about subscribers | 15:07 |
| *** ociuhandu has quit IRC | 15:08 | |
| openstackgerrit | Julien Danjou proposed a change to openstack-infra/config: Gate Gnocchi against MySQL too https://review.openstack.org/97820 | 15:08 |
| sdague | it seems a little odd to add a tag for not a program though, no? | 15:08 |
| * reed blanking out | 15:08 | |
| reed | what's NFV? | 15:08 |
| fungi | it stands for "very confusing acronym" | 15:09 |
| openstackgerrit | A change was merged to stackforge/gertty: Handle multiple child revisions of same parent https://review.openstack.org/97749 | 15:09 |
| ttx | fungi: that would be reed's territory yes | 15:09 |
| fungi | (network function virtualization) | 15:09 |
| rcarrillocruz | network function virtualization iirc | 15:09 |
| russellb | reed: stands for Network Function Virtualization ... a big effort within the telco industry | 15:09 |
| reed | ah | 15:09 |
| reed | thanks | 15:09 |
| ttx | russellb: ^ | 15:09 |
| russellb | of which OpenStack plays an important part | 15:09 |
| russellb | ttx: thanks | 15:09 |
| reed | russellb, Is there a wiki page where that is defined? | 15:09 |
| russellb | NFV group within OpenStack is mainly around collecting use cases, translating them to OpenStack land, and generating requirements, tracking blueprints ... | 15:10 |
| russellb | reed: WIP, just had first meeting, but https://wiki.openstack.org/wiki/Meetings/NFV | 15:10 |
| fungi | it's one of those new acronyms which means exactly the same thing as several other well-established acronyms, but each industry likes to reinvent the naming wheel | 15:10 |
| reed | russellb, col | 15:11 |
| reed | russellb, cool | 15:11 |
| reed | russellb, topics are only active on openstack-dev | 15:11 |
| russellb | ok | 15:12 |
| russellb | that's fine | 15:12 |
| fungi | neat, the term has its own wikipedia page: http://en.wikipedia.org/wiki/Network_Functions_Virtualization | 15:12 |
| russellb | fungi: that means it's legit | 15:12 |
| *** zz_gondoi is now known as gondoi | 15:12 | |
| russellb | updated wiki page with team mission statement | 15:13 |
| fungi | according to the article, about a year and a half ago some telecommunications people had the bright idea that you could have virtual network devices, but this novel concept lacked a good name | 15:13 |
| *** radez_g0n3 is now known as radez | 15:13 | |
| russellb | :) | 15:13 |
| *** spiffxp- has quit IRC | 15:13 | |
| russellb | yeah, basically out with the old traditional hardware networking appliances, and in with sexy VMs that do the same thing. | 15:14 |
| *** SlickNik has quit IRC | 15:14 | |
| *** SlickN1k is now known as SlickNik | 15:14 | |
| russellb | and these telcos are identifying lots of good gaps in OpenStack | 15:14 |
| russellb | and there was lots of distributed, private coordination around these things | 15:14 |
| russellb | so ... just centralizing / coordinating a bit here. | 15:14 |
| fungi | (never mind the fact that i used virtual network devices at a service provider more than a decade ago, we just didn't have the need for a confusing term to go along with that concept) | 15:15 |
| russellb | :) | 15:15 |
| russellb | fungi: so the other thing was making #openstack-nfv official ... | 15:15 |
| *** HenryG has joined #openstack-infra | 15:15 | |
| fungi | russellb: http://git.openstack.org/cgit/openstack-infra/config/tree/modules/openstack_project/files/accessbot/channels.yaml | 15:16 |
| russellb | fungi: ack thanks | 15:16 |
| fungi | russellb: that should get us on track with having appropriate control over the channel (though someone may need to register it with chanserv and set the openstackinfra nick as a founder first) | 15:17 |
| *** sarob has joined #openstack-infra | 15:17 | |
| russellb | fungi: k thanks | 15:17 |
| sdague | fungi: all the failures are in rax iad | 15:17 |
| openstackgerrit | A change was merged to stackforge/gertty: Expand the .gitignore file to ignore .egg files https://review.openstack.org/92891 | 15:17 |
| sdague | http://logstash.openstack.org/#eyJzZWFyY2giOiJcInRpbWVvdXQ6IGludmFsaWQgb3B0aW9uXCIiLCJmaWVsZHMiOltdLCJvZmZzZXQiOjAsInRpbWVmcmFtZSI6IjYwNDgwMCIsImdyYXBobW9kZSI6ImNvdW50IiwidGltZSI6eyJ1c2VyX2ludGVydmFsIjowfSwic3RhbXAiOjE0MDE4OTQ3MzIzNzF9 | 15:18 |
| fungi | russellb: beyond that, if they want channel logging, review notifications, et cetera that's in different files | 15:18 |
| russellb | fungi: sounds good, appreciate the pointer, i can figure that stuff out. i should have guessed it was in infra-config :) | 15:18 |
| russellb | <3 infra-config | 15:18 |
| fungi | russellb: http://ci.openstack.org/irc.html | 15:18 |
| fungi | (for the official rtfm link) | 15:19 |
| fungi | but i'm happy to answer any questions about that stuff if you have them | 15:19 |
| sdague | fungi: any chance that the clock get moverd | 15:19 |
| trinaths | stackers!, I hit with this error http://paste.openstack.org/show/82786/, for a 'recheck no bug' for my patchset. | 15:19 |
| *** eharney has quit IRC | 15:20 | |
| fungi | sdague: it's... possible that something changed the time on the vm before d-g tried to calculate the timeout offset | 15:21 |
| sdague | fungi: it's new, and only on rax-iad | 15:21 |
| sdague | I'm going to put in a failsafe to the calculation | 15:21 |
| fungi | sdague: new as of the past hour? | 15:21 |
| sdague | see logstash link above | 15:21 |
| sdague | new as in last couple days | 15:21 |
| gilliard | What's the process for submitting a patch to something in stackforge? Is there a gerrit? Or a github PR? | 15:21 |
| *** sarob has quit IRC | 15:21 | |
| fungi | maybe something with the devstack-precise image in rax-iad from a little bit ago | 15:22 |
| sdague | 73 hits | 15:22 |
| sdague | started about 48hrs ago | 15:22 |
| fungi | gilliard: gerrit. any stackforge project should have a valid .gitreview file which would configure the git-review utility automagically when you try to use it to submit a patch | 15:22 |
| fungi | sdague: ahh, okay, so maybe it's a recent change to the ubuntu precise base image there | 15:23 |
| *** nati_ueno has joined #openstack-infra | 15:23 | |
| gilliard | thx fungi | 15:24 |
| fungi | phschwartz: do you happen to know if there was a refresh for the "Ubuntu 12.04 LTS (Precise Pangolin)" image in iad roughly 2-3 days ago? | 15:24 |
| *** virmitio has joined #openstack-infra | 15:25 | |
| fungi | phschwartz: we're seeing some oddness when booting from a snapshot there, where it looks like the system time is possibly jumping by a couple hours shortly after boot, but it only started a couple days ago | 15:26 |
| openstackgerrit | Sean Dague proposed a change to openstack-infra/devstack-gate: debug time calculations https://review.openstack.org/97824 | 15:26 |
| *** mmaglana has joined #openstack-infra | 15:26 | |
| sdague | fungi: that might at least help debug | 15:27 |
| sdague | and fail more sanely | 15:27 |
| openstackgerrit | Matthew Gilliard proposed a change to stackforge/gertty: Adds alembic to requirements.txt https://review.openstack.org/97825 | 15:27 |
| fungi | gilliard: looks like you found it! ;) | 15:27 |
| *** IvanBerezovskiy has left #openstack-infra | 15:27 | |
| sdague | fungi: you have a bug for this? | 15:28 |
| fungi | sdague: bug 1326231 | 15:28 |
| uvirtbot | Launchpad bug 1326231 in openstack-ci "Invalid timeout value cusing "gate-tempest-dsvm-full" to fail" [Undecided,New] https://launchpad.net/bugs/1326231 | 15:28 |
| fungi | skimming the new bugs, that one caught my eye as hard to be your run-of-the-mill openstack brokenness | 15:28 |
| sdague | yep | 15:29 |
| sdague | I'll build an er query for it | 15:29 |
| *** gokrokve has quit IRC | 15:29 | |
| *** gokrokve has joined #openstack-infra | 15:29 | |
| fungi | the nodepool image list suggests that we failed to get new devstack-precise and devstack-precise-check images in rax-iad today, but built fresh ones 24 and 48 hours ago, so my guess is that if it's related to the base image it may have been introduced in rax-iad between 48-72 hours ago | 15:30 |
| *** olaph has joined #openstack-infra | 15:30 | |
| openstackgerrit | Sean Dague proposed a change to openstack-infra/elastic-recheck: add query for timeout calculation gone wrong https://review.openstack.org/97826 | 15:30 |
| sdague | fungi: yeh, but the only thing that could be going on is if we jumped time forward by a couple hours after d-g was running | 15:31 |
| fungi | (though really not sure if we got an image 72 hours ago, so it could have actually been like that longer and we just wouldn't have seen it until our successful image updates on monday) | 15:31 |
| fungi | sdague: well, ntp could be lagging, especially if it started up before its gateway was reachable | 15:31 |
| *** rcarrill` has joined #openstack-infra | 15:31 | |
| BobBall | trying to re-run upstream nodepool for the xenserver CI I get a weird error saying a socket has no option for RCVTIMEO - http://paste.openstack.org/show/82790/ - any ideas why this might be? | 15:32 |
| fungi | and then adjusted the system clock just after d-g started | 15:32 |
| *** wenlock has quit IRC | 15:32 | |
| fungi | BobBall: looks familiar. consulting magic 8-ball now | 15:33 |
| BobBall | I did check and zmq is in the right range... | 15:33 |
| BobBall | root@xs-ci-test:~/src/nodepool# pip freeze | grep zmq | 15:33 |
| BobBall | pyzmq==13.1.0 | 15:33 |
| *** rcarrillocruz has quit IRC | 15:34 | |
| fungi | BobBall: conversation starts at 19:33:19 http://eavesdrop.openstack.org/irclogs/%23openstack-infra/%23openstack-infra.2014-04-03.log | 15:35 |
| BobBall | wonderful - thanks fungi | 15:35 |
| pelix | any chance of a review on https://review.openstack.org/#/c/48783/ ? | 15:35 |
| openstackgerrit | Arun Kant proposed a change to openstack/requirements: Adding ldappool module dependency as needed by keystone bug #1320997. https://review.openstack.org/95842 | 15:35 |
| uvirtbot | Launchpad bug 1320997 in keystone "Common Ldap handler connection pooling" [Medium,In progress] https://launchpad.net/bugs/1320997 | 15:35 |
| openstackgerrit | David Shrewsbury proposed a change to openstack-infra/config: Adjust ironic tempest regex for new tests https://review.openstack.org/97830 | 15:35 |
| openstackgerrit | Marc Abramowitz proposed a change to openstack-infra/jenkins-job-builder: Some tweaks to get closer to Python 3 compat https://review.openstack.org/87810 | 15:36 |
| BobBall | fungi: that is soooo ugly! But thanks :) | 15:36 |
| fungi | BobBall: looks like resolution was pyzmq linking against a system installed libzmq1. try uninstalling that | 15:36 |
| BobBall | yup - found that + testing. | 15:36 |
| *** rcarrill` is now known as rcarrillocruz | 15:37 | |
| fungi | BobBall: if that works, maybe a couple-sentence patch to the nodepool docs is in order, to serve as a warning to other hapless adventurers | 15:38 |
| *** markmc has joined #openstack-infra | 15:38 | |
| *** hashar has quit IRC | 15:38 | |
| anteaya | BobBall: this is another citrix gerrit account: NetScalerAts NetScaler TestingSystem | 15:39 |
| *** gyee has joined #openstack-infra | 15:39 | |
| anteaya | BobBall: ring any bells for you? | 15:39 |
| BobBall | anteaya: that one is live + used by the netscaler team (I'm on the XenServer team) | 15:39 |
| BobBall | or at least I believe it's live... | 15:39 |
| BobBall | :) | 15:39 |
| anteaya | BobBall: so citrix-netscaler-ci would be appropriate? | 15:40 |
| *** andreykurilin_ has joined #openstack-infra | 15:40 | |
| BobBall | yup | 15:40 |
| anteaya | thanks | 15:40 |
| BobBall | thanks fungi - that does indeed fix it. | 15:41 |
| *** rcarrill` has joined #openstack-infra | 15:41 | |
| *** markmcclain has joined #openstack-infra | 15:42 | |
| *** dims_ has quit IRC | 15:42 | |
| *** yamahata has quit IRC | 15:43 | |
| *** rcarrillocruz has quit IRC | 15:43 | |
| *** jgallard has quit IRC | 15:45 | |
| openstackgerrit | A change was merged to openstack/requirements: Add xvfbwrapper package for running Selenium tests headless https://review.openstack.org/80393 | 15:45 |
| openstackgerrit | A change was merged to openstack/requirements: Remove amqplib from global-requirements https://review.openstack.org/96983 | 15:46 |
| *** srenatus has quit IRC | 15:46 | |
| *** basha has joined #openstack-infra | 15:46 | |
| *** Sukhdev has joined #openstack-infra | 15:46 | |
| *** srenatus has joined #openstack-infra | 15:46 | |
| *** rgerganov has quit IRC | 15:46 | |
| openstackgerrit | Bob Ball proposed a change to openstack-infra/nodepool: Add warnings about the installation of libzmq1 https://review.openstack.org/97835 | 15:47 |
| fungi | BobBall: thanks! | 15:47 |
| *** basha has quit IRC | 15:48 | |
| fungi | BobBall: git review thought your example error was a commit message header though :/ | 15:48 |
| fungi | hilarious | 15:48 |
| tteggel | hpcloud 1.1 is back to normal scheduling latency. think we got to the root cause this time \o/ | 15:48 |
| fungi | tteggel: awesome!!! thanks for your dogged persistence on that | 15:49 |
| *** ihrachyshka_ has joined #openstack-infra | 15:49 | |
| *** andreykurilin_ has quit IRC | 15:50 | |
| fungi | BobBall: i'm going to test editing that in gerrit's webui | 15:50 |
| openstackgerrit | Jeremy Stanley proposed a change to openstack-infra/nodepool: Add warnings about the installation of libzmq1 https://review.openstack.org/97835 | 15:50 |
| *** gondoi is now known as zz_gondoi | 15:51 | |
| *** jistr has quit IRC | 15:51 | |
| *** ihrachyshka has quit IRC | 15:52 | |
| BobBall | hah - whoops :) | 15:52 |
| fungi | now that i've experienced editing a git commit message in a webui, i feel like i need a shower | 15:54 |
| sdague | heh | 15:55 |
| *** mmaglana has quit IRC | 15:55 | |
| *** ihrachyshka_ has quit IRC | 15:55 | |
| *** mmaglana has joined #openstack-infra | 15:56 | |
| Kiall | fungi: lol | 15:58 |
| *** UtahDave has joined #openstack-infra | 15:58 | |
| reed | ttx, where do we publish the schedules for future releases, beyond Juno (if we do)? | 15:58 |
| openstackgerrit | Mark McLoughlin proposed a change to stackforge/gertty: Refactor duplicated code in dependencies handling https://review.openstack.org/97786 | 15:58 |
| *** rcarrill` is now known as rcarrillocruz | 15:59 | |
| *** nati_ueno has quit IRC | 15:59 | |
| *** mmaglana has quit IRC | 16:00 | |
| *** markmc has quit IRC | 16:02 | |
| *** asadoughi is now known as Swami | 16:02 | |
| *** Swami is now known as asdoughi | 16:02 | |
| *** asdoughi is now known as asadoughi | 16:02 | |
| *** lcheng_ has joined #openstack-infra | 16:03 | |
| *** ociuhandu has joined #openstack-infra | 16:03 | |
| *** e0ne has quit IRC | 16:03 | |
| *** markmcclain has quit IRC | 16:04 | |
| *** esker has quit IRC | 16:04 | |
| openstackgerrit | Jeremy Stanley proposed a change to openstack-infra/config: Update launch script and instructions https://review.openstack.org/97841 | 16:06 |
| *** pblaho has quit IRC | 16:06 | |
| *** afazekas has quit IRC | 16:07 | |
| zaro | morning | 16:08 |
| anteaya | morning zaro | 16:08 |
| *** ominakov has quit IRC | 16:09 | |
| anteaya | dprince: ola | 16:10 |
| *** eharney has joined #openstack-infra | 16:10 | |
| anteaya | dprince: I'm going to have to stick CI on the end of smokestack | 16:10 |
| morganfainberg | mornin! | 16:10 |
| anteaya | morning morganfainberg | 16:10 |
| fungi | zaro: do you happen to know the minimum gerrit acl necessary to create new branches on a project? we've had stackforge projects ask for it in the past and i've seen some with create perms on refs/* or refs/heads/* but have never taken the time to test which actually works | 16:10 |
| dprince | anteaya: whats up? | 16:11 |
| fungi | zaro: similarly, do you know whether branch deletion easier now from an acl perspective (in older gerrit you had to be set as a project owner for that) | 16:11 |
| anteaya | dprince: we have to do a standardized naming format for automated gerrit accounts | 16:11 |
| anteaya | dprince: https://etherpad.openstack.org/p/automated-gerrit-account-naming-format | 16:11 |
| anteaya | dprince: we are going to ensure ci is on the end of any automated account that votes or might vote | 16:12 |
| anteaya | dprince: and bot on accounts that will never vote | 16:12 |
| fungi | dprince: smokestack is now caught in the unrelenting tide of third-party ci ;) | 16:12 |
| anteaya | dprince: I'll need to append ci to smokestack's name | 16:12 |
| anteaya | dprince: yeah, sorry about that | 16:12 |
| *** gokrokve has quit IRC | 16:13 | |
| anteaya | wish I could exempt you but I can't see that I can | 16:13 |
| fungi | the price of being a trail blazer | 16:13 |
| anteaya | yeah | 16:13 |
| dprince | anteaya: you have my blessing | 16:13 |
| anteaya | thank you | 16:13 |
| anteaya | :D | 16:13 |
| *** crc32 has joined #openstack-infra | 16:14 | |
| *** wenlock has joined #openstack-infra | 16:17 | |
| *** jcoufal has quit IRC | 16:17 | |
| *** otherwiseguy has quit IRC | 16:17 | |
| *** ewindisch has quit IRC | 16:18 | |
| *** jlibosva has quit IRC | 16:19 | |
| anteaya | fungi SergeyLukjanov so if you can do a round up of outstanding account requests so that I can add potential usernames and Full Names to my etherpad to facilitate the agreement process, that would make me happy | 16:20 |
| *** aconrad has joined #openstack-infra | 16:22 | |
| *** jooools has quit IRC | 16:25 | |
| *** hemnafk is now known as hemna | 16:26 | |
| *** jpich has quit IRC | 16:28 | |
| *** zz_gondoi is now known as gondoi | 16:29 | |
| *** Guest22618 has quit IRC | 16:30 | |
| openstackgerrit | Khai Do proposed a change to openstack-infra/jenkins-job-builder: refactor cmd processor to make it slightly easier to test https://review.openstack.org/97848 | 16:31 |
| *** otherwiseguy has joined #openstack-infra | 16:32 | |
| *** ewindisch has joined #openstack-infra | 16:32 | |
| *** smarcet has joined #openstack-infra | 16:33 | |
| *** bhuvan has joined #openstack-infra | 16:34 | |
| zaro | fungi: i think all you need is 'Create Reference' permission, https://groups.google.com/d/msg/repo-discuss/2kC1IBQ-n6k/RYjWfz5XKDMJ | 16:36 |
| *** dims_ has joined #openstack-infra | 16:38 | |
| fungi | zaro: ahh, great--thanks! | 16:38 |
| pleia2 | morning | 16:38 |
| fungi | sdague: on bug 1326231 it looks like we're only hitting that on devstack-precise-check nodes, but not devstack-precise (which is odd... those images should in theory be identical) | 16:38 |
| uvirtbot | Launchpad bug 1326231 in openstack-ci "Invalid timeout value cusing "gate-tempest-dsvm-full" to fail" [Undecided,New] https://launchpad.net/bugs/1326231 | 16:38 |
| *** dizquierdo has quit IRC | 16:39 | |
| *** aconrad has quit IRC | 16:40 | |
| *** _nadya_ has quit IRC | 16:40 | |
| anteaya | morning pleia2 | 16:43 |
| *** dims_ has quit IRC | 16:43 | |
| clarkb | morning | 16:43 |
| anteaya | morning clarkb | 16:43 |
| anteaya | jhesketh: so let us know what to do with turbo-hipster: https://etherpad.openstack.org/p/automated-gerrit-account-naming-format | 16:43 |
| anteaya | jhesketh: turbo-hipster-ci? | 16:43 |
| openstackgerrit | A change was merged to stackforge/python-jenkins: Fix hacking docstring warnings https://review.openstack.org/97676 | 16:46 |
| clarkb | rackspace bd migration ci | 16:46 |
| clarkb | imo | 16:46 |
| clarkb | er db | 16:46 |
| *** Alexei_987 has quit IRC | 16:46 | |
| *** fbo is now known as fbo_away | 16:47 | |
| fungi | isviridov: okay, pypi.slave is back on jenkins.o.o now, i've retriggered magnetodb's upload job and it seems to have worked https://pypi.python.org/pypi/magnetodb http://logs.openstack.org/06/06fdbf5dd339163772d6e4b31fa16b45140294ce/release/magnetodb-pypi-upload/72f6116/console.html | 16:47 |
| fungi | isviridov: er, premature. the job hasn't completed yet | 16:47 |
| fungi | isviridov: i'll let you know once it has | 16:47 |
| *** ewindisch has quit IRC | 16:48 | |
| *** ociuhandu has quit IRC | 16:49 | |
| fungi | aha, that 'splains it... forgot the job name is now either .*-pypi-wheel-upload or .*-pypi-both-upload | 16:49 |
| *** dangers is now known as dangers_away | 16:50 | |
| fungi | clarkb: ugh... new bug! http://logs.openstack.org/0606fdbf5dd339163772d6e4b31fa16b45140294ce/release/magnetodb-pypi-both-upload/72f6116/console.html "Uploading magnetodb-2.0.5.tar.gz ... AttributeError: 'str' object has no attribute 'read'" | 16:53 |
| fungi | so something is amiss with pypi-tarball-upload.sh now too | 16:54 |
| anteaya | SergeyLukjanov: right now your system savanna-ci is formatted in such a way that it doesn't require changing, do you want to rename it to sahara-ci? | 16:55 |
| *** derekh_ has quit IRC | 16:56 | |
| reed | my emails to infra mlist failed to spark a conversation, I wonder why | 16:56 |
| anteaya | reed: mostly because the gate is in such a poor state | 16:56 |
| fungi | reed: mostly because we've been firefighting all week | 16:56 |
| reed | oh, that explains it | 16:57 |
| reed | thanks | 16:57 |
| *** mmaglana has joined #openstack-infra | 16:57 | |
| reed | keep firefighting, my stuff can wait | 16:57 |
| anteaya | reed: if we had had a clue during meeting time, we could have at least asked a few questions | 16:57 |
| *** doude has joined #openstack-infra | 16:57 | |
| anteaya | reed: you might consider adding them to next weeks agenda so you are sure they get some attention | 16:57 |
| reed | anteaya, neither of the questions are time sensitive, process when time comes | 16:57 |
| *** arnaud has joined #openstack-infra | 16:58 | |
| pleia2 | reed: I didn't want to hijack your activity board thread, but do you think reviewstats is a good candidate to landing on the new activity server once we have it up? http://git.openstack.org/cgit/openstack-infra/reviewstats/tree/README.rst | 16:58 |
| *** doude has quit IRC | 16:59 | |
| *** rkukura has joined #openstack-infra | 16:59 | |
| pleia2 | trying to figure out where to land it, was going to go with status.o.o but activity seems to make 100x more sense | 16:59 |
| pleia2 | (except for the fact that it doesn't exist in our CI yet) | 16:59 |
| anteaya | reed: they don't have to be time sensitive to get air time during the meeting | 16:59 |
| reed | pleia2, totally, I actually wanted bitergia to add a panel for those stats | 16:59 |
| anteaya | reed: but agenda items ensure we all have time to hear about the idea and ask questions | 16:59 |
| reed | anteaya, adding to the agenda, even if that means logging in the wiki :) | 17:00 |
| pleia2 | reed: a bitergia-powered review script, or that specific one? (we have some reasons for having multiple different bits of code analyzing the review stats) | 17:00 |
| anteaya | reed: yup | 17:00 |
| anteaya | reed: unless you want me to do that for you, I will if you want | 17:01 |
| reed | anteaya, should I delete the June 3 agenda on https://wiki.openstack.org/wiki/Meetings/InfraTeamMeeting ? | 17:01 |
| anteaya | no, just add june 10 to the top and add your item | 17:01 |
| anteaya | the chair de jour will clean it up before the meeting | 17:01 |
| reed | pleia2, I need bitergia to gather some stats for me from the review system, they might as well start producing the charts and plots and tables and json files we actually need | 17:01 |
| anteaya | add your irc nick beside any of your items | 17:02 |
| reed | pleia2, s/we/you | 17:02 |
| pleia2 | :) | 17:02 |
| pleia2 | ok, thanks | 17:02 |
| sdague | fungi: so yeh, we're jumping time a lot - http://logs.openstack.org/11/95911/2/check/check-grenade-dsvm/c91fe2f/logs/syslog.txt.gz | 17:03 |
| clarkb | fungi: nice! | 17:03 |
| clarkb | fungi: did you want to debug the upload script? I am happy to if you already have all the things on your plate this morning | 17:03 |
| fungi | clarkb: i can keep plugging at it. this is a continuation of discovering that now that we have working tarball jobs again, we never rebuilt the pypi slave, and once that was enrolled, remembering that we changed the job name, and now... | 17:04 |
| openstackgerrit | Jarret Raim proposed a change to openstack-infra/config: Adding barbican-specs to infra configs https://review.openstack.org/97616 | 17:05 |
| reed | pleia2, where can I see a sample output of reviewstats.py? | 17:06 |
| pleia2 | reed: russellb has it on his website, I'll dig up link | 17:06 |
| reed | is that russellb's stuff | 17:06 |
| reed | ok, I know that one | 17:06 |
| reed | http://russellbryant.net/openstack-stats/ | 17:07 |
| pleia2 | yeah | 17:07 |
| notmyname | clarkb: jeblair: mordred: is there any reason I can't start populating swift-specs? | 17:07 |
| pleia2 | reed: since the code is now in infra, we want to host it for him :) | 17:07 |
| clarkb | notmyname: I don't think so | 17:07 |
| clarkb | notmyname: we have started populating ours | 17:07 |
| russellb | yeah i keep breaking it | 17:07 |
| notmyname | ok, thanks | 17:07 |
| *** aconrad has joined #openstack-infra | 17:07 | |
| openstackgerrit | A change was merged to openstack-infra/elastic-recheck: add query for timeout calculation gone wrong https://review.openstack.org/97826 | 17:08 |
| mordred_phone | notmyname: it just needs a patch with all the specs-cookiecutter stuff | 17:08 |
| notmyname | mordred_phone: where is the cookiecutter repo? | 17:08 |
| reed | russellb, I noticed :) most files I checked are empty | 17:08 |
| russellb | reed: sorry. | 17:09 |
| russellb | reed: you can run the scripts locally though | 17:09 |
| russellb | in theory | 17:09 |
| russellb | unleess they were busted by the gerrit upgrade | 17:09 |
| mordred_phone | notmyname: openstack-dev/specs-cookiecutter | 17:09 |
| russellb | i'm just back this week after being out for over a month | 17:09 |
| reed | russellb, no need to apologize, I just wanted to get a sample so I can ask Bitergia to make us a dashboard with those | 17:09 |
| notmyname | mordred_phone: thanks | 17:09 |
| sdague | fungi: so on the time jump one, should we try to create a work around, or figure out why that region is weird | 17:10 |
| mordred_phone | notmyname: let me know if you need help, I'm happy to make you a patch or whatnot | 17:10 |
| sdague | maybe they are booting with a different tz? | 17:10 |
| notmyname | mordred_phone: I'll do it today, I think. I'll ask if I have questions. and ping you with a link to the patch I submit | 17:10 |
| sdague | fungi: a 4 hr jump is very suspicious to me | 17:10 |
| fungi | sdague: probably i need to hold a node and check its syslog for signs that ntp is making a huge correction or something | 17:11 |
| sdague | fungi: see url above I posted | 17:11 |
| sdague | that's exactly what's happening | 17:12 |
| fungi | aha! | 17:12 |
| fungi | too many things at once ;) | 17:12 |
| fungi | sdague: i suspect we've got a nova compute in rackspace iad whose time is hours off, and so instances are starting with radically incorrect time | 17:13 |
| fungi | sdague: the start of occurrences doesn't seem to map to a nodepool image update | 17:13 |
| sdague | the fact that it's only happening in one az makes me feel like someone borked the hypervisor config there and it's set to a different timezone | 17:13 |
| fungi | entirely possible | 17:14 |
| sdague | because seed time should be set on boot | 17:14 |
| fungi | sdague: and yeah, it's jumping precisely 4 hours, so almost certainly set to edt (where iad is located) | 17:14 |
| fungi | phschwartz: are you around? | 17:15 |
| sdague | I guess we don't index syslog normally? | 17:18 |
| SpamapS | sdague: I found a hole where something can jump to the gate even if a dependent patch is -1'ed? | 17:18 |
| sdague | SpamapS: yeh, some time ago | 17:18 |
| fungi | SpamapS: you're so good at exposing bugs you can't remember them all any more ;) | 17:18 |
| SpamapS | Ah I was going to add it to my infra-breaking-trophy room ;) | 17:18 |
| *** gokrokve has joined #openstack-infra | 17:18 | |
| sdague | at least my vague memory is that you discovered it | 17:18 |
| SpamapS | It's really that I'm too lazy to test things so I just plod forward expecting them to work intuitively. ;) | 17:19 |
| fungi | SpamapS: change b is dependent on change a, and both have approval votes but b has failed (or not completed) check tests... when a gets enqueued into the gate b comes along for the ride automatically | 17:19 |
| *** Ryan_Lane has joined #openstack-infra | 17:19 | |
| clarkb | sdague: we index syslog | 17:19 |
| fungi | b won't merge, but it will act as a wrecking ball all the way through the pipeline | 17:20 |
| SpamapS | Oh I think I "found" that but I didn't actually debug it.. I just made it happen a few times. :) | 17:20 |
| sdague | clarkb: .... hmmm | 17:20 |
| *** dims_ has joined #openstack-infra | 17:20 | |
| fungi | SpamapS: that's why i like to use the word "expose" | 17:20 |
| clarkb | sdague: we are very behind on indexing last I checked | 17:20 |
| clarkb | I think the tempest logs are more impactful than you expected | 17:21 |
| sdague | clarkb: ok, well we can turn them off | 17:21 |
| *** srenatus has quit IRC | 17:22 | |
| *** shivharis has joined #openstack-infra | 17:23 | |
| *** srenatus has joined #openstack-infra | 17:23 | |
| *** dims_ has quit IRC | 17:23 | |
| shivharis | fungi: ping | 17:23 |
| fungi | shivharis: hi there | 17:23 |
| *** dims_ has joined #openstack-infra | 17:23 | |
| shivharis | fungi: need to get onto "neutron bugs" team, how to do that... | 17:23 |
| shivharis | i have already sent the request, | 17:24 |
| shivharis | in waiting mode for about 2 weeks now | 17:24 |
| sdague | though syslog is running at 3x the # of lines of tempest due to ceilometer | 17:25 |
| fungi | shivharis: an administrator of that group has to approve your request. try contacting one of the people listed as an administrator at https://launchpad.net/~neutron-bugs/+members (possibly via irc or e-mail as they may not be paying attention to lp notifications) | 17:25 |
| clarkb | sdague: oh well that may explain it | 17:26 |
| shivharis | the admins like DanW and Rick C are not active, can we add Kyle Mestery? | 17:26 |
| shivharis | as admin | 17:26 |
| clarkb | sdague: is that swift leaking into syslog every time ceilometer hits it? | 17:26 |
| sdague | clarkb: yep | 17:26 |
| clarkb | sdague: awesome :/ | 17:26 |
| sdague | with good stuff like | 17:26 |
| sdague | Jun 4 16:38:08 devstack-precise-check-rax-iad-63313 proxy-server: Returning cached token | 17:26 |
| sdague | Jun 4 16:38:08 devstack-precise-check-rax-iad-63313 proxy-server: Storing token in cache | 17:26 |
| clarkb | dstufft: is http://logs.openstack.org/0606fdbf5dd339163772d6e4b31fa16b45140294ce/release/magnetodb-pypi-both-upload/72f6116/console.html a twine bug? | 17:27 |
| clarkb | fungi: ^ I wonder if server rebuild updated twine now with new bugs | 17:27 |
| fungi | shivharis: yeah, i suspect mestery not being an administrator on that lp group was just an oversight on markmcclain's part during the ptl changing of the guard. fixed | 17:27 |
| *** BobBall is now known as BobBallAway | 17:27 | |
| fungi | clarkb: either that or we missed puppeting something like credentials | 17:28 |
| shivharis | fungi: thanks, i will request Kyle to admit. Much Thanks!! | 17:28 |
| *** Sukhdev has quit IRC | 17:28 | |
| fungi | clarkb: manually running it is turning up the same error consistently, as expected | 17:28 |
| *** praneshp has joined #openstack-infra | 17:28 | |
| clarkb | fungi: hrm, creds should be .pypirc iirc | 17:28 |
| fungi | yep, and it looks correct | 17:28 |
| fungi | i may try downgrading twine and see where that gets me | 17:29 |
| sdague | clarkb: http://paste.openstack.org/show/82805/ | 17:29 |
| mestery | fungi: +1! I even asked markmcclain to add me, though he's been traveling. | 17:29 |
| fungi | mestery: problem solved now | 17:29 |
| mestery | fungi: You rock man, thanks! | 17:29 |
| fungi | mestery: any time | 17:29 |
| clarkb | fungi: pdb? | 17:30 |
| clarkb | I may give that a go | 17:30 |
| dstufft | clarkb: fungi it's not a bug i'm aware of, but that hardly means much | 17:30 |
| fungi | clarkb: i'm going to stuff older twines in a virtualenv to try to see if it's a regression, so you can pdb the system-installed one all you like | 17:31 |
| clarkb | fungi: why is the source tree checked out there? do we have an unneeded gerrit git prep? | 17:31 |
| fungi | clarkb: i plan to make that a needed ggp at some point soon anyway (so i can compare the tarball to a tag) | 17:32 |
| clarkb | fungi: oh ok I won't propose a change to remove it then :) | 17:32 |
| *** jcoufal has joined #openstack-infra | 17:33 | |
| *** trinaths has left #openstack-infra | 17:33 | |
| dstufft | there hasn't been a new twine in awhile | 17:33 |
| *** jp_at_hp has quit IRC | 17:33 | |
| anteaya | fungi: this is a paste with 6 lines that encompass the format that has come out of my suggestions on renaming gerrit automated accounts: http://paste.openstack.org/show/82807/ | 17:34 |
| *** nati_ueno has joined #openstack-infra | 17:34 | |
| fungi | dstufft: yeah, but we haven't rebuilt the system where we ran it in a while, and may have used an older one | 17:34 |
| *** dangers_away is now known as dangers | 17:34 | |
| dstufft | fungi: gotcha | 17:34 |
| anteaya | fungi: when you get a chance if you can let me know if new accounts can work this this, that would make me happy | 17:34 |
| *** akrivoka has quit IRC | 17:34 | |
| anteaya | s/this this/with this | 17:34 |
| fungi | ...may have used an older one previously (thus wouldn't have noticed at release time) | 17:34 |
| openstackgerrit | Marc Abramowitz proposed a change to openstack-infra/jenkins-job-builder: Some tweaks to get closer to Python 3 compat https://review.openstack.org/87810 | 17:36 |
| clarkb | dstufft: it happens in twine-upload | 17:36 |
| fungi | anteaya: lgtm. as we discussed, we should only enforce the new ssh username format on new accounts (though we should definitely change the display names of all old accounts). the only thing i think might be missing is an upper-bound on the lengths of the username and display name | 17:37 |
| clarkb | and now I have to figure out how to get pdb to run in a subprocess. I guess I can just have pdd run twine-upload direclty | 17:37 |
| dstufft | twine is bad software, but it's less bad tan setup.py upload | 17:37 |
| dstufft | I really need to get around to just making it ``pip publish`` | 17:37 |
| *** harlowja_away is now known as harlowja | 17:39 | |
| anteaya | fungi do you know what those upper bounds are, I do not know | 17:40 |
| *** e0ne has joined #openstack-infra | 17:40 | |
| *** Alexei_987 has joined #openstack-infra | 17:41 | |
| fungi | anteaya: i don't know of any explicit upper bounds in gerrit (probably ~255 bytes allowed in the mysql columns), but we should pick something reasonable which fits the majority of the current systems and doesn't make the ui totally horrible looking | 17:41 |
| anteaya | fungi: I'll see what we have right now | 17:41 |
| fungi | dstufft: clarkb: seems to have been introduced between 1.2.2 and 1.2.3 | 17:42 |
| anteaya | after I eat my sandwich | 17:42 |
| *** e0ne has quit IRC | 17:42 | |
| *** e0ne has joined #openstack-infra | 17:43 | |
| fungi | isviridov: your 2.0.5 release is up on pypi now... https://pypi.python.org/pypi/magnetodb | 17:43 |
| *** masayukig has quit IRC | 17:43 | |
| *** e0ne has quit IRC | 17:43 | |
| fungi | dstufft: clarkb: i was able to upload with 1.2.2 but get that error with 1.2.3 and later | 17:43 |
| *** sarob has joined #openstack-infra | 17:44 | |
| dstufft | apparently i'm a bad person who didn't tag releases | 17:44 |
| fungi | heh | 17:44 |
| openstackgerrit | Sean Dague proposed a change to openstack-infra/os-loganalyze: construct syslog debug level https://review.openstack.org/97863 | 17:44 |
| sdague | clarkb: ^^ is not tested | 17:45 |
| sdague | however, is a proposed idea to address the logs | 17:45 |
| clarkb | dstufft: fields.update({k: (fn, fp.read())}) "/usr/lib/python2.7/dist-packages/requests/models.py", line 376, in send. fp has no read() | 17:45 |
| clarkb | sdague: thanks will look shortly | 17:45 |
| sdague | oh... in the move to hpcloud 1.1 everything is 8 cpu now? | 17:46 |
| dstufft | clarkb: looks like an incompatible requests version? | 17:46 |
| clarkb | dstufft: I don't know enough of requests or twine to sort that out at least not right now | 17:46 |
| clarkb | dstufft: maybe? | 17:46 |
| clarkb | sdague: I guess? | 17:46 |
| dstufft | twine has not been broken for me since 1.2.3, and i use the latest requests whenever I use it | 17:46 |
| sdague | that would explain why the sec group bug is happening more often | 17:47 |
| dstufft | what versionof requests is isntalled | 17:47 |
| *** timrc is now known as timrc-afk | 17:47 | |
| clarkb | dstufft: 0.8.2 | 17:47 |
| dstufft | oh lol | 17:47 |
| dstufft | that's ancient | 17:47 |
| *** lcheng_ has quit IRC | 17:47 | |
| clarkb | dstufft: huh I guess that means something pins requests? | 17:48 |
| *** timrc-afk is now known as timrc | 17:48 | |
| dstufft | that's coming from the OS | 17:48 |
| dstufft | /usr/lib/python2.7/dist-packages | 17:48 |
| dstufft | that's where apt-get installs things | 17:48 |
| *** fanhe has quit IRC | 17:48 | |
| clarkb | oh right | 17:48 |
| fungi | dstufft: clarkb: i get that error with requests (2.3.0) in the virtualenv with twine (1.2.3) and later | 17:48 |
| clarkb | fungi: fun! | 17:49 |
| *** zzelle has quit IRC | 17:49 | |
| fungi | clarkb: try pdb'ing pypi.slave.o.o:~jenkins/twine-1.3.1/bin/twine | 17:49 |
| clarkb | dstufft: https://github.com/dstufft/twine/blob/1.3.1/twine/commands/upload.py#L174 is the twine side of the requests call | 17:49 |
| *** masayukig has joined #openstack-infra | 17:50 | |
| clarkb | fungi: is it a different fail? | 17:50 |
| fungi | clarkb: nope, same error | 17:50 |
| clarkb | I guess the requests code may be differen though /me does that | 17:50 |
| fungi | "AttributeError: 'str' object has no attribute 'read'" | 17:50 |
| Kiall | SergeyLukjanov - if you have a chance, I've updated https://review.openstack.org/#/c/97348/ with you're suggestions (the Designate DevStack job review) :) | 17:51 |
| anteaya | fungi: we have two usernames at 24 characters | 17:51 |
| bodepd | how do I run that spell checker script? ie: where does it live? | 17:52 |
| anteaya | that is also the longest Full Name | 17:52 |
| bodepd | s/spell checker/alpa-checker/ | 17:52 |
| clarkb | dstufft: is it because files=filedata passing a string rather than files? | 17:52 |
| dstufft | I dunno | 17:52 |
| dstufft | doesn't explain why it works for me :| | 17:52 |
| *** _nadya_ has joined #openstack-infra | 17:52 | |
| fungi | clarkb: dstufft: http://paste.openstack.org/show/82814 | 17:53 |
| fungi | dstufft: clarkb: oho! env fail | 17:54 |
| *** Sukhdev has joined #openstack-infra | 17:54 | |
| *** jistr has joined #openstack-infra | 17:55 | |
| fungi | dstufft: clarkb: activating the venv produces different behavior than executing twine directly our of it | 17:55 |
| clarkb | http://docs.python-requests.org/en/latest/user/quickstart/#post-a-multipart-encoded-file there they pass both file objects and strings | 17:55 |
| *** tkelsey has quit IRC | 17:55 | |
| *** ociuhandu has joined #openstack-infra | 17:55 | |
| dstufft | erm | 17:55 |
| dstufft | wat | 17:55 |
| dstufft | I bet this is becuase of my stupid "lol let's do git style dispatch" | 17:55 |
| clarkb | oh right | 17:55 |
| fungi | dstufft: maybe an odd entrypoint behavior? it's definitely not something i'd expect | 17:56 |
| dstufft | it's not entry points | 17:56 |
| clarkb | fungi: twine forks twine-upload | 17:56 |
| clarkb | so the subprocess may derp | 17:56 |
| dstufft | fungi: try to repro with twine-upload instead of twine upload | 17:56 |
| fungi | ooh, and twine-upload may be missing the parent env | 17:56 |
| * fungi tries | 17:56 | |
| fungi | yep! | 17:57 |
| clarkb | that worked? | 17:57 |
| *** andreykurilin_ has joined #openstack-infra | 17:57 | |
| clarkb | if that is happy we can just update requests on that box | 17:57 |
| fungi | so it's the locally-installed requests getting imported by twine-upload when it gets called from twine | 17:57 |
| fungi | calling twine-upload directly from the venv or with the venv activated works | 17:57 |
| clarkb | cool lets just update requests | 17:58 |
| fungi | dstufft: thanks! | 17:58 |
| fungi | clarkb: yes, let's ;) | 17:58 |
| openstackgerrit | Dan Bode proposed a change to openstack-infra/config: Add puppet-openstack project puppet-openstacklib https://review.openstack.org/97357 | 17:58 |
| sdague | fungi / clarkb - how did this get into the gate - https://review.openstack.org/#/c/94813/ ? | 17:58 |
| fungi | dstufft: want a bug for "twine subcommands don't inherit twine's environment"? | 17:58 |
| fungi | sdague: seems to have piggy-backed on the parent 94812 change, via "the SpamapS hole" | 17:59 |
| clarkb | yup | 18:00 |
| sdague | ok, that's another one of those? | 18:00 |
| *** gokrokve has quit IRC | 18:00 | |
| jogo | sdague: look at the top hit in the unclassified gate failrure: http://status.openstack.org/elastic-recheck/data/uncategorized.html hehe | 18:00 |
| dstufft | fungi: that or, "donald is a dolt who has bad ideas and shouldn't have made git style dispatching" | 18:00 |
| sdague | jogo: yeh, so the failure you were tracking isn't the one yuo think | 18:00 |
| sdague | it's not a merge conflict | 18:00 |
| sdague | it's the SpamapS hole | 18:00 |
| jogo | sdague: o_O do tell | 18:01 |
| fungi | you know he's just going to configure his irc client to stop highlighting his nick in this channel any more ;) | 18:01 |
| sdague | he | 18:01 |
| sdague | hehe | 18:01 |
| clarkb | sdague: fungi: going back to slave times. should devstack-gate do ntpdate as the first step in its run? | 18:01 |
| sdague | clarkb: well I was trying to figure out why it seems to happen late here | 18:01 |
| SpamapS | "the SpamapS hole" was my nickname in highschool | 18:02 |
| sdague | jogo: there is some process by which a child change that's approved can pull a bogus parent change into the gate with it | 18:02 |
| SpamapS | shortened it for IRC.. | 18:02 |
| clarkb | also https://jenkins03.openstack.org/job/gate-ceilometer-python27/964/console is cool | 18:02 |
| sdague | honestly, my brain power is kind of drained from diving through the other fixes this morning | 18:03 |
| jogo | sdague: ouch | 18:03 |
| sdague | so I don't fully understand the loophole | 18:03 |
| sdague | jogo: you stable maint? | 18:04 |
| *** e0ne has joined #openstack-infra | 18:04 | |
| jogo | sdague: nope | 18:04 |
| *** sarob has quit IRC | 18:04 | |
| fungi | sdague: jogo: basically when zuul enqueues approved child changes triggered from the enqueuing of a parent change, it happens through a different routine which doesn't pass through the configured criteria | 18:04 |
| sdague | fungi: ah gotcha | 18:04 |
| *** sarob has joined #openstack-infra | 18:04 | |
| jogo | fungi: ahh | 18:05 |
| sdague | yeh, so when we don't have a giant backlog we don't notice it | 18:05 |
| sdague | but with so much in flight | 18:05 |
| fungi | sdague: jogo: and to fix it, that routine needs to have callouts to check that stuff, more or less | 18:05 |
| jogo | yeah, nova has similar issues with live migration | 18:05 |
| jogo | and scheduling | 18:05 |
| clarkb | I am debugging zuul now | 18:05 |
| clarkb | I don't think this is a hard bug to fix | 18:05 |
| fungi | woot! yappi all the way | 18:06 |
| clarkb | (famous last words) | 18:06 |
| clarkb | well I don't need yappi for this | 18:06 |
| clarkb | (famous last works part 2) | 18:06 |
| clarkb | *words | 18:06 |
| *** amcrn has joined #openstack-infra | 18:06 | |
| fungi | clarkb: oh, you said debugging not profiling | 18:06 |
| fungi | you're tracking down the child-enqueuing hole | 18:06 |
| fungi | got it | 18:06 |
| openstackgerrit | A change was merged to openstack-infra/jenkins-job-builder: authorization property: add extended read support https://review.openstack.org/97097 | 18:07 |
| *** esker has joined #openstack-infra | 18:08 | |
| *** praneshp_ has joined #openstack-infra | 18:08 | |
| *** pelix has quit IRC | 18:08 | |
| *** praneshp has quit IRC | 18:09 | |
| *** praneshp_ is now known as praneshp | 18:09 | |
| *** _nadya_ has quit IRC | 18:09 | |
| *** sarob has quit IRC | 18:09 | |
| *** apevec has quit IRC | 18:10 | |
| sdague | ok, need to step away for a bit. | 18:10 |
| fungi | clarkb: sdague: jogo: here i think... http://git.openstack.org/cgit/openstack-infra/zuul/tree/zuul/scheduler.py#n1701 | 18:10 |
| *** e0ne has quit IRC | 18:11 | |
| clarkb | wow transifex spam | 18:11 |
| clarkb | fungi: ya in enqueueChangesBehind | 18:11 |
| clarkb | it checks canMerge but not the pipeline constraints | 18:11 |
| fungi | yep, could trap it there or a few lines later in for other_change in to_enqueue: self.addChange(other_change, quiet) | 18:11 |
| clarkb | arg but the pipeline constraints are event based not change based. this may require more shifting of responsibility than I hoped | 18:15 |
| openstackgerrit | Alex Gaynor proposed a change to openstack-infra/zuul: Simplified the scheduler very slightly https://review.openstack.org/97869 | 18:15 |
| fungi | yeah, i think the infrequency, modest impact and complexity to fix is why this has stuck around as a known issue | 18:16 |
| *** yfried has joined #openstack-infra | 18:16 | |
| clarkb | well we have changeish_filters with approval checks, hrm why isn't this working | 18:21 |
| *** zzelle has joined #openstack-infra | 18:22 | |
| *** nati_uen_ has joined #openstack-infra | 18:23 | |
| fungi | clarkb: do you think we should switch pypi-tarball-upload.sh to just call twine via tox -e venv like we do with setup.py sdist? that would get latest twine and requests and whatever else into play | 18:23 |
| fungi | and then system python-requests deb becomes irrelevant | 18:23 |
| clarkb | we would need to make every project depend on twine | 18:24 |
| clarkb | or I guess you could tox -evenv pip install twine && twine upload | 18:24 |
| fungi | oh, hm... unless we told tox to do that, yeah | 18:24 |
| clarkb | if that works then I think that is a simple way forward | 18:24 |
| clarkb | actually no | 18:24 |
| clarkb | because tox will install the project | 18:24 |
| clarkb | whihc is running arbitrary code | 18:24 |
| clarkb | which is why everything ever is bad | 18:24 |
| fungi | oh, right. poop | 18:24 |
| fungi | so screw tox, just virtualenv directly but otherwise similar story | 18:25 |
| fungi | or does that then introduce new random failures because we're downloading more things? grr... | 18:26 |
| fungi | we need puppet virtualenv manglement | 18:26 |
| *** nati_ueno has quit IRC | 18:26 | |
| fungi | clarkb: do the changeish_filters approval checks just check for approval, or also check for other configured criteria (which are currently only associated with triggers)? | 18:27 |
| fungi | lack of approval isn't the issue. the changes slipping in through this hole are approved with negative or missing check scores | 18:27 |
| clarkb | fungi: its just approval I think | 18:28 |
| fungi | yeah, so that part is working fine | 18:28 |
| clarkb | fungi: basically changeish_filters are the new thing which we are not completely migrated to | 18:28 |
| clarkb | and if we were migrated to them this would work | 18:29 |
| fungi | oh | 18:29 |
| clarkb | but we are still using the old event filters too | 18:29 |
| clarkb | (this is my rudimentary understanding of the problem) | 18:29 |
| clarkb | yeah we depend on require-approval for jenkins votes | 18:29 |
| clarkb | and it is deprecated and only checked on events not changes | 18:30 |
| clarkb | so I will stop working on this now | 18:30 |
| clarkb | the bug will be fixed when we switch to approvals for everything | 18:30 |
| isviridov | fungi, thanks for magnetodb release | 18:30 |
| *** melwitt has joined #openstack-infra | 18:30 | |
| fungi | okay, so completing migration to changeish_filters will in theory solve this hole as a side effect | 18:30 |
| fungi | isviridov: you're welcome. apologies for the delay (you got caught in a chain of several different intertwined bugs there) | 18:31 |
| clarkb | fungi: yup because addChange() checks the changeish_filters | 18:31 |
| bodepd | I sorted the sort stuff. apparently _ comes after [a-z] (surprising...) | 18:32 |
| fungi | clarkb: right, that was begun to solve the testing as a reaction to comments on closed changes problem, and nicely addresses this one as well | 18:32 |
| *** sarob has joined #openstack-infra | 18:32 | |
| clarkb | fungi: I think there is a bug in approvals though | 18:32 |
| clarkb | but we can cross that bridge when we get there | 18:32 |
| fungi | bodepd: yeah, if you set LC_ALL=C in your environment and then run them through the sort utility you can emulate what that job does, hopefully | 18:32 |
| fungi | clarkb: right, that implementation got hurried through to stem the bleeding on the test loop spam problem, but still isn't well tested beyond that one use case | 18:34 |
| fungi | bodepd: basically the sort order for a posix (non-localized) environment follows the ascii byte equivalent ordering | 18:35 |
| fungi | bodepd: oh, except that we set it to a case-insensitive sort, so not entirely | 18:35 |
| fungi | because _ normally falls between A-Z and a-z but i think case-insensitive sort uppercases it all first, which makes _ come after a-z as well | 18:36 |
| openstackgerrit | Dan Bode proposed a change to openstack-infra/config: Add puppet-openstack project puppet-openstacklib https://review.openstack.org/97357 | 18:36 |
| clarkb | well maybe it wasn't a bug | 18:36 |
| bodepd | fungi: that didn't work (to set that to emulate) | 18:37 |
| fungi | bodepd: huh. crazy | 18:37 |
| bodepd | fungi: I just guessed, submitted patches trying both orders | 18:37 |
| bodepd | fungi: and one worked! | 18:37 |
| fungi | and yeah, man sort says "-f, --ignore-case: fold lower case to upper case characters" so that explains why _ comes after a-z | 18:38 |
| bodepd | fungi: ah, I somehow missed the -f | 18:38 |
| clarkb | fungi: no I remember. Approvals are ANDed but require-approvals are OR'd | 18:39 |
| clarkb | fungi: so the mapping isn't quite 1:1 | 18:39 |
| fungi | clarkb: yeah, that'll make translating them somewhat of a challenge | 18:39 |
| clarkb | fungi: yeah especially since in the gate we say a vote of 1 or 2 is valid | 18:40 |
| fungi | clarkb: now you must have a vote of 1 and 2 instead. our code quality can only go up from here! | 18:40 |
| clarkb | fungi: right! | 18:40 |
| Alex_Gaynor | mordred: ping | 18:41 |
| *** jcoufal has quit IRC | 18:42 | |
| mordred_phone | Alex_Gaynor: just boarded a plane ... sup? | 18:43 |
| Alex_Gaynor | mordred_phone: You mentioned the idea of using livegrep as a replication target, instead of needing to pull every 30 minutes. Where can I see some reference code/docs/whatever on how to set that up. Sounds way better than pulling | 18:44 |
| *** denis_makogon_ has joined #openstack-infra | 18:44 | |
| mordred_phone | I think you want to look at our cgit mirrors | 18:45 |
| *** denis_makogon_ is now known as denis_makogon | 18:45 | |
| mordred_phone | pleia2: ^^ | 18:45 |
| mordred_phone | Alex_Gaynor: if you don't have it when I land, I'll send you things | 18:46 |
| Alex_Gaynor | thanks | 18:46 |
| pleia2 | if I understand correctly, we're just using gerrit magic configured in puppet | 18:46 |
| clarkb | sdague: re the os log analyze change. My only concern is that I think those ar ethe only proxy server logs we get | 18:47 |
| clarkb | sdague: because ya | 18:47 |
| pleia2 | or are we talking about something else? | 18:47 |
| Alex_Gaynor | Would that be under the gerrit module then? | 18:47 |
| pleia2 | I'll grab it, sec | 18:47 |
| clarkb | pleia2: I think that + the script to create the repos that cgit uses | 18:47 |
| Alex_Gaynor | thanks | 18:47 |
| clarkb | pleia2: Alex_Gaynor: however, could we just host livegrep on the cgit servers? | 18:48 |
| clarkb | (I don't know what kind of state is needed | 18:48 |
| clarkb | and the cgit servers assume that things are mostly stateless) | 18:48 |
| *** srenatus has quit IRC | 18:48 | |
| Alex_Gaynor | It basically just needs the git repos, it can run in place with no previous prep | 18:48 |
| Alex_Gaynor | You can take a look at the existing puppet module to get a sense of how it works | 18:48 |
| clarkb | oh cool then ya we might be able to just run it on top of the cgit repos | 18:48 |
| *** dprince has quit IRC | 18:48 | |
| *** srenatus has joined #openstack-infra | 18:48 | |
| pleia2 | http://git.openstack.org/cgit/openstack-infra/config/tree/modules/openstack_project/manifests/review.pp#n131 | 18:49 |
| *** e0ne has joined #openstack-infra | 18:49 | |
| clarkb | then you don't haveto worry about how the repos show up | 18:49 |
| pleia2 | ^^ that's where we tell gerrit what to replicate to | 18:49 |
| pleia2 | jeepyb has the script, sec | 18:49 |
| pleia2 | https://git.openstack.org/cgit/openstack-infra/jeepyb/tree/jeepyb/cmd/create_cgitrepos.py | 18:49 |
| clarkb | jogo: for your silent large ops jobs how comfortable are you with just bumping the number now? | 18:50 |
| clarkb | jogo: you must have a reasonable sample at this point | 18:50 |
| lifeless | jogo: hi | 18:51 |
| lifeless | jogo: elastic recheck and nonvoting jobs | 18:51 |
| lifeless | jogo: I haven't checked the review again, thought maybe we needed some realtime bandwidth :) | 18:51 |
| *** arnaud has quit IRC | 18:51 | |
| clarkb | jogo: for the last 24 hours on the non neutron job there are 109 successes to 5 failures | 18:53 |
| *** jistr has quit IRC | 18:55 | |
| *** rcarrillocruz has quit IRC | 19:00 | |
| *** marcoemorais has joined #openstack-infra | 19:02 | |
| clarkb | fungi: for zuul approvals I think we can do something like: https://etherpad.openstack.org/p/zuul-approvals | 19:04 |
| mtreinish | jeblair: https://wiki.openstack.org/wiki/Qa_Infra_Meetup_2014 did you change back without telling anyone? :) | 19:04 |
| fungi | mtreinish: he did. but now he's on a week+long hike into the wilderness, so that you won't be able to interrogate him about it ;) | 19:05 |
| *** reed has quit IRC | 19:06 | |
| *** rcarrillocruz has joined #openstack-infra | 19:06 | |
| clarkb | before I run to grab lunch | 19:06 |
| *** dprince has joined #openstack-infra | 19:07 | |
| clarkb | mtreinish: is there consensus ish on where people are hotelling? | 19:07 |
| clarkb | I don't want to be the ONE person in darmstadt or frankfurt | 19:07 |
| fungi | clarkb: mtreinish: i meant to ask the same | 19:07 |
| mtreinish | I still haven't even really looked for myself yet | 19:07 |
| fungi | knowing someone in the same hotel increases the chances we both make it to the venue successfully via whatever mode of transportation is chosen | 19:08 |
| mtreinish | I was thinking of staying darmstadt though just to make it easier and not having to wake up earlier | 19:08 |
| clarkb | mtreinish: yes I like this plan | 19:08 |
| fungi | that was my preference as well | 19:08 |
| clarkb | also darmstadt appears to have beer++ so bases are covered | 19:08 |
| *** gokrokve has joined #openstack-infra | 19:10 | |
| mtreinish | mkoderer put 2 hotels on the wiki page, both are within a reasonable walking distance | 19:10 |
| fungi | ideal! | 19:11 |
| fungi | also has anyone tried to convince Ajaeger to join us? i think he's only a couple hundred kilometers away, near nuremberg? | 19:12 |
| *** andreaf has joined #openstack-infra | 19:12 | |
| fungi | good opportunity there given he does so much infra-facing automation for the docs team | 19:12 |
| *** andreykurilin_ has quit IRC | 19:13 | |
| mtreinish | btw, before I forget would there be oposition to swapping the qa day with the infra day on Tues.? | 19:13 |
| bknudson | I wonder how this happened -- https://review.openstack.org/#/c/96265/ | 19:14 |
| bknudson | proposal bot got -1 that the requirements don't match | 19:14 |
| fungi | bknudson: laaaaaag | 19:14 |
| bknudson | "recheck no bug" ? | 19:15 |
| bknudson | or is there a laaaaaag bug? | 19:15 |
| *** gokrokve has quit IRC | 19:15 | |
| fungi | bknudson: that was a guess. looking at the log i think it's something more subtle | 19:15 |
| fungi | it's not that it doesn't match, but rather it's rejected for not overlapping with previous requirements | 19:16 |
| fungi | and indeed it does not | 19:16 |
| fungi | the bigger question is how we tunnel ourselves out of this situation... | 19:16 |
| bknudson | ah | 19:16 |
| bknudson | maybe we were slow about merging a previous change | 19:17 |
| fungi | the problem it's attempting to prevent is that sphinx>=1.1.2,<1.2 and sphinx>=1.2.1,<1.3 do not cover any potential versions in common | 19:17 |
| fungi | which makes transitioning from one to the other problematic | 19:17 |
| bknudson | patch set 1 was 05-28 which isn't that long ago. | 19:18 |
| fungi | that new check was only introduced very recently i think, and is possibly being enforced in the wrong place | 19:18 |
| fungi | i thought the intent was to enforce that check on changes to the global-requirements.txt in openstack/requirements, not on individual project reqs changes | 19:19 |
| *** praneshp has quit IRC | 19:20 | |
| fungi | the problem being that when a change to global-reqs is non-overlapping, it leave a window of broken for projects using the restricted mirror since anything behind it in the gate and anything else tested between when it merges and when the mirror updates complete will potentially fail to find viable versions | 19:20 |
| devananda | requirements question and I think folks here might know the answer far better than I | 19:21 |
| bknudson | fungi: I think I remember that discussion from a while back | 19:21 |
| bknudson | so is this going to break the gate for keystoneclient? | 19:22 |
| *** hashar has joined #openstack-infra | 19:22 | |
| devananda | ironic may have an issue with ipmitool <= 1.8.11 and would like to add a warning if that's present, which it is in ubuntu 12.04. the issue is fixed in >= 1.8.12 (which is present in ubuntu 14.04, recent fedoras, etc) | 19:22 |
| bknudson | here's the change in requirements http://git.openstack.org/cgit/openstack/requirements/commit/global-requirements.txt?id=a78540bae70ea75b54ca0aad34fca18dc2ade566 | 19:23 |
| bknudson | sphinx>=1.1.2,<1.2 -> sphinx>=1.2.1,<1.3 | 19:23 |
| bknudson | so those didn't overlap | 19:23 |
| *** otherwiseguy has quit IRC | 19:23 | |
| fungi | enforcement added in https://review.openstack.org/84100 merged 2014-06-01 02:25 utc | 19:24 |
| bknudson | ok, that was recent! | 19:24 |
| fungi | we may want to consider reverting that for now and get it readded to the correct job | 19:24 |
| fungi | sdague: jogo: do you recall the discussion around that? | 19:25 |
| fungi | just want to make sure i'm not completely off base on why this was added where it was | 19:25 |
| fungi | devananda: so what's the question? wrt discovering the ipmitool version installed on the system? | 19:26 |
| devananda | fungi: a) right way to discover that, b) right thing to do if it's old | 19:27 |
| fungi | devananda: or just whether it's okay to depend on a version not in rhel 6 and ubuntu precise? | 19:27 |
| fungi | devananda: if you think you're not going to be integrated prior to rhel 7 (and since ubuntu 14.04 is out), and if nothing else depends on ipmitool besides ironic, seems like just updating ironic's reqs and the global reqs to >= 1.8.12 should be fine | 19:29 |
| SergeyLukjanov | oh, 503 messages in scrollback | 19:30 |
| SergeyLukjanov | Kiall, I'll take a look | 19:30 |
| fungi | devananda: detecting older versions of modules from within python, if you want to still run but present a warning, can be done by importing and then looking in something like .__version__ i think | 19:30 |
| devananda | fungi: sorry, not module -- system package / utility | 19:30 |
| SergeyLukjanov | anteaya, yup, we'd like to rename it to sahara-ci, too lazy to do it ;) | 19:30 |
| fungi | devananda: oh, got it. probably calling ipmitool --version via a shell callout and parsing its stdout, i would guess | 19:31 |
| devananda | fungi: heh, ok. i had imagiend that there might be a cleaner way :) | 19:31 |
| fungi | devananda: since trying to discover it from distro package details is going to get real ugly real fast from a cross-platform perspective | 19:32 |
| devananda | ack | 19:32 |
| *** Sukhdev has quit IRC | 19:33 | |
| *** nati_uen_ has quit IRC | 19:33 | |
| *** esker has quit IRC | 19:35 | |
| *** esker has joined #openstack-infra | 19:35 | |
| SergeyLukjanov | anteaya, I'm here now, google-translating your message re accounts | 19:35 |
| SergeyLukjanov | anteaya, does you mean to provide you a list of new accounts? | 19:36 |
| *** gokrokve has joined #openstack-infra | 19:36 | |
| bknudson | fungi sdague jogo: https://bugs.launchpad.net/openstack-ci/+bug/1326504 | 19:36 |
| uvirtbot | Launchpad bug 1326504 in openstack-ci "Requirements change and check causes proposal failure" [Undecided,New] | 19:36 |
| *** marcoemorais has quit IRC | 19:37 | |
| *** marcoemorais has joined #openstack-infra | 19:38 | |
| *** marcoemorais has quit IRC | 19:38 | |
| *** shivharis has quit IRC | 19:38 | |
| *** reed has joined #openstack-infra | 19:38 | |
| *** Sukhdev has joined #openstack-infra | 19:38 | |
| *** marcoemorais has joined #openstack-infra | 19:38 | |
| *** nati_ueno has joined #openstack-infra | 19:39 | |
| openstackgerrit | Brant Knudson proposed a change to openstack-infra/config: Revert "Checks if requirements overlap between head and branch reqs" https://review.openstack.org/97893 | 19:39 |
| openstackgerrit | Brant Knudson proposed a change to openstack-infra/config: Revert "Checks if requirements overlap between head and branch reqs" https://review.openstack.org/97893 | 19:39 |
| *** esker has quit IRC | 19:40 | |
| *** palar_ has joined #openstack-infra | 19:40 | |
| *** masayukig has quit IRC | 19:40 | |
| *** michchap has quit IRC | 19:40 | |
| *** jp_at_hp has joined #openstack-infra | 19:41 | |
| anteaya | SergeyLukjanov: that could work or are you able to follow this format: | 19:42 |
| anteaya | Ci if it votes or ever might in the future, Bot if it will never ever vote | 19:42 |
| anteaya | Format: | 19:42 |
| anteaya | username --> lowercase, hyphens between words, matches Full Name, -ci or -bot at the end | 19:42 |
| anteaya | Full Name --> capitalize, upper(minimal) and lowercase, matches username, CI, -CI, -ci or Bot, -Bot, -bot at the end | 19:42 |
| anteaya | {company name}-{thing it is testing}-{ci|bot} | 19:42 |
| anteaya | remove test, testing, jenkins, openstack, tempest, user from the names | 19:42 |
| anteaya | paste.o.o is 500'ing | 19:42 |
| Kiall | SergeyLukjanov: thanks :) | 19:42 |
| *** palar has quit IRC | 19:43 | |
| *** marcoemorais has quit IRC | 19:44 | |
| *** marcoemorais has joined #openstack-infra | 19:44 | |
| anteaya | SergeyLukjanov: and fungi asked for an upper bound for length, which I don't know but the longest username and Full Name we have right now is 24 characters | 19:44 |
| SergeyLukjanov | Kiall, it looks ok, but you have a lot of non-voting jobs in gate pipeline, we prefer to avoid it due to the such stuff is totally ignored in gate | 19:44 |
| *** palar_ has quit IRC | 19:44 | |
| Kiall | SergeyLukjanov: yep, I intend to change them to voting as soon as we know they work | 19:45 |
| SergeyLukjanov | anteaya, ack, I'll check acc requests tonight or tomorrow and make a list | 19:45 |
| anteaya | SergeyLukjanov: thanks I appreciate it | 19:45 |
| SergeyLukjanov | Kiall, okay, please, response in CR to avoid new question on it | 19:45 |
| *** masayukig has joined #openstack-infra | 19:45 | |
| Kiall | SergeyLukjanov: Sure | 19:46 |
| anteaya | SergeyLukjanov: I'll let cinder know we are processing requests since we have agreed on a naming format | 19:46 |
| Kiall | SergeyLukjanov: done :) | 19:47 |
| *** e0ne has quit IRC | 19:47 | |
| *** Ryan_Lane has quit IRC | 19:48 | |
| *** esker has joined #openstack-infra | 19:48 | |
| *** Ryan_Lane has joined #openstack-infra | 19:48 | |
| *** resker has joined #openstack-infra | 19:48 | |
| *** sarob has quit IRC | 19:49 | |
| sdague | fungi: reading scrollback | 19:50 |
| SergeyLukjanov | anteaya, just to ack - we agreed to wait for several days to agreed on name pattern? | 19:50 |
| sdague | so the reason we were enforcing overlapping requirements is because of the gate | 19:50 |
| anteaya | SergeyLukjanov: well here we are | 19:50 |
| anteaya | I have a name pattern and am willing and waiting for agreement | 19:50 |
| sdague | fungi: however, that should only be enforcing it on global requirements, not on the projects | 19:51 |
| sdague | so the check seems to be in the wrong place | 19:51 |
| fungi | sdague: okay, that confirms my recollection then | 19:51 |
| anteaya | you and fungi wanted to not adhere to a pattern and create a bunch more accounts | 19:51 |
| *** HenryG has quit IRC | 19:51 | |
| anteaya | which doesn't work for me | 19:51 |
| sdague | because otherwise you leave a wake of broken gate jobs until the mirror updates | 19:51 |
| anteaya | cinder said can you come up with something in 24 hours so you can keep processing accounts, which I have | 19:51 |
| anteaya | and which was reasonable for them to give me that time | 19:52 |
| anteaya | so what are we waiting for? | 19:52 |
| anteaya | if we are waiting for something besides time to elapse, let me know whta it is so I can do it | 19:52 |
| fungi | anteaya: i think the proposed name pattern seems fine to move forward with | 19:52 |
| anteaya | fungi: thanks | 19:52 |
| anteaya | SergeyLukjanov: if you need more, please tell me so I can get it for you | 19:52 |
| *** otherwiseguy has joined #openstack-infra | 19:53 | |
| fungi | sdague: yep, cool. so i guess let's rush bknudson's revert through and then figure out how to get that code transplanted to the correct place | 19:53 |
| *** esker has quit IRC | 19:53 | |
| sdague | fungi: is there a rush on this? | 19:54 |
| sdague | unless it's fixing a gate reset, I don't see the point in the rush. | 19:54 |
| sdague | Because it's going to take 2 days to land | 19:54 |
| sdague | my back of the envelope is that the gate queue is probably really 48hrs deep right now | 19:55 |
| bknudson | there's no rush for me | 19:55 |
| fungi | sdague: well, it's currently broken proposed requirements changes for any project depending on sphinx | 19:56 |
| fungi | which would be... most projects i think | 19:56 |
| fungi | because someone just got a non-overlapping sphinx version bump into global-reqs | 19:57 |
| fungi | causing this to -1 pretty much all the reqs proposal changes which came after | 19:58 |
| bknudson | you want a change to sphinx reqs so they overlap? | 19:58 |
| sdague | bknudson: no | 19:58 |
| clarkb | ok back from lunch | 19:58 |
| *** fbo_away is now known as fbo | 19:58 | |
| sdague | so in the land of new fails - https://jenkins05.openstack.org/job/gate-tempest-dsvm-full/4156/console | 19:58 |
| sdague | 2014-06-04 19:53:07.289 | sudo: /usr/sbin/apache2ctl: command not found | 19:59 |
| sdague | hp region b | 19:59 |
| fungi | sdague: infrequent, or common? | 19:59 |
| clarkb | sdague: devstack should install that if it needs it | 20:00 |
| clarkb | sdague: and iirc the job I saw that did that had a huge lag time between running devstack and that error | 20:00 |
| *** fifieldt_ has joined #openstack-infra | 20:01 | |
| sdague | fungi: searcing | 20:01 |
| *** eharney has quit IRC | 20:01 | |
| sdague | so it's a lot - http://logstash.openstack.org/#eyJzZWFyY2giOiJcIi91c3Ivc2Jpbi9hcGFjaGUyY3RsOiBjb21tYW5kIG5vdCBmb3VuZFwiIiwiZmllbGRzIjpbXSwib2Zmc2V0IjowLCJ0aW1lZnJhbWUiOiI2MDQ4MDAiLCJncmFwaG1vZGUiOiJjb3VudCIsInRpbWUiOnsidXNlcl9pbnRlcnZhbCI6MH0sInN0YW1wIjoxNDAxOTEyMDY4MDA0fQ== | 20:01 |
| *** apevec has joined #openstack-infra | 20:01 | |
| *** apevec has joined #openstack-infra | 20:01 | |
| sdague | but it's not always a failure | 20:01 |
| fungi | looking through the devstack log for that one now... i'm wondering whether it's failing to install apache2? | 20:02 |
| clarkb | fungi: right I think sudo: /usr/sbin/apache2ctl: command not found is just noise | 20:03 |
| fungi | ahh | 20:03 |
| *** nati_ueno has quit IRC | 20:04 | |
| fungi | yep! http://logs.openstack.org/76/79776/15/gate/gate-tempest-dsvm-full/700ee7e/logs/devstacklog.txt.gz#_2014-06-04_19_53_49_918 | 20:04 |
| *** hogepodge has joined #openstack-infra | 20:04 | |
| fungi | it died because curl -O https://pypi.python.org/packages/source/p/pip/pip-1.4.1.tar.gz threw err_trap | 20:04 |
| *** sabari` has joined #openstack-infra | 20:04 | |
| *** fifieldt-afk has quit IRC | 20:04 | |
| fungi | judging from the log | 20:04 |
| *** sabari` is now known as sabari | 20:05 | |
| clarkb | oh fun, why is that curling pypi | 20:05 |
| clarkb | install pip should use the cached file on disk | 20:05 |
| fungi | because /opt/stack/new/devstack/files/pip-1.4.1.tar.gz is missing | 20:05 |
| clarkb | right but we shouldn't use a pip sdist at all | 20:05 |
| clarkb | we should use that crazy self extracting zip file that dstufft has hosted and we cache | 20:06 |
| fungi | right now it's coming from the install_pip_tarball function in devstack | 20:06 |
| *** nati_uen_ has joined #openstack-infra | 20:07 | |
| fungi | oic, the machine system has pip 1.5.6 installed, so devstack decides to uninstall it because it wants 1.4.1 | 20:07 |
| clarkb | oh fun | 20:07 |
| fungi | hilariously, it finds pip-installed pip 1.5.6, freaks out, tries to apt-get remove it... | 20:08 |
| fungi | then goes on to look for a local tarball of 1.4.1 and then, not finding it, downloads one from pypi | 20:09 |
| *** adalbas has quit IRC | 20:09 | |
| clarkb | sdague: ^ is that something you can untangle? I am guessing 1.5.6 comes from the pip zip thing | 20:09 |
| *** nati_ueno has joined #openstack-infra | 20:09 | |
| *** nati_uen_ has quit IRC | 20:10 | |
| fungi | my guess is that this is legacy cruft from trying to UPGRADE pip on systems with too-old distro packages of it | 20:10 |
| fungi | but instead of looking to see if the version of pip is new enough, it just flattens to 1.4.1 or not 1.4.1 | 20:10 |
| sdague | clarkb: any chance I can get hooked up with an hp region b account? I might need to poke. | 20:11 |
| fungi | which would explain the misguided assumption that the distro package tools are removing the non-conforming pip version | 20:11 |
| openstackgerrit | Matt Riedemann proposed a change to openstack-infra/elastic-recheck: Add query for neutron fwaas bug 1326510 https://review.openstack.org/97907 | 20:11 |
| uvirtbot | Launchpad bug 1326510 in neutron "tempest.api.network.test_fwaas_extensions.FWaaSExtensionTestJSON fails with 409" [Undecided,New] https://launchpad.net/bugs/1326510 | 20:11 |
| clarkb | sdague: I can spin up nodes if nothing else | 20:11 |
| sdague | fungi: so the fact that it crokes is weird | 20:11 |
| *** marcoemorais has quit IRC | 20:12 | |
| *** denis_makogon has quit IRC | 20:12 | |
| *** marcoemorais has joined #openstack-infra | 20:12 | |
| fungi | sdague: the croaking/choking looks like maybe a network failure impacting connection from the slave to pypi.python.org | 20:12 |
| fungi | hard to tell without additional detail from curl | 20:12 |
| fungi | but it definitely didn't finish downloading | 20:13 |
| bknudson | I'm not too familiar with the infra config and how it works, but I would expect that to check something for requirements, there should be a separate job for requirements | 20:13 |
| bknudson | so the fix would be to write a new script in modules/openstack_project/files/slave_scripts/ for the code that was added to project-requirements-change.py to verify the nonoverlapping versions | 20:14 |
| sdague | fungi: so nodepool isn't always installing the 1.4.1 tarball? | 20:14 |
| jogo | clarkb: http://jogo.github.io/gate/large-ops-testing.html I am comfortable bumping it if you are. | 20:15 |
| bknudson | and then have a job in requirements to run that new script | 20:15 |
| fungi | bknudson: yeah, or add it to its integration test or to its python unit tests or something, but probably separate job would make most sense | 20:15 |
| jogo | clarkb: i'll push up a patch now | 20:16 |
| jogo | lifeless: pong | 20:16 |
| clarkb | sdague: it is never installing the tarball | 20:16 |
| bknudson | fungi: ah, I'll check for an existing script I can add it to | 20:16 |
| clarkb | sdague: nodepool installs a self extracting zip of pip | 20:16 |
| sdague | clarkb: I mean caching it | 20:16 |
| fungi | bknudson: well, i think that's how we got in this mess ;) | 20:16 |
| clarkb | sdague: yes caching. it caches a different thing | 20:16 |
| clarkb | sdague: and I believe that is where 1.5.6 comes from | 20:16 |
| fungi | bknudson: the integration test is actually running a script out of the openstack-dev/pbr repo (which is sort of strange but for historical reasons), or the unit tests are running things out of the openstack/requirements repo via tox | 20:17 |
| clarkb | sdague: I think there may possibly be a misunderstanding between devstack and nodepool on what should be cached | 20:17 |
| sdague | ok, so install_pip.sh is currently extremely specific on requested version. Think we should just set a floor? | 20:17 |
| clarkb | sdague: I think so | 20:17 |
| sdague | https://github.com/openstack-dev/devstack/blob/master/tools/install_pip.sh#L30 | 20:17 |
| openstackgerrit | Khai Do proposed a change to openstack-infra/jenkins-job-builder: refactor cmd processor to make it slightly easier to test https://review.openstack.org/97848 | 20:18 |
| clarkb | sdague: https://github.com/openstack-dev/devstack/blob/master/tools/install_pip.sh#L47 is what we cache | 20:18 |
| fungi | bknudson: i think maybe a new script in the openstack/requirements repo which can get called as a tox env would be a fairly simple solution, and more easily runnable for devs too | 20:18 |
| clarkb | sdague: so maybe use-get-pip should be default or d-g should set what ever flag is needed for that | 20:18 |
| bknudson | fungi: that makes sense, and sounds pretty easy | 20:19 |
| *** hartsocks has left #openstack-infra | 20:19 | |
| sdague | clarkb: then I'm confused | 20:19 |
| sdague | because if get-pip.py is there it should use that | 20:19 |
| sdague | or you think we're missing the flag? | 20:19 |
| openstackgerrit | Khai Do proposed a change to openstack-infra/jenkins-job-builder: tests for JJB command processing https://review.openstack.org/97908 | 20:19 |
| fungi | bknudson: that way if we add it to the default env list for tox.ini in the reqs project, they can just run 'tox' and see results | 20:19 |
| clarkb | sdague: yes I think missing the flag | 20:19 |
| gilliard | 3 | 20:19 |
| sdague | clarkb: I'm also happy to change the devstack default here if we think that's the better default now | 20:20 |
| fungi | gilliard: i see your 3 and raise you 2 | 20:20 |
| sdague | actually, is there any reason not to use get-pip? | 20:20 |
| clarkb | dstufft: ^ | 20:20 |
| clarkb | sdague: the one issue may be that it is a self extracting zip file | 20:20 |
| clarkb | which may be scary to some people | 20:20 |
| *** mwagner_lap has quit IRC | 20:20 | |
| sdague | that is by far the least scary thing in devstack | 20:20 |
| gilliard | fungi: you win | 20:21 |
| clarkb | sdague: :) | 20:21 |
| clarkb | sdague: it is hosted on a reliable CDN now and so on | 20:21 |
| clarkb | sdague: I think its pretty safe to use | 20:21 |
| sdague | ok, let me propose, at least to get check results. I would like dstufft's pov on it | 20:21 |
| clarkb | ++ | 20:21 |
| anteaya | gilliard: he's bluffing, you should have called | 20:22 |
| openstackgerrit | Joe Gordon proposed a change to openstack-infra/config: Bump LARGE_OPS number from 50=>100 https://review.openstack.org/97910 | 20:22 |
| jogo | clarkb sdague dansmith: ^ | 20:22 |
| fungi | anteaya sees through my poker face | 20:23 |
| jogo | clarkb: are we using HP cloud 1.1 now? | 20:23 |
| jogo | clarkb: because if not we should leave the large-ops testing around for a bit longer | 20:23 |
| fungi | jogo: yes, for large definitions of "using" | 20:23 |
| fungi | jogo: we haven't been running any jobs on hpcloud 1.0 all week | 20:23 |
| anteaya | we have been exersising hpcloud 1.1 | 20:24 |
| anteaya | think hamster wheel | 20:24 |
| clarkb | jogo: we are | 20:24 |
| openstackgerrit | A change was merged to openstack-infra/elastic-recheck: Add query for neutron fwaas bug 1326510 https://review.openstack.org/97907 | 20:24 |
| uvirtbot | Launchpad bug 1326510 in neutron "tempest.api.network.test_fwaas_extensions.FWaaSExtensionTestJSON fails with 409" [Undecided,New] https://launchpad.net/bugs/1326510 | 20:24 |
| jogo | so large-ops-testing has run on there? | 20:24 |
| lifeless | jogo: o/ | 20:25 |
| fungi | jogo: probably a bunch of times, yes | 20:25 |
| openstackgerrit | Khai Do proposed a change to openstack-infra/jenkins-job-builder: tests for JJB command processing https://review.openstack.org/97908 | 20:25 |
| clarkb | fungi: don't forget we need to apply jeblair's nodepool changes and restart nodepool | 20:25 |
| clarkb | fungi: have those changes merged yet? /me looks | 20:25 |
| fungi | clarkb: right, and i think puppet is still disabled there too | 20:26 |
| * jogo checks logstash | 20:26 | |
| jogo | what is the naming scheme for hpcloud 1.1 nodes? | 20:26 |
| fungi | jogo: devstack-precise-hpcloud-b\d-[\d]+ | 20:26 |
| lifeless | jogo: so yeah, elastic-recheck and tripleo jobs - which are all in a nonvoting queue because $reasons | 20:26 |
| clarkb | https://review.openstack.org/#/c/97570/2 it merged | 20:26 |
| lifeless | jogo: but all the tripleo reviewers treat them as voting | 20:27 |
| clarkb | so making sure nodepool puppet is happy first then planning to restart nodepool can happen | 20:27 |
| dstufft | clarkb: sdague get-pip.py should be no different than doing pip install pip | 20:27 |
| jogo | lifeless: so the patch that landed just looks for " (non-voting)" in the jenkins comment | 20:27 |
| lifeless | jogo: which is odd and frustrating but thats what it is | 20:27 |
| jogo | and I don't think the tripleo jobs have that tag | 20:27 |
| dstufft | so if you're OK with pip installing into wherever, it's fine to use | 20:27 |
| fungi | jogo: the integer after the "b" is one of five separate precreated neutron nets we've divvied it up into, so as to account for the network size being a /24 | 20:27 |
| jogo | fungi: so devstack-precise-hpcloud-b2-65572 | 20:27 |
| sdague | dstufft: for devstack, it's probably fine | 20:27 |
| fungi | jogo: like that, right | 20:28 |
| clarkb | jogo: hpcloud-b* | 20:28 |
| sdague | like I said, it's really one of the least scary bits in devstack | 20:28 |
| fungi | s/devstack/blowawayyoursystemstack/ | 20:29 |
| jogo | so in last 48 hours: 16 failed large-ops-tseting | 20:29 |
| gilliard | ugh - terrible connectivity on the train. We believe hpcloud 1.1 issues to be undercontrol now. Thanks for the firedrill everyone. | 20:30 |
| gilliard | It's been a fun week. | 20:30 |
| jogo | and 521 successfull :) | 20:30 |
| fungi | gilliard: thanks for sticking with it! | 20:30 |
| jogo | sound look good numbers to me | 20:30 |
| virmitio | question regarding Jenkins Job Builder. I find in the docs that I can specify 'block-upstream: true' to wait on upstream jobs, but how do I specify an upstream job? | 20:30 |
| jogo | and failed are on both rax and HP 1.1 | 20:30 |
| clarkb | gilliard: yes thank you for hanging in there | 20:30 |
| gilliard | I will pass on your thanks to the team :) | 20:31 |
| jogo | clarkb: don't know if I have told you enough how awesome logstash.o.o is | 20:31 |
| clarkb | jogo: :) | 20:31 |
| *** jamielennox is now known as jamielennox|away | 20:31 | |
| fungi | virmitio: zaro or hashar might know the answer... hopefully one of them is around | 20:32 |
| sdague | fungi: so the invalid timeout bug is #8 on the ER list (the top couple are on non voting jobs, so not really an issue) | 20:33 |
| sdague | thoughts on how we should fix that? | 20:33 |
| fungi | sdague: yeah, i guess clarkb's ntpdate suggestion has merit | 20:33 |
| sdague | do we trust that it's there? | 20:33 |
| clarkb | I am clearly biased but ++ | 20:33 |
| fungi | sdague: also, i should open a ticket with rackspace to let them know they're misconfigured one or more of their nova compute nodes | 20:34 |
| clarkb | sdague: I believe our base node stuff installs ntp machinery | 20:34 |
| clarkb | sdague: so kicking it with ntpdate should make it happy | 20:34 |
| fungi | clarkb: i think ntpdate comes in a separate package | 20:34 |
| fungi | checking | 20:34 |
| clarkb | oh | 20:34 |
| *** dangers is now known as dangers_away | 20:34 | |
| hashar | I am there (virmitio zaro fungi) | 20:34 |
| *** eharney has joined #openstack-infra | 20:35 | |
| sdague | oh, well one of the failed jobs would tell us | 20:35 |
| sdague | fungi: it's a different package on 14.04 | 20:35 |
| fungi | hashar: virmitio has a question about defining jenkins "upstream" jobs in jjb (related to the block-upstream option) | 20:35 |
| sdague | I don't have 12.04 handy | 20:35 |
| fungi | clarkb: sdague: looks like it's a separate package but installed | 20:36 |
| hashar | virmitio: block-upstream block-downstream are generic Jenkins jobs options. They are only used when a job is triggered by another job for example when using the publisher 'trigger' http://ci.openstack.org/jenkins-job-builder/publishers.html#publishers.trigger | 20:36 |
| sdague | fungi: ok, lets risk it | 20:37 |
| fungi | i think it might be preinstalled on rax | 20:37 |
| sdague | fungi: what ntp server are we going to ask for? | 20:37 |
| fungi | because it doesn't look like ntp depends on it, and i don't see it mentioned in our config repo | 20:38 |
| openstackgerrit | Evgeny Fadeev proposed a change to openstack-infra/askbot-theme: fixed ask button on tags and other pages https://review.openstack.org/97920 | 20:38 |
| clarkb | fungi: we use a third party ntp module iirc | 20:38 |
| clarkb | one of pabelangers | 20:38 |
| zaro | hashar: i'm thinking about making a point release of jjb to pypi to fix the missing info on pypi. | 20:38 |
| hashar | virmitio: as an example you can have a job A triggering a job B and want to make sure job A does not run again until the child (B) is finished. You would set job A with block-downstream. | 20:38 |
| zaro | hashar: would like your help to get 2 patches thru to fix that. | 20:39 |
| hashar | zaro: ah sure! | 20:39 |
| virmitio | hashar: Thank you, much appreciated. That wasn't clear when just hunting through the docs on my own. | 20:39 |
| hashar | zaro: sorry I had a strike of reviews a month ago and have been lagging since then. I can surely get those two patches reviewed/checked. | 20:39 |
| sdague | clarkb / fungi - ntp server name to use? or should I grep it out of /etc/ntp.conf? | 20:40 |
| hashar | virmitio: note that you can set A with block-downstream (i.e. wait for B) or set B with block-upstream (i.e. A will wait for B). | 20:40 |
| zaro | virmitio: patches to make the docs more clear would be welcome :) | 20:40 |
| hashar | zaro: what are the tests? I can do that tomorrow morning (europe time) | 20:40 |
| fungi | sdague: clarkb: puppetlabs-ntp 0.2.0 apparently | 20:40 |
| fungi | digging into it now to see whether it's pulling in ntpdate | 20:41 |
| clarkb | sdague: I think you can grep | 20:41 |
| clarkb | hrm though if the daemon is running ntpdate won't update the time | 20:41 |
| clarkb | and we should have the daemon running | 20:42 |
| fungi | sdague: looks like we just use [0123].debian.pool.ntp.org on the current nodes | 20:42 |
| notmyname | mordred_phone: there is a _lot_ more to a specs repo than I expected (I'll leave the "why" questions for later). if you've already got cookiecutter available, can you set it up? | 20:42 |
| virmitio | hashar: I'm afraid your last comment sounds like it conflicts with the docs. Should me placing block-upstream on B cause B to wait for A to next complete? | 20:42 |
| openstackgerrit | A change was merged to openstack-infra/askbot-theme: fixed ask button on tags and other pages https://review.openstack.org/97920 | 20:43 |
| hashar | virmitio: yeah maybe I get it wrong :-/ | 20:44 |
| openstackgerrit | Sean Dague proposed a change to openstack-infra/devstack-gate: force ntpdate before we start the timer https://review.openstack.org/97924 | 20:44 |
| virmitio | hashar: I guess I'll try it and find out. | 20:44 |
| clarkb | I guess that change will test if it works | 20:44 |
| sdague | yep | 20:44 |
| zaro | hashar: ohh just i noticed that you've already approved the fix already. ok new point release should be out soon. | 20:44 |
| hashar | zaro: wonderful! | 20:45 |
| fungi | clarkb: no mention of ntpdate in https://github.com/puppetlabs/puppetlabs-ntp | 20:45 |
| sdague | ok, I'm about to end my day as our lug meeting starts in an hour or so. | 20:45 |
| *** whayutin_ has joined #openstack-infra | 20:45 | |
| clarkb | sdague: have fun | 20:45 |
| clarkb | fungi: huh I wonder if maybe it is in the base image then? | 20:45 |
| sdague | I bet it's in the base image | 20:45 |
| fungi | clarkb: sdague: that's what i'm worried about | 20:46 |
| hashar | zaro: you proposed the removal of a misleading example. Might as well land it in https://review.openstack.org/#/c/96309/ | 20:46 |
| fungi | clarkb: sdague: because if it is, this may insta-break on !rax for example | 20:46 |
| sdague | fungi: well we can add it to local policy | 20:46 |
| zaro | hashar: would be good to get these 2 doc changes in, 96309 and 96279 | 20:46 |
| clarkb | yeah easy to add to puppet | 20:47 |
| sdague | d-g has enough jobs that we should see hp runs on it | 20:47 |
| fungi | sdague: yeah, i think we probably should, and to avoid waiting for image updates, you could make 97924 conditionally opportunistic | 20:47 |
| sdague | clarkb / dstufft - if you want to have an opinion on this - https://review.openstack.org/#/c/97912/ would be appreciated | 20:47 |
| clarkb | looking | 20:47 |
| dstufft | oh I love opinions | 20:47 |
| sdague | that's the get-pip one | 20:48 |
| *** dkliban is now known as dkliban_brb | 20:48 | |
| *** weshay has quit IRC | 20:48 | |
| *** ArxCruz_ has quit IRC | 20:49 | |
| *** sarob has joined #openstack-infra | 20:49 | |
| *** eharney has quit IRC | 20:50 | |
| zaro | hashar: you read my mind, would you mind taking a look at those 2? | 20:50 |
| *** eharney has joined #openstack-infra | 20:50 | |
| hashar | zaro: I have approved 96279 which removes the sample. That is nice. | 20:50 |
| *** tcammann has joined #openstack-infra | 20:50 | |
| hashar | zaro: the other 96309 I have to try it out. | 20:50 |
| openstackgerrit | Khai Do proposed a change to openstack-infra/jenkins-job-builder: tests for JJB command processing https://review.openstack.org/97908 | 20:50 |
| hashar | zaro: was introduced with https://review.openstack.org/#/c/43013/ so I guess the example used to be correct at one point | 20:51 |
| openstackgerrit | A change was merged to openstack-infra/jenkins-job-builder: reference tests as examples https://review.openstack.org/96279 | 20:52 |
| hashar | zaro: I will do the review tomorrow and let you know. | 20:53 |
| *** harlowja has quit IRC | 20:54 | |
| *** harlowja has joined #openstack-infra | 20:54 | |
| *** dizquierdo has joined #openstack-infra | 20:54 | |
| *** mbacchi has quit IRC | 20:54 | |
| zaro | hashar: i believe 43013 added the ability to do subsitution on job lists (i.e. - '{name}-build':) but not on the parameters of the list (i.e. pipeline-next: '{name}-upload') | 20:55 |
| *** tcammann has quit IRC | 20:55 | |
| zaro | hashar: anyways, please take your time. Not in a hurry to do a release. | 20:56 |
| *** tcammann has joined #openstack-infra | 20:56 | |
| *** tcammann has quit IRC | 20:57 | |
| *** masayukig has quit IRC | 20:57 | |
| jogo | sdague: one last review before I am ready to cut hacking 0.9 https://review.openstack.org/#/c/97580/ | 20:58 |
| *** lcostantino has quit IRC | 20:58 | |
| *** pdmars has quit IRC | 20:58 | |
| openstackgerrit | Antoine Musso proposed a change to openstack-infra/zuul: Make swiftclient an optional dependency https://review.openstack.org/97933 | 20:59 |
| jogo | dhellmann: ^ | 21:00 |
| jogo | dhellmann: never mind you already reviewed it. fungi ^ | 21:01 |
| *** marcoemorais has quit IRC | 21:01 | |
| *** marcoemorais has joined #openstack-infra | 21:02 | |
| openstackgerrit | Khai Do proposed a change to openstack-infra/jenkins-job-builder: tests for JJB command processing https://review.openstack.org/97908 | 21:02 |
| *** e0ne has joined #openstack-infra | 21:02 | |
| *** marcoemorais has quit IRC | 21:03 | |
| *** e0ne has quit IRC | 21:03 | |
| *** eharney has quit IRC | 21:03 | |
| sdague | lifeless: re jogo's er change, I think it was more limitted than you realized. It only impacts the real time reporting, not the bulk analysis | 21:03 |
| *** masayukig has joined #openstack-infra | 21:03 | |
| sdague | and the reason for the real time reporting is to tell people the bug unrelated to their code that gave them a -1 | 21:03 |
| sdague | which by definition non-voting jobs can't do | 21:03 |
| *** marcoemorais has joined #openstack-infra | 21:03 | |
| *** marcoemorais has joined #openstack-infra | 21:04 | |
| hashar | zaro: I will take my time. Probably add some more tests to exercise the code while I look at it hehe. | 21:04 |
| jogo | lifeless: yeah, and we can always add whitelists of jobs if needed etc | 21:05 |
| *** mfer has quit IRC | 21:05 | |
| sdague | jogo: well more importantly, I don't think there is ever a valid use of recheck bug for a non voting job | 21:05 |
| sdague | because it didn't -1 you | 21:05 |
| lifeless | sdague: so lets split the discussion in two | 21:06 |
| jogo | sdague: so last I looked because tripleo jobs use a different comment, they don't have the non-voting tag in them | 21:06 |
| *** mfer has joined #openstack-infra | 21:06 | |
| lifeless | sdague: if it looks for (non-voting) its fine, because we have non-voting non-voting jobs | 21:06 |
| lifeless | sdague: *and* non-voting voting jobs | 21:06 |
| lifeless | sdague: the latter, if it fails, its equivalent to a -1 in tripleo. | 21:07 |
| sdague | lifeless: is it equivalent because of people, or is jenkins enforcing it? | 21:07 |
| lifeless | sdague: If I was allowed to, it would vote, and we wouldn't have this friction. | 21:07 |
| sdague | lets step back further | 21:07 |
| lifeless | sdague: its equivalent by review policy, because we're not allowed to make the jobs voting | 21:07 |
| *** amcrn_ has joined #openstack-infra | 21:07 | |
| sdague | recheck exists for 1 reason | 21:07 |
| sdague | jenkins has blocked my code, and I feel it did so incorrectly | 21:08 |
| sdague | because you can't reason with jenkins | 21:08 |
| sdague | he's kind of a douche | 21:08 |
| lifeless | yup | 21:08 |
| lifeless | and our reviewers have a mini-jenkins in their head that says 'if jenkins says a tripleo job failed, don't merge it' | 21:08 |
| sdague | so only in place where jenkins is the one blocking your code is recheck bug ### really a valid thing | 21:08 |
| sdague | lifeless: ok, but you can reason with them, right? | 21:09 |
| *** maxbit has joined #openstack-infra | 21:09 | |
| *** amcrn has quit IRC | 21:09 | |
| *** _nadya_ has joined #openstack-infra | 21:09 | |
| lifeless | sdague: you can certainly argue with them | 21:10 |
| sdague | :) | 21:10 |
| *** resker has quit IRC | 21:11 | |
| sdague | well, I really don't want us solving the problem of virtual jenkins in people's heads. Because there is actually a solution there, people being reasonable | 21:11 |
| *** mfer has quit IRC | 21:11 | |
| jogo | this sounds a bit like we are going in a circle | 21:12 |
| sdague | the solution without recheck and jenkins is having to upload another patch that changes some whitespace | 21:12 |
| fungi | there is an entire profession focused on solving the problems of virtual things in people's heads, in fact | 21:12 |
| sdague | :) | 21:12 |
| *** sarob_ has joined #openstack-infra | 21:12 | |
| fungi | also a sharp blow with a blunt instrument can often sort that out | 21:12 |
| anteaya | or cause it | 21:13 |
| * fungi follows the flintstones school of psychology | 21:13 | |
| *** _nadya_ has quit IRC | 21:14 | |
| hashar | if tripleo are non voting but should be considered maybe make them voting or highlight them in <b></b> ? | 21:14 |
| morganfainberg | jeblair, i keep almost selecting the wrong review option in gertty because -1/-2 are at the top, not the bottom (used to the web interface). not sure if there is any real need to change that... or you know... I might need to l2readbetter :P | 21:14 |
| openstackgerrit | German Eichberger proposed a change to openstack-infra/gear: adds code to not block when using eventlet (incorporated Clark's suggestions) https://review.openstack.org/97533 | 21:14 |
| fungi | morganfainberg: proposing a patch might get more traction. jeblair's out for a week or more | 21:15 |
| *** eharney has joined #openstack-infra | 21:15 | |
| morganfainberg | fungi, yeah.. not sure if i think it should be changed. | 21:15 |
| fungi | otherwise, you probably want to leave yourself a reminder to ask again | 21:15 |
| morganfainberg | fungi, is the point.... | 21:15 |
| *** sarob has quit IRC | 21:15 | |
| morganfainberg | fungi, i might just stew on it and see how many times i make the same mistake :P | 21:15 |
| fungi | seems reasonable. i'm not convinced that replicating gerrit's ui design choices is a great way to make an application more usable | 21:16 |
| morganfainberg | fungi, I tend to agree... | 21:16 |
| sdague | fungi: though being gratuitous in changing them also makes it hard for people to use things situationaly | 21:17 |
| *** dkliban_brb is now known as dkliban | 21:18 | |
| *** ArxCruz has joined #openstack-infra | 21:18 | |
| morganfainberg | woohoo. just crashed gerrty | 21:18 |
| * morganfainberg tries to figure out why now. | 21:19 | |
| *** derekh_ has joined #openstack-infra | 21:20 | |
| anteaya | ArxCruz: hey | 21:21 |
| anteaya | ArxCruz: I need some of your time, let me know when you are available | 21:21 |
| fungi | morganfainberg: using the tip of master? jeblair just merged a bunch of fixes right before he left | 21:22 |
| morganfainberg | fungi, checking that now | 21:22 |
| fungi | morganfainberg: in particular there were some rendering bugs which would cause it to jettison the reactor core | 21:22 |
| morganfainberg | fungi, yep, still crashing. - KeyError: /COMMIT_MSG | 21:23 |
| morganfainberg | fungi, /me goes and pokes at it | 21:23 |
| *** ArxCruz has quit IRC | 21:23 | |
| *** marcoemorais has quit IRC | 21:23 | |
| *** marcoemorais has joined #openstack-infra | 21:23 | |
| openstackgerrit | A change was merged to openstack-dev/hacking: Update localization checks to understand separate catalogs https://review.openstack.org/97580 | 21:24 |
| hashar | fungi: morganfainberg any clue where to report gertty bugs beside james? | 21:24 |
| *** smarcet has quit IRC | 21:25 | |
| morganfainberg | hashar, propose a fix in gerrit for it? | 21:25 |
| hashar | NotImplementedError: op.f() feature requires SQLAlchemy 0.9.4 or greater. | 21:26 |
| hashar | :D | 21:26 |
| fungi | hashar: https://storyboard.openstack.org/#!/project/698 | 21:26 |
| hashar | some weir dependency issue hehe | 21:26 |
| morganfainberg | fungi, ah storyboard! | 21:26 |
| *** nati_ueno has quit IRC | 21:26 | |
| hashar | awesome | 21:26 |
| fungi | though i like morganfainberg's idea. i need a bug tracker which only lets you open a bug by proposing a patch ;) | 21:27 |
| morganfainberg | fungi, totally want to do that now! | 21:27 |
| sdague | fungi: um, isn't that just gerrit | 21:28 |
| fungi | sdague: that was the quiet joke there, yes | 21:28 |
| *** eharney has quit IRC | 21:29 | |
| hashar | fungi: yeah that is named Gerrit isn't it ? :-D | 21:31 |
| *** julim has quit IRC | 21:32 | |
| openstackgerrit | Joe Gordon proposed a change to openstack/requirements: Bump pep8 from 1.5.6 to 1.5.7 https://review.openstack.org/97944 | 21:35 |
| *** marcoemorais1 has joined #openstack-infra | 21:36 | |
| jogo | sdague: random question: do you think its worth holding off cutting hacking 0.9 until we switch over to pep8 1.5.7? | 21:37 |
| jogo | pep8 changelog https://pypi.python.org/pypi/pep8 | 21:37 |
| *** Sukhdev has quit IRC | 21:38 | |
| jogo | fungi: ^ | 21:38 |
| jogo | I am thinking its worth it | 21:38 |
| jogo | (waiting that is) | 21:38 |
| fungi | jogo: i really have just about zero opinion on pep8 1.5.7. is 1.5.6 broken, or is this just the endless march of progress? | 21:39 |
| jogo | fungi: a few bug fixes | 21:40 |
| *** andreaf has quit IRC | 21:40 | |
| *** marcoemorais1 has quit IRC | 21:40 | |
| *** marcoemorais has quit IRC | 21:40 | |
| jogo | pep8 1.5 was extremely buggy | 21:40 |
| jogo | not surprisingly | 21:40 |
| *** andreykurilin_ has joined #openstack-infra | 21:41 | |
| jogo | fungi: the main bug that cought my eye is: Check the last line even if it does not end with a newline. | 21:42 |
| *** hashar has quit IRC | 21:42 | |
| *** eharney has joined #openstack-infra | 21:43 | |
| fungi | ha | 21:43 |
| fungi | okay, that's warranted | 21:43 |
| *** radez is now known as radez_g0n3 | 21:45 | |
| jogo | fungi: oh never mind flake raises a differnt error in that case | 21:46 |
| mikal | Hello... Where does the openstack governance RST stuff get published to? | 21:46 |
| jogo | so we cannot have any of those in our code base | 21:46 |
| jogo | mikal: this isn't the answer but https://github.com/openstack/governance/blob/master/resolutions/20131106-ceilometer-and-heat-official-names.rst ;) | 21:47 |
| anteaya | mikal: I don't know that is is currently published anywhere | 21:48 |
| fungi | mikal: eventually www.openstack.org | 21:48 |
| fungi | mikal: currently git.openstack.org | 21:48 |
| jogo | just like specs aren't published yet either | 21:48 |
| mikal | Hmmm, ok | 21:49 |
| mikal | I wanted to link to a resolution somewhere nice | 21:49 |
| mikal | But git it is! | 21:49 |
| melwitt | hi, is there a bug where gerrit isn't linking people's patches to launchpad? I've seen this happen a few times so far lately | 21:49 |
| *** timrc is now known as timrc-afk | 21:50 | |
| anteaya | melwitt: it sometimes depends on the commit message | 21:51 |
| anteaya | melwitt: the commit message has to mention the bug in the first patchset | 21:51 |
| *** timrc-afk is now known as timrc | 21:51 | |
| fungi | melwitt: there are several factors which can cause it. the automation account has to be a member (directly or indirectly) of the bug supervisor group for that project *if* the bug is not yet assigned to the patch committer, the gerrit change has to be targeting the same project and branch as an existing task on the bug, the bug header has to appear in the first patchset of the change which addresses | 21:51 |
| fungi | it | 21:51 |
| fungi | oh, and launchpad's api has to not randomly fail | 21:51 |
| anteaya | melwitt: https://wiki.openstack.org/wiki/GitCommitMessages#Including_external_references | 21:52 |
| melwitt | anteaya: okay. I have seen ones with the Closes-bug: # but it's possible it wasn't there for the initial patchset | 21:52 |
| *** mburned is now known as mburned_out | 21:52 | |
| clarkb | melwitt: do you have an exmaple change? | 21:52 |
| fungi | melwitt: right, the hook which does that doesn't maintain any state, so to prevent spamming the bug over and over it only fires for the very first patchset of a changhe | 21:53 |
| melwitt | clarkb: I just saw this one get mentioned https://review.openstack.org/#/c/96293/ https://bugs.launchpad.net/nova/+bug/1324277 | 21:53 |
| uvirtbot | Launchpad bug 1324277 in nova "Use of finally/return considered harmful" [Medium,Confirmed] | 21:53 |
| *** andreykurilin_ has quit IRC | 21:53 | |
| *** mriedem has quit IRC | 21:54 | |
| *** ArxCruz has joined #openstack-infra | 21:54 | |
| fungi | melwitt: it looks like there's a chance the patch was uploaded for review so quickly after the bug was opened on lp that the api wasn't returning it yet | 21:56 |
| fungi | they were within the same minute | 21:56 |
| lifeless | sdague: sorry, distracted | 21:56 |
| lifeless | sdague: yes, you can argue with a reviewer. | 21:56 |
| lifeless | sdague: but these reviewers will say 'recheck the test run with the relevant bug #' | 21:56 |
| melwitt | fungi: interesting. let me check the others I saw for similar | 21:56 |
| sdague | lifeless: so that seems extremely wasteful | 21:57 |
| lifeless | sdague: why? we don't want regressions to land. | 21:57 |
| lifeless | sdague: and something that fails at step X due to a know bug may fail at step X+N due to a bug in the patch. | 21:57 |
| *** markmcclain has joined #openstack-infra | 21:58 | |
| fungi | melwitt: it would be good to correlate, for sure. i've seen some lag before between the lp webui and the api, which i think i remember hearing is intentional (lifeless?) | 21:58 |
| *** jgrimm has quit IRC | 21:59 | |
| *** homeless has joined #openstack-infra | 21:59 | |
| lifeless | fungi: readonly queries via the webui will be served from slave DB servers unless the session has done a write in the last N seconds (30 IIRC) | 21:59 |
| lifeless | fungi: but DB replication lag is typically a few seconds at most | 22:00 |
| sdague | lifeless: well, I'm basically off for the night now. Maybe we can pick it up tomorrow. | 22:00 |
| lifeless | sdague: ok | 22:00 |
| sdague | there was a real issue with people rechecking blindly with non-voting bugs | 22:00 |
| fungi | lifeless: okay, so if the bug is opened through the webui and then queried and attempted to be updated via the api seconds later, that should be okay? | 22:00 |
| lifeless | fungi: always | 22:01 |
| fungi | lifeless: thanks! | 22:01 |
| fungi | melwitt: so i guess it's not backend latency causing that | 22:01 |
| sdague | so we can figure out something. but it would also be good to figure out if there is a way to make the tripleo jobs just vote anyway, instead of putting the extra load on people. | 22:01 |
| lifeless | sdague: I would love that. | 22:01 |
| clarkb | the tripleo jobs can vote using that thing jhesketh is doing | 22:02 |
| fungi | sdague: isn't the idea that we do something for that pipeline similar to what ironic's wanting for their non-binding jobs? | 22:02 |
| *** ArxCruz has quit IRC | 22:02 | |
| melwitt | fungi: another example, patch upload about one minute after lp created https://bugs.launchpad.net/python-novaclient/+bug/1324857 https://review.openstack.org/#/c/96715/ | 22:02 |
| uvirtbot | Launchpad bug 1324857 in python-novaclient "floating-ip-list does not list attached server anymore" [Medium,Confirmed] | 22:02 |
| clarkb | fungi: yup | 22:02 |
| lifeless | fungi: clarkb: thats still nonbinding though right? So no different. | 22:02 |
| clarkb | lifeless: I don't follow | 22:02 |
| clarkb | lifeless: its just not jenkins voting | 22:02 |
| clarkb | but it can be a real vote | 22:02 |
| lifeless | clarkb: will zuul merge on such a failure if someone asks it to ? | 22:03 |
| clarkb | lifeless: yes | 22:03 |
| clarkb | but that is a different issue | 22:03 |
| lifeless | clarkb: if the answer is yes, the burden is still held by human reviewers, which is exactly the thing sdague said it would be good to fix. | 22:03 |
| lifeless | clarkb: and the thing I would love to fix. | 22:03 |
| fungi | melwitt: that one looks like it was uploaded a couple minutes later | 22:03 |
| *** sarob_ has quit IRC | 22:04 | |
| *** sarob has joined #openstack-infra | 22:04 | |
| fungi | but i would still call that a fairly tight correlation | 22:05 |
| melwitt | fungi: yeah, 10:30:53 is the timestamp on the lp. I agree | 22:05 |
| *** dizquierdo has quit IRC | 22:05 | |
| * jogo just pushed the hacking 0.9.0 tag | 22:05 | |
| jogo | I think its funny how check queue is shorter then the gate queue | 22:06 |
| openstackgerrit | Khai Do proposed a change to openstack-infra/jenkins-job-builder: tests for JJB command processing https://review.openstack.org/97908 | 22:06 |
| fungi | melwitt: if you find that the majority which failed to update were uploaded very, very soon after the bugs were opened, then that very well could indicate some sort of something there | 22:06 |
| *** mkoderer has quit IRC | 22:06 | |
| fungi | jogo: oh, i think that may not end up on pypi without manual help, which i can apply after dinner | 22:07 |
| fungi | jogo: i still need to decide how we're getting a newer version of the requests module onto the new pypi slave | 22:07 |
| fungi | so the upload job is going to die in flames | 22:07 |
| clarkb | we should update the gerrit ACLs to prevent any tags from being pushed when we are in this state | 22:08 |
| clarkb | fungi: I don't think it is too terrible to just upgrade requests globablly | 22:08 |
| fungi | jogo: i just uploaded it now | 22:08 |
| fungi | clarkb: yeah, though so far it's just been these two | 22:09 |
| fungi | (based on what i see workspaces created for) | 22:09 |
| fungi | clarkb: so right now we're installing python-requests from the distro on all slaves. is the suggestion that we globally install it from pip instead? | 22:10 |
| clarkb | hrm do we know why we are installing it? | 22:10 |
| fungi | that is a very good question. looking for clues | 22:10 |
| clarkb | oh for pushing files to swift according to the comment | 22:11 |
| clarkb | in that case I would expect newer requests to work as well | 22:11 |
| *** mmaglana has quit IRC | 22:11 | |
| *** mkoderer has joined #openstack-infra | 22:11 | |
| *** dkliban is now known as dkliban_afk | 22:11 | |
| *** sarob has quit IRC | 22:11 | |
| jogo | fungi: sanity check: hacking revs don't automatically get rolled out to projects right? | 22:11 |
| *** jamespd has quit IRC | 22:11 | |
| *** gondoi is now known as zz_gondoi | 22:11 | |
| clarkb | jogo: correct hacking should be pinned in every project | 22:12 |
| fungi | jogo: right, unless they fail to pin | 22:12 |
| *** jamespd has joined #openstack-infra | 22:12 | |
| *** doug-fish has left #openstack-infra | 22:12 | |
| jogo | clarkb fungi: thanks, I confirmed they all pinned a while back | 22:12 |
| fungi | clarkb: yeah, so the trick here is to ensure absent the deb/rpm version *and* install the pip one... oh! the deb package name will not be the same as the pip package name, so all good | 22:12 |
| jogo | fungi: you manually uploaded hacking 0.9.0 to pypi? | 22:12 |
| clarkb | fungi: ya should be straightfowrad. we should confirm with jhesketh that pip requests works with his script but I would expect it to | 22:13 |
| jogo | fungi: if not, I'll hold off sending the email until its up | 22:13 |
| fungi | jogo: sort of. i reran the failing command from the job using a virtualenv on that slave which contained a newer requests installed | 22:13 |
| fungi | jogo: so it should be there, but it was really only sort of manual | 22:13 |
| fungi | clarkb: okay, change coming, then i'll disappear for dinner | 22:14 |
| *** SpamapS has quit IRC | 22:14 | |
| *** YorikSar has quit IRC | 22:15 | |
| jogo | fungi: hmm I don't see it http://pypi.openstack.org/openstack/hacking/ | 22:15 |
| jogo | well its on pypi.python.org so thats good | 22:15 |
| *** YorikSar has joined #openstack-infra | 22:15 | |
| lifeless | devananda: have you considered asking for the fixs (or the nova revert) to be promoted the next time the gate queue resets? | 22:16 |
| lifeless | devananda: (nova-bm was broken too..) | 22:16 |
| clarkb | jogo: you will have to wait for a mirror rebuild | 22:16 |
| jogo | clarkb: ack, thanks | 22:16 |
| devananda | lifeless: i thought nova-bm was fixed by that patch (not broken) | 22:17 |
| *** signed8bit has quit IRC | 22:17 | |
| devananda | lifeless: and i'm not aware that it's possible to bump a patcjh to the top of the queue | 22:17 |
| devananda | clarkb: ^ ? | 22:17 |
| *** signed8b_ has joined #openstack-infra | 22:17 | |
| clarkb | devananda: it is, but we only do it for gate breakers really | 22:17 |
| devananda | clarkb: ah. ironic's gate has been broken for >12hr now | 22:18 |
| *** SpamapS has joined #openstack-infra | 22:18 | |
| *** SpamapS has quit IRC | 22:18 | |
| *** SpamapS has joined #openstack-infra | 22:18 | |
| clarkb | devananda: what change is this? | 22:18 |
| lifeless | devananda: it broke the hostmanager for both nova-bm and Ironic | 22:18 |
| devananda | clarkb: https://review.openstack.org/#/c/97447/4 should unwedge our gate | 22:18 |
| devananda | lifeless: oh. nice | 22:18 |
| *** ociuhandu has quit IRC | 22:19 | |
| devananda | oh great | 22:19 |
| devananda | https://review.openstack.org/#/c/97757/ is a revert on the nova side | 22:19 |
| devananda | if both of those land, ironic will be broken /again/ | 22:19 |
| jogo | devananda: it looks like the patch that we asked you guys to look at broke you :/ | 22:19 |
| lifeless | clarkb: devananda: see https://review.openstack.org/#/c/97447/too | 22:19 |
| lifeless | where the author says that the ironic patch has to be stopped if the nova revert goes through | 22:19 |
| devananda | jogo: yep. I missed the one line changing the definition of __init__ | 22:19 |
| clarkb | devananda: thats the first ironic change | 22:19 |
| lifeless | since it will break in the opposite direction | 22:19 |
| devananda | jogo: and focused on the functional change to the get_host_stats | 22:19 |
| clarkb | so I don't think promoting it changes anything | 22:19 |
| openstackgerrit | Sean Dague proposed a change to openstack-infra/devstack-gate: force ntpdate before we start the timer https://review.openstack.org/97924 | 22:20 |
| lifeless | clarkb: it would let checks succeeed | 22:20 |
| *** masayukig has quit IRC | 22:20 | |
| jogo | devananda: o_O I missed that too | 22:20 |
| devananda | clarkb: well, it's buried under a lot of other non-related changes | 22:20 |
| lifeless | clarkb: in ironic and tripleo | 22:20 |
| clarkb | lifeless: there is nothing in check | 22:20 |
| clarkb | oh in tripleo too | 22:20 |
| lifeless | clarkb: but they cannot succeed while ironic-dvsm fails | 22:20 |
| lifeless | clarkb: and that fails because trunk is broken | 22:20 |
| clarkb | right, I am just trying to reason about whether or not we gain anything by promoting this change | 22:20 |
| clarkb | and it sounds like we do | 22:20 |
| lifeless | clarkb: tripleo has a local revert of the nova patch to make tripleo deploy tests work, but that doesn't help the dvsm-ironic jobs | 22:21 |
| lifeless | with check failing we cannot ascertain whether other patches are good or not | 22:21 |
| devananda | clarkb: if we promote the ironic change(97447) and cancel the nova one (97757), then i think we're ok. lifeless - is there a corresponding fix for nova_bm up yet? | 22:21 |
| clarkb | so someone needs to decide on which change should die | 22:21 |
| lifeless | dprince: hi | 22:21 |
| clarkb | because I am not doing anything until that is sorted | 22:22 |
| devananda | I agree that the nova change which caused this was the right direction. I would rather not revert it, because we'll just need to do it again eventually | 22:22 |
| devananda | jogo: agree yes/no ^ ? | 22:22 |
| jogo | devananda: agree | 22:22 |
| jogo | devananda: although if that is easer for whatever reason, I am fine with the revert. | 22:23 |
| openstackgerrit | Jeremy Stanley proposed a change to openstack-infra/config: Install newer requests package for twine https://review.openstack.org/97962 | 22:23 |
| fungi | clarkb: that ^ i guess | 22:23 |
| *** bknudson has quit IRC | 22:23 | |
| lifeless | so there are three patches in flight here | 22:23 |
| devananda | lifeless: what's the third? | 22:24 |
| lifeless | https://review.openstack.org/#/c/97757/ (revert nova change) | 22:24 |
| lifeless | https://review.openstack.org/#/c/97447/ (fix ironic) | 22:24 |
| lifeless | https://review.openstack.org/#/c/97806/ (fix nova) | 22:24 |
| *** amcrn_ has quit IRC | 22:24 | |
| lifeless | with either need 97757 to land, or both 97447 and 97806 to land | 22:24 |
| *** mmaglana has joined #openstack-infra | 22:24 | |
| clarkb | fungi: I am +2. hopefully jhesketh will pop up and can confirm it is ok | 22:24 |
| devananda | lifeless: ++ | 22:25 |
| *** masayukig has joined #openstack-infra | 22:25 | |
| clarkb | 97806 has not been approved | 22:25 |
| clarkb | that leaves you with 97757 | 22:25 |
| lifeless | so, since the rollforward hasn't been approved, my suggesetion is to let the revert roll in - and promote it to unbreak everything | 22:26 |
| jogo | yeah still isn't passing jenkins yet | 22:26 |
| lifeless | and pull 97447 out of the gate | 22:26 |
| jogo | lifeless: works for me | 22:26 |
| lifeless | yes we need to do the change at some point, but it can hopefully be done by adding new api, migrating all users, then removing old one. | 22:27 |
| sdague | jogo: 97806 bounced off the devstack get-pip fix | 22:27 |
| lifeless | devananda: ? | 22:27 |
| sdague | jogo: realistically we've got about 20 coupled race bugs right now | 22:27 |
| sdague | which is why we're grinding | 22:27 |
| devananda | I may have another trivial fix, but it'd depend on someone in nova approving it | 22:28 |
| devananda | the problem is the change to __init__ on L112 here https://review.openstack.org/#/c/94043/2/nova/scheduler/host_manager.py | 22:28 |
| * fungi is going to go grind on some food for a while | 22:28 | |
| *** prad has quit IRC | 22:28 | |
| sdague | clarkb: is there any chance that hp 1.1 nodes have 16 cores? | 22:28 |
| lifeless | devananda: and the missing stats, no ? | 22:28 |
| sdague | http://logs.openstack.org/12/97912/1/check/check-grenade-dsvm-partial-ncpu/6f8f858/console.html | 22:29 |
| fungi | clarkb: we can do the nodepool restart when i get back if you're still okay with that | 22:29 |
| sdague | clarkb: there definitely are 8 testr processes there | 22:29 |
| devananda | lifeless: ah, right - it's not addign those :( | 22:29 |
| devananda | yes, revert | 22:29 |
| lifeless | clarkb: ok we have consensus from nova core and ironic core | 22:29 |
| lifeless | clarkb: the revert needs to go through and we need to pull the ironic patch | 22:29 |
| clarkb | sdague: no 8 cores | 22:30 |
| * devananda -2's the ironic patch | 22:30 | |
| *** otherwiseguy has quit IRC | 22:30 | |
| clarkb | devananda: even better is pushing a new patchset but whatever | 22:30 |
| sdague | clarkb: are we sure? | 22:30 |
| clarkb | sdague: yes I just booted one for testing | 22:30 |
| clarkb | it has 8 | 22:30 |
| *** thedodd has quit IRC | 22:30 | |
| sdague | in b5? | 22:30 |
| clarkb | ya b5 is just a logical name | 22:30 |
| lifeless | clarkb: if you can promote https://review.openstack.org/#/c/97757/ when the gate next resets, that would be low disruption and very useful | 22:30 |
| clarkb | lifeless: will do | 22:30 |
| devananda | clarkb: k, doing that instead then | 22:31 |
| lifeless | devananda: we should be able to add the stats now though as part of prep work | 22:31 |
| lifeless | devananda: AIUI | 22:31 |
| devananda | lifeless: yep | 22:31 |
| zaro | anteaya: create reference permission isn't necessary? https://review.openstack.org/#/c/90281/2/modules/openstack_project/files/gerrit/acls/openstack/dib-utils.config | 22:32 |
| devananda | oh, nvm | 22:32 |
| *** mburned_out is now known as mburned | 22:32 | |
| *** maxbit has quit IRC | 22:33 | |
| jhesketh | anteaya: At the moment turbo-hipster is only doing database migrations CI and in our deployment that's all it is likely to do | 22:34 |
| jhesketh | well it might do other database tests against datasets | 22:35 |
| devananda | lifeless: so I don't have time to figure out the right way to clean up 97447 before totally missing this appointment :( | 22:35 |
| jhesketh | so maybe a name like 'db-datasets-ci'? | 22:35 |
| devananda | clarkb: if I block 97447, it will stop it from merging, even though it's in the queue already, right? | 22:35 |
| clarkb | devananda: if you -2 it will prevent merging. if you push a new patchste it will rip it out of the gate | 22:36 |
| *** whayutin_ has quit IRC | 22:36 | |
| lifeless | devananda: want me to push up a no-op change to 97447 ? | 22:36 |
| lifeless | devananda: you go, I got this | 22:36 |
| jhesketh | clarkb: what did you need me to look at/confirm? | 22:36 |
| *** fbo is now known as fbo_away | 22:36 | |
| *** markwash has joined #openstack-infra | 22:37 | |
| clarkb | jhesketh: https://review.openstack.org/97962 change to how we install python requests on slaves | 22:37 |
| lifeless | clarkb: is a commit message only change sufficient ? | 22:37 |
| devananda | lifeless: thanks! | 22:37 |
| clarkb | jhesketh: twine (which we use to upload packages to pypi) needs newer requests | 22:37 |
| openstackgerrit | Sean Dague proposed a change to openstack-infra/devstack-gate: capture cpu information when we setup_host https://review.openstack.org/97965 | 22:37 |
| clarkb | lifeless: yes, but maybe not via gerrit UI | 22:37 |
| clarkb | I can't remember if that does a new patchset | 22:37 |
| sdague | lifeless: yes, commit change is sufficient | 22:37 |
| lifeless | done | 22:38 |
| sdague | even via the gerrit UI I believe | 22:38 |
| lifeless | switched the metadata header ordering | 22:38 |
| anteaya | zaro, correct under refs/tags/* the create line is no longer required | 22:38 |
| sdague | typically I leave a comment that just explains I've removed it from the gate, for whatever reason | 22:38 |
| anteaya | zaro the pushSignedTag line is though | 22:38 |
| sdague | so people that are surprised about it later know why | 22:38 |
| *** jp_at_hp has quit IRC | 22:39 | |
| anteaya | zaro and refs/tags/* is not required for every acl file, only if that project wants to push tags | 22:39 |
| jhesketh | clarkb: right, I have no idea of the implications of that (in terms of what is supported etc), but I don't see any obvious problems with using pip | 22:39 |
| anteaya | jhesketh: if you like db-datasets-ci I will add that to the renaming etherpad | 22:40 |
| anteaya | jhesketh: https://etherpad.openstack.org/p/automated-gerrit-account-naming-format | 22:40 |
| clarkb | jhesketh: cool. was mostly worried about your swift uploader script | 22:40 |
| zaro | anteaya: i believe it used to be required, is that due to never version of gerrit? | 22:40 |
| *** amcrn has joined #openstack-infra | 22:40 | |
| *** dprince has quit IRC | 22:40 | |
| jhesketh | anteaya: sounds good to me | 22:40 |
| zaro | anteaya: or does it get handled by All-Projects? | 22:40 |
| jhesketh | mikal: Do you think 'db-datasets-ci' is appropriate for turbo-hipster ^ | 22:41 |
| anteaya | jhesketh: line 149 | 22:41 |
| mikal | Pardon? | 22:41 |
| jhesketh | clarkb: well so long as requests still works :p | 22:41 |
| jhesketh | clarkb, fungi: I do have a question for 97962 which I left on the review | 22:41 |
| clarkb | jhesketh: it should but its version 0.8.2 vs 2.3.0 | 22:41 |
| anteaya | zaro correct, create used to be required with Gerrit 2.4, but is no longer requried with Gerrit 2.8 | 22:41 |
| mtreinish | sdague: I think we just forgot to do the concurrency = nprocs/2 in grenade and that's why you're seeing 8 workers there | 22:41 |
| jhesketh | mikal: we need to make turbo-hipster's name more obvious when it leaves votes | 22:41 |
| *** arnaud has joined #openstack-infra | 22:42 | |
| mtreinish | but I'm coming in a little late so I'm probably missing what the problem is | 22:42 |
| anteaya | jhesketh mikal more like the standardized naming found here: https://etherpad.openstack.org/p/automated-gerrit-account-naming-format | 22:42 |
| zaro | anteaya: ok. sounds good. | 22:42 |
| anteaya | zaro: thanks | 22:42 |
| jerryz | clarkb: are there any network interruption on jenksin slaves? or gem repo is down? http://logs.openstack.org/48/97948/1/check/gate-cookbook-openstack-common-chef-style/bdc6688/console.html | 22:42 |
| *** marcoemorais has joined #openstack-infra | 22:42 | |
| anteaya | jhesketh mikal don't get me wrong I love turbo-hipster | 22:42 |
| sdague | mtreinish: oh, really? | 22:43 |
| anteaya | but bringing sembelance to the masses means you have to leave out some fun stuff | 22:43 |
| anteaya | :( | 22:43 |
| clarkb | jerryz: not that I know of in IAD. and you have to ask gem people about their end | 22:43 |
| jhesketh | I've got to run out for a little bit guys, be back soon | 22:43 |
| *** gokrokve has quit IRC | 22:43 | |
| mtreinish | sdague: yeah because we call tempest from inside grenade not devstack gate right | 22:44 |
| anteaya | jhesketh: oh and since you are here, what should I propose a session on? | 22:44 |
| sdague | mtreinish: right | 22:44 |
| mtreinish | but, the concurrency stuff is set in devstack gate | 22:44 |
| anteaya | jhesketh: when you get back | 22:44 |
| jhesketh | :-) | 22:44 |
| mtreinish | so it'll just use the default in grenade | 22:44 |
| lifeless | ok 97447 is out of the queue | 22:44 |
| *** jhesketh has quit IRC | 22:44 | |
| mtreinish | sdague: I'll push out a fix for it | 22:45 |
| *** zzelle has quit IRC | 22:46 | |
| sdague | mtreinish: I have it | 22:46 |
| lifeless | now we just wait 8 hours or for a failure | 22:47 |
| *** ArxCruz has joined #openstack-infra | 22:47 | |
| mikal | I have no problem renaming it | 22:47 |
| lifeless | clarkb: actually, is it possible to promote to the next-window-in-the-predictive-queue? | 22:47 |
| mikal | We were also meant to link to a wiki page in our votes, did that ever get done? | 22:47 |
| lifeless | clarkb: e.g to where 97126,2 is? | 22:47 |
| clarkb | lifeless: it isn't unfortunately | 22:47 |
| clarkb | lifeless: I was going to do it after some of these merge if no failures happen | 22:48 |
| lifeless | nuts | 22:48 |
| lifeless | cool | 22:48 |
| *** maxbit has joined #openstack-infra | 22:48 | |
| sdague | mtreinish: that would actually explain a lot about the fact that grenade was failing more often than the other jobs | 22:49 |
| sdague | 8 way tempest exposes way too many other issues | 22:49 |
| clarkb | lifeless: around 92739 hitting head of queue | 22:49 |
| anteaya | mikal: we are working on wikipage set up for automated accounts | 22:49 |
| *** derekh_ has quit IRC | 22:50 | |
| anteaya | mikal: I will assign you one | 22:50 |
| anteaya | mikal: plus I will be bugging you to fill out one for nova as well | 22:50 |
| mikal | anteaya: IIRC Josh needed to add a reporter tweak to get it done in zuul, but I thought he would be working on that already | 22:50 |
| mikal | He appears to have dropped off IRC, so I can't ask | 22:50 |
| anteaya | he may be | 22:50 |
| clarkb | mikal: I think that merged | 22:51 |
| *** dims_ has quit IRC | 22:51 | |
| anteaya | let's pile up questions for when he returns | 22:51 |
| *** sarob has joined #openstack-infra | 22:51 | |
| clarkb | sdague: its like a salt shaker with bugs in it | 22:51 |
| anteaya | mikal: then he will know we missed him | 22:51 |
| mikal | clarkb: Oh, he might just need to turn it on then? | 22:51 |
| mikal | I have to run to a meeting in a minute, so someone please hassle him about that | 22:51 |
| mtreinish | sdague: yeah, it just stresses a devstack-gate node a bit too much | 22:51 |
| *** ArxCruz has quit IRC | 22:52 | |
| mikal | Heh | 22:52 |
| mattoliverau | Morning all | 22:53 |
| anteaya | morning mattoliverau | 22:53 |
| sdague | clarkb: if you are taking promotes - https://review.openstack.org/#/c/97845/ | 22:56 |
| sdague | is the backport of a fix that's #7 in the er bugs | 22:56 |
| clarkb | ok | 22:58 |
| *** prad has joined #openstack-infra | 22:59 | |
| notmyname | what are the restrictions on force-pushing to branches in gerrit? I know -core can push merge commits. and it seems ttx can force push to milestone-proposed. what about a feature branch? specifically, I'm thinking about swift's feature/ec branch | 23:00 |
| clarkb | I think we do our best to avoid force pushing to anything | 23:01 |
| *** olaph has quit IRC | 23:02 | |
| notmyname | clarkb: yes I know. but that doesn't answer the question ;-) | 23:02 |
| clarkb | are you trying to change history (badbadbad) or just merge something on top of history | 23:02 |
| clarkb | for changing history never. for merging something on top of history without going through the gate we only do that when the gate has managed to get itself into a situation where the only practical way out is to merge without testing | 23:02 |
| jogo | sdague: ahh I see now http://jogo.github.io/gate/ we have some really high failure rates | 23:03 |
| sdague | jogo: yes | 23:03 |
| sdague | realistically, I think we've been pretty bad for a while, clean check was masking it | 23:04 |
| sdague | but a 500 element check queue meant a giant flush | 23:04 |
| *** sarob_ has joined #openstack-infra | 23:04 | |
| clarkb | notmyname: what are we trying to accomplish? | 23:04 |
| notmyname | clarkb: I understand and am aware of the dangers of force pushing code. but I'm curious about what roles have the ability to do it | 23:05 |
| *** aconrad has quit IRC | 23:05 | |
| *** bhuvan_ has joined #openstack-infra | 23:06 | |
| clarkb | notmyname: the only role I am aware that actively has the ability to do it is the project creator account so that it can seed repos | 23:06 |
| notmyname | ok | 23:06 |
| *** sarob has quit IRC | 23:07 | |
| notmyname | clarkb: there's a few reasons I'm asking, but the most immediate one is because the feature/ec branch is done (and has the wrong history). we are interested in reusing it for actual erasure code work | 23:07 |
| *** maxbit has quit IRC | 23:07 | |
| clarkb | notmyname: I consider that bad | 23:08 |
| *** markmcclain has quit IRC | 23:08 | |
| clarkb | you can do what the milestone branches used to do (which we are going to stop doing because it is bad) and delete the branch and recreate it with a different head | 23:08 |
| clarkb | the thing to keep in mind is that anyone that had the branch locally will have to reset --hard to catch the update | 23:08 |
| *** bhuvan has quit IRC | 23:08 | |
| *** dims_ has joined #openstack-infra | 23:08 | |
| openstackgerrit | K Jonathan Harker proposed a change to openstack-infra/config: Move variables in manifests/site.pp to top of file https://review.openstack.org/97659 | 23:09 |
| *** maxbit has joined #openstack-infra | 23:09 | |
| *** atiwari has quit IRC | 23:09 | |
| clarkb | notmyname: this is similar to a force push but less scary with gerrit ACLs iirc | 23:09 |
| clarkb | (you can delete branches via the web ui without proper force push abilities) | 23:10 |
| notmyname | clarkb: ah yes. a delete and recreate makes sense | 23:10 |
| notmyname | oh really? | 23:10 |
| notmyname | clarkb: what web ui? | 23:10 |
| clarkb | the gerrit web ui | 23:11 |
| clarkb | lifeless: sdague changes promoted | 23:11 |
| clarkb | a reset happened | 23:11 |
| openstackgerrit | K Jonathan Harker proposed a change to openstack-infra/config: Move variables in manifests/site.pp to top of file https://review.openstack.org/97659 | 23:12 |
| *** timrc is now known as timrc-afk | 23:13 | |
| notmyname | clarkb: on https://review.openstack.org/#/admin/projects/openstack/swift,branches I don't see anything do delete branches | 23:13 |
| clarkb | notmyname: I think you may need additional permissions to do it | 23:14 |
| notmyname | clarkb: ah ok | 23:14 |
| clarkb | they are just more restrictive than allowing force push iirc | 23:14 |
| clarkb | let me figure that out | 23:14 |
| notmyname | clarkb: ok, thanks. no rush. I can ping you on it tomorrow | 23:14 |
| *** virmitio has quit IRC | 23:15 | |
| *** mrodden has quit IRC | 23:15 | |
| clarkb | oh its attached to "owner" | 23:16 |
| *** jhesketh_ has joined #openstack-infra | 23:17 | |
| anteaya | jhesketh_: welcome back | 23:17 |
| anteaya | mikal wants you to do a thing | 23:17 |
| jhesketh_ | thanks anteaya | 23:17 |
| anteaya | and asked me to ensure you were bugged about it | 23:17 |
| jhesketh_ | I'm catching onto mikal's tricks of making me do work.... | 23:18 |
| anteaya | I consider my job done | 23:18 |
| anteaya | ha ha ha | 23:18 |
| jhesketh_ | heh, he's also good at getting others to bug me | 23:18 |
| anteaya | my favourite memory of summit is him giggling for a solid 20 minutes | 23:18 |
| anteaya | that was the best | 23:18 |
| anteaya | so yeah, back to me | 23:19 |
| anteaya | what am I to propose for a talk? | 23:19 |
| clarkb | jhesketh_: did you manage to get your pet rock through customs? | 23:19 |
| anteaya | ha ha ha | 23:19 |
| jhesketh_ | heh, I didn't list it, so just walked straight through | 23:19 |
| anteaya | ha ha ha | 23:19 |
| jhesketh_ | so I plugged it in... turns out, it works as advertised | 23:19 |
| anteaya | ha ha ha | 23:19 |
| anteaya | was the blank book of instructions helpful | 23:20 |
| jhesketh_ | exceptionally.. it's important to take good care of your pets | 23:20 |
| anteaya | ha ha ha | 23:21 |
| *** jhesketh_ is now known as jhesketh | 23:21 | |
| *** ArxCruz has joined #openstack-infra | 23:21 | |
| *** aconrad has joined #openstack-infra | 23:21 | |
| *** sarob_ has quit IRC | 23:23 | |
| *** mrda-away is now known as mrda | 23:24 | |
| *** ArxCruz has quit IRC | 23:25 | |
| dstufft | wat is this redhat ci stuff i'm getting emails from | 23:25 |
| *** lcostantino has joined #openstack-infra | 23:26 | |
| anteaya | dstufft: what do the email say? | 23:26 |
| anteaya | s/email/emails | 23:26 |
| dstufft | anteaya: some failure stuff | 23:27 |
| dstufft | I'm just confused why openstack stuff is sending me redhat ci emails o.O | 23:27 |
| anteaya | dstufft: like on any of these patches? https://review.openstack.org/#/q/reviewer:%22Red+Hat+CI+%253Credhatci%2540redhat.com%253E%22,n,z | 23:28 |
| lifeless | dstufft: it is from gerrit? | 23:28 |
| dstufft | https://review.openstack.org/#/c/97912/ this patch | 23:28 |
| dstufft | lifeless: yea | 23:28 |
| dstufft | is there like | 23:28 |
| dstufft | multiple CI systems now | 23:28 |
| dstufft | oh man there's some xenserver stuff there too | 23:29 |
| lifeless | dstufft: so 3rd partt checks into gerrit | 23:29 |
| lifeless | gerrit tells you | 23:29 |
| lifeless | we all cry | 23:29 |
| dstufft | I guess I need to get around to setting up some email filters | 23:30 |
| apevec | dstufft, redhat ci is 3rd party testing devstack, ianw is the contact | 23:32 |
| dstufft | I guess I should pay more attention or something, I had no idea ther was third party testing :] | 23:32 |
| dstufft | is it new? | 23:32 |
| clarkb | dstufft: its not new, but has picked up more steam during the icehouse cycle | 23:33 |
| clarkb | smokestack was a third party test system that ran for years | 23:33 |
| anteaya | we liked it | 23:33 |
| dstufft | do they test everything? or just certain stuff? | 23:33 |
| anteaya | then the crowds came | 23:33 |
| anteaya | they should only test certain stuff | 23:33 |
| anteaya | but some folks don't know how to irc or email list yet | 23:34 |
| anteaya | so they tend to do wild things before we can contain them | 23:34 |
| dstufft | probably why I never noticed it then, I must not have been paying attention to anything that was tested by hird party things | 23:34 |
| anteaya | we should just clone dprince | 23:34 |
| lifeless | dstufft: they test all the stuff relevant to their tech | 23:34 |
| lifeless | dstufft: e.g. a switch vendor might test neutron jobs only | 23:34 |
| dstufft | makes sense | 23:35 |
| dstufft | gerrit should totally have a way to record test results that doesn't add more emails | 23:35 |
| dstufft | (also :sadtrombone: this'll be the first thing I filter out of my inbox and don't read) | 23:36 |
| *** mrodden has joined #openstack-infra | 23:36 | |
| openstackgerrit | Joe Gordon proposed a change to openstack-dev/hacking: Make import checks properly detect blank lines and comments https://review.openstack.org/97988 | 23:39 |
| jogo | fungi: looks like I will need to cut a hacking 0.9.1 :/ | 23:40 |
| ianw | dstufft: there's moves to standardize the names of CI accounts, which will mean possibility to filter and write some javascript to hide it | 23:42 |
| *** praneshp has joined #openstack-infra | 23:42 | |
| *** Sukhdev has joined #openstack-infra | 23:45 | |
| *** maxbit has quit IRC | 23:45 | |
| Sukhdev | anteaya: ping | 23:45 |
| anteaya | hello Sukhdev | 23:46 |
| anteaya | what is on your mind | 23:46 |
| *** maxbit has joined #openstack-infra | 23:46 | |
| Sukhdev | anteaya: I mentioned to you in the morning that I will ping you later….. | 23:46 |
| Sukhdev | anteaya: wanted to let you know that I did not forget - day has been crazy | 23:46 |
| anteaya | yes, we were talking about Arista Testing becoming Arista CI | 23:46 |
| anteaya | Sukhdev: I understand, thanks for following up | 23:47 |
| Sukhdev | anteaya: Yes, but, can I ask a favor - can I ping you regarding this tomorrow and go over it. | 23:47 |
| anteaya | Sukhdev: sure | 23:47 |
| anteaya | you don't have to do anything | 23:47 |
| Sukhdev | anteaya: I want to understand it correctly so that I do not mess up - and now I am up against the clock | 23:47 |
| anteaya | someone else I was talking to thought they had to do something and they created more accounts | 23:48 |
| Sukhdev | anteaya: wanted to just let you know that I did not forget - just things have been hectic | 23:48 |
| anteaya | please don't do that | 23:48 |
| anteaya | fine | 23:48 |
| *** unicell has quit IRC | 23:48 | |
| *** jaypipes has quit IRC | 23:48 | |
| anteaya | just keep in mind, you don't have to do anything | 23:48 |
| Sukhdev | anteaya: cool - thanks, will ping you tomorrow | 23:48 |
| *** michchap has joined #openstack-infra | 23:48 | |
| anteaya | I'll talk to you then, have a good night | 23:48 |
| anteaya | Sukhdev: don't do anything | 23:49 |
| Sukhdev | anteaya: ah thats even better - I do not have to do anything :-) | 23:49 |
| anteaya | no | 23:49 |
| anteaya | you don't have to do anythign | 23:49 |
| Sukhdev | anteaya: I will ping you tomorrow - thanks for your understanding. | 23:49 |
| anteaya | np | 23:49 |
| Sukhdev | anteaya: bye - have a wonderful night | 23:49 |
| anteaya | thanks Sukhdev you too | 23:49 |
| *** Sukhdev has quit IRC | 23:51 | |
| *** Sukhdev has joined #openstack-infra | 23:52 | |
| *** bhuvan_ has quit IRC | 23:53 | |
| anteaya | ianw yes https://etherpad.openstack.org/p/neutron-juno-mid-cycle-meeting | 23:53 |
| anteaya | ianw: efforts ongoing in that etherpad | 23:53 |
| anteaya | sorry wrong etherpad | 23:54 |
| anteaya | this one: https://etherpad.openstack.org/p/automated-gerrit-account-naming-format | 23:54 |
| *** signed8b_ is now known as signed8bit | 23:54 | |
| *** Sukhdev has quit IRC | 23:56 | |
| fungi | jogo: poo | 23:59 |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!
