| bodepd | mordred: did you know there is a jjb module on the forge? | 00:08 |
|---|---|---|
| bodepd | crazy :) | 00:08 |
| mordred | what? | 00:08 |
| mordred | omg | 00:08 |
| *** gyee has quit IRC | 00:09 | |
| bodepd | https://github.com/kickstandproject/kickstandproject-jenkins_job_builder | 00:09 |
| mordred | oh! that's pabelanger | 00:09 |
| bodepd | who is that? | 00:10 |
| bodepd | the name os familiar | 00:10 |
| bodepd | is | 00:10 |
| bodepd | also, the plugin code I used is blatently stolen from here: | 00:11 |
| bodepd | https://github.com/jenkinsci/puppet-jenkins/blob/master/manifests/plugin.pp | 00:11 |
| mordred | omg | 00:11 |
| mordred | there is a puppet-jenkins module run by the jenkinsci people? | 00:12 |
| bodepd | duh :) | 00:12 |
| pabelanger | mordred, bodepd yup | 00:13 |
| bodepd | I borrowed some bits from that, and some from your guys for my franken-fork | 00:13 |
| mordred | bodepd: nice | 00:13 |
| mordred | so - | 00:13 |
| *** bnemec has quit IRC | 00:15 | |
| *** bnemec has joined #openstack-infra | 00:15 | |
| *** michchap has quit IRC | 00:15 | |
| *** michchap has joined #openstack-infra | 00:16 | |
| bodepd | looks like I got everything installed, but I'm not convined it works :) | 00:20 |
| *** ladquin has joined #openstack-infra | 00:21 | |
| melwitt | is this the right channel to talk about devstack? | 00:23 |
| openstackgerrit | James E. Blair proposed a change to openstack-infra/config: Further tuning of lists https://review.openstack.org/39610 | 00:24 |
| jeblair | i'm going to stop puppet on lists and apply those changes manually | 00:24 |
| *** sdake_ has quit IRC | 00:25 | |
| clarkb | melwitt: depends | 00:27 |
| clarkb | melwitt: if you are interested in how we use devstack to test and gate the projects then yes | 00:28 |
| clarkb | melwitt: but if you are wondering why $feature isn't included or something along those lines you will probably have better luck asking dtroyer directly | 00:28 |
| clarkb | melwitt: and if you aren't sure feel free to ask. we tend to not bite | 00:29 |
| jeblair | http://cacti.openstack.org/cacti/graph_image.php?action=view&local_graph_id=216&rra_id=2 | 00:30 |
| jeblair | lists cpu usage increase with the openstack list is dramatic. | 00:30 |
| clarkb | that is cool | 00:30 |
| *** andrew_plunk has joined #openstack-infra | 00:30 | |
| pabelanger | jeblair, http://cacti.openstack.org/cacti/ is password protected? | 00:31 |
| openstackgerrit | A change was merged to openstack-dev/pbr: Add skip_pip_install to setup.cfg https://review.openstack.org/39363 | 00:31 |
| jeblair | pabelanger: not really; we just need to get around to adding a redirect to the root | 00:31 |
| jeblair | pabelanger: http://cacti.openstack.org/cacti/graph_view.php?action=tree&tree_id=1 | 00:32 |
| jeblair | pabelanger: try that | 00:32 |
| melwitt | clarkb: thanks, makes sense. I bumped into something that didn't work on rhel and commenting out one line in stack.sh made it work. I'll find their launchpad or list where I can see if it can help others | 00:32 |
| pabelanger | jeblair, cool, that works | 00:32 |
| *** cybertron_ has joined #openstack-infra | 00:33 | |
| *** bnemec has quit IRC | 00:33 | |
| clarkb | jeblair: the tuning lgtm, I did have one question inline though | 00:33 |
| *** adalbas has joined #openstack-infra | 00:33 | |
| jeblair | clarkb: the answer is yes (i mentioned that in the commit msg) | 00:34 |
| clarkb | melwitt: I would definitely submit a bug, should be the devstack project on launchpad | 00:34 |
| clarkb | jeblair: bah, you know that is one behavior of gerrit that drives me nuts. It always scrolls down to the files so you skip the commit message | 00:35 |
| jeblair | hehe | 00:35 |
| openstackgerrit | Dan Bode proposed a change to openstack-infra/config: Ensure python pip is installed before zuul https://review.openstack.org/39611 | 00:35 |
| melwitt | clarkb: k, will do. thanks | 00:35 |
| clarkb | jeblair: I just noticed recently that it does that every time I open a change | 00:35 |
| *** UtahDave has quit IRC | 00:36 | |
| jeblair | the queue is currently 130313 and falling at a rate of about 500/min (which gives us a bit more than 4 hours to git it to a reasonable level) | 00:36 |
| jeblair | (though it's only up to 43 out of the 50 runners so far) | 00:36 |
| openstackgerrit | Dan Bode proposed a change to openstack-infra/config: Ensure /var/lib/zuul is owned by zuul https://review.openstack.org/39612 | 00:38 |
| openstackgerrit | Dan Bode proposed a change to openstack-infra/config: Update zuul to be compatible with the latest apache module https://review.openstack.org/39613 | 00:41 |
| bodepd | I'm not sure what to do about the last chnage | 00:42 |
| bodepd | I updated zuul to work with the latest version of apache | 00:42 |
| bodepd | but that is just a small part of the change that needs to happen | 00:42 |
| jeblair | bumped the the number of queue runners to 70 | 00:42 |
| openstackgerrit | Dan Bode proposed a change to openstack-infra/config: Add jenkins::plugin defined resource type https://review.openstack.org/39615 | 00:49 |
| jeblair | ok, that was probably too much; load average is 4.5 now (on a 2vcpu machine) | 00:50 |
| *** dina_belova has joined #openstack-infra | 00:50 | |
| bodepd | those are the changed I had to make today. feel free to ask questions | 00:51 |
| jeblair | otoh, it's now falling at 1000/min, so i'll leave it for now. | 00:55 |
| *** dina_belova has quit IRC | 00:55 | |
| jeblair | the load avg is high, but the cpu stats are good | 00:56 |
| *** changbl has joined #openstack-infra | 00:59 | |
| *** dkranz has joined #openstack-infra | 01:00 | |
| fungi | i've had responsive exim servers with load >50 and only 2x cpu. can actually be okay with sufficient ram and i/o bandwidth as long as that's the only thing the server needs to handle | 01:01 |
| clarkb | does each queue run in its own process? the new config stuff implies that is the case | 01:03 |
| fungi | each delivery gets farmed out to an available runner | 01:04 |
| fungi | similar to how apache handles requests | 01:05 |
| *** reed has quit IRC | 01:13 | |
| pabelanger | jeblair, When you have time, can you do you want to register openstack.org with freenum.org, so we can do ISN dialing: http://freenum.org/cookbook/ | 01:21 |
| pabelanger | jeblair, I can setup the asterisk side, but need valid ITAD info | 01:21 |
| openstackgerrit | Russell Bryant proposed a change to openstack-infra/config: pbx: update SIP config to help deal with NAT issues https://review.openstack.org/39616 | 01:24 |
| *** vipul is now known as vipul-away | 01:26 | |
| *** sandywalsh has quit IRC | 01:33 | |
| clarkb | mordred: I noticed earlier today that you don't use gerrit email. I am going to give it a shot and see if it makes reading list mail easier | 01:38 |
| mordred | clarkb: going to give no gerrit mail a shot? | 01:39 |
| mordred | yeah - I pretty much just poll the UI all day long | 01:39 |
| mordred | the biggest downside is the ui is bad at telling me that someone has made comments I should look at | 01:39 |
| mordred | or that someone has voted -1 on a change if someone else has voted positively | 01:39 |
| mordred | but - I believe I still keep up with it better than I would with list mail | 01:40 |
| mordred | I also disabled all bug email | 01:40 |
| mordred | that one I don't think is the right choice | 01:40 |
| mordred | and I need to undo | 01:40 |
| clarkb | ya I tend to poll gerrit pretty frequently especially when I am in a review binge | 01:40 |
| mordred | yah | 01:42 |
| *** nibalizer has quit IRC | 01:43 | |
| *** pcrews has quit IRC | 01:43 | |
| *** bnemec has joined #openstack-infra | 01:45 | |
| *** michchap has quit IRC | 01:45 | |
| *** cybertron_ has quit IRC | 01:46 | |
| *** sandywalsh has joined #openstack-infra | 01:49 | |
| clarkb | I think we broke the zuul jobs launched per hour | 01:50 |
| clarkb | *per hour graph on the zuul status page | 01:50 |
| *** dina_belova has joined #openstack-infra | 01:52 | |
| fungi | clarkb: the change to track job results via statsd requires an update to the zuul status page | 01:53 |
| fungi | s/./_/ mostly, i think? | 01:54 |
| clarkb | fungi: it added the pipelines to the path too | 01:56 |
| clarkb | fungi: but I thought that required a zuul restart. is that why jeblair was rechecking jobs? | 01:56 |
| *** dina_belova has quit IRC | 01:56 | |
| fungi | i probably flew under my radar while i was out at dinner, if so | 01:57 |
| *** thomasm has joined #openstack-infra | 01:59 | |
| fungi | last zuul restart was almost 20 hours ago | 01:59 |
| *** melwitt has quit IRC | 02:03 | |
| *** mriedem has joined #openstack-infra | 02:05 | |
| openstackgerrit | A change was merged to openstack-infra/config: Add branch tarball job for tempest. https://review.openstack.org/38910 | 02:08 |
| *** jrex_laptop has quit IRC | 02:11 | |
| *** michchap has joined #openstack-infra | 02:23 | |
| *** anteaya has quit IRC | 02:26 | |
| *** andrew_plunk has quit IRC | 02:26 | |
| *** HenryG has joined #openstack-infra | 02:34 | |
| *** dguitarbite has joined #openstack-infra | 02:34 | |
| *** vipul-away is now known as vipul | 02:35 | |
| *** HenryG has quit IRC | 02:42 | |
| *** erfanian has joined #openstack-infra | 02:43 | |
| lifeless | jeblair: when you are around, can we talk about https://review.openstack.org/#/c/38543/ ? perhaps a voice call is needed to get more bandwidth on it? | 02:45 |
| *** michchap has quit IRC | 02:46 | |
| *** SergeyLukjanov has joined #openstack-infra | 02:50 | |
| *** dina_belova has joined #openstack-infra | 02:51 | |
| SpamapS | The authenticity of host '[review.openstack.org]:29418 ([198.101.231.251]:29418)' can't be established. | 02:56 |
| SpamapS | RSA key fingerprint is ee:2f:ac:1b:f8:25:d0:39:be:55:02:c7:76:5e:39:53. | 02:56 |
| SpamapS | oh.. n/m .. my chroot doesn't have my ~/.ssh/known_hosts | 02:57 |
| *** michchap has joined #openstack-infra | 02:57 | |
| mordred | SpamapS: :) | 02:57 |
| SpamapS | Need to figure out how to just do a git review -d in the local cached version of a repo | 02:58 |
| *** yaguang has joined #openstack-infra | 02:58 | |
| SpamapS | (guessing that is high on the list of things to make di-b useful for CI/CD anyway) | 02:58 |
| *** michchap has quit IRC | 02:59 | |
| *** michchap has joined #openstack-infra | 03:00 | |
| *** mriedem has quit IRC | 03:00 | |
| mordred | well, not for ci | 03:03 |
| mordred | it's probably useful for dev workflow though | 03:03 |
| *** jjmb has quit IRC | 03:04 | |
| lifeless | SpamapS: uhm, WAT ? | 03:04 |
| lifeless | SpamapS: I think you're pushing on the wrong knob. | 03:04 |
| *** SergeyLukjanov has quit IRC | 03:05 | |
| *** yaguang has quit IRC | 03:05 | |
| *** dina_belova has quit IRC | 03:15 | |
| *** yaguang_ has joined #openstack-infra | 03:19 | |
| *** UtahDave has joined #openstack-infra | 03:20 | |
| *** michchap has quit IRC | 03:23 | |
| *** michchap has joined #openstack-infra | 03:24 | |
| zaro | mordred, jeblair, clarkb, fungi: WIP vote patches have been submitted to upstream. | 03:28 |
| zaro | https://gerrit-review.googlesource.com/#/c/48270/ & https://gerrit-review.googlesource.com/#/c/48290/ | 03:29 |
| fungi | zaro: awesome! | 03:30 |
| fungi | i recall submitting patches to google's gerrit... their workflow is interesting, or at least was at the time | 03:31 |
| zaro | fungi: how so? | 03:32 |
| * fungi looks back at his notes | 03:32 | |
| *** michchap has quit IRC | 03:33 | |
| fungi | eh, mostly that they do push via https into gerrit, even though released gerrit didn't support that at the time (maybe it does now?) | 03:36 |
| zaro | i think so, at least the doc indicates so.. https://gerrit-review.googlesource.com/Documentation/config-gerrit.html | 03:39 |
| *** bnemec has quit IRC | 03:40 | |
| *** bnemec has joined #openstack-infra | 03:40 | |
| * fungi looks forward to yet another new option there! | 03:41 | |
| *** erfanian has quit IRC | 04:02 | |
| *** pcrews has joined #openstack-infra | 04:03 | |
| *** vogxn has joined #openstack-infra | 04:06 | |
| *** nati_ueno has quit IRC | 04:14 | |
| *** bnemec has quit IRC | 04:16 | |
| *** bnemec has joined #openstack-infra | 04:16 | |
| *** amotoki_ has joined #openstack-infra | 04:18 | |
| *** UtahDave has quit IRC | 04:19 | |
| *** amotoki has quit IRC | 04:19 | |
| *** nicedice has quit IRC | 04:23 | |
| *** dkliban_afk has quit IRC | 04:23 | |
| *** emagana has joined #openstack-infra | 04:28 | |
| *** afazekas has joined #openstack-infra | 04:34 | |
| *** markmcclain has quit IRC | 04:36 | |
| *** nayward has joined #openstack-infra | 04:41 | |
| *** nayward has quit IRC | 04:46 | |
| *** toddmorey has quit IRC | 04:52 | |
| *** rcleere has quit IRC | 04:54 | |
| *** yaguang_ has quit IRC | 05:00 | |
| *** toddmorey has joined #openstack-infra | 05:01 | |
| *** nayward has joined #openstack-infra | 05:12 | |
| *** dina_belova has joined #openstack-infra | 05:15 | |
| *** dina_belova has quit IRC | 05:16 | |
| *** toddmorey has quit IRC | 05:17 | |
| *** dina_belova has joined #openstack-infra | 05:25 | |
| *** rcleere has joined #openstack-infra | 05:33 | |
| *** koolhead11|away has joined #openstack-infra | 05:33 | |
| *** koolhead11|away is now known as koolhead11 | 05:34 | |
| *** nayward has quit IRC | 05:37 | |
| *** nayward has joined #openstack-infra | 05:40 | |
| *** jhesketh has quit IRC | 05:41 | |
| *** jhesketh has joined #openstack-infra | 05:41 | |
| *** toddmorey has joined #openstack-infra | 05:48 | |
| *** nayward has quit IRC | 05:49 | |
| *** dina_belova has quit IRC | 05:52 | |
| *** toddmorey1 has joined #openstack-infra | 05:55 | |
| *** toddmorey has quit IRC | 05:55 | |
| *** dguitarbite has quit IRC | 05:55 | |
| *** michchap has joined #openstack-infra | 05:56 | |
| *** michchap has quit IRC | 05:59 | |
| *** michchap has joined #openstack-infra | 06:00 | |
| *** toddmorey1 has quit IRC | 06:11 | |
| *** pabelanger has quit IRC | 06:15 | |
| *** odyssey4me has joined #openstack-infra | 06:15 | |
| *** dina_belova has joined #openstack-infra | 06:20 | |
| *** michchap has quit IRC | 06:21 | |
| *** yaguang has joined #openstack-infra | 06:21 | |
| *** nayward has joined #openstack-infra | 06:22 | |
| *** dina_belova has quit IRC | 06:25 | |
| *** nayward has quit IRC | 06:27 | |
| *** pabelanger has joined #openstack-infra | 06:32 | |
| *** emagana has quit IRC | 06:34 | |
| *** mestery_ has joined #openstack-infra | 06:34 | |
| *** mestery has quit IRC | 06:36 | |
| *** amotoki_ has quit IRC | 06:37 | |
| marun | anybody around? | 06:38 |
| *** jjmb has joined #openstack-infra | 06:43 | |
| *** dina_belova has joined #openstack-infra | 06:44 | |
| *** bnemec has quit IRC | 06:49 | |
| *** dina_belova has quit IRC | 06:50 | |
| *** bnemec has joined #openstack-infra | 06:50 | |
| *** dina_belova has joined #openstack-infra | 06:55 | |
| *** jpich has joined #openstack-infra | 06:58 | |
| *** bnemec has quit IRC | 06:59 | |
| *** bnemec has joined #openstack-infra | 07:00 | |
| *** vogxn has quit IRC | 07:06 | |
| *** dina_belova has quit IRC | 07:08 | |
| *** jhesketh has quit IRC | 07:09 | |
| *** jhesketh has joined #openstack-infra | 07:09 | |
| *** vogxn has joined #openstack-infra | 07:11 | |
| openstackgerrit | David Caro proposed a change to openstack-infra/jenkins-job-builder: Added post-task publisher https://review.openstack.org/37924 | 07:11 |
| jhesketh | Do any of the jenkins workers run tests against databases other than sqlite? | 07:12 |
| *** Ryan_Lane has quit IRC | 07:16 | |
| openstackgerrit | David Caro proposed a change to openstack-infra/jenkins-job-builder: Added some more scm options https://review.openstack.org/39298 | 07:20 |
| openstackgerrit | David Caro proposed a change to openstack-infra/jenkins-job-builder: Added more options to the Build Timeout plugin https://review.openstack.org/39329 | 07:22 |
| *** rcleere has quit IRC | 07:26 | |
| *** vogxn has quit IRC | 07:30 | |
| *** amotoki has joined #openstack-infra | 07:32 | |
| *** nayward has joined #openstack-infra | 07:35 | |
| *** toddmorey has joined #openstack-infra | 07:41 | |
| *** yaguang has quit IRC | 07:46 | |
| *** sdake has quit IRC | 07:46 | |
| *** toddmorey has quit IRC | 07:58 | |
| *** dina_belova has joined #openstack-infra | 07:59 | |
| *** SergeyLukjanov has joined #openstack-infra | 07:59 | |
| *** yaguang has joined #openstack-infra | 08:06 | |
| *** jjmb has quit IRC | 08:07 | |
| *** hashar has joined #openstack-infra | 08:08 | |
| *** dina_belova has quit IRC | 08:10 | |
| *** derekh has joined #openstack-infra | 08:27 | |
| *** fifieldt has quit IRC | 08:27 | |
| *** dina_belova has joined #openstack-infra | 08:28 | |
| *** vogxn has joined #openstack-infra | 08:28 | |
| *** nayward has quit IRC | 08:49 | |
| *** nayward has joined #openstack-infra | 08:51 | |
| *** toddmorey has joined #openstack-infra | 08:58 | |
| *** nayward has quit IRC | 08:58 | |
| *** toddmorey has quit IRC | 09:02 | |
| *** nayward has joined #openstack-infra | 09:03 | |
| *** toddmorey has joined #openstack-infra | 09:03 | |
| *** bnemec has quit IRC | 09:05 | |
| *** bnemec has joined #openstack-infra | 09:06 | |
| *** nayward has quit IRC | 09:13 | |
| *** toddmorey has quit IRC | 09:19 | |
| *** nayward has joined #openstack-infra | 09:24 | |
| *** nayward has quit IRC | 09:28 | |
| *** carlp-away is now known as carlp | 09:30 | |
| openstackgerrit | Emilien Macchi proposed a change to openstack-infra/jeepyb: Add puppet-heat in launchpad project list https://review.openstack.org/39655 | 09:30 |
| *** nayward has joined #openstack-infra | 09:43 | |
| *** dina_belova has quit IRC | 09:57 | |
| *** vandermerwe has joined #openstack-infra | 10:02 | |
| *** toddmorey has joined #openstack-infra | 10:11 | |
| *** toddmorey has quit IRC | 10:27 | |
| *** nayward has quit IRC | 10:29 | |
| *** vandermerwe has quit IRC | 10:34 | |
| *** vandermerwe has joined #openstack-infra | 10:38 | |
| *** SergeyLukjanov has quit IRC | 10:41 | |
| *** yaguang has quit IRC | 10:53 | |
| *** bnemec has quit IRC | 10:57 | |
| *** bnemec has joined #openstack-infra | 10:57 | |
| *** weshay has joined #openstack-infra | 10:57 | |
| *** dina_belova has joined #openstack-infra | 10:58 | |
| *** sdake_ has joined #openstack-infra | 10:59 | |
| *** sdake_ has joined #openstack-infra | 10:59 | |
| *** dina_belova has quit IRC | 11:02 | |
| *** dina_belova has joined #openstack-infra | 11:11 | |
| *** SergeyLukjanov has joined #openstack-infra | 11:11 | |
| openstackgerrit | Sean Dague proposed a change to openstack/requirements: update jsonschema requirements to match reality https://review.openstack.org/39670 | 11:20 |
| *** jjmb has joined #openstack-infra | 11:26 | |
| *** toddmorey has joined #openstack-infra | 11:27 | |
| *** vogxn has quit IRC | 11:27 | |
| *** odyssey4me has quit IRC | 11:28 | |
| *** vogxn has joined #openstack-infra | 11:29 | |
| *** hashar has quit IRC | 11:30 | |
| *** boris-42 has joined #openstack-infra | 11:31 | |
| *** toddmorey has quit IRC | 11:31 | |
| *** odyssey4me has joined #openstack-infra | 11:36 | |
| *** toddmorey has joined #openstack-infra | 11:39 | |
| sdague | mordred: so global requirements repo tests are pretty fubar | 11:41 |
| sdague | https://review.openstack.org/#/c/39670/ and http://logs.openstack.org/61/39461/1/gate/gate-requirements-install/811ea23/ | 11:41 |
| *** rfolco has joined #openstack-infra | 11:41 | |
| openstackgerrit | Sean Dague proposed a change to openstack/requirements: make sqla requirements safe for nova https://review.openstack.org/39461 | 11:42 |
| *** mriedem has joined #openstack-infra | 11:42 | |
| openstackgerrit | Sean Dague proposed a change to openstack/requirements: update jsonschema requirements to match reality https://review.openstack.org/39670 | 11:42 |
| lifeless | mordred: https://review.openstack.org/#/c/30522/32/pbr/extra_files.py <- no feedback from you on those suggestions | 11:43 |
| lifeless | mordred: you actioned one, the other not? | 11:43 |
| *** vogxn has quit IRC | 11:45 | |
| *** vsergeyev has joined #openstack-infra | 11:48 | |
| *** hashar has joined #openstack-infra | 11:49 | |
| *** zul has quit IRC | 11:54 | |
| *** vandermerwe has quit IRC | 11:55 | |
| *** toddmorey has quit IRC | 11:55 | |
| *** zul has joined #openstack-infra | 11:57 | |
| *** HenryG has joined #openstack-infra | 11:58 | |
| *** sandywalsh has quit IRC | 11:58 | |
| *** dkranz has quit IRC | 12:00 | |
| *** bnemec has quit IRC | 12:02 | |
| *** lcestari has joined #openstack-infra | 12:06 | |
| *** dkliban_afk has joined #openstack-infra | 12:08 | |
| *** sandywalsh has joined #openstack-infra | 12:11 | |
| *** vandermerwe has joined #openstack-infra | 12:12 | |
| *** carlp is now known as carlp-away | 12:15 | |
| *** weshay has quit IRC | 12:15 | |
| *** weshay has joined #openstack-infra | 12:15 | |
| *** vandermerwe has quit IRC | 12:20 | |
| *** vandermerwe has joined #openstack-infra | 12:20 | |
| *** CaptTofu has joined #openstack-infra | 12:20 | |
| *** vandermerwe is now known as cthulhup | 12:25 | |
| *** zul has quit IRC | 12:27 | |
| *** zul has joined #openstack-infra | 12:27 | |
| *** woodspa has joined #openstack-infra | 12:35 | |
| *** toddmorey has joined #openstack-infra | 12:35 | |
| *** zul has quit IRC | 12:50 | |
| *** zul has joined #openstack-infra | 12:51 | |
| *** cthulhup has quit IRC | 13:02 | |
| *** zul has quit IRC | 13:02 | |
| *** zul has joined #openstack-infra | 13:03 | |
| *** yaguang has joined #openstack-infra | 13:03 | |
| *** cthulhup has joined #openstack-infra | 13:05 | |
| *** vogxn has joined #openstack-infra | 13:13 | |
| *** dkliban_afk has quit IRC | 13:18 | |
| *** jjmb has quit IRC | 13:20 | |
| *** fbo_away is now known as fbo | 13:29 | |
| *** changbl has quit IRC | 13:29 | |
| *** zul has quit IRC | 13:30 | |
| *** CaptTofu has quit IRC | 13:32 | |
| *** krtaylor has quit IRC | 13:32 | |
| *** amotoki has quit IRC | 13:32 | |
| *** CaptTofu has joined #openstack-infra | 13:32 | |
| *** CaptTofu has quit IRC | 13:36 | |
| mordred | lifeless: purely for api symmetry. having a setter and then attribute access for the getter just feel weird | 13:40 |
| sdague | mordred: so... the requirements check job, any idea what's up with it? | 13:45 |
| openstackgerrit | Monty Taylor proposed a change to openstack-infra/config: Put the needed -r in https://review.openstack.org/39706 | 13:46 |
| mordred | sdague: ^^ | 13:46 |
| mordred | that | 13:46 |
| *** dkranz has joined #openstack-infra | 13:47 | |
| *** pentameter has joined #openstack-infra | 13:47 | |
| openstackgerrit | A change was merged to openstack-infra/config: Put the needed -r in https://review.openstack.org/39706 | 13:49 |
| mordred | sdague: k. when puppet kicks, that job should be fixed. sorry | 13:49 |
| *** shardy is now known as shardy_afk | 13:49 | |
| mordred | sdague: also, we discussed yesterday that that job can go away once the pbr/jeepyb gate on requirements lands - and once the devstack requirements gate lands | 13:50 |
| *** cthulhup has quit IRC | 13:51 | |
| *** anteaya has joined #openstack-infra | 13:52 | |
| *** prad_ has joined #openstack-infra | 13:52 | |
| koolhead11 | mordred, morning | 13:56 |
| mordred | morning koolhead11 | 13:57 |
| *** dhellmann_ is now known as dhellmann | 13:57 | |
| koolhead11 | mordred, dm | 13:59 |
| *** SergeyLukjanov has quit IRC | 14:01 | |
| *** amotoki has joined #openstack-infra | 14:03 | |
| *** vijendar has joined #openstack-infra | 14:03 | |
| sdague | mordred: cool, let me know when you think rechecks are safe | 14:04 |
| sdague | the night's sleep helped a lot in understanding why we were failing. I was completely lost yesterday afternoon, and this morning after 2 sips of coffee I saw it | 14:05 |
| mordred | sdague: sleep and coffee are often the bug | 14:06 |
| sdague | heh | 14:06 |
| *** hashar_ has joined #openstack-infra | 14:06 | |
| saper | [A | 14:06 |
| *** mrodden has joined #openstack-infra | 14:07 | |
| *** hashar has quit IRC | 14:08 | |
| *** hashar_ is now known as hashar | 14:08 | |
| mordred | sdague: I rechecked that one alreadyu | 14:08 |
| *** dkliban_afk has joined #openstack-infra | 14:09 | |
| *** hashar has left #openstack-infra | 14:09 | |
| *** mrodden has quit IRC | 14:10 | |
| *** pabelanger has quit IRC | 14:11 | |
| *** mrodden has joined #openstack-infra | 14:11 | |
| *** toddmorey has quit IRC | 14:11 | |
| *** markmcclain has joined #openstack-infra | 14:11 | |
| *** CaptTofu has joined #openstack-infra | 14:11 | |
| BobBall | sdague - any thoughts on how you'd like devstack to handle the badness in removal-of-python-lxml? | 14:11 |
| BobBall | Do you think we need to use venv? | 14:12 |
| sdague | we really can't use a venv in the gate | 14:12 |
| *** amotoki has quit IRC | 14:12 | |
| BobBall | oh - why? just for my understanding | 14:12 |
| *** markmcclain has quit IRC | 14:12 | |
| *** mrodden has quit IRC | 14:12 | |
| *** markmcclain has joined #openstack-infra | 14:12 | |
| *** mrodden has joined #openstack-infra | 14:13 | |
| mordred | BobBall: also, could you remind me what the removal-of-lxml problem is? | 14:14 |
| BobBall | it force removes Xen | 14:14 |
| mordred | ah yes. that one | 14:15 |
| BobBall | I hit my head a few times for that | 14:16 |
| BobBall | couldn't understand it... | 14:16 |
| BobBall | xen was installed | 14:16 |
| BobBall | I ran devstack | 14:16 |
| BobBall | and then things started breaking | 14:16 |
| BobBall | but I didn't tie it back to running devstack because I didn't imagine that it would be removing xen! | 14:16 |
| mordred | BobBall, sdague: so, what if we wrap the remove lxml in an ifndef virt=xen? | 14:17 |
| mordred | or ... | 14:17 |
| sdague | yeh, can we just not do it in the xen case? | 14:17 |
| BobBall | but the problem with wanting to remove lxml exists for xen as well as non-xen | 14:17 |
| BobBall | so if we did that there might be random breakages when running devstack with xen | 14:18 |
| mordred | hrm | 14:18 |
| mordred | oh right - this is the thing that caused me to start beating my head against my desk the other day | 14:18 |
| BobBall | Sorry... and I hope your desk is padded :D | 14:18 |
| mordred | nope. but my head is not veryuseful | 14:19 |
| mordred | ok - here's a suggestion... (it's bad) | 14:19 |
| BobBall | Fair enough! | 14:19 |
| mordred | what if... | 14:19 |
| *** krtaylor has joined #openstack-infra | 14:19 | |
| mordred | we changed this line: | 14:19 |
| mordred | if [[ $TRACK_DEPENDS = True ]]; then | 14:19 |
| mordred | to | 14:19 |
| *** avtar has joined #openstack-infra | 14:20 | |
| mordred | if [[ $TRACK_DEPENDS = True -o "$VIRT_DRIVER" = 'xenserver' ]]; then | 14:20 |
| BobBall | so if we're using XS then we always use a venv? | 14:20 |
| BobBall | and we then also prevent removal of python-lxml in that case | 14:20 |
| mordred | we'd also have to wrap the lxml thing | 14:20 |
| mordred | right | 14:20 |
| BobBall | That might work | 14:20 |
| mordred | other option... | 14:21 |
| BobBall | What's the purpose of TRACK_DEPENDS? is it roughly FORCE_VENV or could I add that as a param? | 14:21 |
| mordred | figure out how to get an lxml version new enough onto the system via rpm | 14:21 |
| *** cthulhup has joined #openstack-infra | 14:22 | |
| mordred | so that pip install -U lxml is a no-op | 14:22 |
| BobBall | it should be a no-op ATM | 14:22 |
| BobBall | I think the version in CentOS is new enough | 14:22 |
| BobBall | oh | 14:22 |
| BobBall | hang on | 14:22 |
| mordred | then are we protecting against a problem that's only theoretical? | 14:22 |
| BobBall | maybe I'm thinking of a diff one | 14:22 |
| *** cthulhup has quit IRC | 14:22 | |
| BobBall | theoretical in this CentOS case - let me check | 14:22 |
| mordred | :) | 14:22 |
| *** cthulhup has joined #openstack-infra | 14:22 | |
| *** yaguang has quit IRC | 14:24 | |
| BobBall | Boo :( I'm wrong. pip freeze shows lxml=3.2.3 but only python-lxml 2.2.3 is available | 14:24 |
| *** zul has joined #openstack-infra | 14:25 | |
| dtroyer | BobBall: I think I just realized something…you're working on a xen-on-RHEL6 here right? Not a XenServer (or whatever the self-contained product is)? | 14:26 |
| BobBall | this seems to be a CentOS issue then - things like F19 have 3.2.1 | 14:26 |
| BobBall | xenserver-core on CentOS 6, right. Not XenServer itself - but the XAPI packages etc | 14:26 |
| BobBall | I'm trying to make devstack run in dom0 (otherwise devstack is running in a VM which is always fine because it's seperate to xen and XAPI so python-lxml isn't needed for us) | 14:27 |
| dtroyer | BobBall: ok, thanks…I've been imagining XenServer all this time…not sure what that changes except my assumptions... | 14:27 |
| *** andrew_plunk has joined #openstack-infra | 14:27 | |
| dtroyer | BobBall: IIRC this is one reason its in domU there | 14:28 |
| BobBall | devstack in dom0 on XenServer is much harder because XS is stripped down - loads of packages would be missing... | 14:28 |
| BobBall | Let me try adding a config "FORCE_VENV" to devstack which would pick parts of the TRACK_DEPENDS code (and TRACK_DEPENDS would add FORCE_VENV) - that might be good enough | 14:29 |
| BobBall | Why can't we use venv in the gate though? | 14:29 |
| *** datsun180b has joined #openstack-infra | 14:29 | |
| *** luis_ has joined #openstack-infra | 14:30 | |
| dtroyer | BobBall: thinking out loud: if we 'fix' the python path issue (or work around it) on RHEL this all goes away because pip-installed packages co-exist and take priority over rpm-installed packages | 14:30 |
| *** _TheDodd_ has joined #openstack-infra | 14:30 | |
| luis_ | hi i have some changes that didn't get picked up by jenkins here is one: https://review.openstack.org/#/c/39589/ | 14:31 |
| luis_ | any help is appreciated | 14:31 |
| BobBall | What is the generic python path in RHEL problem? | 14:31 |
| dtroyer | (from memory) pip installs packages to the same paths as rpm, overwriting the rpm-managed files. | 14:32 |
| BobBall | why is that RHEL specific though | 14:33 |
| dtroyer | I forget which one is wrong, but IIRc it is a RHEL-specific config issue | 14:33 |
| BobBall | ah haha | 14:33 |
| dtroyer | dunno why, fedora works | 14:33 |
| mordred | BobBall: mainly because we want to _catch_ these issues and not gloss over them in the normal case | 14:33 |
| openstackgerrit | David Caro proposed a change to openstack-infra/jenkins-job-builder: Added pre-scm-buildstep wrapper https://review.openstack.org/39338 | 14:34 |
| *** adalbas has quit IRC | 14:34 | |
| sdague | TRACK_DEPENDS doesn't actually work, btw | 14:35 |
| BobBall | poo | 14:35 |
| BobBall | haha | 14:35 |
| BobBall | thanks sdague | 14:35 |
| sdague | I tried it yesterday | 14:36 |
| sdague | it's probably fixable | 14:36 |
| sdague | it just doesn't work as is in the tree | 14:36 |
| openstackgerrit | David Caro proposed a change to openstack-infra/jenkins-job-builder: Added pre-scm-buildstep wrapper https://review.openstack.org/39338 | 14:36 |
| sdague | BobBall: the reason we can't use venv in the gate is because we need to be able to have a system that works without them | 14:37 |
| sdague | because ubuntu and redhat and others need to be able to package this all at the end of the day | 14:37 |
| mordred | yah | 14:37 |
| sdague | so our gate needs to be able to produce a system, with python libs installed globally, that can run | 14:37 |
| mordred | also - ubuntu and redhat have both promised to provide backport system packages if needed | 14:37 |
| BobBall | But it might be OK for some jobs to run outside of venv and some to run inside? e.g. perhaps xenserver-core could gate using a venv as long as the existing devstack runs don't? | 14:38 |
| *** dina_belova has quit IRC | 14:39 | |
| *** changbl has joined #openstack-infra | 14:40 | |
| clarkb | luis_ It looks like that got lost in the zuul + jenkins log shuffle yesterday. you can leave a comment on the change with 'recheck no bug' to rerun the tests | 14:43 |
| *** psedlak has joined #openstack-infra | 14:43 | |
| luis_ | clarkb, ah ic, great thx | 14:43 |
| clarkb | jhesketh: yes all of the unittest slaves have mysql, postgresql, and mongodb available to them | 14:44 |
| *** cthulhup has quit IRC | 14:44 | |
| sdague | BobBall: for live testing we really can't use a venv | 14:45 |
| *** adalbas has joined #openstack-infra | 14:46 | |
| sdague | mordred: hmmm.... so puppet run yet? | 14:46 |
| sdague | oh there it is | 14:46 |
| BobBall | okay - then I don't think that TRACK_DEPENDS or FORCE_VENV would work - I don't want a solution that would prevent us from adding a gate job in the future | 14:48 |
| fungi | jhesketh: if you're still wondering about databases tested in our ci, we do have for example opportunistic tests against mysql, postgres and mongodb in some projects, though the scope of those tests is fairly limited they do interact with actual databases | 14:49 |
| clarkb | we remove python-lxml? I thought projects depend on it? | 14:50 |
| BobBall | we remove it then pip reinstalls it | 14:50 |
| sdague | clarkb: just in RHEL, because, rhel is *fun* | 14:51 |
| *** burt has joined #openstack-infra | 14:51 | |
| fungi | luis_: was 39589 originally uploaded as a draft? if so, tests don't run until a new (non-draft) patchset is uploaded or the published draft is rechecked with a review comment. also your change needs to be rebased onto the tip of master to eliminate existing merge conflicts between them before it can be tested | 14:51 |
| *** adalbas has quit IRC | 14:51 | |
| clarkb | is this related to the discussion mordred had in #openstack-dev? we need a different local pip root on rhel and friends | 14:51 |
| BobBall | I thought we always removed python-lxml - due to the RHEL issue - but it's removed everywhere? | 14:51 |
| clarkb | BobBall on debian dervatives system and pip packages install side by side | 14:52 |
| *** andrew_p_ has joined #openstack-infra | 14:52 | |
| fungi | luis_: nevermind, clarkb seems to have replied to you already | 14:52 |
| clarkb | not sure if we need to forcefully remove things there | 14:52 |
| mordred | sdague: I just wrote this: https://review.openstack.org/39725 | 14:52 |
| BobBall | agree we probably don't _need_ to - but we do it anyway :) | 14:52 |
| mordred | sdague: because the world is an evil place | 14:52 |
| sdague | heh | 14:52 |
| *** michchap has joined #openstack-infra | 14:52 | |
| sdague | ok, I'll look | 14:52 |
| *** michchap has quit IRC | 14:52 | |
| * fungi needs to not respond to people while reading scrollback | 14:52 | |
| mordred | yes | 14:53 |
| mordred | it's related to ... wait for it ... wait for it ... | 14:53 |
| sdague | mordred: can you land sqla now that it passes requirements | 14:53 |
| clarkb | fungi its ok, that is what I did. | 14:53 |
| mordred | redhat instaling local packages into /usr | 14:53 |
| clarkb | fungi I just happened to do it first :) | 14:53 |
| *** michchap has joined #openstack-infra | 14:53 | |
| *** hashar has joined #openstack-infra | 14:53 | |
| mordred | thereby completely removing all logic and reason around package installed vs. source installed thigns | 14:53 |
| clarkb | mordred the lxml thing? | 14:53 |
| mordred | everything | 14:54 |
| *** andrew_plunk has quit IRC | 14:54 | |
| mordred | everything | 14:54 |
| mordred | python setup.py install installs into /usr | 14:54 |
| clarkb | :( | 14:54 |
| luis_ | fungi, actually i think that may have been the problem, i had tried to push the patch out as a draft but it failed, so i ended up pushing it as non-draft, and marking it as wip, but maybe that is why it wasnt picked up | 14:54 |
| mordred | seriously, who at redhat can I horsewhip | 14:54 |
| luis_ | thanks though, the recheck worked | 14:54 |
| BobBall | dprince? ;) | 14:54 |
| mordred | this is a completely unworkable and daft thing | 14:54 |
| mordred | he's not in channel | 14:54 |
| mordred | russellb: can you find me someone at redhat to yell at and throw things please? | 14:55 |
| BobBall | It was an opportune time to go on holiday then | 14:55 |
| mordred | well, I'm betting this is old and deep and I'm going to get nowhere | 14:55 |
| mordred | sdague: on it | 14:55 |
| sdague | so very close on global requirements.... so very close | 14:56 |
| fungi | sdague: reapproved that one now | 14:56 |
| mordred | fungi: could you do the jsonschema one too? | 14:56 |
| sdague | dtroyer: can you take a look at this - https://review.openstack.org/#/c/39725/1/lib/infra ? | 14:57 |
| fungi | mordred: yup was just looking at it | 14:57 |
| sdague | because my bash-fu starts to glaze over on that | 14:57 |
| mordred | sdague, dtroyer: that's not for the requirements thing - it's for the other | 14:57 |
| sdague | mordred: right, I know | 14:57 |
| dtroyer | am testing fedora now…not sure it has the same problem RHEL does | 14:57 |
| russellb | mordred: sorry. i accept your horsewhip on behalf of red hat. | 14:58 |
| mordred | dtroyer: it does - I spun up a fedora node and verified the /usr install behavior | 14:58 |
| * mordred horsewhips russellb | 14:58 | |
| mordred | mm. that was fun :) | 14:58 |
| russellb | ha | 14:58 |
| mordred | https://bugzilla.redhat.com/show_bug.cgi?id=662034 | 14:58 |
| uvirtbot | mordred: Error: Could not parse XML returned by bugzilla.redhat.com: HTTP Error 404: Not Found | 14:58 |
| sdague | man, the comments in devstack keep getting more and more fun :) | 14:58 |
| *** SergeyLukjanov has joined #openstack-infra | 14:59 | |
| fungi | we need some kernel-grade snarkisms in there | 14:59 |
| dtroyer | fungi: on it | 14:59 |
| sdague | heh | 14:59 |
| fungi | nothing brightens my day like slogging through dull code review only to stumble across a humorous/sarcastic comment | 15:00 |
| mordred | alternately: https://bugzilla.redhat.com/show_bug.cgi?id=662034#c4 | 15:00 |
| uvirtbot | mordred: Error: Could not parse XML returned by bugzilla.redhat.com: HTTP Error 404: Not Found | 15:00 |
| sdague | I was pretty close to putting the whole of who's on first in the unfubar_setuptools function | 15:00 |
| fungi | ha | 15:00 |
| mordred | omg uvirtbot. those are not launchpad urls | 15:00 |
| *** andrew_p_ has quit IRC | 15:00 | |
| *** mrodden has quit IRC | 15:01 | |
| fungi | uvirtbot is hoping rh might tack an lp api onto their bz | 15:01 |
| uvirtbot | fungi: Error: "is" is not a valid command. | 15:01 |
| *** andrew_plunk has joined #openstack-infra | 15:01 | |
| fungi | also, hoping that i might say something actionable, apparently | 15:01 |
| *** cp16net is now known as cp16net|away | 15:02 | |
| *** krtaylor has quit IRC | 15:03 | |
| openstackgerrit | A change was merged to openstack/requirements: make sqla requirements safe for nova https://review.openstack.org/39461 | 15:03 |
| mordred | russellb: how do I find the source for the python spec file for the official rpms for fedora? and how do I submit a patch? | 15:07 |
| *** cp16net|away is now known as cp16net | 15:07 | |
| openstackgerrit | A change was merged to openstack/requirements: update jsonschema requirements to match reality https://review.openstack.org/39670 | 15:07 |
| russellb | mordred: http://pkgs.fedoraproject.org/cgit/python.git/ | 15:08 |
| russellb | mordred: patches submitted to bugzilla.redhat.com | 15:08 |
| *** adalbas has joined #openstack-infra | 15:08 | |
| russellb | mordred: so master is rawhide, and then a branch for each fedora release | 15:09 |
| *** pabelanger has joined #openstack-infra | 15:10 | |
| mordred | russellb: thanks! | 15:11 |
| russellb | sure np | 15:11 |
| mordred | russellb: I'm going to attempt to just submit the patch to unbreak python | 15:11 |
| mordred | russellb: rather than asking other people to | 15:11 |
| russellb | nice moves | 15:11 |
| *** boris-42 has quit IRC | 15:11 | |
| *** mrodden has joined #openstack-infra | 15:12 | |
| russellb | mordred: or you want to try a test build of a package with your changes, you should be able to do it with the 'mock' utility | 15:13 |
| russellb | or there's a 'fedpkg' thing that wraps a bunch of stuff and can invoke mock correctly for ou | 15:13 |
| *** vogxn has quit IRC | 15:14 | |
| russellb | s/or you/if you/ | 15:14 |
| *** shardy_afk is now known as shardy | 15:14 | |
| mordred | russellb: neat! | 15:14 |
| russellb | fedpkg mockbuild | 15:14 |
| russellb | builds the package in a chroot populated with the deps from the spec file | 15:15 |
| BobBall | That's great for when it comes down/if it gets accepted - but to work around in devstack could we use pip install --install-option="--prefix=..."? | 15:15 |
| *** toddmorey has joined #openstack-infra | 15:15 | |
| russellb | mordred: gif of the week, seems applicable here: http://d3j5vwomefv46c.cloudfront.net/photos/large/795272903.gif?1375176386 | 15:16 |
| russellb | right sdague? :-) | 15:16 |
| *** toddmorey has quit IRC | 15:17 | |
| *** dina_belova has joined #openstack-infra | 15:19 | |
| *** jjmb has joined #openstack-infra | 15:19 | |
| clarkb | poor Hal^H^H^Hmordred | 15:19 |
| *** toddmorey has joined #openstack-infra | 15:21 | |
| *** toddmorey has quit IRC | 15:23 | |
| *** sparkycollier has joined #openstack-infra | 15:23 | |
| fungi | i feel like that *all* the time | 15:25 |
| fungi | i think there's something i started to fix about 20 years ago and what i'm doing now is just the other prerequisite work which has cropped up | 15:25 |
| sdague | russellb: right on man :) | 15:26 |
| *** cppcabrera has joined #openstack-infra | 15:26 | |
| *** cppcabrera has left #openstack-infra | 15:27 | |
| *** toddmorey has joined #openstack-infra | 15:27 | |
| *** vogxn has joined #openstack-infra | 15:29 | |
| openstackgerrit | Thierry Carrez proposed a change to openstack-infra/odsreg: Fix module paths in recent additions https://review.openstack.org/39735 | 15:29 |
| *** vijendar has quit IRC | 15:31 | |
| mrodden | so the manage_projects.py in jeepyb is designed to be run on the Gerrit server locally? | 15:32 |
| mrodden | i might hack that up to be run as remotely as a jenkins job or something... | 15:33 |
| *** UtahDave has joined #openstack-infra | 15:35 | |
| openstackgerrit | A change was merged to openstack-infra/odsreg: Fix module paths in recent additions https://review.openstack.org/39735 | 15:35 |
| clarkb | mrodden: it is but the only local thing it does is create the local repo replica. should be possible to run remotely without that | 15:36 |
| clarkb | I believe mordred may have done this before? | 15:36 |
| mrodden | clarkb: yep, thats what i concluded as well... not sure why the local project needs to be made by manage_projects, i think Gerrit does that anyways? | 15:37 |
| mrodden | sorry local git repo | 15:37 |
| *** rnirmal has joined #openstack-infra | 15:37 | |
| clarkb | gerrit does not do it | 15:37 |
| mrodden | really? | 15:38 |
| mrodden | i always end up with a git repo in Gerrit when i used the web ui to make a project | 15:38 |
| clarkb | replication will fail and you get a lot of stacktraces. gerrit does create the non replica repos under its review site dir though | 15:38 |
| mordred | what did I do? | 15:38 |
| mrodden | oh a replica | 15:38 |
| clarkb | yes, we use a local replica that apache serves to reduce load on the gerrit werver | 15:39 |
| notmyname | the pep8 spec has been updated (first time since 2001). I hope this doesn't cause a lot of stupid patches in openstack projects http://hg.python.org/peps/rev/fb24c80e9afb | 15:40 |
| clarkb | it shouldn't if we continue to pin pep8 each release | 15:41 |
| mrodden | clarkb: that makes sense, i thought that project_dir was pointing to the git repo under gerrit_home/git/ | 15:41 |
| clarkb | notmyname can we vote and gate on the swift functional tests? | 15:41 |
| notmyname | clarkb: hmm...have they been causing any problems? | 15:42 |
| clarkb | notmyname: not on our end. I was hoping you would know :) | 15:42 |
| notmyname | clarkb: actually, I've been somewhat disconnected recently (oscon + another one day conf this week). today, I'm WFH to get caught up on the openstack stuff | 15:42 |
| notmyname | clarkb: I hope to be able to give you a better answer tomorrow | 15:43 |
| *** hashar has quit IRC | 15:43 | |
| clarkb | that works, thanks | 15:43 |
| notmyname | clarkb: in general, I don't see it as a pressing issue, so figuring that out hasn't been at the top of my priority list. but it's good to have (IMO), so sooner is better than later :-) | 15:44 |
| pleia2 | good morning | 15:44 |
| clarkb | notmyname: definitely not pressing, but trying to avoid tests that only run in the check queue that eventually always fail... | 15:45 |
| clarkb | pleia2 good morning | 15:46 |
| *** dina_belova has quit IRC | 15:47 | |
| EmilienM | I'm not sure to be at the right place, but I have a question. I've done a bunch of monitoring scripts for Nagios (working with other tools like Sensu). They are here : http://goo.gl/vZyWkd and also on Nagios Exchange. I was wondering if it could make sense to create a dedicated repository on StackForge ? | 15:47 |
| mordred | notmyname: we will not update the pep8 pin until the next release cycle | 15:48 |
| *** SergeyLukjanov has quit IRC | 15:48 | |
| notmyname | mordred: sure sure. but even so, updates like "line length of 79 is preferred, but up to 99 is ok" could lead to all sorts of silly patch sets whenever the pin is moved :-/ | 15:49 |
| mordred | notmyname: well, I doubt anyone will lengthen the lines | 15:50 |
| mordred | notmyname: but I know that line will make jeblair quite happy | 15:50 |
| notmyname | not that I think updating pep8 for the first time in 11 years is bad :-) | 15:50 |
| mordred | reading through the updates right now, so far they all seem great | 15:50 |
| mordred | ah - the explicit relative imports rule has been relaxed | 15:52 |
| anteaya | I don't know where else to ask this question, so feel free to redirect me if there is a better place, the topic is storyboard | 15:53 |
| anteaya | if I invoke the django shell with pythonn manage.py shell and load up the story and task models from: https://github.com/openstack-infra/storyboard/blob/master/storyboard/stories/models.py#L23-L56 | 15:53 |
| *** krtaylor has joined #openstack-infra | 15:54 | |
| jeblair | mordred, notmyname: "i look forward to reducing the number of silly line cont" | 15:54 |
| jeblair | "inuations." | 15:54 |
| mordred | jeblair: :) | 15:54 |
| mordred | "Always use a def statement instead of assigning a lambda expression to a name." | 15:54 |
| mordred | they had to put that in? | 15:54 |
| anteaya | I should get all the stories if I execute Story.objects.all() - which I do - and I also should get all the tasks if I execute Task.objects.all() should I not? | 15:55 |
| clarkb | mordred: that was my favorite change :) | 15:55 |
| clarkb | anteaya: do you not get all tasks? | 15:56 |
| clarkb | anteaya if you connect to the db directly do you see tasks? | 15:57 |
| anteaya | clarkb: no I get an error | 15:57 |
| marun | mordred: have you seen this?: https://bugs.launchpad.net/devstack/+bug/1207162 | 15:57 |
| uvirtbot | Launchpad bug 1207162 in devstack ""Unbreaking setuptools" fails under Precise" [Undecided,New] | 15:57 |
| anteaya | clarkb: I would only see the schema, but I can look at the schema, I will paste both | 15:57 |
| mordred | marun: how could it fail under precise? | 15:57 |
| marun | mordred: It's failing for me, virgin install | 15:58 |
| mordred | marun: we run all of the devstack-gate under precise? | 15:58 |
| burt | I reported that bug | 15:58 |
| * mordred is so confused | 15:58 | |
| mordred | how does it break? | 15:58 |
| * mordred goes to read bug | 15:58 | |
| marun | VersionConflict: (netaddr 0.7.5 (/usr/lib/python2.7/dist-packages), Requirement.parse('netaddr>=0.7.6')) | 15:58 |
| marun | mordred ^ | 15:58 |
| marun | nova can't load any plugins due to this conflict | 15:58 |
| *** zul has quit IRC | 15:58 | |
| burt | In short /usr/local/lib/python2.7/dist-packages/easy-install.pth gets /usr/lib/python2.7/dist-packages instead of /usr/local/lib/python2.7/dist-packages due to that change | 15:58 |
| *** jjmb has quit IRC | 15:59 | |
| anteaya | clarkb: https://gist.github.com/anteaya/5e9f0d7a06af456756eb | 15:59 |
| * mordred sobs | 15:59 | |
| burt | (and I think the gate isn't updated yet, isn't it a clone from https://review.openstack.org/p/openstack-infra/devstack-gate ?) | 16:00 |
| mordred | no | 16:00 |
| anteaya | clarkb I am tracking down an error, a newly created task is not viewed, I am trying to find out it if is in the db | 16:00 |
| mordred | devstack is run from trunk ever time | 16:00 |
| burt | that's a wtf moment for me then too | 16:00 |
| mordred | on a pristine precise machine | 16:00 |
| jeblair | mordred: nice stuff in there about exception chaining. | 16:01 |
| mordred | jeblair: yah | 16:01 |
| *** sarob has joined #openstack-infra | 16:01 | |
| mordred | burt, marun: well, I'm working on the unbreak-setuptools for fedora | 16:01 |
| mordred | where redhat has decided to do CRAZY things with python | 16:01 |
| mordred | I'll stick this on the list for after taht | 16:01 |
| burt | I think this is higher priority IMHO | 16:01 |
| anteaya | clarkb: so I can see that the stories_task table is created, but I can't see the tasks, I am not sure if I should be able to see them in the shell independently of the story model or not, not enough django foo yet | 16:01 |
| clarkb | anteaya: looks like the source and pyc are out of sync maybe? | 16:01 |
| anteaya | okay let me look at that | 16:02 |
| anteaya | how do I fix it? | 16:02 |
| anteaya | nuke the pyc? | 16:02 |
| clarkb | ya I would try that | 16:02 |
| anteaya | okay thanks | 16:02 |
| marun | mordred: I would vote with burt. The fact that this is happening despite the gate passing is a serious concern. | 16:02 |
| anteaya | clarkb: so maybe when I git pull changes I have to nuke the storyboard.db file and the pyc files | 16:03 |
| anteaya | okay | 16:03 |
| mordred | I think it's actually the same block of code- so it's likely we can fix both with the same patch | 16:03 |
| clarkb | anteaya: ya I would do that | 16:03 |
| anteaya | okay thanks | 16:03 |
| marun | mordred: I'm trying to create a precise deployment to work on the bug that is preventing neutron from being in the gate, and I can't even get devstack to complete on precise. | 16:03 |
| clarkb | mordred you know we have image building issues because openvz | 16:04 |
| mordred | that is true | 16:04 |
| mordred | anyway - I'm looing in to it right now | 16:04 |
| marun | mordred: awesome, thank you! | 16:04 |
| *** michchap has quit IRC | 16:11 | |
| *** gyee has joined #openstack-infra | 16:15 | |
| *** andrew_plunk has left #openstack-infra | 16:16 | |
| *** david-lyle has joined #openstack-infra | 16:17 | |
| openstackgerrit | A change was merged to openstack-infra/config: Further tuning of lists https://review.openstack.org/39610 | 16:17 |
| *** vijendar has joined #openstack-infra | 16:18 | |
| anteaya | clarkb: w00t, just learned about pyclean | 16:18 |
| anteaya | pyclean storyboard worked | 16:19 |
| *** dina_belova has joined #openstack-infra | 16:20 | |
| *** zul has joined #openstack-infra | 16:24 | |
| *** SergeyLukjanov has joined #openstack-infra | 16:25 | |
| *** avtar has quit IRC | 16:28 | |
| mordred | marun: ok. I just ran by hand in a node that is what a current devstack-gate node would look like | 16:36 |
| mordred | so _something_ is happening in devstack-gate node prep that is making the node be in a different state than a pristine precise node | 16:37 |
| mordred | I wiil now track down what that is | 16:37 |
| *** CaptTofu has quit IRC | 16:37 | |
| *** BobBall is now known as BobBall_Away | 16:38 | |
| *** jjmb has joined #openstack-infra | 16:38 | |
| fungi | mordred: looking at the devstack image build failures, do we need to blow away the current cached snapshot and let it rebuild from a vanilla provider image? | 16:40 |
| jeblair | fungi: if we can't build the image, blowing away the image would be a bad idea | 16:40 |
| fungi | something is persisting in trying to grab the openvz tarball for 11.10 even though that's fixed in devstack since | 16:40 |
| jeblair | fungi: is it fixed in all branches? | 16:40 |
| fungi | oh! i bet that's it | 16:40 |
| jeblair | fungi: i believe mordred knew that and was doing a patch | 16:41 |
| fungi | ahh, okay. i was willing to backport his patch from yesterday if nobody was working on that | 16:41 |
| jeblair | fungi: if you remove the cached images, no nodes will be able to be built at all and everything will stop in about 5 minutes. | 16:41 |
| fungi | touche | 16:41 |
| jeblair | (it would likely succeed in raising the priority of the fix) | 16:41 |
| fungi | heh | 16:42 |
| jeblair | :) | 16:42 |
| jeblair | fungi: but it's designed to handle this case gracefully -- it tries to build a new image, and only if that succeeds, does it become the 'current' image for new nodes | 16:43 |
| clarkb | fungi I almost prefer removing the openvz bits at this point | 16:43 |
| clarkb | or hosting the image ourselves | 16:43 |
| jeblair | fungi: and any images that are not the 'current' image still stick around for 24 hours | 16:43 |
| clarkb | otherwisw we go through this again in 9 months | 16:43 |
| anteaya | clarkb: thanks, that worked, I can now see that creating a new storyboard task does not create an object in the database. Now to find out why. | 16:44 |
| jeblair | fungi: so if the new image breaks in an exciting way, you have 24 hours to manually remove it and the previous one will be used. | 16:44 |
| *** andrew_plunk has joined #openstack-infra | 16:44 | |
| *** cp16net is now known as cp16net|away | 16:44 | |
| fungi | jeblair: right it will keep using the old image. just wondering about cases where he old image is no longer capable of being used as a base for the new image (or does it always build from a fresh provider image rather than iterating from the previous one?) | 16:44 |
| jeblair | fungi: always builds from scratch | 16:44 |
| * fungi rescinds his misconceptions | 16:45 | |
| fungi | so yes, almost certainly just blocked on missing devstack stable backports now | 16:45 |
| *** fbo is now known as fbo_away | 16:47 | |
| mordred | burt: the easy-install.pth files that you pasted into the bug | 16:49 |
| mordred | burt: are those from /usr/local/lib/python2.7/dist-packages/easy-install.pth ? | 16:49 |
| *** sarob has quit IRC | 16:49 | |
| *** emagana has joined #openstack-infra | 16:49 | |
| burt | mordred: yep | 16:49 |
| mordred | burt: any chance you can show me what they look like _before_ a devstack run? | 16:49 |
| burt | yup | 16:49 |
| *** sarob has joined #openstack-infra | 16:49 | |
| burt | give me 2 minutes to rebuild the boxes | 16:49 |
| mordred | burt: because here's what I get: | 16:50 |
| mordred | http://paste.openstack.org/show/42876/ | 16:50 |
| mordred | which does not match either of yours | 16:50 |
| burt | .. hm. | 16:51 |
| burt | there's no /usr/local/lib/python2.7/dist-packages/easy-install.pth | 16:51 |
| mordred | ok. that was the state my system was in before I ran devstack | 16:52 |
| mordred | so that's good | 16:52 |
| *** pcrews has quit IRC | 16:52 | |
| mordred | I mean- in that we started from the same state | 16:52 |
| burt | heh | 16:52 |
| burt | you are starting from a clean precise 12.04 ? | 16:52 |
| mordred | burt: I'm starting from a devstack-gate node which has things precached (so the state of the nodes from the gate) | 16:54 |
| *** sarob has quit IRC | 16:54 | |
| burt | hmm. | 16:54 |
| mordred | burt: which starts as clean precise, but then some things do happen to prep it | 16:54 |
| mordred | oh - actually | 16:55 |
| mordred | burt: when you said doesn't work - in what way does it fail? | 16:55 |
| mordred | nevermind. I see the error in the bug | 16:55 |
| burt | yup | 16:55 |
| burt | mordred: lunch, brb in an hour | 16:56 |
| mordred | burt, marun: this is with you guys running with neutron enabled? | 16:57 |
| burt | afaik yes -- I don't disable/enable anything in the localrc but take the default | 16:57 |
| mordred | ok. no, then you don't get neutron | 16:57 |
| burt | er no then | 16:57 |
| *** burt is now known as burt-afk | 16:57 | |
| mordred | I don't see any attempts to stevedore netaddr in my logs | 16:58 |
| *** vogxn has quit IRC | 16:58 | |
| marun | mordred: I was trying to run with neutron | 16:59 |
| *** vogxn has joined #openstack-infra | 16:59 | |
| marun | mordred: does neutron have some os dependencies that are breaking things? | 16:59 |
| mordred | marun: unclear - but it's breaking for burt without neutron | 17:00 |
| marun | mordred: ah, ok | 17:00 |
| marun | mordred: how long has neutron been out of gate | 17:00 |
| marun | ? | 17:00 |
| mordred | marun: several weeks at least | 17:00 |
| marun | mordred: oy | 17:01 |
| *** koolhead11 has quit IRC | 17:01 | |
| mordred | marun: it was flaky/breaking everything for a WHILE before that | 17:01 |
| marun | mordred: :( | 17:02 |
| mordred | omg. my new test node is the best thing in the world | 17:02 |
| mordred | marun: I'm going to start clean and just run that code and see what it does to the system | 17:03 |
| marun | mordred: ok | 17:04 |
| *** nati_ueno has joined #openstack-infra | 17:04 | |
| *** boris-42 has joined #openstack-infra | 17:05 | |
| *** ^d has joined #openstack-infra | 17:06 | |
| *** ladquin has quit IRC | 17:07 | |
| mordred | jeblair, clarkb: devstack-gate nodes have setuptools and pip pre-installed on them because I believe we're doing that in our base template now | 17:09 |
| jeblair | mordred: i believe that is the case | 17:10 |
| mordred | jeblair: I think we need to move it up from outside of that - because it's pre-doing something outside of devstack that devstack is also trying to do | 17:10 |
| mordred | so we're not catching errors in devstack's attempt | 17:10 |
| jeblair | mordred: okay; this is starting to feel icky; it is apparently required for absolutely everything we do... | 17:11 |
| mordred | yeah. agree | 17:11 |
| jeblair | mordred: because pbr doesn't work without it | 17:11 |
| mordred | it might be time to step back and re-look at the whole problem | 17:11 |
| jeblair | mordred: yeah; keep in mind that devstack nodes aren't just used for devstack now | 17:11 |
| mordred | and it's not just pbr - it's anything in the world that tries to install distribute as part of its depends via pip | 17:11 |
| jeblair | mordred: so if you fix it for devstack, it may break for something else | 17:12 |
| mordred | ++ | 17:12 |
| *** koolhead11 has joined #openstack-infra | 17:13 | |
| mordred | jeblair: so - there is another option here... | 17:14 |
| mordred | the big reason to attempt to upgrade setuptools out of band is to prevent anything that depends on distribute from breaking the system | 17:15 |
| *** jpich has quit IRC | 17:15 | |
| mordred | pbr used to depend on d2to1 which depended on distribute | 17:15 |
| dstufft | pip 1.4 internally translates distribute to setuptools IIRC | 17:16 |
| *** derekh has quit IRC | 17:16 | |
| dstufft | and distribute <= 0.7 translates setuptools to distribute | 17:16 |
| dstufft | :V | 17:16 |
| mordred | dstufft: are you saying that if I upgrade to pip 1.4 _first_ that the distribute translation might be caught? | 17:16 |
| mordred | by pip? | 17:16 |
| dstufft | supposidily | 17:17 |
| dstufft | I didn't write or test that | 17:17 |
| dstufft | but that's what i'm told | 17:17 |
| dstufft | mordred: https://github.com/pypa/pip/issues/1033 | 17:18 |
| *** thomasbiege has joined #openstack-infra | 17:18 | |
| mordred | dstufft: nope. doesn't work | 17:20 |
| *** sarob has joined #openstack-infra | 17:20 | |
| dstufft | welp | 17:20 |
| dstufft | Marcus sits on a throne of lies | 17:20 |
| *** jjmb has quit IRC | 17:20 | |
| mordred | http://paste.openstack.org/show/42877/ | 17:20 |
| *** vipul is now known as vipul-away | 17:21 | |
| *** jjmb has joined #openstack-infra | 17:21 | |
| fungi | sounds like a comfortable seat, all things considered | 17:21 |
| dstufft | mordred: oh, maybe I just have details of the fix wrong | 17:21 |
| jeblair | starting puppet on lists.o.o | 17:22 |
| clarkb | jeblair: ok | 17:22 |
| jeblair | that will take exim back to 50 queue runners. | 17:22 |
| clarkb | mordred: when you are next in seattle you must show me the kexec setup, because I am really interested but probably won't have time to sort it out any time soon | 17:22 |
| jeblair | 70 worked, but let's see if we can do 50. | 17:22 |
| clarkb | mordred: and for removing the setuptools fix stuff from the base node... ugh | 17:23 |
| clarkb | mordred: I don't think that fix is on the d-g images yet | 17:23 |
| clarkb | mordred: because we haven't been able to build a new image since my fix went in | 17:23 |
| *** vogxn has quit IRC | 17:23 | |
| jeblair | i'm thinking we really need to get back to a state where our starting point is just os-packages (possibly with add-on archives) | 17:24 |
| mordred | jeblair: I probably agree - but can you define 'our' there? | 17:24 |
| *** nicedice has joined #openstack-infra | 17:24 | |
| jeblair | all of our nodes and all of our test processes | 17:24 |
| *** rcleere has joined #openstack-infra | 17:24 | |
| jeblair | so that step 1 isn't "pip install something globally" | 17:25 |
| mordred | you mean test all of openstack from distro packages and not using pip virtualenvs? | 17:25 |
| *** reed has joined #openstack-infra | 17:25 | |
| jeblair | i do not mean that | 17:25 |
| mordred | ok | 17:25 |
| mordred | phew | 17:25 |
| *** sarob has quit IRC | 17:25 | |
| luis_ | hi, does anybody know what could be the problem here: http://logs.openstack.org/90/39590/2/check/gate-neutron-pep8/952fbaf/console.html | 17:25 |
| jeblair | we have somehow conspired to move the starting point for all of our test processes to _before_ the start of our test processes. | 17:25 |
| luis_ | setup seems to be failing due to: error in setup command: Error parsing /home/jenkins/workspace/gate-neutron-pep8/setup.cfg: ImportError: neutron.hooks.setup_hook | 17:26 |
| jeblair | mordred: did you propose the openvz change to the other devstack branches? | 17:27 |
| mordred | jeblair: not the second one, no | 17:27 |
| jeblair | clarkb, fungi: maybe you want to take that on then? | 17:27 |
| mordred | luis_: processing setup.cfg is trying to import neutron.hooks.setup_hook | 17:27 |
| clarkb | ya, I can do that | 17:27 |
| mordred | luis_: this causes neutron/__init__.py to be imported | 17:27 |
| mordred | which I see you are changing in this pach | 17:27 |
| fungi | clarkb: i can take it if you're not at a good stopping point on something | 17:28 |
| mordred | so I'm guessing that something about this causes it to be impossible to import that file anymore outside of a venv with all the deps | 17:28 |
| *** andrew_plunk has quit IRC | 17:28 | |
| clarkb | fungi: I just got to my desk so perfect time to start something. I am good | 17:28 |
| mordred | I would strongly suggest not doing the gettext install in neutron/__init__ as it will cause issues | 17:28 |
| luis_ | mrodden, thanks for the pointers, i will try to see if doing it somewhere else would work | 17:29 |
| jeblair | clarkb, fungi: graphite is about to run out of disk space (though i can recover some by deleting the old trees)... | 17:30 |
| jeblair | but it also could potentially use a cpu/memory bump. | 17:30 |
| jeblair | it looks _okay_, but i think one more doubling of resources wouldn't hurt it. | 17:30 |
| clarkb | jeblair: wfm | 17:31 |
| jeblair | (let me check and see where the cpu bump thresholds are) | 17:31 |
| clarkb | jeblair: there are times I have asked it to graph a thing and I have had to wait a while | 17:31 |
| *** Ryan_Lane has joined #openstack-infra | 17:31 | |
| jeblair | clarkb: yeah | 17:31 |
| *** vipul-away is now known as vipul | 17:31 | |
| jeblair | we're at 2g/2cpu; we'd have to go to 8g/4cpu to get a cpu increase. | 17:32 |
| jeblair | clarkb, fungi: does that sound reasonable, or should i just get a nova volume for the data? | 17:32 |
| jeblair | http://cacti.openstack.org/cacti/graph_view.php?action=tree&tree_id=1&leaf_id=25 | 17:33 |
| jeblair | i believe the load is low now because of the broken graph; i expect it to increase again when we fix it | 17:33 |
| jeblair | (though perhaps we should make an optimized statsd entry just for that graph) | 17:34 |
| *** avtar has joined #openstack-infra | 17:36 | |
| clarkb | the history cpu usage is pretty consistent | 17:37 |
| clarkb | *historic | 17:37 |
| clarkb | is graphite capable of using more cpu time? | 17:38 |
| clarkb | mordred: jeblair fungi https://review.openstack.org/#/c/39757/ and https://review.openstack.org/#/c/39758/ backport the devstack openvz image fix | 17:39 |
| fungi | jeblair: it looks like most of the cpu usage on graphite.o.o is iowait, so i'm not sure additonal cpus will help? does it need faster storage? or more cache? (the memory graph suggests it very well may benefit there) | 17:40 |
| fungi | jeblair: maybe just doubling the ram and disk will actually help cpu impact if so | 17:41 |
| fungi | especially if rackspace also scales i/o bandwidth with the flavor | 17:41 |
| *** dina_belova has quit IRC | 17:43 | |
| fungi | sometimes interpreting resource utilization trends feels a bit like reading tea leaves | 17:43 |
| clarkb | fungi: indeed | 17:43 |
| clarkb | I think your comment about more cache is probably valid though | 17:44 |
| *** fbo_away is now known as fbo | 17:45 | |
| *** fbo is now known as fbo_away | 17:45 | |
| *** dkliban_afk is now known as dkliban | 17:46 | |
| *** jjmb has quit IRC | 17:47 | |
| pleia2 | Ryan_Lane: available for coffee/lunch later to chat openstack workshop? | 17:48 |
| openstackgerrit | Sergey Lukjanov proposed a change to openstack-dev/hacking: Extract configs-related code to the configs helper https://review.openstack.org/39143 | 17:49 |
| openstackgerrit | Sergey Lukjanov proposed a change to openstack-dev/hacking: Import exceptions list is now configurable https://review.openstack.org/39140 | 17:49 |
| *** pcrews has joined #openstack-infra | 17:51 | |
| Ryan_Lane | pleia2: yep. I was hoping you'd want to meet today | 17:51 |
| clarkb | its worth noting on my devstack backports I used the first commit's change id and commit message but I applied both the first commit and the commit that fixes the first commit | 17:51 |
| clarkb | they are squashed together | 17:51 |
| pleia2 | Ryan_Lane: cool, any time is good for me really, so just let me know and I'll pop downstairs | 17:51 |
| *** sarob has joined #openstack-infra | 17:51 | |
| *** UtahDave has quit IRC | 17:52 | |
| Ryan_Lane | I have a couple meetings, but will be available afterwards | 17:52 |
| Ryan_Lane | maybe 1:00? | 17:52 |
| pleia2 | Ryan_Lane: sounds good | 17:52 |
| pleia2 | Ryan_Lane: just ping me then | 17:52 |
| *** sparkycollier has quit IRC | 17:53 | |
| Ryan_Lane | will do | 17:53 |
| *** fbo_away is now known as fbo | 17:54 | |
| *** ladquin has joined #openstack-infra | 17:54 | |
| fungi | clarkb: yep, i actually checked to make sure | 17:54 |
| fungi | clarkb: it's what i would have done as well | 17:54 |
| reed | did you get your hotels for New Orleans? W or Sheraton? | 17:54 |
| reed | or Westin, and which of the two W? :) | 17:55 |
| *** koolhead11 has quit IRC | 17:55 | |
| reed | don't want to be by myself | 17:55 |
| reed | :) | 17:55 |
| clarkb | jeblair: russellb pabelanger is https://review.openstack.org/#/c/38651/2 safe to apply without the CID functionality? eg the logger won't die or otherwise disagree with the lack of info? | 17:56 |
| *** sarob has quit IRC | 17:56 | |
| russellb | clarkb: it will just be blank | 17:57 |
| russellb | those fields in the CSV file that is | 17:57 |
| russellb | no big deal | 17:57 |
| clarkb | russellb: ok | 17:58 |
| *** vipul is now known as vipul-away | 17:58 | |
| russellb | clarkb: and it will be populated in the case that someone calls in via SIP directly | 17:58 |
| russellb | (with whatever they want it to say, heh) | 17:58 |
| *** Ryan_Lane has quit IRC | 18:01 | |
| *** toddmorey has quit IRC | 18:01 | |
| *** ^d has quit IRC | 18:02 | |
| openstackgerrit | A change was merged to openstack-infra/config: Add a CDR log file https://review.openstack.org/38651 | 18:03 |
| *** vipul-away is now known as vipul | 18:03 | |
| *** Ryan_Lane has joined #openstack-infra | 18:04 | |
| *** odyssey4me has quit IRC | 18:05 | |
| *** jjmb has joined #openstack-infra | 18:07 | |
| *** changbl has quit IRC | 18:08 | |
| *** burt-afk is now known as burt | 18:08 | |
| *** koolhead11 has joined #openstack-infra | 18:09 | |
| *** lcheng has quit IRC | 18:09 | |
| mgagne | With jenkins scp and console timestamps plugins: when copying console log to a scp site, how do you make it include console timestamps too? | 18:09 |
| clarkb | mgagne: the console timestamps are part of the console log so it just happens | 18:10 |
| *** lcheng has joined #openstack-infra | 18:10 | |
| mgagne | clarkb: unfortunately, it's not the case for me =( | 18:10 |
| clarkb | mgagne: you do have to enable the timestamps on a per job basis is that missing? | 18:11 |
| *** cp16net|away is now known as cp16net | 18:12 | |
| mgagne | clarkb: I enabled timestamps for my jobs and added a config to copy console log to a scp site. So I'm wondering if I'm missing a config | 18:12 |
| *** andrew_plunk has joined #openstack-infra | 18:14 | |
| *** jjmb has quit IRC | 18:16 | |
| *** andrew_plunk has quit IRC | 18:16 | |
| *** andrew_plunk has joined #openstack-infra | 18:16 | |
| clarkb | mgagne: do the timestamps show up in the jenkins served console logs? | 18:18 |
| clarkb | mordred: https://review.openstack.org/#/c/39427/ I did not approve that or +2 because I wasn't a part of the conversation taht went on around that | 18:19 |
| mgagne | clarkb: yes, I have an option box in the sidebar to show/hide console timestamps. Could it be I installed the wrong plugin or you guys have an older one? I don't see this option on jenkins.o.o. | 18:19 |
| clarkb | mordred: but you do have the needed votes if you want to approve now | 18:19 |
| *** hashar has joined #openstack-infra | 18:19 | |
| clarkb | mgagne: yeah that doesn't sound the same | 18:19 |
| clarkb | mgagne: https://wiki.jenkins-ci.org/display/JENKINS/Timestamper is what we use | 18:20 |
| mgagne | clarkb: I use the same, v1.5.4. :-/ | 18:20 |
| *** mestery_ is now known as mestery | 18:20 | |
| clarkb | let me see what version we are running | 18:20 |
| mgagne | clarkb: don't tell me 1.0beta =) | 18:21 |
| *** gyee has quit IRC | 18:21 | |
| clarkb | mgagne: 1.3.1 | 18:21 |
| mgagne | clarkb: ok, I'll start from there and see what I can do to get the same behaviour. Could be a change in the format used to store the console log. | 18:22 |
| *** sarob has joined #openstack-infra | 18:22 | |
| bodepd | I would like to split the pip module off into it's own repo | 18:23 |
| bodepd | (probably not a bad place to start :) ) | 18:23 |
| clarkb | bodepd: like a forge module? | 18:23 |
| bodepd | is there by-chance a doc that just says how to do it :) | 18:23 |
| bodepd | clarkb: could be eventually. I just want a repo puppet-pip to exist at | 18:24 |
| bodepd | clarkb: git://github.com/openstack-infra/puppet-pip | 18:24 |
| bodepd | clarkb: and then for config to use that repo | 18:24 |
| bodepd | then eventually, the same needs to happen for zuul, jenkins, etc. | 18:25 |
| bodepd | for now, those are the only ones I need | 18:26 |
| clarkb | this is something we want to do, but isn't something we have really done. so blazing new territory | 18:26 |
| *** CaptTofu has joined #openstack-infra | 18:26 | |
| bodepd | oh good! | 18:26 |
| mgagne | bodepd: there's some modules available on https://forge.puppetlabs.com/openstackci | 18:26 |
| clarkb | mgagne: ya those are ones that mordred forked for reasons... | 18:26 |
| mgagne | bodepd: not sure how releases are managed. | 18:26 |
| bodepd | if pip was released up there, I could live with that | 18:27 |
| clarkb | bodepd: making modules that are available on the forge would be the easiest way for us to reconsume them | 18:27 |
| *** sarob has quit IRC | 18:27 | |
| mgagne | clarkb: forked or not, I'm more interested in the workflow to manage/release such modules =) | 18:27 |
| bodepd | me too. I just need to be able to consume them | 18:27 |
| clarkb | it is manual :( | 18:27 |
| bodepd | and I want to use the same code infra does | 18:28 |
| bodepd | there are auto-release tools the forge team is looking at | 18:28 |
| bodepd | I haven't had a chance to look at them | 18:28 |
| bodepd | but I am sure they assume: 1 module, 1 repo | 18:28 |
| bodepd | (well, it seems likely they would) | 18:29 |
| *** melwitt has joined #openstack-infra | 18:29 | |
| *** sarob has joined #openstack-infra | 18:29 | |
| mgagne | bodepd: having multiple modules in the same repo would be a challenge from a release pov as IMO, releases should be tagged =) | 18:29 |
| *** zul has quit IRC | 18:29 | |
| clarkb | ya repo per module is sane | 18:30 |
| bodepd | which goes back to the original question, would you be open to a patch to decouple pip | 18:30 |
| *** sarob has quit IRC | 18:30 | |
| bodepd | and where do I start :) | 18:30 |
| fungi | bodepd: familiar with git filter-branch? | 18:30 |
| clarkb | bodepd: I would be open to it. The where to start is tough. we could filter branch out the pip module | 18:30 |
| *** sarob has joined #openstack-infra | 18:31 | |
| fungi | heh | 18:31 |
| bodepd | that preserves the history :) | 18:31 |
| clarkb | import it into gerrit, publish to forge then remove it from the config repo | 18:31 |
| clarkb | mordred: jeblair ^ | 18:31 |
| fungi | that's the order i'd do it in. also probably would need to add it to the modules list we pull in, presumably | 18:31 |
| bodepd | does it need to be initially published to forge? | 18:31 |
| bodepd | yeah. I can figure out how to patch the config repo | 18:32 |
| clarkb | bodepd: yes so that we can use it | 18:32 |
| mgagne | bodepd: check install_modules.sh | 18:32 |
| bodepd | I'm a little less certain about how to create it as a new project | 18:32 |
| clarkb | or we figure out submodules or puppet librarian if it stopped sucking | 18:32 |
| bodepd | puppet-librarian-simple :) | 18:32 |
| mgagne | clarkb: haha :D | 18:32 |
| fungi | i would say we need to publish it to the forge before we rip it out of our main repo, but that can happen in multiple stages and we can just put changes to that module on hold while it's being worked through | 18:32 |
| jeblair | clarkb: i agree with what you have been saying | 18:32 |
| mgagne | bodepd: you are biased :P | 18:33 |
| bodepd | absolutely | 18:33 |
| jeblair | clarkb: submodules -- | 18:33 |
| bodepd | mgagne: as soon as it support forge installs, it *will* conquer the world | 18:33 |
| bodepd | I'm such a submodule hater | 18:33 |
| mordred | no | 18:33 |
| * fungi ph33rs git submodules | 18:33 | |
| mordred | oops | 18:33 |
| mordred | ah. good. I was saying no to submodules - extra complex, no need | 18:33 |
| bodepd | yeah | 18:34 |
| fungi | mordred: nice save | 18:34 |
| bodepd | use a Puppetfile before you use submodules | 18:34 |
| mgagne | bodepd: and then we will have the same problem again and puppet-librarian-dummy will be created ;) | 18:34 |
| mordred | bodepd: I still don't know how to do that | 18:34 |
| mordred | althogh you showed me one | 18:34 |
| mordred | is that the librarian thing? | 18:34 |
| * mordred actually would not mind using a Puppetfile instead of install_modules btw | 18:34 | |
| bodepd | https://github.com/stackforge/puppet-openstack/blob/master/Puppetfile | 18:34 |
| mordred | yeahyeah. I've _seen_ a Puppetfile | 18:35 |
| bodepd | the problem is that librarian-puppet really does suck | 18:35 |
| mordred | I don't know how to use it | 18:35 |
| mordred | so that is librarian-puppet | 18:35 |
| bodepd | librarian-puppet install | 18:35 |
| clarkb | bodepd: I know :) | 18:35 |
| bodepd | yeahhttps://github.com/bodepd/librarian-puppet-simple | 18:35 |
| mgagne | mordred: I like the way install_modules.sh works as it does not require an external tool but it requires modules to be available on forge =) | 18:35 |
| mordred | yeah. I believe it needs versions we don't use or something crazy | 18:35 |
| bodepd | that is where it gets painful | 18:35 |
| bodepd | the format is good, but the tool is so bad I wrote my own | 18:35 |
| mordred | oh - well... | 18:36 |
| mordred | well, clearly we're not going to put github references in there | 18:36 |
| bodepd | anyways, focus b/c we don't have to solve that today :) | 18:36 |
| mordred | :) | 18:36 |
| mordred | dude, I'm just taking a quick break from setuptools hell | 18:36 |
| bodepd | sound fun :) | 18:36 |
| clarkb | bodepd: I think we can start by splitting it out at least | 18:37 |
| bodepd | how do I create a new project under openstack-infra ? | 18:37 |
| bodepd | yep | 18:37 |
| bodepd | and pip is actually a pretty easy one to start with | 18:37 |
| clarkb | bodepd: http://ci.openstack.org/stackforge.html but s/stackforge/openstack-infra/ | 18:37 |
| mgagne | mordred: upstream needs discipline and tag their releases as they should have been doing since the start. | 18:37 |
| jeblair | clarkb: i think graphite will be able to use more cpus because much of it is apache mod wsgi (which are sep procs) there's a single carbon-cache which will not scale and be a bottleneck :(, and then there's node.js for statsd. so in general, some reasonable ability to use a few processors. | 18:37 |
| jeblair | fungi: ^ | 18:37 |
| clarkb | jeblair: I think we should bump mem and cpu then | 18:38 |
| mordred | mgagne: discipline to tag your releases becomes much easier when tagging your releases creates your releases... :) | 18:38 |
| mgagne | mordred: I have nothing to add to that comment =) | 18:39 |
| *** cthulhup has joined #openstack-infra | 18:39 | |
| fungi | jeblair: clarkb: yeah, given that, i agree that turning the flavor knob up two clicks instead of the one makes sense | 18:39 |
| jeblair | resize button clicked | 18:40 |
| annegentle | hey pleia2 anteaya, did you work out who's going to present to Codechix? | 18:40 |
| anteaya | not yet | 18:41 |
| pleia2 | annegentle: Ryan_Lane and I are, also looking for someone else (there are lots of people around, shouldn't be an issue, but I would prefer another woman just because of the nature of the thing) | 18:41 |
| anteaya | I was going to talk to pleia2 about it next week | 18:41 |
| *** NobodyCam has joined #openstack-infra | 18:42 | |
| anteaya | annegentle: did you have any thoughts or suggestions? | 18:42 |
| pleia2 | if anteaya can't make it, thinking of putting feelers out on -women linkedin group if that's ok | 18:43 |
| anteaya | pleia2: I should have a better sense of whether I can make it or not next week | 18:43 |
| openstackgerrit | Sergey Lukjanov proposed a change to openstack-dev/hacking: Extract configs-related code to the configs helper https://review.openstack.org/39143 | 18:43 |
| openstackgerrit | Sergey Lukjanov proposed a change to openstack-dev/hacking: Import exceptions list is now configurable https://review.openstack.org/39140 | 18:43 |
| NobodyCam | may the new zuul page looks very nice! congrats -infra | 18:43 |
| pleia2 | anteaya: great, thanks :) | 18:43 |
| annegentle | anteaya: pleia2: would it make sense for Rackspace to sponsor CodeChix for those types of workshops? Not to fly people in necessarily, but for Rupa to use as she wishes. | 18:43 |
| NobodyCam | gah | 18:43 |
| *** dina_belova has joined #openstack-infra | 18:43 | |
| NobodyCam | * may I say ... | 18:44 |
| Ryan_Lane | pleia2: I'm happy to just be a TA and have you and another woman present | 18:44 |
| anteaya | annegentle: I can't comment, I have just been asked to present | 18:44 |
| pleia2 | annegentle: perhaps! I know she's always looking for sponsors for various things (also planning on doing some codechix events in SF soon, not sure if she's sorted out space - rackspace has an office here now if I recall?) | 18:44 |
| anteaya | I don't know anything about the direction and mandate of the group | 18:44 |
| annegentle | pleia2: that's right, we have an office in SF | 18:45 |
| anteaya | Ryan_Lane: it would be lovely to meet you, I hope I can go | 18:45 |
| jeblair | NobodyCam: thanks :) | 18:45 |
| Ryan_Lane | I've presented at codechix before. there's no requirement for the presenter to be a woman, but I'm sure it's not a bad goal :) | 18:45 |
| annegentle | Ryan_Lane: you're awesome, you know that. Don't feel like you have to take a back seat at all | 18:45 |
| pleia2 | annegentle: +1 | 18:45 |
| annegentle | Ryan_Lane: :) it's always nice to show that OpenStack has women presenters! | 18:45 |
| *** vipul is now known as vipul-away | 18:45 | |
| Ryan_Lane | annegentle: ageeed | 18:45 |
| Ryan_Lane | *agreed | 18:46 |
| annegentle | ok I'll see what I can do to help, just wanted to hear your latest-n-greatest | 18:46 |
| pleia2 | thanks for checking in :) | 18:46 |
| mordred | pleia2: anteaya should be able to go if we can figure out all of her travel accounts first | 18:46 |
| pleia2 | mordred :D | 18:46 |
| annegentle | mordred: Awesome. | 18:46 |
| anteaya | I couldn't say anything yet | 18:46 |
| anteaya | but mordred can | 18:46 |
| clarkb | fungi: jeblair mordred https://review.openstack.org/#/c/39576/ | 18:48 |
| jeblair | clarkb: fungi: https://review.openstack.org/#/c/39585/ | 18:48 |
| jeblair | clarkb: fungi: can you look that over, even though it's not passing tests? | 18:48 |
| clarkb | previously I think we asked ptls to delegate by adding members to the ptl group. that still the case? | 18:48 |
| clarkb | jeblair: sure | 18:48 |
| *** dina_belova has quit IRC | 18:49 | |
| jeblair | clarkb: fungi: i'd like to merge the breaking change, and then retest that, and then merge it and restart zuul to pick it all up (in a relatively short period) | 18:49 |
| mordred | clarkb: for the client libs, I think we have other projects using core | 18:49 |
| jeblair | mordred: i'm not sure we should... | 18:49 |
| fungi | clarkb: yes, the .*-ptl groups were meant to be "project lead plus delegates for git repo permission purposes" | 18:49 |
| mordred | ah. I'm with all of you now | 18:50 |
| jeblair | i'm really troubled by the idea that lots of people would automatically get the ability to irrecovably screw up the release process | 18:50 |
| mordred | I'd be most onboard with them just adding people to the -plt | 18:50 |
| mordred | ptl | 18:50 |
| mordred | grou | 18:50 |
| mordred | gah | 18:50 |
| clarkb | I will update my review with info about using the ptl group | 18:50 |
| *** vijendar has quit IRC | 18:51 | |
| jeblair | maybe we should rename them at some point, but it doesn't seem terribly important now. | 18:51 |
| *** sdake has joined #openstack-infra | 18:52 | |
| *** sdake has quit IRC | 18:52 | |
| *** sdake has joined #openstack-infra | 18:52 | |
| * fungi will be mostly afk for a few hours starting around 19:30z today. time again for the monthly osug here in the triangle | 18:52 | |
| mgagne | clarkb: It seems -Dtimestamper-consolenotes=true is required for console log to be stored in the old format and for console log copied by scp to include timestamps. Will be able to test and confirm in 1h. Might worth noting if you are planning on upgrading this plugin. | 18:53 |
| *** dina_belova has joined #openstack-infra | 18:54 | |
| *** vijendar has joined #openstack-infra | 18:56 | |
| reed | jeblair, fungi: we're kicking off the community/user group portal project today | 18:57 |
| openstackgerrit | James E. Blair proposed a change to openstack-infra/zuul: Support multiple triggers https://review.openstack.org/39186 | 18:57 |
| openstackgerrit | James E. Blair proposed a change to openstack-infra/zuul: Add a timer trigger https://review.openstack.org/39187 | 18:57 |
| mriedem | mtreinish: launchpad doesn't seem to have changed this bug status after the fix was merged? https://bugs.launchpad.net/tempest/+bug/1207066 | 18:57 |
| uvirtbot | Launchpad bug 1207066 in tempest "Tempest is dumping debug-level messages to console output" [High,Triaged] | 18:57 |
| fungi | reed: i will make sure to mention it at our user group meeting today, in that case! | 18:57 |
| mordred | clarkb, sdague, jeblair: https://jenkins.openstack.org/job/gate-tempest-devstack-vm-full/38321/ <--- that's a job where I made it go back to just doing the upgrade of setuptols | 18:57 |
| reed | we'll need two machines, one for staging and one later for production | 18:57 |
| reed | and a couple of hostnames, too | 18:58 |
| reed | not sure how we'll call this thing | 18:58 |
| jeblair | reed: django app, right? | 18:58 |
| reed | jeblair, drupal | 18:58 |
| fungi | reed: oh, by kicking off you mean starting development. i'll hold off plugging it to the ug just yet in that case ;) | 18:59 |
| reed | fungi, yes, it's just a start of development | 18:59 |
| mordred | drupal? | 18:59 |
| * mordred goes afk | 18:59 | |
| jeblair | reed: do the devs have an existing puppet setup or expertise to contribute to that? | 18:59 |
| *** cthulhup has quit IRC | 19:00 | |
| reed | jeblair, I think they do, and if not they're willing to learn | 19:00 |
| mtreinish | mriedem: yes it merged that's weird that it didn't update the status though | 19:00 |
| mriedem | mtreinish: i have another one in nova like that | 19:00 |
| mriedem | merged 3 weeks ago | 19:00 |
| reed | I'll ask them to integrate in our workflow | 19:01 |
| mriedem | mtreinish: maybe it's only bugs i open? | 19:01 |
| jeblair | reed: cool, then let's try to start with a basic puppet install that we can help them iterate on, so it's not overwhelming | 19:01 |
| reed | ok | 19:01 |
| jeblair | reed: just let us know the hostname you want. :) | 19:02 |
| reed | i'll file a ticket after the meeting today | 19:02 |
| *** andrew_plunk has quit IRC | 19:04 | |
| sdague | reed: who's driving the drupal thing? I've actually do a bit of that for some local non profits, so might have some helpful input | 19:04 |
| jeblair | sdague: do you use puppet with that by any chance? :) | 19:04 |
| reed | sdague, mrmarton will be the main dev on the team with colleague | 19:05 |
| sdague | I don't | 19:05 |
| jeblair | shucks | 19:05 |
| sdague | well, not yet. I've been thinking about it for one of the sites, just haven't gotten there yet | 19:05 |
| jeblair | reed: when's the meeting? | 19:05 |
| reed | mrmarton and his pal have done lots of drupal development too, that's what made them win | 19:05 |
| reed | jeblair, today, 2pm PDT | 19:05 |
| reed | on skype though | 19:05 |
| jeblair | reed: and are they going to be there and talking about | 19:06 |
| jeblair | oh | 19:06 |
| jeblair | well, um, have fun i guess. | 19:06 |
| *** changbl has joined #openstack-infra | 19:06 | |
| jeblair | you know where we are if you want help from anyone working on the project. :) | 19:06 |
| reed | yeah ... alternative that would not cost € to the guys in Hungary? | 19:06 |
| openstackgerrit | A change was merged to openstack-infra/zuul: Support multiple triggers https://review.openstack.org/39186 | 19:06 |
| jeblair | reed: i thought there was an irc meeting | 19:06 |
| reed | irc is too slow | 19:07 |
| reed | we'll have irc meetings later | 19:07 |
| openstackgerrit | A change was merged to openstack-infra/zuul: Add a timer trigger https://review.openstack.org/39187 | 19:07 |
| *** sarob has quit IRC | 19:09 | |
| openstackgerrit | James E. Blair proposed a change to openstack-infra/config: Add queue_smtp_domains as a parameter https://review.openstack.org/39770 | 19:14 |
| *** toddmorey has joined #openstack-infra | 19:14 | |
| jeblair | i'm going to approve that now, it's a brown-bag syntax error ^ | 19:14 |
| clarkb | go for it | 19:16 |
| jeblair | so actually, i'm going to do more than that | 19:17 |
| fungi | no valid value found for dictionary value @ data['pipelines'][0]['trigger'] | 19:18 |
| fungi | indeed | 19:18 |
| clarkb | pleia2: re https://review.openstack.org/#/c/36593/ were you planning on updating that change or would you like me to? | 19:18 |
| *** pabelanger has quit IRC | 19:18 | |
| jeblair | stopping puppet on zuul | 19:18 |
| clarkb | pleia2: either way is fine, I just want to avoid stepping on toes and duplicating work | 19:18 |
| *** vipul-away is now known as vipul | 19:18 | |
| jeblair | i'm going to merge the layout change, then the puppet fix, then run puppet manually on zuul | 19:19 |
| *** boris-42 has quit IRC | 19:19 | |
| jeblair | then restart zuul | 19:19 |
| clarkb | jeblair: ok, that sounds safe | 19:19 |
| pleia2 | clarkb: I can take care of it, just been wrapped up in other things these past couple days | 19:19 |
| *** boris-42 has joined #openstack-infra | 19:20 | |
| clarkb | jeblair: https://review.openstack.org/#/c/38565/ is there value in removing the race there just to be extra cautious or should I abandon that change? | 19:20 |
| clarkb | pleia2: no problem. I know you have been busy lately | 19:20 |
| dtroyer | mordred, sdague: this will be helpful for the pip stuffs on fedora/centos: https://review.openstack.org/39771 | 19:20 |
| jeblair | clarkb: i'd like to avoid merging that change in favor of continuing the major overhaul. | 19:21 |
| openstackgerrit | A change was merged to openstack-infra/config: Specify gerrit triggers https://review.openstack.org/39585 | 19:21 |
| sdague | mordred., clarkb: https://review.openstack.org/#/c/35472/1 does that solve the thing you were talking about in -qa ? | 19:21 |
| sdague | dtroyer: looking... | 19:21 |
| sdague | dtroyer: oh.... right | 19:22 |
| sdague | dtroyer: you tested on fedora? | 19:22 |
| clarkb | jeblair: ok | 19:22 |
| dtroyer | yes | 19:23 |
| sdague | dtroyer: to the gate it goes | 19:23 |
| openstackgerrit | A change was merged to openstack-infra/config: Add queue_smtp_domains as a parameter https://review.openstack.org/39770 | 19:23 |
| openstackgerrit | A change was merged to openstack-dev/pbr: Hierarchical sphinx API documentation generation https://review.openstack.org/39427 | 19:24 |
| jeblair | restarting zuul | 19:25 |
| jeblair | done, and rechecks left | 19:26 |
| *** Ryan_Lane has quit IRC | 19:27 | |
| jeblair | that should be it for the invasive zuul changes for a bit. | 19:27 |
| *** jjmb has joined #openstack-infra | 19:27 | |
| *** dina_belova has quit IRC | 19:29 | |
| harlowja | thx jeblair | 19:30 |
| *** jjmb has quit IRC | 19:32 | |
| openstackgerrit | A change was merged to openstack-infra/config: Handle html log annotations. https://review.openstack.org/39547 | 19:32 |
| *** sandywalsh has quit IRC | 19:33 | |
| sdague | jeblair: I noticed earlier today that stable/grizzly sits in the same pipeline as master | 19:33 |
| openstackgerrit | A change was merged to openstack-infra/config: Handle log paths in logstash gearman client better https://review.openstack.org/39597 | 19:33 |
| sdague | is that intentional? | 19:34 |
| jeblair | sdague: yes; it'll become particularly important once we have bidirectional grenade testing, and is also useful for compat testing for client libs, etc. | 19:34 |
| sdague | ok, fair | 19:35 |
| hashar | jeblair: hi! I finally completed our little status page https://integration.wikimedia.org/zuul/ :] | 19:35 |
| sdague | now I just need to figure out where mordred ran off to :) Because the land of magical ponies is close | 19:35 |
| hashar | jeblair: thank you very much for all the inspiration! | 19:35 |
| *** rnirmal has quit IRC | 19:36 | |
| sdague | jeblair: also did you notice the zuul jobs launched per hour stopped updating yesterday | 19:36 |
| *** pabelanger has joined #openstack-infra | 19:36 | |
| jeblair | hashar: great! i added progress bars to ours, have you seen? http://status.openstack.org/zuul/ | 19:36 |
| hashar | jeblair: yeah I have been following your zuul commits | 19:37 |
| harlowja | qq for ya'll, is there anyway to force a dependency Y for a review X, without having to make code changes to said review X | 19:37 |
| hashar | will hopefully migrate to gear in october. | 19:37 |
| jeblair | hashar: (i still need to add those to the page in the repo that's based on yours) | 19:37 |
| hashar | that looks very nice | 19:37 |
| jeblair | hashar: cool, i think some other people are setting up a new system with gear now (bodepd), so hopefully it'll be pretty smooth by then | 19:38 |
| jeblair | sdague: yes, i'll fix that soon (i moved some statsd keys around) | 19:38 |
| hashar | jeblair: will have a look at gear, and might end up writing a bunch of tiny workers for the most basic tasks (such as running pep8) | 19:39 |
| jeblair | hashar: great! i'd like to move in that direction too! | 19:39 |
| mrodden | clarkb: its a bit more involved than i thought to be able to run manage-projects remotely. when updating the ACLs it has to build the groups file, which then needs to look up the group UUIDs in Gerrit which currently requires parsing the gerrit.config file for the database connection information | 19:40 |
| jeblair | hashar: i was brainstorming with mordred, and we came up with the idea of trying to use linux containers to separate the gear worker from where the job is actually running | 19:40 |
| jeblair | hashar: we haven't tried that, but it sounds like it could be useful for privilege separation | 19:40 |
| clarkb | mrodden: hmm, we could probably split out the gerrit DB config info into a manage projects config file | 19:40 |
| clarkb | mrodden: but ya that would break you if you need the other file to be local | 19:40 |
| clarkb | harlowja: are you asking if you can force a dependency during the check and gate tests? | 19:41 |
| mrodden | clarkb: yeah that was my thought too, although, it is possible to do SQL queries on gerrit through the SSH api | 19:41 |
| mrodden | oddly enough | 19:41 |
| clarkb | mrodden: it is, but we have decided that using direct DB connections is better beacuse you get sanitized db inputs when you use python-mysql or psycopg2 for example | 19:42 |
| *** dkliban has quit IRC | 19:42 | |
| mrodden | yeah. plus its nasty to have to parse query results from an SSH channel... | 19:42 |
| harlowja | clarkb mainly putting up https://review.openstack.org/#/c/39768/, have active review https://review.openstack.org/#/c/29862/ which is solved by the previous bug, but tried to add dependency between them (after the second one is commited) without changing the code in the second one seems like its not possible, if that makes sense | 19:42 |
| openstackgerrit | Sergey Lukjanov proposed a change to openstack-infra/config: Add projects filter to zuul dashboard https://review.openstack.org/39773 | 19:43 |
| clarkb | harlowja: ya, this has to do with the way git works. The sha1 of your parent in git is included in the calculation of the current commit's sha1 | 19:44 |
| *** dina_belova has joined #openstack-infra | 19:44 | |
| clarkb | harlowja: this makes fetches really fast but means changing a parent changes the child (makes sense, but does force an update in gerrit) | 19:44 |
| harlowja | k, i can find something to change, checkout first review, then apply second review (with small modification) ontop and be all set | 19:45 |
| clarkb | you don't need to change anything | 19:45 |
| *** UtahDave has joined #openstack-infra | 19:46 | |
| clarkb | unless I am mistaken. Simply creating the dependency in the git DAG is sufficient | 19:46 |
| *** sandywalsh has joined #openstack-infra | 19:46 | |
| harlowja | k, let me try that again | 19:46 |
| fungi | yeah, gerrit will note that you've made no changes to the parent and won't bother to update it, but will update the child to depend on it | 19:48 |
| fungi | i do that all the time | 19:48 |
| openstackgerrit | Sean Dague proposed a change to openstack-infra/config: add devstack gate to requirements changes https://review.openstack.org/39774 | 19:48 |
| openstackgerrit | Sean Dague proposed a change to openstack-infra/config: add missing neutron-full job to neutron gate https://review.openstack.org/39775 | 19:48 |
| openstackgerrit | Sean Dague proposed a change to openstack-infra/config: make oslo.config gating on devstack/tempest runs https://review.openstack.org/39508 | 19:48 |
| * fungi disappears for a few hours... bbl | 19:49 | |
| clarkb | jeblair: sdague https://review.openstack.org/#/q/I64b03ed387f5205a885427498e8cf8c2a5b32b91,n,z can we get those in so that d-g image updates can be run in one AZ before hitting the others? | 19:49 |
| clarkb | once that is happy I think we can consider https://review.openstack.org/#/c/33204/ | 19:49 |
| clarkb | fungi: if you haven't left yet | 19:50 |
| jeblair | clarkb: i +2'd those | 19:50 |
| clarkb | fungi: is subunit2html still causing problems? | 19:50 |
| clarkb | jeblair: ty | 19:50 |
| sdague | clarkb: sure, though man zuul is pretty backed up right now | 19:50 |
| clarkb | sdague: yeah, that is what happens when jeblair restarts all the things | 19:51 |
| jeblair | my fault | 19:51 |
| fungi | clarkb: subunit2html is not directly. but testr is (i have a change up to fix that) | 19:51 |
| sdague | ok, well while I wait for the world to clear, clarkb do you understand the crux of the update.py thing? | 19:51 |
| fungi | clarkb: specifically, we were invoking testr and overriding the interpreter in ways it ignored | 19:51 |
| clarkb | fungi: awesome I will look for it, currently trying hard to do code review | 19:51 |
| *** mgagne has quit IRC | 19:51 | |
| fungi | i figure i will be 100% code review when i get home this evening | 19:52 |
| * fungi really leaves now | 19:52 | |
| * jeblair gets lunch | 19:52 | |
| clarkb | sdague: https://review.openstack.org/#/c/35472/1 <- that what you are talking about? | 19:52 |
| sdague | yes | 19:52 |
| sdague | hmmm... gate is also bouncing on things like this - https://jenkins.openstack.org/job/gate-tempest-devstack-vm-postgres-full/38010/console | 19:53 |
| sdague | that just because of jenkins load? | 19:53 |
| sdague | clarkb: you think that's actually our solution? | 19:53 |
| clarkb | sdague: yes jenkins load | 19:54 |
| clarkb | sdague: the indicator is the inprogress job failed because we don't have enough executors to run all of the inprogress jobs that start when we restart all the things :( | 19:54 |
| *** ladquin has quit IRC | 19:55 | |
| clarkb | so back to update.py I feel like I am missing history somewhere | 19:55 |
| jeblair | clarkb: it should never fail -- it's failing because it's taking jenkins more than 10 minutes to execute an api call to modify a node, which is ridiculous. | 19:55 |
| jeblair | which is why i'm adding more jenkinses | 19:55 |
| clarkb | I read that last sentence in Gollum's voice | 19:56 |
| jeblair | i said it in Gollum's voice | 19:56 |
| pleia2 | haha | 19:56 |
| jeblair | actually, that one wasn't a timeout. i'm going to look into it after lunch | 19:57 |
| sdague | hehe | 19:57 |
| anteaya | so I can create a new task in the django shell and it shows up in the gui | 19:57 |
| clarkb | sdague: it says restore merging the two files, but it doesn't actually merge the two files | 19:57 |
| clarkb | sdague: what am I missing | 19:57 |
| anteaya | I can't successfully query all tasks from the django shell | 19:57 |
| clarkb | sdague: oh I get it | 19:58 |
| sdague | I think it allows content from either file | 19:58 |
| anteaya | and I can't create a task in the django gui | 19:58 |
| clarkb | sdague: ya, which is wrong | 19:58 |
| sdague | you know, let me actually test it | 19:58 |
| *** mgagne has joined #openstack-infra | 19:58 | |
| sdague | clarkb: well, only sort of | 19:58 |
| clarkb | sdague: instead, we should just have one file | 19:58 |
| sdague | clarkb: ok | 19:59 |
| clarkb | we shouldn't be so split brained that we allow duplicate and potentially conflicting requirements in different places (not sure if that script allows conflicts) | 19:59 |
| clarkb | instead we should have the one list to rule them all and in its power bind them | 19:59 |
| sdague | the new version of update.py doesn't allow it | 19:59 |
| sdague | but I get the point | 19:59 |
| sdague | so if we do that, then we could actually do the following | 20:00 |
| sdague | mv requirements.txt -> global-requirements.txt | 20:00 |
| sdague | put test-requirements.txt => global-requirements.txt | 20:00 |
| sdague | then have actual requirements.txt be the requirements for that repo | 20:00 |
| sdague | which means the unit testing could land without a pbr change | 20:01 |
| clarkb | sdague: I like that | 20:02 |
| openstackgerrit | lin-hua-cheng proposed a change to openstack/requirements: Add support for Keystone V3 Auth in Horizon. https://review.openstack.org/39779 | 20:02 |
| clarkb | and unittesting wasn't something I had considered would be enabled but ya that sounds good | 20:02 |
| sdague | yeh... let me tell you, seeing how unit testing exploded on this was *fun* :) | 20:02 |
| *** Ryan_Lane has joined #openstack-infra | 20:03 | |
| *** cp16net is now known as cp16net|away | 20:06 | |
| *** dina_belova has quit IRC | 20:06 | |
| Ryan_Lane | pleia2: ok, ready | 20:08 |
| *** ladquin has joined #openstack-infra | 20:08 | |
| Ryan_Lane | pleia2: any where good to meet? | 20:08 |
| *** dina_belova has joined #openstack-infra | 20:08 | |
| sdague | clarkb: actually.... I should just merge this into the unit test patch | 20:09 |
| pleia2 | Ryan_Lane: hey, can meet at wikimedia street entrance, I'll be down in a minute | 20:09 |
| *** koolhead11 has quit IRC | 20:09 | |
| Ryan_Lane | cool. heading down there. | 20:09 |
| *** Ryan_Lane has quit IRC | 20:09 | |
| clarkb | sdague: logstash should be ready for your annotation stuff now, any idea why it 500s instead of 404ing when you request non existant files? | 20:10 |
| *** cp16net|away is now known as cp16net | 20:12 | |
| *** afazekas has quit IRC | 20:13 | |
| lifeless | mordred: module level getters are weird | 20:16 |
| lifeless | mordred: perhaps the setter shouldn't be a setter but a helper | 20:16 |
| *** Ryan_Lane has joined #openstack-infra | 20:17 | |
| *** nati_ueno has quit IRC | 20:17 | |
| *** psedlak has quit IRC | 20:17 | |
| *** nati_ueno has joined #openstack-infra | 20:18 | |
| *** dina_belova has quit IRC | 20:18 | |
| *** rnirmal has joined #openstack-infra | 20:19 | |
| *** nati_ueno has quit IRC | 20:22 | |
| *** dkranz has quit IRC | 20:24 | |
| *** prad_ has quit IRC | 20:25 | |
| anteaya | I am getting closer in figuring out what is happening with storyboard tasks, there seems to be a dependency on milestones/branches/projects and stories but I am uncertain which way to proceed because I don't have the vision of what relationships are supposed to be dependencies | 20:27 |
| anteaya | guess I am waiting for ttx | 20:27 |
| openstackgerrit | A change was merged to openstack-infra/config: Additional path patterns for log readme files. https://review.openstack.org/39116 | 20:27 |
| *** nati_ueno has joined #openstack-infra | 20:29 | |
| clarkb | anteaya: after reading that sentence I would wait for ttx too :) | 20:29 |
| anteaya | clarkb: ha ha ha | 20:29 |
| anteaya | thanks | 20:30 |
| *** HenryG has quit IRC | 20:30 | |
| openstackgerrit | Alejandro Cabrera proposed a change to openstack-infra/config: Add Python 3.3 gate for Marconi + Client https://review.openstack.org/39790 | 20:31 |
| clarkb | jeblair: mordred https://review.openstack.org/#/c/39580/ | 20:32 |
| *** cppcabrera has joined #openstack-infra | 20:32 | |
| openstackgerrit | Sean Dague proposed a change to openstack/requirements: (WIP) add update.py testing to requirements https://review.openstack.org/39358 | 20:32 |
| anteaya | in other news it appears that if I run `testr run --parallel tempest.api.identity` on a brand new devstack install (or a refresh with ./unstack.sh && ./stack.sh) the tests complete very quickly but if the vm instance has been running away (running tempest tests) that same command takes far longer to complete | 20:32 |
| anteaya | s/away/awhile | 20:33 |
| anteaya | after this run is finished I was going to unstack and stack again to be sure I am finding the same thing again | 20:33 |
| clarkb | anteaya: interesting | 20:34 |
| cppcabrera | Hey. I heard that py33 gates were added recently to our CI backend! I'm happy to hear it. Who reviews/approves patches for project looking to start using the py33 gate? :) | 20:34 |
| clarkb | cppcabrera: mordred jeblair fungi and myself | 20:34 |
| anteaya | clarkb: I find it to be so | 20:34 |
| clarkb | cppcabrera: I will take a look in a bit | 20:34 |
| cppcabrera | I submitted one a moment ago for the marconi project: https://review.openstack.org/#/c/39790/ | 20:34 |
| cppcabrera | Thanks, clarkb! | 20:34 |
| anteaya | not that it really matters since in most use cases the devstack/tempest set-up is single use disposable | 20:35 |
| clarkb | anteaya: yeah, but understanding the behavior is probably a good thing | 20:36 |
| mordred | lifeless: it should _probably be a proper object | 20:36 |
| anteaya | clarkb: I concur | 20:36 |
| *** whoops has joined #openstack-infra | 20:36 | |
| anteaya | whoops!! | 20:36 |
| whoops | drop something? | 20:37 |
| anteaya | ha ha ha | 20:37 |
| anteaya | good to see you | 20:37 |
| clarkb | it is times like this I feel like I did poorly when choosing a nick | 20:37 |
| *** Ryan_Lane has quit IRC | 20:38 | |
| *** boris-42 has quit IRC | 20:38 | |
| mordred | right? | 20:38 |
| openstackgerrit | Alejandro Cabrera proposed a change to openstack-infra/config: Add Python 3.3 gate for Marconi + Client https://review.openstack.org/39790 | 20:39 |
| anteaya | clarkb: I like ghost tooth | 20:39 |
| anteaya | you could go back to ghost tooth and I would talk to you | 20:39 |
| lifeless | mordred: so I don't want to polish an irrelevancy ;) - I will check for any other issues and +1 shortly | 20:39 |
| anteaya | to be fair his last name is hoops | 20:39 |
| whoops | yeah this nick sorta fell into my lap | 20:40 |
| clarkb | anteaya: so you are saying that my real problem is my parents provided me with a poor name for IRC? | 20:40 |
| mordred | lifeless: thanks - it's a good note in general - but I do think I'd like to probably make that an object in a later refactor | 20:40 |
| anteaya | clarkb: heck no | 20:40 |
| clarkb | :P | 20:41 |
| clarkb | its ok, I complain to them that they gave me a boring name | 20:41 |
| anteaya | they gave you access to a great language with stuff like ghost tooth in it | 20:41 |
| *** Ryan_Lane has joined #openstack-infra | 20:41 | |
| anteaya | okay that is your prerogative | 20:41 |
| anteaya | not piling on that fight | 20:41 |
| anteaya | mine came about because my nephews wanted to play computer games at my place | 20:41 |
| * mordred always thinks of clarkb as 'ghost tooth' | 20:42 | |
| anteaya | so I said okay if they showed me how | 20:42 |
| anteaya | ha ha ha | 20:42 |
| *** cp16net is now known as cp16net|away | 20:43 | |
| clarkb | cppcabrera: the change looks good to me, but quick question. Are you expecting those tests to pass soon? For many projects I think we expect to spend a lot of time dealing with dependencies before we even worry about testing | 20:44 |
| *** prad_ has joined #openstack-infra | 20:44 | |
| sdague | clarkb: so that's not right -https://review.openstack.org/39358, but my brain is breaking on why the test-requirements.txt part doesn't | 20:45 |
| sdague | work | 20:45 |
| clarkb | cppcabrera: in any case happy to make you a guinea pig | 20:45 |
| clarkb | sdague: looking | 20:45 |
| sdague | you can actually run tox on that tree | 20:46 |
| sdague | and it does a thing | 20:46 |
| cppcabrera | I expect them to pass once our deps pass, as well, so hopefully soon. I decided to make those gates non-voting after checking with the rest of team marconi, but yeah, we're happy to start gating on that, clarkb. :) | 20:47 |
| clarkb | sdague: but not the expected thing? | 20:47 |
| sdague | well, update.py doesn't seem to work on the test-requirements.txt file | 20:48 |
| *** cp16net|away is now known as cp16net | 20:48 | |
| *** thomasbiege has quit IRC | 20:48 | |
| sdague | hmmm... actually it works outside of the repo | 20:49 |
| sdague | lifeless: in testr, setUp is called in the children? | 20:50 |
| *** vipul is now known as vipul-away | 20:52 | |
| *** dkliban has joined #openstack-infra | 20:52 | |
| lifeless | sdague: huh? testr doesn't have any impact on what is called in the backends | 20:52 |
| lifeless | sdague: the only coupling is on test selection (for parallelism) and the output from the children (subunit) | 20:52 |
| *** woodspa has quit IRC | 20:53 | |
| mordred | sdague: setUp is called in each test runner for each test case | 20:53 |
| sdague | mordred: ok, like I thought | 20:53 |
| lifeless | if the test runner does setUp :) - testr doesn't care | 20:53 |
| sdague | well I've gone code blind on why that link above doesn't work | 20:53 |
| mordred | looking | 20:53 |
| mordred | sdague: it says success... | 20:54 |
| sdague | mordred: pull the repo | 20:54 |
| sdague | and run tox | 20:54 |
| *** hashar has quit IRC | 20:54 | |
| sdague | I'm doing the merge of the requirements files into the patch where I added unit tests | 20:55 |
| mordred | sdague: btw - why the use-site-packages and the -U ? | 20:55 |
| mordred | cool | 20:55 |
| mordred | AttributeError: 'NoneType' object has no attribute 'utf_8_decode' | 20:55 |
| mordred | mmm | 20:55 |
| mordred | IOError: [Errno 2] No such file or directory: 'tests/files/gr-base.txt' | 20:55 |
| mordred | is that more the error you're seeing? | 20:55 |
| sdague | no | 20:55 |
| *** rcleere has quit IRC | 20:56 | |
| mordred | or this MismatchError: 'testtools>=0.9.32' not in ['# Imported by ldapdns so required to generate', '# the sample configuration file', 'setuptools_git>=0.4'] | 20:56 |
| sdague | that one | 20:56 |
| mordred | gotcha | 20:56 |
| *** rfolco has quit IRC | 20:57 | |
| *** pcrews has quit IRC | 20:58 | |
| *** SergeyLukjanov has quit IRC | 21:00 | |
| clarkb | fungi: https://review.openstack.org/#/c/39580/ I left a comment on that and would be curious to know what you think | 21:01 |
| openstackgerrit | A change was merged to openstack-infra/config: Add gerritbot for openstack-glance channel https://review.openstack.org/38855 | 21:01 |
| clarkb | mordred: https://review.openstack.org/#/c/39583/ | 21:01 |
| * fungi is mostly on silent running, but will take a look | 21:02 | |
| openstackgerrit | A change was merged to openstack-infra/config: Add Python 3.3 gate for Marconi + Client https://review.openstack.org/39790 | 21:02 |
| fungi | michael dehaan from ansibleworks is presenting on openstack automation at the moment | 21:02 |
| openstackgerrit | James E. Blair proposed a change to openstack-infra/config: Update jobs launched graph with new paths https://review.openstack.org/39797 | 21:02 |
| marun | sdague: ping | 21:03 |
| clarkb | fungi: no rush | 21:03 |
| clarkb | fungi: I don't think we want to merge that change when stuff is busy (because paranoia) | 21:03 |
| fungi | clarkb: commented | 21:05 |
| *** vipul-away is now known as vipul | 21:05 | |
| marun | sdague: I'm seeing a problem with this change: https://review.openstack.org/#/c/39528/ | 21:05 |
| marun | sdague: I was hoping you could explain why it was necessary so I could figure out the best way to approach a solution. | 21:06 |
| mordred | marun: I'm still working on that one :) | 21:07 |
| mordred | marun: this is what happens: https://jenkins.openstack.org/job/gate-tempest-devstack-vm-full/38321/console | 21:07 |
| marun | mordred: is that the same issue? | 21:07 |
| clarkb | lifeless: https://review.openstack.org/#/c/39580/1/modules/jenkins/files/slave_scripts/run-tox.sh is fungi's comment there a bug with testr? | 21:07 |
| mordred | marun: yup | 21:07 |
| marun | ah, ffs | 21:08 |
| marun | mordred: it breaks rhel, too. | 21:08 |
| marun | mordred: for different reasons | 21:08 |
| mordred | marun: yah | 21:08 |
| mordred | so, my current approach | 21:08 |
| mordred | is find a different way to fix that | 21:08 |
| mordred | since that one is doing the'right' thing | 21:08 |
| mordred | and the other thing is a workaround | 21:08 |
| mordred | but since the workaround is breaking other things | 21:08 |
| notmyname | mordred: I found your picture on the internets! http://openstackreactions.enovance.com/2013/08/the-way-i-see-the-openstack-infra-team/ | 21:09 |
| mordred | also - once sdague and I start running update.py in things - it's possible that this will sort itself | 21:10 |
| mordred | marun: so it's possible that it's all part of one giant breakage | 21:10 |
| mordred | notmyname: nice! | 21:10 |
| mordred | notmyname: wait - which one is me, and which one is jeblair? | 21:10 |
| notmyname | mordred: I think the answer to anything regarding that pictures is "wat?" | 21:10 |
| mordred | notmyname: fair | 21:11 |
| clarkb | I'm not sure how to feel about that | 21:11 |
| clarkb | EmilienM: are you trying to say we are just one weird party? | 21:11 |
| marun | mordred: so as a short-term solution to keep working in the meantime, is it safe to just comment out unfubar_setuptools? | 21:11 |
| fungi | i hope i'm the one with the headband | 21:12 |
| mordred | marun: on rhel, sure, go for it | 21:12 |
| mordred | marun: if that fixes things for you | 21:13 |
| mordred | for the gate in general, nope, that would break EVERYTHING (sadly) | 21:13 |
| marun | mordred: what breakage is unfubar intended to fix? | 21:13 |
| mordred | the link I just pasted you | 21:13 |
| mordred | https://jenkins.openstack.org/job/gate-tempest-devstack-vm-full/38321/console | 21:13 |
| marun | ok | 21:13 |
| mordred | that's what happens without that function | 21:13 |
| mordred | this is all related to the distribute/setuptools remerge | 21:13 |
| marun | ah, so this is crap raining from above? | 21:14 |
| mordred | yes | 21:14 |
| marun | nuts | 21:14 |
| clarkb | because thing depend on distribute | 21:14 |
| clarkb | which breaks other things | 21:14 |
| mordred | once we get past it, the world will be better | 21:14 |
| mordred | but for now, egads | 21:14 |
| marun | *sigh* | 21:14 |
| marun | if it's not one thing, it's another. | 21:15 |
| mordred | marun: believe me. I could not possibly agree with you more | 21:15 |
| marun | ok, well at least i understand the enormity of the situation now | 21:15 |
| mordred | marun: sdague and I have been inside of this rabbit hole for a _while_ now | 21:15 |
| mordred | and I'm starting to get stockholm syndrome | 21:15 |
| marun | lol | 21:15 |
| mordred | sdague: the file does not seem to be being copied well | 21:15 |
| mordred | sdague: | 21:15 |
| mordred | sdague: I added some addDetails calls: | 21:16 |
| mordred | http://paste.openstack.org/show/42896 | 21:16 |
| clarkb | jeblair: mordred: zaro: https://review.openstack.org/#/c/37924/ any chance I can get more reviews on that? I would like to flush out the JJB change queue so that new changes coming in can be required to have tests | 21:16 |
| marun | mordred: i don't get it - how on earth is this supposed to be upgradeable? | 21:18 |
| openstackgerrit | James E. Blair proposed a change to openstack-infra/config: Add a test periodic job https://review.openstack.org/39805 | 21:18 |
| marun | mordred: it seems so incompatible they should have just changed the name of the project | 21:19 |
| mordred | well - there's a few different things | 21:19 |
| mordred | if you upgrade it by itself, it works | 21:19 |
| jeblair | clarkb: can you approve https://review.openstack.org/39805 ? | 21:19 |
| openstackgerrit | A change was merged to openstack-infra/jenkins-job-builder: Added post-task publisher https://review.openstack.org/37924 | 21:19 |
| mordred | unless there are system versions in some cases | 21:19 |
| jeblair | sdague: https://review.openstack.org/#/c/39797/ should fix the graph | 21:19 |
| marun | the system version screws everything up | 21:19 |
| mordred | marun: but part of the problem is that what happened was a few years ago, setuptools was forked and called distribute | 21:20 |
| mordred | but distribute installed itself as setuptools | 21:20 |
| clarkb | jeblair: reviewing and can approve if all looks well | 21:20 |
| mordred | and in fact, monkeypatched any existing setuptools you happened to have | 21:20 |
| marun | too clever by half | 21:20 |
| marun | grrr | 21:20 |
| mordred | then all the distros decided to ship distribute as setuptools | 21:20 |
| mordred | because it was more maintained and setuptools was 'dead' | 21:20 |
| mordred | then, a few months ago | 21:20 |
| mordred | they reconciled | 21:20 |
| mordred | and merged distribute back in to setuptools | 21:21 |
| mordred | and made distribute go away | 21:21 |
| jeblair | clarkb: (i plan on replacing that almost immediately with actual the jobs and setting the timer to midnight utc if it all checks out) | 21:21 |
| mordred | dstufft: marun isn't too pleased with the setuptools/distribute merge | 21:21 |
| clarkb | jeblair: ok, is that why the name isn't very verbose? | 21:21 |
| *** thomasm has quit IRC | 21:21 | |
| mordred | marun: actually, most of this would work fine | 21:21 |
| mordred | EXCEPT | 21:21 |
| clarkb | I can live with it if that is part of the plan | 21:21 |
| *** pabelanger has quit IRC | 21:21 | |
| jeblair | clarkb: yep. i'm writing the patch that removes the job now. | 21:22 |
| mordred | marun: for projects that have explicit dependencies in their install_requires on distribute | 21:22 |
| zaro | i'm not sure what to do about this.. https://review.openstack.org/#/c/38520/ | 21:22 |
| jeblair | clarkb: i just want to make sure that things more or less work before i start spamming stable maint | 21:22 |
| zaro | do we need to update the bug or something? | 21:22 |
| marun | mordred: it is a disaster for anything installed from os packages | 21:22 |
| clarkb | jeblair: it failed layout testing | 21:22 |
| marun | mordred: which is an awful lot of the world | 21:22 |
| clarkb | jeblair: looking at that now | 21:22 |
| mordred | marun: well - only for things installed with os packages that are then overlaying pip packages on top | 21:22 |
| mordred | and it's _worse_ in redhat | 21:22 |
| marun | mordred: very much worse :( | 21:23 |
| mordred | because of the way redhat has broken python | 21:23 |
| mordred | by making setup.py install install into /usr by default | 21:23 |
| clarkb | jeblair: I think you need to quote the cron spec string? | 21:23 |
| mordred | overlaying 3rd party and system things | 21:23 |
| marun | mordred: I don't understand that one at all. | 21:23 |
| mordred | so there are about 5 bad thigns all forming a perfect storm here | 21:23 |
| mordred | or, to be fair, 2 bad things | 21:23 |
| jeblair | clarkb: not a bad idea | 21:23 |
| mordred | but 2 bad thigns are enough | 21:23 |
| jeblair | zaro: i'll leave another comment in that review | 21:23 |
| mordred | marun: I'm going to fix redhat's python or die trying | 21:23 |
| burt | mordred: when's the wake? | 21:24 |
| marun | mordred: i'd be happy to participate in that effort | 21:24 |
| *** prad_ has quit IRC | 21:24 | |
| marun | mordred: it's unlikely to change in 6.x, but maybe in 7 | 21:24 |
| dtroyer | mordred, marun: I extracted the unfubar_setuptools() stuff to a stand-alone script to try to get a handle on it away from devstack…it ugly but does basically the same thing on precise, f18 and rhel6: https://gist.github.com/dtroyer/6135224 | 21:25 |
| mordred | dtroyer: awesome! | 21:25 |
| *** openstackgerrit has quit IRC | 21:25 | |
| *** openstackgerrit has joined #openstack-infra | 21:25 | |
| marun | mordred: so as a stopgap, the only way I can see rhel working with devstack after this change is having a way to install distribute from scratch, no os packages involved. | 21:25 |
| dtroyer | it's designed to go into devstack/tools | 21:25 |
| marun | dtroyer: ah, install from source. brilliant | 21:26 |
| openstackgerrit | James E. Blair proposed a change to openstack-infra/config: Add a test periodic job https://review.openstack.org/39805 | 21:26 |
| dtroyer | marun: that's basically the approach I took, except using the vendor packages to bootstrap it. that has the side-effect of also working on systems with stuff already installed | 21:26 |
| jeblair | clarkb: ^ | 21:26 |
| dtroyer | marun: it's mordred's bits unrolled a little | 21:26 |
| clarkb | jeblair: do we know why the graphite graph on status isn't updating? | 21:26 |
| mordred | dtroyer: you are a magical pony! | 21:27 |
| jeblair | clarkb: https://review.openstack.org/#/c/39797/ | 21:27 |
| dtroyer | I'm doing a bit more sanity-checking on fresh installs then I'll propose it... | 21:27 |
| clarkb | jeblair: and I will approve the periodic test if the layout test apsses | 21:27 |
| jeblair | clarkb: this channel is busy today; you can't keep up with gerritbot! | 21:27 |
| dtroyer | once we're happy with how it works, stack.sh can call it like install_prereqs works | 21:27 |
| jeblair | clarkb: thx | 21:27 |
| mordred | dtroyer: I thnk you want to make https://gist.github.com/dtroyer/6135224#file-install_pip-sh-L80 just be an fi | 21:28 |
| clarkb | jeblair: I have had a hard time keeping up all week | 21:28 |
| mordred | or even skip it | 21:28 |
| jeblair | clarkb: (though you can approve it before it passes :) | 21:28 |
| mordred | hrm. no. maybe you're right | 21:28 |
| mordred | nevermind. ignore me - you're totally right | 21:28 |
| clarkb | jeblair: I approved the graphite fix as its already broken and seems low impact | 21:28 |
| dtroyer | mordred: we don't have the setuptools tarball there yet…only distribute.zip gets downloaded | 21:28 |
| clarkb | jeblair: ok will approve periodic test | 21:28 |
| *** dina_belova has joined #openstack-infra | 21:29 | |
| dtroyer | that's why we do it again at line 92 | 21:29 |
| mordred | dtroyer: this is truly amazing isn't it? | 21:29 |
| dtroyer | of course, if leaving distribute there is ok, the bits at 92 can go away | 21:29 |
| openstackgerrit | A change was merged to openstack-infra/config: Update jobs launched graph with new paths https://review.openstack.org/39797 | 21:29 |
| dtroyer | mordred: I'll get shot around here for saying this, but python packaging makes CPAN look like a dream | 21:30 |
| mordred | dtroyer: no you won't | 21:30 |
| mordred | dtroyer: dstufft, who runs pypi, might even agree with you | 21:30 |
| mordred | although, fwiw, we _are_ working on trying to fix it | 21:30 |
| dtroyer | fortuantely I didn't know he ran that when I met him ;) | 21:30 |
| openstackgerrit | A change was merged to openstack-infra/config: Add a test periodic job https://review.openstack.org/39805 | 21:31 |
| dstufft | I didn't run it when we met :) | 21:31 |
| *** dkliban has quit IRC | 21:31 | |
| dstufft | I've become an admin since then | 21:31 |
| dstufft | Also what's PyPI doing wrong now | 21:31 |
| dtroyer | well then, you've taken on quite the challenge! | 21:31 |
| marun | dtroyer: have you tested with centos/rhel? | 21:31 |
| * mordred goes to test on fedora | 21:31 | |
| dtroyer | I'm getting ready to on fresh installs. I did run it on already-devstack-ed vms | 21:31 |
| marun | dtroyer: does 'python setup.py' have any dependency on having setuptools installed? | 21:32 |
| marun | (sorry if that's a dumb question) | 21:33 |
| mordred | marun: it does | 21:33 |
| dstufft | yes | 21:33 |
| dstufft | Wheels don't | 21:33 |
| marun | :( | 21:33 |
| dstufft | marun: We're trying to make it better? ;/ | 21:33 |
| mordred | dstufft: nothing - we're battling setuptools right now | 21:33 |
| *** dina_belova has quit IRC | 21:33 | |
| mordred | marun: oh - wait | 21:33 |
| mordred | marun: not the setup.py for setuptools | 21:34 |
| mordred | marun: it just uses distribute/itself | 21:34 |
| mordred | gah | 21:34 |
| dstufft | oh | 21:34 |
| dstufft | right | 21:34 |
| mordred | distutils | 21:34 |
| dstufft | it uses itself | 21:34 |
| mordred | dstufft: we've found GLORIOUS issues | 21:34 |
| mordred | dstufft: also - because I'm yelling at everyone ... | 21:34 |
| dstufft | Jason is on vacation in scandanavia | 21:34 |
| mordred | dstufft: redhat has changed python so that it installs into /usr by default | 21:34 |
| sdague | mordred: ok.... sorry had phone things | 21:35 |
| marun | mordred: …because installing things not from packages is bad, mmkay? and not just because we've made it dangerous! | 21:35 |
| sdague | mordred: so I'm doing this - shutil.copy("tests/files/test-project.txt", self.proj_test_file) | 21:35 |
| burt | dtroyer: whoops, tried to source it. | 21:35 |
| dtroyer | script fails spectacularly on fresh installs | 21:35 |
| sdague | how can that not be working? | 21:35 |
| dstufft | mordred: hrm? | 21:35 |
| dstufft | you mean /usr/lib/python2.7/site-packages | 21:36 |
| dstufft | ? | 21:36 |
| dstufft | or w/e the fuck the url is | 21:36 |
| mordred | yes | 21:36 |
| dstufft | That's actually where Python installs by default | 21:36 |
| dstufft | Debian patches their python | 21:36 |
| mordred | ah! | 21:36 |
| mordred | I've been yelling the wrong things then | 21:36 |
| dstufft | so it installs to /usr/local/lib/python2.7/whatever | 21:36 |
| mordred | yeah. | 21:36 |
| mordred | so - the problem this uncovered today | 21:36 |
| mordred | is that "yum install python-pip ; pip install -U pip ; yum remove python-pip" breaks things | 21:37 |
| openstackgerrit | Matthew Treinish proposed a change to openstack-dev/pbr: Add option to run testr serially https://review.openstack.org/39811 | 21:37 |
| dtroyer | I've been thinking that the redhat issue was mixing site-packages and dist-packages, or maybe not using dist-packages when we expect them to | 21:37 |
| mordred | since pip is overlaying what yum is doing | 21:37 |
| *** cppcabrera has left #openstack-infra | 21:37 | |
| dstufft | dist-packages is a strictly debian thing | 21:37 |
| mordred | dist-packages is another debian thing - which I _really_ don't understand | 21:37 |
| dstufft | well ubuntu and shit too, but that's because of debian | 21:37 |
| dtroyer | iirc there was a long argument about that somewhere I foudn a while back and the result was "this is the way it is" | 21:37 |
| fungi | clarkb: jeblair: i had a couple comments to that effect in the timer triggers change... basically recap cron's field order and also document that sepcial sixth "seconds" field | 21:37 |
| fungi | er, special | 21:37 |
| dstufft | mordred: I think plan was apt-get would be dist-packages, and pip install would be site-packages | 21:38 |
| dstufft | but it got fucked up | 21:38 |
| dstufft | or something | 21:38 |
| mordred | yeah. | 21:38 |
| mordred | so - that would work and help | 21:38 |
| dstufft | Do you need to pip install things at the syste level? | 21:38 |
| dstufft | system | 21:38 |
| jeblair | fungi: yeah, i'm not sure i want to document seconds; i just put it in there so the unit tests don't take >1 min; though i suppose we could. | 21:38 |
| mordred | like, globally? | 21:38 |
| dstufft | yea | 21:38 |
| mordred | yes | 21:38 |
| mordred | that's how all of the devstack gate works | 21:38 |
| dstufft | ok | 21:39 |
| dstufft | I should probably know this stuff by now | 21:39 |
| dstufft | :[ | 21:39 |
| mordred | :) | 21:39 |
| marun | dstufft: hard to run things as root if they're not installed at the system level | 21:39 |
| mordred | there are a LOT of moving pieces | 21:39 |
| mordred | dstufft: basically, openstack gave up on distro packages as a basis for deployment 2 years ago ish | 21:39 |
| dstufft | mordred: so why not just uninstall python-pip and install from get-pip.py | 21:40 |
| mordred | dstufft: that's essentially what we're working on now | 21:40 |
| dstufft | and the setuptools equivilant | 21:40 |
| clarkb | jeblair: mordred fungi I am running the image update job in az3 now. The devstack changes appear to have merged | 21:40 |
| mordred | dstufft: but, I mean, holy crap this is all broke as hell | 21:40 |
| reed | jeblair, can you join the discussion on #openstack-community? | 21:40 |
| mordred | clarkb: excellent | 21:40 |
| *** mrmartin has joined #openstack-infra | 21:40 | |
| *** fifieldt has joined #openstack-infra | 21:40 | |
| dstufft | mordred: Well Linux will just say you shouldn't be using pip to install things globally | 21:41 |
| marun | wait, why isn't get-pip.py being used? | 21:41 |
| marun | dstufft: linux who? | 21:41 |
| dstufft | marun: any of the distros | 21:41 |
| marun | dstufft: it works on ubuntu and debian (except for this mess) | 21:41 |
| dstufft | The people who work on them yell at me all the time saying pip shouldn't fuck with system packages | 21:41 |
| sdague | dstufft: the reason we need to in the gate, is to ensure that the resultant could be packaged | 21:41 |
| dstufft | to which I tell them idc | 21:41 |
| marun | dstufft: that's the point of isolating pip install to /usr/local | 21:41 |
| dstufft | but that's what they say :) | 21:42 |
| marun | don't mess with system packages | 21:42 |
| * fungi disappears again for a bit | 21:42 | |
| sdague | mordred: ok, can I steal you back for a minute? | 21:42 |
| mordred | sdague: YES! | 21:42 |
| sdague | so... those unit tests | 21:42 |
| sdague | is there a better way to copy the files over? | 21:42 |
| dstufft | marun: yea, Debian did that to prevent these kind of issues afaik, not because they like the fact people are doing it | 21:42 |
| sdague | because I just don't understand why that would fail | 21:43 |
| mordred | dstufft: well, sure - but they all have a distro view | 21:43 |
| mordred | dstufft: having an upstream view is much different, it turns out :) | 21:43 |
| marun | dstufft: yes, the distro view. but not everything gets os packages | 21:43 |
| marun | dstufft: at least they gave the rest of us an out | 21:43 |
| marun | dstufft: rh, not so much | 21:43 |
| dstufft | marun: Yea I bitch at Nick from time to time about it | 21:43 |
| dstufft | I think that's compiled in :[ | 21:44 |
| burt | dtroyer: should I just replace the unfubar line with install_pip.sh in stack.sh to test this? | 21:44 |
| *** pentameter has quit IRC | 21:44 | |
| mordred | sdague: sorry - still poking trying to figure out why it's not copying | 21:44 |
| dtroyer | burt: not yet. it doesn't work on fresh installs | 21:44 |
| burt | okey dokey. | 21:44 |
| mordred | dstufft: at the start of openstack, we made distro packages for everything and did our installs using them | 21:44 |
| mordred | dstufft: then ubuntu got mad at us, saying that was their job | 21:44 |
| dstufft | lol | 21:44 |
| dstufft | EVERYONES GETS MAD AT EVERYONE | 21:44 |
| dstufft | :[ | 21:44 |
| mordred | dstufft: and redhat agreed with them, saying it was their job | 21:44 |
| mordred | so we quit | 21:44 |
| mordred | and adopted pip | 21:44 |
| marun | lol | 21:45 |
| mordred | I'm 3 seconds away from telling everyone to shove it, we're going to start making packages for all the distros and hosting our own apt/yum repo... | 21:45 |
| sdague | mordred: you should make ebuilds | 21:45 |
| dtroyer | ok, so I've got a process to bootstrap pip/distribute on win***s, we want setuptools instead right? | 21:45 |
| * sdague ducks | 21:45 | |
| marun | so why can't get-pip.py be used? | 21:45 |
| mordred | sdague: any time someone can show me how to run a stable gentoo system ... | 21:45 |
| sdague | mordred: now you're hunting unicorns | 21:46 |
| dtroyer | marun: I think that's what I'm looking for | 21:46 |
| marun | https://raw.github.com/pypa/pip/master/contrib/get-pip.py | 21:46 |
| dstufft | dtroyer: you pretty much always want setuptool now | 21:46 |
| dstufft | dstribiute is merged back in and dead | 21:46 |
| dtroyer | ok, thanks | 21:46 |
| mordred | dtroyer, marun: if we use git-pip or direct downloading... | 21:46 |
| mordred | we'll wnat to make sure we can re-direct it at our mirror | 21:46 |
| dstufft | hm | 21:46 |
| mordred | and/or pre-downloaded things | 21:46 |
| dstufft | get-pip uses a bundled copy of pip | 21:46 |
| dstufft | in order to install from pypi | 21:47 |
| mordred | does it honor ~/.pip/pip.conf ? | 21:47 |
| dstufft | don't think it supports command line args, probably still supports the config file and env vars | 21:47 |
| mordred | we can have the image build job do get-pip | 21:47 |
| mordred | and stash the results locally | 21:47 |
| mordred | so that devstack can just consume them | 21:47 |
| dstufft | oh maybe it does | 21:48 |
| dstufft | https://github.com/pypa/pip/blob/develop/pip/__init__.py#L151-L155 | 21:48 |
| dstufft | that's what it runs | 21:48 |
| dstufft | just unpacked a local pip and runs pip.bootstrap() | 21:48 |
| dtroyer | dstufft: is that the place we should get it from? | 21:48 |
| dtroyer | I mean, in scripts and all | 21:48 |
| dstufft | you'll want to make sure any system package is uninstalled before running it | 21:48 |
| marun | …and make the system cough if someone tries to install os packages on top... | 21:49 |
| dtroyer | mordred: we need to handle that for devs too, that's why I wanted the stand-alone script | 21:49 |
| marun | *boom* | 21:49 |
| dstufft | http://www.pip-installer.org/en/latest/installing.html#using-get-pip | 21:49 |
| *** burt has quit IRC | 21:49 | |
| dstufft | well | 21:49 |
| marun | it may require setuptools already installed | 21:49 |
| dstufft | yea it does | 21:49 |
| dstufft | https://pypi.python.org/pypi/setuptools/0.9.8#unix-based-systems-including-mac-os-x | 21:50 |
| marun | ez_setup? | 21:50 |
| clarkb | mgagne: you about? https://review.openstack.org/#/c/39383/1 does puppet default to using root as the owoner? | 21:50 |
| mordred | dtroyer: totally. I just want to make sure that we always have the opportunity to pre-fetch anything that devstack wants to grab from the internets | 21:50 |
| mordred | and/or direct things to our mirror | 21:50 |
| dtroyer | 'zactly | 21:51 |
| marun | install to custom locations!! https://pypi.python.org/pypi/setuptools/0.9.8#id106 | 21:51 |
| mgagne | clarkb: puppet defaults to the user running puppet, most of the time it's root. | 21:51 |
| marun | oops | 21:51 |
| marun | https://pypi.python.org/pypi/setuptools/0.9.8#advanced-installation | 21:51 |
| mordred | sdague: I'm not sure it's a file copy problem | 21:51 |
| sdague | mordred: ok | 21:51 |
| mordred | sdague: I think it might actually be a bug in update.py | 21:51 |
| mordred | like, I think the test might be telling us something :) | 21:51 |
| sdague | ok, I'm happy with that as the answer | 21:52 |
| sdague | but like I said, I think I went code blind :) | 21:52 |
| mordred | sdague: can you explain to me what you did with global-requirements and test-requirements ? | 21:52 |
| sdague | so I needed second eyes | 21:52 |
| dstufft | I don't know if it helps at all | 21:52 |
| dstufft | are you aware of --user? | 21:52 |
| sdague | copy requirements.txt global-requirements | 21:52 |
| sdague | cat test-requirements >> global-requirements | 21:52 |
| sdague | plus a couple of comment lines seperating | 21:52 |
| mordred | dstufft: yeah - but we're installing clouds here- we're trying to test that a system level install of a cloud works | 21:53 |
| dstufft | mordred: ok | 21:53 |
| mordred | we _explicitly_ want to install system wide | 21:53 |
| *** bnemec has joined #openstack-infra | 21:53 | |
| dstufft | alterantively install your own Python in /opt :V | 21:53 |
| dstufft | and tell RedHat's to go fuck off | 21:53 |
| sdague | dstufft: then we need to own python updates | 21:53 |
| sdague | which is not desirable | 21:54 |
| mordred | right. we're trying to test that people can install this in a general way | 21:54 |
| mordred | not that we can instal it knowing special things | 21:54 |
| sdague | and that the distros can actually build a packaged version | 21:54 |
| mordred | well, I'm caring less and less about that ... | 21:54 |
| mordred | but yeah | 21:54 |
| sdague | I still think that's *very* important from an ecosystem perspective | 21:55 |
| sdague | but that's beer talk | 21:55 |
| sdague | lets get back to update.py bug | 21:55 |
| sdague | because I think it's all that stands between us and magical ponies | 21:55 |
| mordred | it may not look like it - but I am actually trying to step through it | 21:55 |
| mordred | yes. I agree | 21:55 |
| sdague | :) | 21:55 |
| mordred | so many ponies | 21:55 |
| mordred | millions of dancing ponies | 21:55 |
| mordred | (now imagine millions - like, actually _millions_ of ponies dancing all in a mob) | 21:55 |
| dstufft | mordred: if you have contacts at RH tell them to go adopt the ddebian patch :D (not it fixes the problem right this second but) | 21:56 |
| EmilienM | Hi, I need some review to push puppet-heat into StackForge > https://review.openstack.org/#/c/39594/ & https://review.openstack.org/#/c/39655/ | 21:56 |
| mordred | dstufft: I'm going to | 21:56 |
| mordred | dstufft: I'll be calling ... people | 21:56 |
| clarkb | mgagne: but it looks like it won't change it from the existing value if the file exists? | 21:56 |
| clarkb | mgagne: I did a quick puppet agent --noop test and it didn't seem to want to change those permissions | 21:56 |
| mordred | dstufft: I'd try to convince python to change its behavior... | 21:56 |
| dstufft | mordred: in the future this will be.. better even if they refuse to. | 21:56 |
| mordred | dstufft: but something tells me that even if I were successful... | 21:57 |
| mordred | dstufft: it would still onlybe 2.5 | 21:57 |
| dstufft | mordred: It would only land in 3.4 even if you were successful *today* | 21:57 |
| clarkb | jeblair: can we reenable puppet on jenkins.o.o? | 21:57 |
| mordred | dstufft: what if pip just magically started installing things into /usr/local ... | 21:57 |
| mordred | dstufft: you have the power to change the world... abuse it!!! | 21:57 |
| dstufft | mordred: the problem is we just ask python where it should install things at, this is how the debian patch works (it patches Python), and it's how virtualenv works, and how installing python into /opt works etc | 21:58 |
| mordred | yah | 21:58 |
| dstufft | We'd have to detect if we're using a global python, and reason about if /usr/local made sense, and that sounds like it has massive breakage potentional | 21:58 |
| * marun is boggled that zip and unzip are separate commands/packages in rhel | 21:59 | |
| clarkb | mordred: safe to approve https://review.openstack.org/#/c/38540/2 ? | 21:59 |
| clarkb | mordred: I am not up to speed on todays "The State of PBR" | 21:59 |
| dstufft | I'll send an email to Nick too whine again | 22:00 |
| openstackgerrit | A change was merged to openstack-infra/config: Set ownership of JJB config file to root https://review.openstack.org/39383 | 22:00 |
| dstufft | to* | 22:00 |
| mgagne | clarkb: could be that puppet won't change the ownership of a file if it exists already but would default to root if it has to create it. Want me to enforce root ownership? | 22:00 |
| clarkb | mgagne: no, I think the non enforcement is fine | 22:00 |
| clarkb | mgagne: as it will grandfather in people using jenkins | 22:00 |
| clarkb | mordred: I am going to approve it, you can -2 if it will break the world | 22:02 |
| mordred | clarkb: totally +2/aprv | 22:02 |
| mordred | totally safe | 22:02 |
| dstufft | mordred: It's really not a Python problem either, Python expects to own it's tree (which it does if you install from anywhere other than a package manager). Linux Distros takes away control over part of it's tree and RH doesn't provide a non packager owned place. (ACtually discovering this was a big revalation as to why the RH people I knew got so angry about people using pip at the global level) | 22:02 |
| mordred | dstufft: that's an excellent point | 22:03 |
| *** bnemec has quit IRC | 22:03 | |
| jeblair | mrmartin: meet clarkb, mordred, and fungi, the other core infrastructure people | 22:03 |
| clarkb | mrmartin: hello | 22:03 |
| *** bnemec has joined #openstack-infra | 22:03 | |
| mrmartin | hi guys | 22:04 |
| jeblair | mrmartin: will be working on the drupal user group system with reed | 22:04 |
| jeblair | in #openstack-community, i was explaining how to get started with our system | 22:04 |
| jeblair | it sounds very promising | 22:04 |
| clarkb | cool | 22:04 |
| jeblair | mrmartin: sdague is also interested in this project; he knows a lot about our infrastructure and does some things with drupal in his spare time | 22:04 |
| datsun180b | installing to /usr/local/lib wouldn't be a problem if sys.path in python didn't somehow kick /usr/local/lib to below /opt/stack/* and /usr/lib/, from what i'm seeing in my own environment | 22:05 |
| mrmartin | the first sprint of the work will involve the ci system, we want to integrate drupal and use the same processes as python projects. | 22:05 |
| jeblair | mrmartin: we love running tests, and lots of them. :) | 22:05 |
| datsun180b | context: i work in trove and i'm very silly in that i'm building things with vagrant too, so what i'm saying may be at right angles to the rest of you | 22:05 |
| marun | dstufft: translation: rh people angry at themselves for making a mess of things. | 22:06 |
| mtreinish | jeblair, fungi, mordred: can I get a quick review on: https://review.openstack.org/#/c/39583/ so we can save tempest logs from runs | 22:06 |
| dstufft | datsun180b: on Debian /usr/local/ is before /usr/ | 22:07 |
| dstufft | ['', '/usr/lib/python2.7', '/usr/lib/python2.7/plat-linux2', '/usr/lib/python2.7/lib-tk', '/usr/lib/python2.7/lib-old', '/usr/lib/python2.7/lib-dynload', '/usr/local/lib/python2.7/dist-packages', '/usr/local/lib/python2.7/dist-packages/setuptools-0.6c11-py2.7.egg-info', '/usr/lib/python2.7/dist-packages', '/usr/lib/python2.7/dist-packages/PIL'] | 22:07 |
| mrmartin | ok, I start to prepare the project, and I'll spend the next two weeks with you guys finding out how to integrate this platform. | 22:07 |
| datsun180b | dstufft: https://gist.github.com/ed-/062857f85de2079f9629 | 22:07 |
| *** reed has quit IRC | 22:07 | |
| mrmartin | thanks for your support. | 22:07 |
| jeblair | mtreinish: done | 22:08 |
| dstufft | datsun180b: hrm, well mine's Ubuntu specifically | 22:08 |
| mtreinish | jeblair: cool thanks | 22:08 |
| clarkb | EmilienM: the puppet-heat change lgtm | 22:09 |
| EmilienM | clarkb: thx | 22:09 |
| EmilienM | clarkb: if you could also review https://review.openstack.org/#/c/39655/ | 22:10 |
| datsun180b | hrm indeed | 22:10 |
| clarkb | EmilienM: done | 22:10 |
| EmilienM | clarkb: cool | 22:11 |
| *** beagles has quit IRC | 22:11 | |
| EmilienM | clarkb: I have another question. Does it make sense to submit a new repo in StackForge with monitoring Scripts for OpenStack, which could be use by many monitoring system like Nagios or other tools ? | 22:12 |
| clarkb | mordred: sdague https://review.openstack.org/#/c/39508/2/modules/openstack_project/files/zuul/layout.yaml I thought oslo.config was consumed through releases and not git | 22:12 |
| *** _TheDodd_ has quit IRC | 22:12 | |
| clarkb | mordred: sdague was that changed during the great requirements upheaval? | 22:12 |
| sdague | clarkb: yes | 22:12 |
| sdague | think of it like the python-*clients | 22:12 |
| *** whoops has quit IRC | 22:12 | |
| clarkb | sdague: ok | 22:12 |
| sdague | also see my long email on list :) | 22:13 |
| clarkb | EmilienM: if you feel that the infrastructure behind stackforge would be useful then yes | 22:13 |
| dstufft | mordred: so I emailed nick asking what it would take to get RH to do what debian did | 22:13 |
| clarkb | EmilienM: stackforge has been called infrastructure as a service... If that is useful to you then yes, If you end up with a singel reviewer and use the gate-noop jobs then probably not | 22:13 |
| dstufft | That's about my only RH contact I think :[ | 22:14 |
| mordred | dstufft: well, he's not a bad one in this world | 22:14 |
| dstufft | yea | 22:14 |
| dstufft | specially since he's involved in packaging on the python side too :D | 22:14 |
| clarkb | sdague: in that change is there a reason we included neutron but not postgres? | 22:14 |
| *** changbl has quit IRC | 22:14 | |
| mordred | dstufft: next pycon, we should really sit down and try to work through some of the uptream vs. distro stuff from a python pov | 22:15 |
| *** rnirmal has quit IRC | 22:15 | |
| dstufft | Maybe CPython can make some sort of policy or document for people packaging it that they should do /usr/local | 22:15 |
| mordred | yeah | 22:15 |
| mordred | exactly | 22:15 |
| mordred | if we had a policy that made sense taking both sides in to account | 22:15 |
| * clarkb jumps into the gray. Didn't arch start unifying /usr | 22:15 | |
| dstufft | doesn't stop the pain today, but might make it better in the future | 22:15 |
| sdague | clarkb: because there shouldnt' be any difference between mysql & postgresql jobs with that | 22:16 |
| clarkb | sdague: I see | 22:16 |
| *** mrmartin has quit IRC | 22:17 | |
| *** lcestari has quit IRC | 22:19 | |
| clarkb | mordred: https://review.openstack.org/#/c/39208/ any idea why unittests failed there? | 22:19 |
| clarkb | doesn't seem related to the change | 22:19 |
| jeblair | clarkb: puppet restarted on zuul and jenkins | 22:19 |
| clarkb | https://jenkins.openstack.org/job/gate-swift-docs/3218/console so that is happening | 22:20 |
| clarkb | jeblair: thank you | 22:20 |
| clarkb | Alex_Gaynor: ^ does that look familiar to you? | 22:20 |
| mordred | clarkb: it's an oops in the dependent package | 22:21 |
| mordred | clarkb: it's something in the unittests - the functionality works | 22:22 |
| clarkb | mordred: in cffi | 22:22 |
| mordred | what? | 22:22 |
| clarkb | oh you are referrng to pbr | 22:22 |
| mordred | yah | 22:22 |
| clarkb | I am context switched into broken xattr beacuse the gate is going boom! | 22:22 |
| clarkb | mordred: look at the gate-swift-docs thing above | 22:24 |
| mordred | clarkb: it's a distutils/setuptools thing | 22:24 |
| mordred | how did that happen on one of our nodes? | 22:24 |
| clarkb | they just pushed a new release | 22:24 |
| mordred | are we not using up to date tox everywhere? | 22:24 |
| clarkb | its not setuptools its cffi | 22:25 |
| clarkb | or at least new cffi released just now is tickling a problem | 22:25 |
| mordred | oh. nop | 22:25 |
| clarkb | https://pypi.python.org/pypi/cffi/0.7.1 | 22:25 |
| mordred | I'm sorry, you're right | 22:25 |
| clarkb | so uh ya | 22:25 |
| clarkb | we should find a way for maybe the mirror to let us know when this happens | 22:25 |
| clarkb | should we have it email the infra list when it fails to install a thing? | 22:26 |
| mordred | assert backend.__version__ == __version__ | 22:26 |
| * clarkb tries to reproduce locally to see if debugging can be done | 22:26 | |
| mordred | clarkb: probably | 22:26 |
| jeblair | clarkb, mordred: er what? | 22:27 |
| jeblair | i'm pretty sure emailing someone is not the right kind of automation | 22:27 |
| clarkb | mordred: hmm is that complaining that libffi version != cffi's version | 22:27 |
| clarkb | jeblair: well we need something that tells us it may have broken | 22:27 |
| mordred | clarkb: you've got to be kidding me | 22:27 |
| clarkb | jeblair: but we want it to update the mirror anyways | 22:27 |
| clarkb | mordred: going to test | 22:27 |
| *** mrodden has quit IRC | 22:27 | |
| lifeless | clarkb: hi, looking | 22:27 |
| lifeless | clarkb: (I had to drop C off to kindy then get my monthly allergy shot) | 22:27 |
| *** mriedem has quit IRC | 22:28 | |
| dhellmann | clarkb: we don't have an existing repo with pycadf code; we're waiting for this one to be approved so we can avoid adding the code to the ceilometer code base | 22:28 |
| mtreinish | clarkb: yeah arch moved things from /bin and /sbin into /usr: https://mailman.archlinux.org/pipermail/arch-dev-public/2012-March/022625.html | 22:29 |
| clarkb | pip install cffi works | 22:29 |
| mtreinish | but /usr/local still exists | 22:29 |
| *** plomakin has quit IRC | 22:29 | |
| jeblair | clarkb: i'm not up to speed enough to say anything helpful, but i would like to avoid having "email someone" be a design goal of our automation. what kind of breakage are we talking about? | 22:29 |
| mordred | sdague: found it | 22:29 |
| clarkb | dhellmann: mtreinish gotcha | 22:29 |
| *** emagana has quit IRC | 22:29 | |
| dhellmann | clarkb: after the repo is created, I'll set up the packaging stuff as the first commit | 22:29 |
| *** plomakin has joined #openstack-infra | 22:29 | |
| lifeless | clarkb: no, it's not a testr bug | 22:29 |
| clarkb | jeblair: https://jenkins.openstack.org/job/gate-swift-docs/3218/console is breaking because the mirror scripts pushed cffi 0.7.1 | 22:29 |
| mordred | sdague: gr-base.txt is a file from before you added test-requirements to requirements.txt | 22:30 |
| jeblair | clarkb: did the mirror scripts to the wrong thing? | 22:30 |
| sdague | mordred: oh... ffs | 22:30 |
| clarkb | jeblair: I think so, I haven't confirmed but it did the last time cffi broke | 22:30 |
| sdague | yep | 22:30 |
| sdague | ok, I'll fix it | 22:30 |
| lifeless | clarkb: the .testr.conf specifies ${PYTHON:-python} as the thing to run. | 22:30 |
| clarkb | lifeless: aha | 22:30 |
| clarkb | jeblair: https://jenkins.openstack.org/job/periodic-mirror-python27/159/console yes the error occured in the mirror script | 22:31 |
| jeblair | clarkb: what do you want the mirror script to do instead? | 22:31 |
| lifeless | clarkb: so either having the venv activated (so python will be on the path) or specifying the actual binary to run, is needed. | 22:31 |
| clarkb | jeblair: make it known that something potentially broke | 22:32 |
| *** vijendar has quit IRC | 22:32 | |
| clarkb | I cannot reproduce locally | 22:32 |
| lifeless | jeblair: so | 22:32 |
| mordred | sdague: I've got a fix in tree... | 22:32 |
| lifeless | jeblair: is now an ok time to talk about the mirror list thingy? | 22:33 |
| *** krtaylor has quit IRC | 22:33 | |
| jeblair | clarkb: so you don't want to alter the behavior of the mirror script? | 22:33 |
| lifeless | jeblair: and/or rsync or something access to the mirror ? | 22:33 |
| *** ogelbukh has quit IRC | 22:33 | |
| jeblair | clarkb: as in, you don't want to either upload or not upload something differently than what's happening now? | 22:33 |
| clarkb | jeblair: if possible I would like to not upload the one package that failed | 22:33 |
| clarkb | jeblair: but pip makes that very hard aiui | 22:33 |
| mordred | sdague: http://paste.openstack.org/show/42903 | 22:34 |
| jeblair | clarkb: so istr that we wanted not to gate the world | 22:34 |
| mordred | jeblair: right, but we do want to make our mirror a good thing | 22:34 |
| jeblair | clarkb: and that if upstream packagers broke our expectations, we wanted that to propogate to the projects | 22:34 |
| jeblair | mordred: is it bad? | 22:34 |
| mordred | jeblair: and we're hitting problems now where a package will update | 22:34 |
| mordred | we'll get _half_ of it | 22:34 |
| clarkb | jeblair: it is bad to not upgrade everything else because xattr and cffi are derping | 22:34 |
| mordred | and one of its deps will not build on our mirror builder | 22:35 |
| mordred | but we'll still upload part of the upgraded thing | 22:35 |
| mordred | even though things failed | 22:35 |
| clarkb | oh, I can reproduce, you have to install xattr not cffi | 22:35 |
| clarkb | xattr has not updated recently, so I think cffi update may have broken xattr | 22:35 |
| jeblair | mordred, clarkb: if we were not using a mirror, would this be a problem? | 22:35 |
| clarkb | jeblair: yes | 22:36 |
| mordred | ah - my bad | 22:36 |
| mordred | I thought the other | 22:36 |
| mordred | in that cas,e I agree with jeblair | 22:36 |
| clarkb | oh you mean the badness and not xattr, in that case no | 22:36 |
| dstufft | mordred: ok, got the path forward from Nick, I'm gonna have to join a fedora mailing list (exactly what I was hoping for!) | 22:36 |
| jeblair | it doesn't sound like a problem with the mirror. | 22:36 |
| mordred | I thought this was a mirror builder issue that was producing a state in the mirror that was not what was happening in the world | 22:36 |
| mordred | dstufft: yay! | 22:36 |
| *** weshay has quit IRC | 22:36 | |
| mordred | jeblair: agree | 22:37 |
| clarkb | mordred: jeblair: what is happening is the mirror is letting through brokeness | 22:37 |
| clarkb | it should not do that | 22:37 |
| mordred | it should | 22:37 |
| mordred | it should mirror the world | 22:37 |
| mordred | if xattr is now broken | 22:37 |
| clarkb | I thought the whole point of the mirror was to protect against stuff like this? | 22:37 |
| mordred | we need for swift to know to pin it | 22:37 |
| mordred | nope | 22:37 |
| mordred | it's to protect us against flaky network connections | 22:37 |
| mordred | and stuff | 22:37 |
| clarkb | ok, so back to the email suggestion. We want to know when the mirror lets this stuff through | 22:37 |
| mordred | our mirror having content different than the world at large is bad | 22:38 |
| clarkb | we don't need to wait for swift, xattr does not install something should set off a klaxon | 22:38 |
| jeblair | clarkb: the thing that's like what you're suggestion that the mirror should do is protect us from putting _ourselves_ in this situation by having bad deps | 22:38 |
| jeblair | but if someone else makes them bad, we need to know about it. | 22:38 |
| mordred | yes | 22:38 |
| mordred | that I agree with | 22:38 |
| jeblair | mordred: what do you agree with? | 22:38 |
| lifeless | 'that' | 22:38 |
| jeblair | mordred: ping | 22:39 |
| mordred | jeblair: that we want the mirror to proect us from putting _ourselves_ in this situation | 22:39 |
| jeblair | ok | 22:39 |
| mordred | lifeless: cinder has run in to this: http://logs.openstack.org/62/29862/37/check/gate-cinder-python26/360b9f0/console.html.gz | 22:40 |
| mordred | lifeless: which I'm pretty sure is due to the test having an id attribute | 22:40 |
| clarkb | jeblair: should the mirror try to be more proactive in some way when it knows the world is broken? | 22:40 |
| jeblair | clarkb: i think the current notification mechanism is very effective. i'm not interested in getting emails when something fails to install. | 22:41 |
| clarkb | jeblair: or will we just let it fall through? | 22:41 |
| clarkb | ok | 22:41 |
| *** bnemec has quit IRC | 22:41 | |
| lifeless | mordred: all tests have an id function, it's part of the contract of being 'a test' | 22:41 |
| clarkb | Alex_Gaynor: xattr is broken with new cffi on precise | 22:41 |
| mordred | lifeless: but I've been asked by harlowja why it's bombing there | 22:41 |
| mordred | lifeless: no, I mean they have a attribute called id | 22:41 |
| Alex_Gaynor | clarkb: grumble, thanks for the heads up, investigating | 22:41 |
| mordred | that is not the function | 22:41 |
| lifeless | mordred: oh, you can't do that. It will break things. | 22:41 |
| lifeless | its reserved. | 22:41 |
| jeblair | clarkb: most of the time fixing something like this involves a requirements change. i really don't want to be seen (or actually be) responsible for fixing things every time some pypi upstream package breaks. | 22:41 |
| jgriffith | mordred: lifeless I think the point is why did it trigger suddenly here? | 22:41 |
| mordred | yes. that's what I said | 22:41 |
| jgriffith | mordred: haha | 22:41 |
| harlowja | ya, seems odd | 22:42 |
| mordred | lifeless: the question is what jgriffith asked | 22:42 |
| jeblair | clarkb: we have empowered everyone in the project to fix this situation. | 22:42 |
| jgriffith | mordred: sorry.. not to cut in, but I'm extremely curious | 22:42 |
| jgriffith | :) | 22:42 |
| lifeless | oh, well if you were using a runner that happened to not call id() it would not show up | 22:42 |
| mordred | jgriffith: take it away... I'm just channel you | 22:42 |
| *** bnemec has joined #openstack-infra | 22:42 | |
| clarkb | Alex_Gaynor: if I pin cffi to 0.7.0 before installing xattr that works | 22:42 |
| jgriffith | haha | 22:42 |
| mordred | lifeless: I believe they've been on testr/testtools for a while | 22:42 |
| harlowja | who am i channeling? | 22:42 |
| jeblair | clarkb: if it should email anyone, it should probably email openstack-dev. | 22:42 |
| Alex_Gaynor | clarkb: go ahead and do that for now, i tlooks like the whole 0.7.1 release is broken | 22:42 |
| * harlowja must channel someone to | 22:42 | |
| lifeless | same as if you broke self.fail() but didn't happened to call it | 22:42 |
| mordred | Alex_Gaynor: isn't this the second time in like 2 weeks that cffi broke something? | 22:43 |
| jgriffith | lifeless: not an explanation | 22:43 |
| dtroyer | mordred, dstufft, marun: dinnertime, but before I do here's a vastly simplified install_pip.sh: https://gist.github.com/dtroyer/6135224. It needs to get the downloaded scripts locally first if available so they can be pre-cached and I'm not happy with the version detection. but it works a hell of a lot better. | 22:43 |
| clarkb | jeblair: I don't want to be responsible either. I am on board with this. I really thought other people thought differently and was one of the reasons I didn't want to use the mirror for strong enforcement of requirements | 22:43 |
| jgriffith | lifeless: https://review.openstack.org/#/c/29862/40/cinder/tests/test_volume.py | 22:44 |
| clarkb | but I am fully grokking now (some months later apparently) | 22:44 |
| jgriffith | lifeless: without this patch, tests run fine | 22:44 |
| lifeless | jgriffith: so before the test is run the runner asks for the id(); I don't think it asks for it again unless you trigger a failure. | 22:44 |
| jgriffith | lifeless: with this patch... barf | 22:44 |
| jgriffith | lifeless: ahhh | 22:44 |
| jgriffith | lifeless: now that helps | 22:44 |
| lifeless | jgriffith: the patch is broken, and it's triggering a cascade error which shows up the fact that you'd been violating the contract. | 22:44 |
| Alex_Gaynor | mordred: I don't believe so, no. | 22:44 |
| *** ogelbukh has joined #openstack-infra | 22:44 | |
| mordred | or was it ffi | 22:44 |
| harlowja | lifeless just odd that the cascade would show up in a different test | 22:45 |
| jgriffith | lifeless: thanks | 22:45 |
| lifeless | now - its ugly that the Test contract impinges on your objects, but it is what it is. | 22:45 |
| mordred | clarkb: didn't we have a broken release a few weeks ago with something? | 22:45 |
| jgriffith | harlowja: since we're trying to be fancy and paralell etc it's not so odd I don't think | 22:45 |
| harlowja | k | 22:45 |
| harlowja | possibly | 22:45 |
| *** datsun180b has quit IRC | 22:45 | |
| harlowja | thread locals for id? | 22:45 |
| lifeless | harlowja: hmmm, it doesn't does it ? | 22:45 |
| clarkb | mordred: yes xattr switched to cffi and there were two problems. One cffi requires libffi-dev and two it didn't build on our linuxes | 22:46 |
| dstufft | mordred: 5 minutes in and Nick already told me Openstack shouldn't use global pip and should sue a software collection instead! lol | 22:46 |
| harlowja | lifeless ok, let me see what i can do to fix that, interesting to know | 22:47 |
| *** mrodden has joined #openstack-infra | 22:47 | |
| marun | dstufft: Nick who? | 22:49 |
| lifeless | harlowja: we can't easiy see what test it broke on, because we're in a try:except: block in the framework | 22:49 |
| dstufft | marun: Nick Coghlan | 22:49 |
| harlowja | gotcha | 22:49 |
| lifeless | harlowja: there is no reason to think it is failing and reporting in different tests - that would be super odd | 22:50 |
| dstufft | marun: He's a friend who works @ RH, and is agaisnt using pip at the global level, needless to say I don't really agree with him :) | 22:50 |
| dstufft | but thems the breaks between linux distros and language specific packagers ;P | 22:51 |
| marun | dstufft: I work at RH, and I think railing against global installation with pip is conflating a symptom with the disease | 22:51 |
| dstufft | marun: well then ! | 22:51 |
| dstufft | marun: I'm going to be emailing uh, fedora-python@lists.fedorahoted.org about why they should patch their python to install to /usr/local | 22:51 |
| dstufft | later on tonight probably | 22:52 |
| marun | +1 | 22:52 |
| locke105 | where are the scheduled downtimes for -infra systems posted? i can't seem to find them... | 22:52 |
| clarkb | harlowja: I believe mordred has a quick one liner that will run each test in its own test runner one at a time | 22:52 |
| clarkb | harlowja: may help in finding the problem test | 22:52 |
| clarkb | locke105: right here | 22:52 |
| harlowja | oh, good idea | 22:52 |
| dstufft | for now i'm going to get some dinner | 22:52 |
| clarkb | locke105: usually in this channel, and #openstack-dev and #openstack. Planned outages will be annoucned to the openstack-dev and possibly openstack mailing lists depending on impact | 22:53 |
| clarkb | locke105: and I suppose a prolonged unplanned outage would get mail list treatment as well | 22:53 |
| locke105 | clarkb: ok, was looking for like a schedule for future downtimes | 22:53 |
| locke105 | unless there isn't one | 22:53 |
| harlowja | mordred do u have said script memorized? :-P | 22:54 |
| clarkb | locke105: we typically don't need to plan that far in advance as long as we avoid milestone and release candidate weeks :) | 22:54 |
| clarkb | locke105: you need an outage for the pypi-mirror rename right? | 22:54 |
| locke105 | yeah was wondering when that was happening | 22:54 |
| openstackgerrit | James E. Blair proposed a change to openstack-infra/config: Support periodic jobs in log custom function https://review.openstack.org/39829 | 22:54 |
| jeblair | clarkb: can you aprv that ^ ? | 22:55 |
| *** pcrews has joined #openstack-infra | 22:55 | |
| clarkb | jeblair: yup | 22:55 |
| clarkb | jeblair: thoughts on outage time for locke105 ? | 22:55 |
| jeblair | clarkb: that's the cause of the test jobs not running (and backing up, and throwing exceptions in the zuul log) | 22:55 |
| jeblair | clarkb, locke105: i think mordred was going to think on that, but i believe we've all been overwhelmed this week so far. | 22:56 |
| locke105 | on an unrelated note, i can't seem to reach http://ci-puppetmaster.openstack.org/ | 22:56 |
| mordred | locke105: yeah - sorry man - it's been a week in setuptools hell | 22:56 |
| clarkb | jeblair: should line 6 be an elif? | 22:56 |
| locke105 | lol np | 22:56 |
| jeblair | locke105: it has no services, but you can see the dashboard at http://puppet-dashboard.openstack.org:3000/ | 22:56 |
| jeblair | clarkb: yes, thanks. :) | 22:56 |
| openstackgerrit | James E. Blair proposed a change to openstack-infra/config: Support periodic jobs in log custom function https://review.openstack.org/39829 | 22:57 |
| jeblair | clarkb: i'm clearly about burned out for the day myself. | 22:57 |
| clarkb | jeblair: me too | 22:57 |
| locke105 | jeblair: oh nice, was just wondering if were aware that URL wasn't working or not | 22:57 |
| clarkb | this week has been weird | 22:57 |
| clarkb | I think getting up at 0530 yesterday didn't help | 22:57 |
| clarkb | jeblair: approved | 22:58 |
| locke105 | who would have thought setuptools could cause so much chaos... | 22:59 |
| mordred | jeblair: +1000 | 22:59 |
| openstackgerrit | A change was merged to openstack-infra/config: Support periodic jobs in log custom function https://review.openstack.org/39829 | 23:00 |
| *** zaro0508 has joined #openstack-infra | 23:02 | |
| clarkb | Alex_Gaynor: I have confirmed that import cffi ; cffi.FFI() is sufficient to trigger that assertion. cffi seems broken | 23:02 |
| jeblair | periodic jobs are running now | 23:02 |
| clarkb | Alex_Gaynor: will you guys be pinning upstream? | 23:02 |
| clarkb | jeblair: \o/ | 23:02 |
| clarkb | mordred: https://jenkins.openstack.org/job/devstack-update-vm-image-hpcloud-az3/174/console image built | 23:03 |
| openstackgerrit | James E. Blair proposed a change to openstack-infra/config: Set periodic trigger to midnight UTC https://review.openstack.org/39832 | 23:04 |
| clarkb | mordred: https://jenkins.openstack.org/job/gate-tempest-devstack-vm-full/38447/console does that look like a broken image to you? | 23:04 |
| Alex_Gaynor | clarkb: a new release has been issued which resolves it | 23:04 |
| jeblair | clarkb: one more speedy aprv for today please, and then i'm going to defer non-fixup changes till tomorrow: https://review.openstack.org/39832 | 23:04 |
| Alex_Gaynor | clarkb: purging 0.7.1 from any chcaces should be sufficient | 23:05 |
| clarkb | jeblair: ko | 23:05 |
| *** avtar has quit IRC | 23:05 | |
| clarkb | Alex_Gaynor: good to know I will rerun the mirror updates now to get new xattr then I can kill new cffi | 23:05 |
| clarkb | after I approve jeblair's thing | 23:05 |
| clarkb | jeblair when you have a moment can you take a look at that tempest job | 23:05 |
| jeblair | clarkb: http://logs.openstack.org/periodic/periodic-test/ seems to work btw | 23:06 |
| jeblair | clarkb: looking | 23:06 |
| clarkb | jeblair: can we shift the periodic jobs to alter in the day? | 23:06 |
| clarkb | jeblair: our quiet time is from about 11pm to 2 or 3am PST | 23:06 |
| clarkb | maybe run it at 6UTC? | 23:06 |
| jeblair | clarkb: sure, let's do in the change where we actually add the jobs though | 23:07 |
| clarkb | ok | 23:07 |
| jeblair | clarkb: i mostly want to just stop running the zuul tests every 5 mins now. | 23:07 |
| clarkb | approved | 23:07 |
| *** emagana has joined #openstack-infra | 23:08 | |
| *** bnemec has quit IRC | 23:08 | |
| clarkb | mirror jobs running | 23:08 |
| jeblair | clarkb: have any other jobs run on az3? | 23:08 |
| *** bnemec has joined #openstack-infra | 23:08 | |
| openstackgerrit | Dan Bode proposed a change to openstack-infra/config: Add puppet-pip https://review.openstack.org/39833 | 23:08 |
| clarkb | jeblair: I am not sure | 23:09 |
| jeblair | 2013-08-01 22:39:29.076 | 2013-08-01 22:39:29 error: invalid command 'egg_info' | 23:09 |
| jeblair | clarkb: i mean, that looks fishy. | 23:09 |
| clarkb | https://jenkins.openstack.org/job/gate-tempest-devstack-vm-full/38457/ | 23:09 |
| bodepd | let me know if you guys have questions about that commit | 23:09 |
| openstackgerrit | A change was merged to openstack-infra/config: Set periodic trigger to midnight UTC https://review.openstack.org/39832 | 23:10 |
| clarkb | ^ ignore that one | 23:10 |
| uvirtbot | clarkb: Error: "ignore" is not a valid command. | 23:10 |
| bodepd | I don't have permission to release things to the forge under openstackci | 23:10 |
| bodepd | so I will need a little help to get this done | 23:10 |
| clarkb | jeblair: https://jenkins.openstack.org/job/gate-tempest-devstack-vm-full/38454/ however that may be on an old image | 23:10 |
| *** dims_ has joined #openstack-infra | 23:11 | |
| *** bnemec has quit IRC | 23:12 | |
| clarkb | cffi has released a newer version... | 23:12 |
| clarkb | Alex_Gaynor: new version appears to fix the bug | 23:14 |
| jeblair | clarkb: when was the new image in place? | 23:14 |
| * clarkb goes back to sorting out az3 | 23:14 | |
| clarkb | jeblair: ~2155 | 23:14 |
| *** bnemec has joined #openstack-infra | 23:14 | |
| jeblair | clarkb: we should have d-g log which image a machine was build from | 23:14 |
| jeblair | clarkb: https://jenkins.openstack.org/job/gate-tempest-devstack-vm-full/38454/consoleFull | 23:17 |
| jeblair | clarkb: that was launched at 22:20:16 | 23:17 |
| jeblair | clarkb: and seems to be working | 23:17 |
| clarkb | ok | 23:17 |
| clarkb | I see it in syslog | 23:18 |
| clarkb | and the timestamps indicate we are good | 23:18 |
| clarkb | I am checking the node with the failure now | 23:18 |
| jeblair | bodepd: i think we will be happy to help when we are not slammed :) | 23:18 |
| bodepd | jeblair: no worries | 23:18 |
| bodepd | jeblair: let me know | 23:18 |
| clarkb | 177734 was also launched after the new image so potentially related | 23:19 |
| jeblair | clarkb: oh, the failure you pointed out is on stable/grizzly | 23:19 |
| clarkb | jeblair: the id numbers would indicate that hosts created after the new image by and large work | 23:20 |
| jeblair | clarkb: perhaps the new image doesn't play well with that branch? | 23:20 |
| clarkb | jeblair: oh interesting. | 23:20 |
| clarkb | that is possible | 23:20 |
| jeblair | clarkb: https://review.openstack.org/#/c/27956/ | 23:21 |
| *** bnemec has quit IRC | 23:22 | |
| jeblair | clarkb: there's a 1:1 correlation with az3 nodes and that failure for that change, and the converse for !az3 and not having that failure | 23:22 |
| *** bnemec has joined #openstack-infra | 23:22 | |
| clarkb | yup | 23:22 |
| jeblair | clarkb: i think it breaks stable/grizzly, and we should revert the change to puppet | 23:22 |
| clarkb | https://review.openstack.org/#/c/39511/ may give us more info | 23:23 |
| clarkb | jeblair: ok I will whip that up. I think we should approach this by refactoring the d-g base out of the same base of the rest of our servers which is what you were talking about this morning | 23:23 |
| *** changbl has joined #openstack-infra | 23:24 | |
| jeblair | clarkb: yeah, we should let devstack handle this kind of thing and keep the bare slave more bare. :) | 23:24 |
| openstackgerrit | Clark Boylan proposed a change to openstack-infra/config: Revert "Ensure latest setuptools after installing pip." https://review.openstack.org/39837 | 23:25 |
| jeblair | clarkb: i think we can remove this pip stuff from our base class, and put it just where it's needed for our servers for now (jeepyb and zuul) | 23:25 |
| jeblair | clarkb: then put it in the jenkins slave manifest for bare==false | 23:26 |
| jeblair | so the unit test slaves get it | 23:26 |
| jeblair | then count on devstack to do this for d-g runs | 23:26 |
| jeblair | clarkb: i also rechecked 39511 (even tho it's in the gate queue) because i expect that to finish before the gate jobs. | 23:27 |
| clarkb | jeblair: sounds good | 23:28 |
| clarkb | jeblair: we can do a quick grep for provider => pip and add it there | 23:28 |
| jeblair | it has 3/5 runs in az3 | 23:28 |
| *** pabelanger has joined #openstack-infra | 23:30 | |
| jeblair | clarkb: all the az3 runs have failed with that error, non az3 are continuing | 23:30 |
| ianw | dtroyer: i'm going to investigate making a dummy rpm dependency, that's about the best thing i can think of? | 23:30 |
| jeblair | clarkb: i approved the change; you want to kick off a new image build when it merges? | 23:31 |
| *** toddmorey has quit IRC | 23:31 | |
| openstackgerrit | A change was merged to openstack-infra/config: Revert "Ensure latest setuptools after installing pip." https://review.openstack.org/39837 | 23:32 |
| jeblair | clarkb: ^ | 23:32 |
| jeblair | #status alert testing for stable/grizzly branch is broken on some devstack-gate nodes, fix in progress | 23:33 |
| jeblair | meh | 23:33 |
| dtroyer | ianw: I just looked at my really-really-bare centos image and removing python-setuptools even took cloud-ini with it. Ugh … | 23:35 |
| *** bnemec has quit IRC | 23:35 | |
| jeblair | clarkb: i kicked off an image build for az3 | 23:35 |
| *** bnemec has joined #openstack-infra | 23:36 | |
| clarkb | jeblair: thank you, I got distracted | 23:37 |
| dstufft | dtroyer: lol | 23:37 |
| dstufft | dtroyer: I promise this is going to get better, atleast on the python side :[ | 23:38 |
| clarkb | would a single giant venv manage to do what the ditros need as far as packages goes? | 23:38 |
| clarkb | everything is known to work with the system C libs, all the packages installed together work, but you avoid the global root? | 23:39 |
| *** pcrews has quit IRC | 23:39 | |
| dstufft | won't tell you if versions clash with system packages, but IMO that's the distro's problem to solve | 23:40 |
| ianw | clarkb: how do you get all the daemons to run in the virtualenv? | 23:40 |
| dstufft | ianw: /path/to/venv/bin/daemon-name | 23:40 |
| clarkb | ianw: at the beginning of devstack's run it creates a virtualenv and source the activate script | 23:40 |
| dstufft | or that | 23:41 |
| dstufft | all you need to do is use the things installed in the venv's bin dir | 23:41 |
| dstufft | which activating just messes with $PATH | 23:42 |
| *** pcrews has joined #openstack-infra | 23:42 | |
| dtroyer | the downside is that creates an even larger departure from our testing to the packaged environment | 23:43 |
| ianw | i guess i mean do things like init scripts respect that | 23:43 |
| dstufft | looks like that's basically what a software collection is lol | 23:43 |
| dtroyer | can we do a hybrid and install pip 1.4 and setuptools to a non-system location and still use it to install packages as usual? | 23:43 |
| dstufft | uhh | 23:43 |
| dstufft | maybe | 23:44 |
| dstufft | with --prefix or --root or whatever the option is | 23:44 |
| clarkb | ianw: devstack shouldn't care about that | 23:44 |
| dstufft | dtroyer: FWIW the typical answer to packaged environments in python world is "let the packagers figure it out", not the nicest answer but dealing with system level deps is painful and alot of the distros historically haven't been willing to make using pip less painful | 23:45 |
| *** jrex_laptop has joined #openstack-infra | 23:46 | |
| ianw | clarkb: but does it? a separate virtualenv certainly seems right, but i just wonder how many issues there will be around things that don't respect the virtualenv, or need help figuring it out | 23:47 |
| dtroyer | dstufft: right…and when they diverge it makes our life even better. whether or not it is the default, rhel letting anything install over packaged files out of the box is broken. I'm getting tired of working around that brokenness. and that's hard to admit because I've been a RH homie since the red disk beta | 23:47 |
| clarkb | ianw: devstack is for development, it doesn't care about init scripts for the services it is installing from source and so on | 23:48 |
| clarkb | and as long as everything is started through devstack and it has activated the virtualenv all children should pick up that environment too | 23:50 |
| dstufft | dtroyer: yea | 23:50 |
| dstufft | dtroyer: I'm going to try and get fedora-python to adopt a solution like Debian does | 23:51 |
| dstufft | I'm not particularly hopeful | 23:51 |
| dstufft | Becuase arguing on distutils-sig isn't enough for me! I need new and exciting places to argue | 23:51 |
| clarkb | dstufft: does rhel do the same thing with gems? | 23:52 |
| dtroyer | IIRC they've been over that more than once, I think it has become a sore point for them | 23:52 |
| clarkb | or is gems sufficiently different that it is a non issue for taht community | 23:52 |
| dtroyer | fwiw, perl has the distinction between system libs and local libs baked in. it seemed to work well enough | 23:53 |
| dstufft | clarkb: unsure about for ``gem``, for bundler the gems don't get installed into a sys.path like location, they get installed outside of that and then bundler adds them to the sys.path | 23:53 |
| dstufft | dtroyer: that's essentially what the debian patch does | 23:53 |
| ianw | clarkb: "everything started via devstack it works" <- i think that's a departure from status quo. maybe an acceptable one, however | 23:53 |
| clarkb | ianw: that is how the gate works | 23:54 |
| *** bnemec has quit IRC | 23:54 | |
| clarkb | then in theory the rhel packagers make native packages for you when each release comes out | 23:54 |
| clarkb | if you are doing continuous deployment type things then the disk image builder do your own distro type thing is probably prudent | 23:55 |
| *** bnemec has joined #openstack-infra | 23:55 | |
| dstufft | you can make init scripts start things inside of a venv pretty easily if you're doing the CD thing and want to use pip or a virtualenv | 23:55 |
| *** morganfainberg has joined #openstack-infra | 23:59 | |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!
