| *** tqtran is now known as tqttran_afk | 00:00 | |
| *** yolandamobile has joined #openstack-infra | 00:00 | |
| *** puranamr has quit IRC | 00:01 | |
| *** yolandamobile has quit IRC | 00:01 | |
| *** hdd has quit IRC | 00:01 | |
| *** olaph has quit IRC | 00:01 | |
| *** olaph has joined #openstack-infra | 00:02 | |
| *** geoffarnold has joined #openstack-infra | 00:03 | |
| openstackgerrit | Jeremy Stanley proposed openstack-infra/project-config: Override constraints job to use proposal worker https://review.openstack.org/195360 | 00:04 |
|---|---|---|
| fungi | lifeless: mordred: pleia2: and that ^ too | 00:04 |
| *** yolandamobile has joined #openstack-infra | 00:04 | |
| *** geoffarn_ has joined #openstack-infra | 00:04 | |
| fungi | 2015-06-24 06:00:02,909 DEBUG zuul.Gearman: Function build:propose-requirements-constraints-master:bare-trusty is not registered | 00:04 |
| fungi | (from the zuul debug log) | 00:04 |
| *** claudiub has quit IRC | 00:05 | |
| *** salv-orlando has joined #openstack-infra | 00:05 | |
| openstackgerrit | Christopher Aedo proposed openstack-infra/puppet-apps_site: Enable CORS for app-catalog https://review.openstack.org/194875 | 00:06 |
| *** yolandamobile has quit IRC | 00:06 | |
| *** david-ly_ has joined #openstack-infra | 00:06 | |
| *** geoffarnold has quit IRC | 00:07 | |
| *** david-lyle has quit IRC | 00:08 | |
| *** yolandamobile has joined #openstack-infra | 00:09 | |
| *** geoffarn_ has quit IRC | 00:09 | |
| *** puranamr has joined #openstack-infra | 00:10 | |
| *** yolandamobile has quit IRC | 00:10 | |
| *** fallenpegasus has joined #openstack-infra | 00:11 | |
| *** wenlock has joined #openstack-infra | 00:11 | |
| *** salv-orlando has quit IRC | 00:12 | |
| *** Longgeek has joined #openstack-infra | 00:12 | |
| jeblair | pleia2, docaedo: pabelanger said he was going to try to look into the centos6 timeouts tomorrow | 00:12 |
| *** yolandamobile has joined #openstack-infra | 00:12 | |
| jeblair | until then, basically "if the centos6 job failed and it took ~30 minutes to do so, recheck" is the word | 00:13 |
| *** dims_ has joined #openstack-infra | 00:14 | |
| *** yolandamobile has quit IRC | 00:14 | |
| *** wenlock has quit IRC | 00:15 | |
| *** esker has quit IRC | 00:15 | |
| *** dims has quit IRC | 00:16 | |
| docaedo | jeblair: thanks! | 00:16 |
| *** yolandamobile has joined #openstack-infra | 00:17 | |
| *** AzherKhan has joined #openstack-infra | 00:17 | |
| *** linkedinyou has joined #openstack-infra | 00:17 | |
| *** gyee_ has quit IRC | 00:17 | |
| *** weshay has quit IRC | 00:20 | |
| *** salv-orlando has joined #openstack-infra | 00:21 | |
| *** puranamr has quit IRC | 00:22 | |
| *** yolandamobile has quit IRC | 00:22 | |
| *** Longgeek has quit IRC | 00:23 | |
| jeblair | mordred: can you review https://review.openstack.org/194119 please | 00:25 |
| *** esker has joined #openstack-infra | 00:25 | |
| *** yolandamobile has joined #openstack-infra | 00:25 | |
| *** salv-orlando has quit IRC | 00:27 | |
| *** yolandamobile has quit IRC | 00:27 | |
| *** marun has quit IRC | 00:27 | |
| openstackgerrit | Merged openstack-infra/infra-specs: Apps site spec is implemented https://review.openstack.org/192289 | 00:27 |
| *** shaleh has quit IRC | 00:28 | |
| *** yamamoto has joined #openstack-infra | 00:29 | |
| *** yolandamobile has joined #openstack-infra | 00:30 | |
| bdemers | Is there a wiki that shows how to add an “openstack-packages” repo ? | 00:30 |
| *** rossella_s has quit IRC | 00:30 | |
| *** puranamr has joined #openstack-infra | 00:31 | |
| *** rossella_s has joined #openstack-infra | 00:31 | |
| *** zz_dimtruck is now known as dimtruck | 00:31 | |
| *** dboik has quit IRC | 00:32 | |
| *** weshay has joined #openstack-infra | 00:33 | |
| *** yamamoto has quit IRC | 00:34 | |
| *** otter768 has quit IRC | 00:35 | |
| *** yolandamobile has quit IRC | 00:35 | |
| *** yolandamobile has joined #openstack-infra | 00:36 | |
| *** dprince has joined #openstack-infra | 00:37 | |
| *** yolandamobile has quit IRC | 00:37 | |
| *** puranamr has quit IRC | 00:37 | |
| *** rwsu has quit IRC | 00:38 | |
| *** puranamr has joined #openstack-infra | 00:39 | |
| *** yolandamobile has joined #openstack-infra | 00:40 | |
| *** geoffarnold has joined #openstack-infra | 00:41 | |
| *** yolandamobile has quit IRC | 00:41 | |
| *** larainema has joined #openstack-infra | 00:42 | |
| *** puranamr has quit IRC | 00:42 | |
| *** lhcheng has quit IRC | 00:42 | |
| *** boris-42 has quit IRC | 00:42 | |
| *** ddieterly has joined #openstack-infra | 00:43 | |
| *** husanu has quit IRC | 00:43 | |
| *** husanu has joined #openstack-infra | 00:45 | |
| *** yolandamobile has joined #openstack-infra | 00:45 | |
| *** mriedem has quit IRC | 00:45 | |
| *** yolandamobile has quit IRC | 00:47 | |
| *** r-daneel has quit IRC | 00:48 | |
| *** yolandamobile has joined #openstack-infra | 00:49 | |
| mordred | jeblair: looking | 00:49 |
| lifeless | mordred: also see the three for fungi higher up :) | 00:51 |
| *** yolandamobile has quit IRC | 00:51 | |
| davideagnello | Hello, this review https://review.openstack.org/#/c/194768/ needs one more review | 00:52 |
| Clint | ew, function exports | 00:53 |
| *** yolandamobile has joined #openstack-infra | 00:55 | |
| *** sigmavirus24 is now known as sigmavirus24_awa | 00:58 | |
| *** yolandamobile has quit IRC | 00:58 | |
| *** geoffarnold has quit IRC | 01:00 | |
| *** yolandamobile has joined #openstack-infra | 01:01 | |
| *** amotoki has joined #openstack-infra | 01:01 | |
| * mordred is actally not looking at anything -nobody hold their breath | 01:03 | |
| *** yolandamobile has quit IRC | 01:03 | |
| * Clint exhales. | 01:03 | |
| *** yolandamobile has joined #openstack-infra | 01:04 | |
| *** Rockyg has joined #openstack-infra | 01:05 | |
| *** tjones1 has joined #openstack-infra | 01:05 | |
| *** amotoki has quit IRC | 01:05 | |
| *** yolandamobile has quit IRC | 01:05 | |
| *** AzherKhan has quit IRC | 01:06 | |
| *** AzherKhan has joined #openstack-infra | 01:08 | |
| *** yolandamobile has joined #openstack-infra | 01:09 | |
| *** olaph has quit IRC | 01:10 | |
| *** yolandamobile has quit IRC | 01:10 | |
| *** olaph has joined #openstack-infra | 01:11 | |
| *** enikanorov2_ has quit IRC | 01:12 | |
| *** yolandamobile has joined #openstack-infra | 01:13 | |
| *** baoli has quit IRC | 01:13 | |
| jhesketh | hmm, anybody with puppet knowledge able to help clarify http://git.openstack.org/cgit/openstack-infra/puppet-jenkins/tree/manifests/jenkinsuser.pp#n57 for me? | 01:14 |
| *** yolandamobile has quit IRC | 01:14 | |
| jhesketh | I don't understand the second ensure absent part | 01:15 |
| *** DericHorn-HP has joined #openstack-infra | 01:16 | |
| *** yolandamobile has joined #openstack-infra | 01:17 | |
| jhesketh | does puppet just resolve that to say "delete any keys other than jenkins-master-2014-04-24"? | 01:18 |
| openstackgerrit | Merged openstack-infra/project-config: Do not set ZUUL_REF to a valid value on devstack checkout https://review.openstack.org/195185 | 01:20 |
| *** yolandamobile has quit IRC | 01:20 | |
| *** jasondotstar has joined #openstack-infra | 01:21 | |
| *** dprince has quit IRC | 01:22 | |
| *** puranamr has joined #openstack-infra | 01:23 | |
| *** whoops has quit IRC | 01:23 | |
| *** sputnik13 has quit IRC | 01:24 | |
| *** yolandamobile has joined #openstack-infra | 01:25 | |
| *** laughterwym_ has joined #openstack-infra | 01:25 | |
| *** yolandamobile has quit IRC | 01:25 | |
| *** olaph has quit IRC | 01:26 | |
| *** olaph has joined #openstack-infra | 01:28 | |
| *** sdake has joined #openstack-infra | 01:28 | |
| *** laughterwym has quit IRC | 01:29 | |
| *** abitha has quit IRC | 01:29 | |
| *** fallenpe_ has joined #openstack-infra | 01:29 | |
| *** yolandamobile has joined #openstack-infra | 01:29 | |
| *** sdake_ has joined #openstack-infra | 01:29 | |
| fungi | jhesketh: i think it means that at one point we inadvertently created an authorized key for the jenkins user with a key-id of "/home/jenkins/.ssh/authorized_keys" | 01:31 |
| *** yolandamobile has quit IRC | 01:31 | |
| fungi | and that's lingering cleanup | 01:31 |
| *** DericHorn-HP has quit IRC | 01:31 | |
| fungi | though i would need to go digging into the source for the ssh_authorized_key resource to see if passing a file path in place of a key-id has some magic meaning | 01:32 |
| *** fallenpegasus has quit IRC | 01:32 | |
| *** yolandamobile has joined #openstack-infra | 01:33 | |
| *** sdake has quit IRC | 01:33 | |
| jhesketh | fungi: ah I see, thanks | 01:33 |
| *** david-ly_ is now known as david-lyle | 01:36 | |
| *** yolandamobile has quit IRC | 01:36 | |
| *** EricGonczer_ has joined #openstack-infra | 01:36 | |
| *** yolandamobile has joined #openstack-infra | 01:37 | |
| *** ramashri has quit IRC | 01:38 | |
| *** tqttran_afk has quit IRC | 01:38 | |
| *** AzherKhan has quit IRC | 01:38 | |
| *** rlandy has joined #openstack-infra | 01:40 | |
| *** yolandamobile has quit IRC | 01:40 | |
| *** spzala has joined #openstack-infra | 01:42 | |
| *** yolandamobile has joined #openstack-infra | 01:43 | |
| *** rfolco has quit IRC | 01:45 | |
| *** yolandamobile has quit IRC | 01:45 | |
| *** dims_ has quit IRC | 01:45 | |
| *** ivar-lazzaro has joined #openstack-infra | 01:46 | |
| *** ivar-lazzaro has quit IRC | 01:47 | |
| *** ivar-lazzaro has joined #openstack-infra | 01:48 | |
| *** yolandamobile has joined #openstack-infra | 01:49 | |
| *** dims has joined #openstack-infra | 01:50 | |
| *** tjones1 has quit IRC | 01:52 | |
| *** dimtruck is now known as zz_dimtruck | 01:54 | |
| ayoung | mordred, with shade, do you have any way of knowing that cloud-init has finished running on a new VM | 01:55 |
| *** dontalton has quit IRC | 01:56 | |
| mordred | ayoung: I do not run cloud-init | 01:56 |
| mordred | ayoung: but no - there is nothing about shade currently that facilitates anything like that | 01:57 |
| ayoung | mordred, crud, | 01:57 |
| ayoung | mordred, I was hoping my base VM could at least have updated packages on it | 01:57 |
| ayoung | IOK..thanks anyway | 01:57 |
| mordred | ayoung: oh - yeah - so, we build our own base images with diskimage-builder | 01:58 |
| *** tjones1 has joined #openstack-infra | 01:58 | |
| mordred | ayoung: so in my world view, step one is "build base image" step two is "boot it", step three is "run config management" | 01:58 |
| ayoung | mordred, yeah. I still have a delay between booting a VM and seeing the static IP on it. | 01:59 |
| ayoung | I'd rather not busy wait, but I kindof need that info for later in the pipeline | 02:00 |
| mordred | ayoung: so - there is a useful thing in ansible - which is that you can wait on port 22 being ready | 02:00 |
| mordred | ayoung: and yeah - openstack itself does not provide any capabilities in this area | 02:00 |
| ayoung | ok...that makes sense...I guess I need to do that in ansible itself, then | 02:00 |
| mordred | yah | 02:00 |
| mordred | ayoung: http://docs.ansible.com/wait_for_module.html | 02:01 |
| ayoung | I really want your updated Ansible stuff | 02:01 |
| mordred | me too | 02:02 |
| *** asettle is now known as asettle-afk | 02:02 | |
| mordred | ayoung: after it's all landed, my next step is rewriting long-lived node creation for infra using it | 02:03 |
| mordred | ayoung: so you and I are working on a very similar problem :) | 02:03 |
| *** dims has quit IRC | 02:04 | |
| *** bhunter71 has joined #openstack-infra | 02:07 | |
| *** ddieterly has quit IRC | 02:07 | |
| *** puranamr has quit IRC | 02:09 | |
| *** yolandamobile has quit IRC | 02:09 | |
| *** ddieterly has joined #openstack-infra | 02:09 | |
| *** earlephilhower has joined #openstack-infra | 02:10 | |
| *** dims has joined #openstack-infra | 02:10 | |
| ayoung | mordred, I think I also need to do something like create a vm, then query the floating IP assigned to it, and then add that to the inventory | 02:10 |
| *** akerr has joined #openstack-infra | 02:10 | |
| patrickeast | hi everyone, a few of us running 3rd party ci’s for cinder have noticed nodepool failing to build images today during prepare_tempest_testerpo.py, the error looks like http://paste.openstack.org/show/320645/ | 02:10 |
| *** yolandamobile has joined #openstack-infra | 02:11 | |
| *** DericHorn-HP has joined #openstack-infra | 02:11 | |
| mordred | ayoung: yes. so ... shade will give you the IP, as will os_server | 02:11 |
| mordred | ayoung: then you want add_host in your playbook | 02:12 |
| *** DericHorn-HP has quit IRC | 02:12 | |
| patrickeast | is this a symptom of needing to update oslo libs on the nodepool machine building dib? or the actual element scripts need to be updated? | 02:12 |
| mordred | so that you can use it ... I'm afk, but I've got a playbook with the pattern I'll send | 02:12 |
| *** sdake has joined #openstack-infra | 02:13 | |
| *** yamamoto has joined #openstack-infra | 02:14 | |
| *** ddieterly has quit IRC | 02:14 | |
| *** DericHorn-HP has joined #openstack-infra | 02:14 | |
| *** linkedinyou has quit IRC | 02:15 | |
| *** yolandamobile has quit IRC | 02:15 | |
| *** linkedinyou has joined #openstack-infra | 02:16 | |
| *** linkedinyou has quit IRC | 02:16 | |
| *** sdake_ has quit IRC | 02:17 | |
| *** unicell has quit IRC | 02:17 | |
| *** DericHorn-HP has quit IRC | 02:17 | |
| *** yolandamobile has joined #openstack-infra | 02:17 | |
| *** rm_work is now known as rm_work|away | 02:18 | |
| *** yolandamobile has quit IRC | 02:18 | |
| *** puranamr has joined #openstack-infra | 02:18 | |
| *** yolandamobile has joined #openstack-infra | 02:19 | |
| *** yolandamobile has quit IRC | 02:21 | |
| *** Rockyg has quit IRC | 02:22 | |
| *** patrickeast has quit IRC | 02:22 | |
| *** yolandamobile has joined #openstack-infra | 02:25 | |
| *** akerr has quit IRC | 02:25 | |
| *** bhunter71 has quit IRC | 02:25 | |
| *** yolandamobile has quit IRC | 02:27 | |
| *** spzala has quit IRC | 02:27 | |
| *** puranamr has quit IRC | 02:27 | |
| *** yamahata has quit IRC | 02:28 | |
| *** tjones1 has quit IRC | 02:28 | |
| *** puranamr has joined #openstack-infra | 02:28 | |
| *** salv-orl_ has joined #openstack-infra | 02:29 | |
| *** yolandamobile has joined #openstack-infra | 02:30 | |
| *** armax has joined #openstack-infra | 02:32 | |
| *** yolandamobile has quit IRC | 02:32 | |
| *** fallenpe_ has quit IRC | 02:32 | |
| *** salv-orl_ has quit IRC | 02:34 | |
| *** yolandamobile has joined #openstack-infra | 02:34 | |
| *** laughterwym__ has joined #openstack-infra | 02:35 | |
| *** yolandamobile has quit IRC | 02:35 | |
| *** laughterwym_ has quit IRC | 02:36 | |
| *** otter768 has joined #openstack-infra | 02:36 | |
| *** yolandamobile has joined #openstack-infra | 02:37 | |
| *** otter768 has quit IRC | 02:41 | |
| *** yolandamobile has quit IRC | 02:41 | |
| *** yolandamobile has joined #openstack-infra | 02:42 | |
| *** mestery has joined #openstack-infra | 02:44 | |
| *** yolandamobile has quit IRC | 02:44 | |
| *** rlandy has quit IRC | 02:44 | |
| *** yolandamobile has joined #openstack-infra | 02:46 | |
| openstackgerrit | Steve Kowalik proposed openstack-infra/puppet-zanata: Update Zanata client to 3.7.1 https://review.openstack.org/195390 | 02:48 |
| *** yolandamobile has quit IRC | 02:48 | |
| *** AzherKhan has joined #openstack-infra | 02:48 | |
| *** zz_dimtruck is now known as dimtruck | 02:50 | |
| *** yolandamobile has joined #openstack-infra | 02:50 | |
| *** ddieterly has joined #openstack-infra | 02:52 | |
| *** yolandamobile has quit IRC | 02:52 | |
| *** hdd has joined #openstack-infra | 02:52 | |
| *** yolandamobile has joined #openstack-infra | 02:56 | |
| *** mestery has quit IRC | 02:57 | |
| *** yolandamobile has quit IRC | 02:57 | |
| *** mestery has joined #openstack-infra | 02:57 | |
| *** dims has quit IRC | 02:58 | |
| *** EricGonczer_ has quit IRC | 03:00 | |
| *** yolandamobile has joined #openstack-infra | 03:01 | |
| dougwig | lifeless: fyi, the requirements job is spewing a bunch of these: "2015-06-25 02:54:43.740 | Ignoring unparseable requirement in non-strict mode: python-ldap>=2.4;python_version=='2.7'" | 03:01 |
| *** harlowja_at_home has joined #openstack-infra | 03:01 | |
| *** Sukhdev has joined #openstack-infra | 03:01 | |
| *** cindyo has joined #openstack-infra | 03:02 | |
| *** amotoki has joined #openstack-infra | 03:02 | |
| *** stevemar has joined #openstack-infra | 03:03 | |
| *** enikanorov2 has joined #openstack-infra | 03:03 | |
| *** yolandamobile has quit IRC | 03:03 | |
| *** yolandamobile has joined #openstack-infra | 03:04 | |
| openstackgerrit | Doug Wiegley proposed openstack/requirements: WIP - Add neutron, so that child repos can reference in requirements https://review.openstack.org/195392 | 03:05 |
| *** yolandamobile has quit IRC | 03:05 | |
| *** amotoki has quit IRC | 03:06 | |
| *** yolandamobile has joined #openstack-infra | 03:08 | |
| *** yolandamobile has quit IRC | 03:10 | |
| *** amit213 has quit IRC | 03:12 | |
| *** zul has quit IRC | 03:13 | |
| *** amit213 has joined #openstack-infra | 03:13 | |
| *** yolandamobile has joined #openstack-infra | 03:13 | |
| *** cody-somerville has quit IRC | 03:15 | |
| *** stevemar has quit IRC | 03:15 | |
| *** asettle-afk is now known as asettle | 03:15 | |
| *** yolandamobile has quit IRC | 03:16 | |
| *** zul has joined #openstack-infra | 03:17 | |
| *** yolandamobile has joined #openstack-infra | 03:18 | |
| lifeless | dougwig: yes, thats why I mailed the list | 03:19 |
| lifeless | dougwig: last night my time | 03:19 |
| dougwig | lifeless: ahh, missed it, all good then. | 03:20 |
| *** Longgeek has joined #openstack-infra | 03:20 | |
| *** Longgeek has quit IRC | 03:21 | |
| *** yolandamobile has quit IRC | 03:21 | |
| *** Longgeek has joined #openstack-infra | 03:21 | |
| *** yolandamobile has joined #openstack-infra | 03:23 | |
| *** fallenpegasus has joined #openstack-infra | 03:25 | |
| *** yolandamobile has quit IRC | 03:25 | |
| *** puranamr has quit IRC | 03:26 | |
| *** yolandamobile has joined #openstack-infra | 03:27 | |
| *** cody-somerville has joined #openstack-infra | 03:28 | |
| *** yolandamobile has quit IRC | 03:28 | |
| dougwig | isn't it 4:30am in ireland? :) | 03:29 |
| lifeless | dougwig: I don't know. | 03:29 |
| lifeless | dougwig: buts its plausible depending on their DST status | 03:29 |
| *** yolandamobile has joined #openstack-infra | 03:29 | |
| Clint | libc can tell you | 03:29 |
| *** macjack has joined #openstack-infra | 03:30 | |
| *** mtanino has quit IRC | 03:31 | |
| *** yolandamobile has quit IRC | 03:31 | |
| dougwig | i'm going insane, for some reason i thought you were there. nothing new about that, i suppose. | 03:31 |
| *** puranamr has joined #openstack-infra | 03:33 | |
| *** yolandamobile has joined #openstack-infra | 03:33 | |
| *** puranamr has quit IRC | 03:35 | |
| *** yolandamobile has quit IRC | 03:35 | |
| *** puranamr has joined #openstack-infra | 03:35 | |
| *** dannywil_ has joined #openstack-infra | 03:36 | |
| *** yolandamobile has joined #openstack-infra | 03:37 | |
| *** yolandamobile has quit IRC | 03:40 | |
| *** dannywilson has quit IRC | 03:40 | |
| *** dannywil_ has quit IRC | 03:40 | |
| *** ivar-lazzaro has quit IRC | 03:42 | |
| *** yolandamobile has joined #openstack-infra | 03:44 | |
| *** yolandamobile has quit IRC | 03:46 | |
| lifeless | dougwig: :) | 03:46 |
| lifeless | dougwig: NZ | 03:46 |
| *** ayoung is now known as ayoung-ZZZzzz__ | 03:46 | |
| dougwig | lifeless: oh, sweet. | 03:47 |
| *** puranamr_ has joined #openstack-infra | 03:47 | |
| *** puranamr has quit IRC | 03:47 | |
| *** yolandamobile has joined #openstack-infra | 03:48 | |
| *** linkedinyou has joined #openstack-infra | 03:53 | |
| *** yolandamobile has quit IRC | 03:53 | |
| *** olaph has quit IRC | 03:55 | |
| *** yolandamobile has joined #openstack-infra | 03:57 | |
| *** sarob has quit IRC | 03:57 | |
| *** dimtruck is now known as zz_dimtruck | 03:58 | |
| *** yolandamobile has quit IRC | 03:59 | |
| *** olaph has joined #openstack-infra | 04:02 | |
| *** yolandamobile has joined #openstack-infra | 04:03 | |
| *** abitha has joined #openstack-infra | 04:03 | |
| *** yolandamobile has quit IRC | 04:03 | |
| *** coolsvap|away is now known as coolsvap | 04:04 | |
| *** yolandamobile has joined #openstack-infra | 04:05 | |
| *** olaph has quit IRC | 04:06 | |
| *** yolandamobile has quit IRC | 04:08 | |
| *** yolandamobile has joined #openstack-infra | 04:09 | |
| *** ddieterly has quit IRC | 04:10 | |
| *** amotoki has joined #openstack-infra | 04:10 | |
| *** yolandamobile has quit IRC | 04:12 | |
| *** yolandamobile has joined #openstack-infra | 04:15 | |
| *** yolandamobile has quit IRC | 04:17 | |
| *** esker has quit IRC | 04:18 | |
| *** esker has joined #openstack-infra | 04:18 | |
| *** mestery_ has joined #openstack-infra | 04:19 | |
| *** mestery has quit IRC | 04:19 | |
| *** yolandamobile has joined #openstack-infra | 04:19 | |
| *** puranamr has joined #openstack-infra | 04:21 | |
| *** yolandamobile has quit IRC | 04:21 | |
| *** puranamr_ has quit IRC | 04:21 | |
| *** esker has quit IRC | 04:23 | |
| *** achanda has joined #openstack-infra | 04:23 | |
| *** yolandamobile has joined #openstack-infra | 04:23 | |
| *** yolandamobile has quit IRC | 04:26 | |
| *** fallenpegasus has quit IRC | 04:26 | |
| *** fallenpe_ has joined #openstack-infra | 04:26 | |
| *** yolandamobile has joined #openstack-infra | 04:27 | |
| *** DericHorn-HP has joined #openstack-infra | 04:28 | |
| *** puranamr_ has joined #openstack-infra | 04:29 | |
| *** puranamr has quit IRC | 04:29 | |
| *** laughterwym__ has quit IRC | 04:30 | |
| *** laughterwym has joined #openstack-infra | 04:30 | |
| *** laughterwym_ has joined #openstack-infra | 04:30 | |
| *** laughterwym_ has quit IRC | 04:31 | |
| *** fallenpe_ has quit IRC | 04:31 | |
| *** yamada-h has joined #openstack-infra | 04:31 | |
| *** ajmiller has quit IRC | 04:32 | |
| *** cindyo has quit IRC | 04:33 | |
| *** puranamr_ has quit IRC | 04:34 | |
| *** puranamr has joined #openstack-infra | 04:34 | |
| *** otter768 has joined #openstack-infra | 04:37 | |
| *** stevemar2 has joined #openstack-infra | 04:37 | |
| *** Longgeek has quit IRC | 04:37 | |
| *** Longgeek has joined #openstack-infra | 04:37 | |
| *** unicell has joined #openstack-infra | 04:40 | |
| *** salv-orlando has joined #openstack-infra | 04:41 | |
| *** otter768 has quit IRC | 04:42 | |
| *** sarob has joined #openstack-infra | 04:44 | |
| *** Longgeek has quit IRC | 04:44 | |
| *** sarob has quit IRC | 04:47 | |
| *** sarob has joined #openstack-infra | 04:47 | |
| *** salv-orlando has quit IRC | 04:47 | |
| *** SlickN1k has quit IRC | 04:50 | |
| *** puranamr has quit IRC | 04:50 | |
| *** puranamr has joined #openstack-infra | 04:51 | |
| *** tjones1 has joined #openstack-infra | 04:53 | |
| *** yolandamobile has quit IRC | 04:53 | |
| *** yolandamobile has joined #openstack-infra | 04:56 | |
| *** sarob has quit IRC | 04:57 | |
| *** yolandamobile has quit IRC | 04:57 | |
| *** sarob has joined #openstack-infra | 04:57 | |
| *** yolandamobile has joined #openstack-infra | 04:59 | |
| *** Sukhdev has quit IRC | 05:01 | |
| *** yolandamobile has quit IRC | 05:01 | |
| *** yolandamobile has joined #openstack-infra | 05:03 | |
| *** yolandamobile has quit IRC | 05:05 | |
| *** achanda has quit IRC | 05:06 | |
| *** yolandamobile has joined #openstack-infra | 05:07 | |
| *** dannywilson has joined #openstack-infra | 05:07 | |
| *** mestery_ has quit IRC | 05:09 | |
| *** yolandamobile has quit IRC | 05:09 | |
| *** puranamr_ has joined #openstack-infra | 05:10 | |
| *** ddieterly has joined #openstack-infra | 05:10 | |
| *** tjones1 has quit IRC | 05:12 | |
| *** BharatK has joined #openstack-infra | 05:12 | |
| *** puranamr has quit IRC | 05:13 | |
| *** rm_work|away is now known as rm_work | 05:13 | |
| *** sarob has quit IRC | 05:13 | |
| *** yolandamobile has joined #openstack-infra | 05:14 | |
| *** sarob has joined #openstack-infra | 05:14 | |
| *** sarob has quit IRC | 05:15 | |
| *** yolandamobile has quit IRC | 05:15 | |
| *** ghostpl has joined #openstack-infra | 05:15 | |
| *** ddieterly has quit IRC | 05:15 | |
| *** Longgeek has joined #openstack-infra | 05:17 | |
| *** puranamr_ has quit IRC | 05:18 | |
| openstackgerrit | David Shrewsbury proposed openstack-infra/shade: Update keypair APIs to latest standards https://review.openstack.org/195318 | 05:18 |
| *** yolandamobile has joined #openstack-infra | 05:19 | |
| *** esker has joined #openstack-infra | 05:19 | |
| openstackgerrit | David Shrewsbury proposed openstack-infra/shade: Update keypair APIs to latest standards https://review.openstack.org/195318 | 05:20 |
| *** yolandamobile has quit IRC | 05:20 | |
| *** yolandamobile has joined #openstack-infra | 05:22 | |
| *** ghostpl has quit IRC | 05:23 | |
| *** unicell1 has joined #openstack-infra | 05:24 | |
| *** yolandamobile has quit IRC | 05:24 | |
| *** ghostpl has joined #openstack-infra | 05:24 | |
| Shrews | mordred: that ^^^ should be what we need so i can fix up the keypair module | 05:24 |
| *** unicell has quit IRC | 05:25 | |
| *** yolandamobile has joined #openstack-infra | 05:25 | |
| *** ildikov has quit IRC | 05:26 | |
| *** hdd has quit IRC | 05:29 | |
| *** yolandamobile has quit IRC | 05:29 | |
| *** dannywilson has quit IRC | 05:30 | |
| *** dannywilson has joined #openstack-infra | 05:31 | |
| *** puranamr has joined #openstack-infra | 05:31 | |
| *** puranamr has quit IRC | 05:31 | |
| *** yolandamobile has joined #openstack-infra | 05:32 | |
| *** esker has quit IRC | 05:32 | |
| *** yolandamobile has quit IRC | 05:32 | |
| *** puranamr has joined #openstack-infra | 05:33 | |
| *** yolandamobile has joined #openstack-infra | 05:33 | |
| *** flepied has quit IRC | 05:35 | |
| *** dannywilson has quit IRC | 05:35 | |
| *** puranamr has quit IRC | 05:37 | |
| *** yolandamobile has quit IRC | 05:37 | |
| *** yolandamobile has joined #openstack-infra | 05:37 | |
| *** yolandamobile has quit IRC | 05:41 | |
| *** markvoelker has quit IRC | 05:43 | |
| *** yolandamobile has joined #openstack-infra | 05:45 | |
| *** armax has quit IRC | 05:49 | |
| *** Krinkle has joined #openstack-infra | 05:51 | |
| *** asettle has quit IRC | 05:51 | |
| Krinkle | Hey. I'm running into an error with jenkins-job-builder and can't for the life of me figure out how to fix it. | 05:52 |
| Krinkle | it's giving CERTIFICATE_VERIFY_FAILED on update commands. | 05:52 |
| Krinkle | it worked earlier today. | 05:52 |
| Krinkle | Already did a clean re-install of jenkins-job-builder, openssl (openssl-1.0.2c; homebrew) and python (2.7.10; homebrew) | 05:53 |
| Krinkle | Tried using both sudo python setup.py install and with pip. Same same. | 05:53 |
| Krinkle | full stack: https://gist.githubusercontent.com/Krinkle/45490f91459086e5378d/raw | 05:53 |
| Krinkle | The server in question has a valid certificate as far as I'm aware. – https://integration.wikimedia.org/ci/ | 05:55 |
| *** kushal has joined #openstack-infra | 05:58 | |
| *** mrunge has joined #openstack-infra | 05:58 | |
| openstackgerrit | Steve Kowalik proposed openstack-infra/project-config: Add a script to populate a Zanata project https://review.openstack.org/195421 | 05:58 |
| openstackgerrit | Steve Kowalik proposed openstack-infra/project-config: Add a script to populate a Zanata project https://review.openstack.org/195421 | 05:59 |
| *** esker has joined #openstack-infra | 05:59 | |
| openstackgerrit | OpenStack Proposal Bot proposed openstack-infra/project-config: Normalize projects.yaml https://review.openstack.org/195423 | 06:01 |
| *** laughterwym_ has joined #openstack-infra | 06:02 | |
| *** DericHorn-HP has quit IRC | 06:04 | |
| *** samueldmq has quit IRC | 06:04 | |
| *** ericksonsantos has quit IRC | 06:05 | |
| *** larainema has quit IRC | 06:05 | |
| *** laughterwym has quit IRC | 06:05 | |
| *** BharatK has quit IRC | 06:07 | |
| *** olaph has joined #openstack-infra | 06:07 | |
| *** ghostpl has quit IRC | 06:10 | |
| *** ddieterly has joined #openstack-infra | 06:11 | |
| *** yolandamobile has quit IRC | 06:11 | |
| *** abregman has joined #openstack-infra | 06:11 | |
| *** kushal has quit IRC | 06:12 | |
| *** ildikov has joined #openstack-infra | 06:13 | |
| *** yolandamobile has joined #openstack-infra | 06:14 | |
| *** ParsectiX has joined #openstack-infra | 06:15 | |
| *** yolandamobile has quit IRC | 06:15 | |
| *** ghostpl has joined #openstack-infra | 06:15 | |
| *** jtomasek has joined #openstack-infra | 06:15 | |
| *** Guest93518 has quit IRC | 06:15 | |
| *** changbl_ has quit IRC | 06:15 | |
| *** esker has quit IRC | 06:15 | |
| *** ddieterly has quit IRC | 06:16 | |
| *** yolandamobile has joined #openstack-infra | 06:16 | |
| *** marekd has joined #openstack-infra | 06:18 | |
| *** ghostpl has quit IRC | 06:18 | |
| *** ghostpl has joined #openstack-infra | 06:18 | |
| marekd | Good morning everyone. Could I ask for a pair of eyes on the change https://review.openstack.org/#/c/190631/ ? It's already got supportive +1 from Morgan. | 06:19 |
| *** BharatK has joined #openstack-infra | 06:19 | |
| *** deepakcs has joined #openstack-infra | 06:21 | |
| *** yolandamobile has quit IRC | 06:21 | |
| *** yolandamobile has joined #openstack-infra | 06:22 | |
| *** flepied has joined #openstack-infra | 06:24 | |
| *** yolandamobile has quit IRC | 06:24 | |
| AJaeger | clarkb, mordred, jhesketh, SergeyLukjanov, fungi, jeblair: Our proposal slave run out of space building the horizon-propose-translation-update. Additional jobs run fine. Could one of you check this, please? https://jenkins.openstack.org/job/horizon-propose-translation-update/50/console | 06:26 |
| *** ghostpl has quit IRC | 06:26 | |
| *** yolandamobile has joined #openstack-infra | 06:28 | |
| *** abitha has quit IRC | 06:30 | |
| *** yolandamobile has quit IRC | 06:30 | |
| *** yolandamobile has joined #openstack-infra | 06:31 | |
| *** claudiub has joined #openstack-infra | 06:33 | |
| *** yolandamobile has quit IRC | 06:36 | |
| *** katyafervent is now known as katyafervent_awa | 06:37 | |
| *** yolandamobile has joined #openstack-infra | 06:37 | |
| *** otter768 has joined #openstack-infra | 06:38 | |
| *** scheuran has joined #openstack-infra | 06:39 | |
| *** yolandamobile has quit IRC | 06:39 | |
| *** mkoderer_cloud has joined #openstack-infra | 06:39 | |
| *** esker has joined #openstack-infra | 06:41 | |
| *** jtomasek has quit IRC | 06:42 | |
| *** yolandamobile has joined #openstack-infra | 06:42 | |
| *** otter768 has quit IRC | 06:43 | |
| *** markvoelker has joined #openstack-infra | 06:43 | |
| *** fabo has quit IRC | 06:44 | |
| *** katyafervent_awa is now known as katyafervent | 06:44 | |
| *** achanda has joined #openstack-infra | 06:44 | |
| *** fabo has joined #openstack-infra | 06:44 | |
| *** nihilifer has joined #openstack-infra | 06:45 | |
| *** yolandamobile has quit IRC | 06:45 | |
| StevenK | AJaeger: WRT https://review.openstack.org/195421 , my plan is pleia2 or I or *someone* runs it for each project when we need it -- the reason it's in project-config is so we can use common_translations_update.sh and friends like create-zanata-xml.py | 06:45 |
| *** legoktm has joined #openstack-infra | 06:48 | |
| Krinkle | o/ | 06:48 |
| legoktm | \o | 06:48 |
| *** markvoelker has quit IRC | 06:49 | |
| *** yolandamobile has joined #openstack-infra | 06:49 | |
| *** jtomasek has joined #openstack-infra | 06:50 | |
| *** soren has joined #openstack-infra | 06:50 | |
| *** AJaeger has quit IRC | 06:50 | |
| *** ParsectiX has quit IRC | 06:52 | |
| *** yolandamobile has quit IRC | 06:52 | |
| *** yolandamobile has joined #openstack-infra | 06:54 | |
| *** enikanorov2 has quit IRC | 06:54 | |
| *** salv-orlando has joined #openstack-infra | 06:54 | |
| *** salv-orlando has quit IRC | 06:56 | |
| *** yolandamobile has quit IRC | 06:56 | |
| *** salv-orlando has joined #openstack-infra | 06:56 | |
| *** esker has quit IRC | 06:56 | |
| *** markus_z has joined #openstack-infra | 06:57 | |
| openstackgerrit | Merged openstack-infra/puppet-cgit: Fix in rsyslog haproxy path https://review.openstack.org/190142 | 06:57 |
| *** linkedinyou has quit IRC | 06:57 | |
| *** yolandamobile has joined #openstack-infra | 06:58 | |
| *** achanda has quit IRC | 06:58 | |
| *** _nadya_ has joined #openstack-infra | 06:58 | |
| *** sshnaidm has joined #openstack-infra | 07:00 | |
| *** ihrachyshka has joined #openstack-infra | 07:00 | |
| *** _nadya_ has quit IRC | 07:00 | |
| *** _nadya_ has joined #openstack-infra | 07:00 | |
| *** yolandamobile has quit IRC | 07:00 | |
| *** bradjones has quit IRC | 07:02 | |
| *** yolandamobile has joined #openstack-infra | 07:02 | |
| *** lennyb has quit IRC | 07:03 | |
| *** yolandamobile has quit IRC | 07:03 | |
| openstackgerrit | Merged openstack-infra/project-config: Normalize projects.yaml https://review.openstack.org/195423 | 07:03 |
| *** bradjones has joined #openstack-infra | 07:04 | |
| *** bradjones has quit IRC | 07:04 | |
| *** bradjones has joined #openstack-infra | 07:04 | |
| *** _nadya_ has quit IRC | 07:05 | |
| *** yolandamobile has joined #openstack-infra | 07:06 | |
| *** leanderthal has joined #openstack-infra | 07:06 | |
| *** root_ has joined #openstack-infra | 07:07 | |
| *** yolandamobile has quit IRC | 07:07 | |
| *** fallenpegasus has joined #openstack-infra | 07:07 | |
| *** achanda has joined #openstack-infra | 07:08 | |
| *** sarob has joined #openstack-infra | 07:08 | |
| *** root_ has quit IRC | 07:09 | |
| *** sshnaidm has quit IRC | 07:09 | |
| *** linkedinyou has joined #openstack-infra | 07:10 | |
| *** larainema has joined #openstack-infra | 07:10 | |
| *** yolandamobile has joined #openstack-infra | 07:11 | |
| *** lennyb has joined #openstack-infra | 07:12 | |
| *** sarob has quit IRC | 07:12 | |
| *** yolandamobile has quit IRC | 07:12 | |
| *** ddieterly has joined #openstack-infra | 07:12 | |
| *** stevemar2 has quit IRC | 07:13 | |
| *** k4n0 has joined #openstack-infra | 07:14 | |
| *** linkedinyou has quit IRC | 07:15 | |
| *** yolandamobile has joined #openstack-infra | 07:15 | |
| *** ParsectiX has joined #openstack-infra | 07:16 | |
| *** AJaeger has joined #openstack-infra | 07:16 | |
| *** AJaeger has joined #openstack-infra | 07:16 | |
| *** yolandamobile has quit IRC | 07:16 | |
| *** ddieterly has quit IRC | 07:17 | |
| *** yolandamobile has joined #openstack-infra | 07:20 | |
| *** shardy has joined #openstack-infra | 07:20 | |
| AJaeger | StevenK: please discuss with pleia2 what's the best place for the script. jenkins/scripts is for scripts that are run by Jenkins. I'm not sure what's the best place it - and if your proposal is the best one, I'm fine. Just not sure about it. | 07:21 |
| *** unicell has joined #openstack-infra | 07:22 | |
| *** yolandamobile has quit IRC | 07:22 | |
| *** sshnaidm has joined #openstack-infra | 07:22 | |
| *** esker has joined #openstack-infra | 07:23 | |
| *** unicell1 has quit IRC | 07:23 | |
| *** yolandamobile has joined #openstack-infra | 07:24 | |
| *** yolandamobile has quit IRC | 07:26 | |
| *** yolandamobile has joined #openstack-infra | 07:27 | |
| openstackgerrit | Merged openstack-infra/project-config: Add temporary non voting multinode smoke jobs https://review.openstack.org/195259 | 07:28 |
| *** yolandamobile has quit IRC | 07:28 | |
| openstackgerrit | Merged openstack-infra/project-config: Grant keystone core pushMerge on keystoneclient feature branches https://review.openstack.org/194801 | 07:28 |
| *** unicell has quit IRC | 07:29 | |
| *** enikanorov2 has joined #openstack-infra | 07:29 | |
| *** cdent has joined #openstack-infra | 07:29 | |
| *** camunoz is now known as camunoz_gone | 07:30 | |
| *** samuelBartel has joined #openstack-infra | 07:32 | |
| *** yolandamobile has joined #openstack-infra | 07:32 | |
| *** doude has joined #openstack-infra | 07:33 | |
| *** yolandamobile has quit IRC | 07:33 | |
| *** Hal has joined #openstack-infra | 07:33 | |
| *** Hal is now known as Guest27822 | 07:34 | |
| *** ParsectiX has quit IRC | 07:35 | |
| *** ParsectiX has joined #openstack-infra | 07:35 | |
| *** dizquierdo has joined #openstack-infra | 07:36 | |
| *** yolandamobile has joined #openstack-infra | 07:36 | |
| *** esker has quit IRC | 07:38 | |
| *** yolandamobile has quit IRC | 07:38 | |
| *** mpaolino has joined #openstack-infra | 07:38 | |
| *** arxcruz has joined #openstack-infra | 07:38 | |
| *** mpaolino has quit IRC | 07:39 | |
| *** BharatK has quit IRC | 07:39 | |
| *** yolandamobile has joined #openstack-infra | 07:42 | |
| *** Ala has joined #openstack-infra | 07:45 | |
| *** yolandamobile has quit IRC | 07:45 | |
| *** larainema has quit IRC | 07:47 | |
| *** yolandamobile has joined #openstack-infra | 07:48 | |
| *** Krinkle is now known as Krinkle_ | 07:49 | |
| *** dtantsur|afk is now known as dtantsur | 07:50 | |
| *** dkehn has quit IRC | 07:50 | |
| *** BharatK has joined #openstack-infra | 07:51 | |
| *** yolandamobile has quit IRC | 07:51 | |
| *** dkehn has joined #openstack-infra | 07:51 | |
| *** ghostpl has joined #openstack-infra | 07:53 | |
| *** larainema has joined #openstack-infra | 07:53 | |
| *** mpavone has joined #openstack-infra | 07:53 | |
| *** yolandamobile has joined #openstack-infra | 07:55 | |
| *** hashar has joined #openstack-infra | 07:57 | |
| *** yolandamobile has quit IRC | 07:57 | |
| *** BharatK has quit IRC | 07:58 | |
| *** BharatK has joined #openstack-infra | 07:59 | |
| *** pblaho has joined #openstack-infra | 08:00 | |
| *** yolandamobile has joined #openstack-infra | 08:00 | |
| *** yolandamobile has quit IRC | 08:02 | |
| *** ihrachyshka_ has joined #openstack-infra | 08:02 | |
| *** yolandamobile has joined #openstack-infra | 08:03 | |
| *** husanu has quit IRC | 08:03 | |
| *** esker has joined #openstack-infra | 08:04 | |
| *** ihrachyshka has quit IRC | 08:04 | |
| *** mpavone has quit IRC | 08:05 | |
| *** husanu has joined #openstack-infra | 08:05 | |
| *** liam__ has joined #openstack-infra | 08:06 | |
| *** yolandamobile has quit IRC | 08:06 | |
| *** liam__ has quit IRC | 08:06 | |
| *** sshnaidm has quit IRC | 08:07 | |
| *** abregman has quit IRC | 08:07 | |
| *** yolandamobile has joined #openstack-infra | 08:08 | |
| *** abregman has joined #openstack-infra | 08:09 | |
| *** harlowja_still_a has joined #openstack-infra | 08:10 | |
| *** yolandamobile has quit IRC | 08:10 | |
| *** abregman has quit IRC | 08:11 | |
| *** abregman has joined #openstack-infra | 08:11 | |
| *** ddieterly has joined #openstack-infra | 08:13 | |
| *** yolandamobile has joined #openstack-infra | 08:14 | |
| *** harlowja_at_home has quit IRC | 08:14 | |
| *** coolsvap is now known as coolsvap|away | 08:16 | |
| *** deepakcs has quit IRC | 08:17 | |
| *** mpavone has joined #openstack-infra | 08:17 | |
| *** yolandamobile has quit IRC | 08:17 | |
| *** ddieterly has quit IRC | 08:18 | |
| *** u_glide has left #openstack-infra | 08:18 | |
| *** coolsvap|away is now known as coolsvap | 08:19 | |
| *** esker has quit IRC | 08:20 | |
| *** sshnaidm has joined #openstack-infra | 08:20 | |
| *** yolandamobile has joined #openstack-infra | 08:20 | |
| *** dguerri` is now known as dguerri | 08:21 | |
| *** yolandamobile has quit IRC | 08:21 | |
| *** yolandamobile has joined #openstack-infra | 08:23 | |
| *** isviridov_away is now known as isviridov | 08:27 | |
| *** HeOS has quit IRC | 08:28 | |
| *** rossella_s has quit IRC | 08:31 | |
| *** yolandamobile has quit IRC | 08:31 | |
| *** rossella_s has joined #openstack-infra | 08:31 | |
| *** notnownikki has joined #openstack-infra | 08:31 | |
| *** e0ne has joined #openstack-infra | 08:32 | |
| *** yolandamobile has joined #openstack-infra | 08:32 | |
| *** markvoelker has joined #openstack-infra | 08:32 | |
| *** _nadya_ has joined #openstack-infra | 08:33 | |
| *** ihrachyshka_ has quit IRC | 08:33 | |
| *** yfried|afk has quit IRC | 08:35 | |
| *** e0ne has quit IRC | 08:37 | |
| *** markvoelker has quit IRC | 08:37 | |
| *** _nadya_ has quit IRC | 08:38 | |
| *** _nadya_ has joined #openstack-infra | 08:38 | |
| *** otter768 has joined #openstack-infra | 08:39 | |
| *** fallenpegasus has quit IRC | 08:41 | |
| *** laughterwym_ has quit IRC | 08:42 | |
| *** _nadya_ has quit IRC | 08:43 | |
| Kennan | hi | 08:43 |
| Kennan | anyone knows this tox: error: unrecognized arguments: --concurrency=1 | 08:43 |
| Kennan | ? | 08:43 |
| Kennan | in jenkins job failed for that | 08:43 |
| openstackgerrit | Davide Guerri proposed openstack-infra/shade: Fix AttributeError in keystone functional tests https://review.openstack.org/195451 | 08:43 |
| openstackgerrit | Davide Guerri proposed openstack-infra/shade: Skip Swift functional tests if needed https://review.openstack.org/195452 | 08:43 |
| *** otter768 has quit IRC | 08:44 | |
| *** BharatK has quit IRC | 08:45 | |
| *** yolandamobile has quit IRC | 08:45 | |
| *** esker has joined #openstack-infra | 08:46 | |
| *** jcoufal has joined #openstack-infra | 08:46 | |
| *** BobBall_AWOL has quit IRC | 08:47 | |
| *** katyafervent is now known as katyafervent_awa | 08:47 | |
| *** viktors has joined #openstack-infra | 08:48 | |
| *** yolandamobile has joined #openstack-infra | 08:48 | |
| *** doude has quit IRC | 08:49 | |
| *** doude has joined #openstack-infra | 08:49 | |
| *** derekh has joined #openstack-infra | 08:51 | |
| *** yolandamobile has quit IRC | 08:51 | |
| *** yamada-h has quit IRC | 08:51 | |
| *** yolandamobile has joined #openstack-infra | 08:52 | |
| *** mpavone_ has joined #openstack-infra | 08:53 | |
| *** yolandamobile has quit IRC | 08:53 | |
| *** salv-orl_ has joined #openstack-infra | 08:53 | |
| *** mpavone has quit IRC | 08:54 | |
| *** BobBall has joined #openstack-infra | 08:54 | |
| viktors | folks, can someone +2A patch https://review.openstack.org/#/c/195020/ to unblock oslo.messaging gates ? Thanks! | 08:56 |
| *** yolandamobile has joined #openstack-infra | 08:56 | |
| *** salv-orlando has quit IRC | 08:57 | |
| *** salv-orl_ has quit IRC | 08:58 | |
| *** yolandamobile has quit IRC | 08:58 | |
| *** yolandamobile has joined #openstack-infra | 08:59 | |
| *** masayukig_ has quit IRC | 09:00 | |
| *** esker has quit IRC | 09:01 | |
| *** yolandamobile has quit IRC | 09:01 | |
| *** yolandamobile has joined #openstack-infra | 09:03 | |
| *** masayukig_ has joined #openstack-infra | 09:04 | |
| *** pelix has joined #openstack-infra | 09:05 | |
| *** yolandamobile has quit IRC | 09:05 | |
| *** yolandamobile has joined #openstack-infra | 09:06 | |
| *** masayukig_ has quit IRC | 09:09 | |
| *** e0ne has joined #openstack-infra | 09:10 | |
| *** fhubik has joined #openstack-infra | 09:10 | |
| *** yolandamobile has quit IRC | 09:13 | |
| *** shardy has quit IRC | 09:14 | |
| *** masayukig_ has joined #openstack-infra | 09:14 | |
| *** ddieterly has joined #openstack-infra | 09:14 | |
| *** shardy has joined #openstack-infra | 09:15 | |
| *** yolandamobile has joined #openstack-infra | 09:16 | |
| *** e0ne is now known as e0ne_ | 09:16 | |
| *** mrmartin has joined #openstack-infra | 09:17 | |
| *** yolandamobile has quit IRC | 09:17 | |
| *** ddieterly has quit IRC | 09:18 | |
| *** yolandamobile has joined #openstack-infra | 09:20 | |
| *** AJaeger has quit IRC | 09:20 | |
| openstackgerrit | Davide Guerri proposed openstack-infra/shade: Do not use environment for Swift unit tests https://review.openstack.org/195474 | 09:21 |
| *** e0ne_ has quit IRC | 09:22 | |
| openstackgerrit | Davide Guerri proposed openstack-infra/shade: Add Neutron/Nova Floating IP attach/detach https://review.openstack.org/183693 | 09:22 |
| *** yolandamobile has quit IRC | 09:22 | |
| openstackgerrit | Davide Guerri proposed openstack-infra/shade: Fix available_floating_ip when using Nova network https://review.openstack.org/195476 | 09:22 |
| *** yolandamobile has joined #openstack-infra | 09:25 | |
| *** abregman is now known as abregman_|numnum | 09:28 | |
| *** yolandamobile has quit IRC | 09:28 | |
| *** esker has joined #openstack-infra | 09:28 | |
| *** yolandamobile has joined #openstack-infra | 09:29 | |
| *** AJaeger has joined #openstack-infra | 09:30 | |
| *** AJaeger has joined #openstack-infra | 09:30 | |
| *** yolandamobile has quit IRC | 09:30 | |
| *** masayukig_ has quit IRC | 09:33 | |
| *** e0ne has joined #openstack-infra | 09:33 | |
| *** yolandamobile has joined #openstack-infra | 09:34 | |
| *** masayukig_ has joined #openstack-infra | 09:36 | |
| *** laughterwym has joined #openstack-infra | 09:36 | |
| *** mrmartin has quit IRC | 09:36 | |
| *** HeOS has joined #openstack-infra | 09:37 | |
| openstackgerrit | Brian Exelbierd proposed openstack-infra/yaml2ical: start_date now supported in schedules https://review.openstack.org/195262 | 09:37 |
| *** katyafervent_awa is now known as katyafervent | 09:38 | |
| *** BharatK has joined #openstack-infra | 09:40 | |
| *** bexelbie has joined #openstack-infra | 09:40 | |
| *** masayukig_ has quit IRC | 09:43 | |
| *** yolandamobile has quit IRC | 09:43 | |
| *** fhubik is now known as fhubik_afk | 09:43 | |
| *** esker has quit IRC | 09:44 | |
| *** yolandamobile has joined #openstack-infra | 09:44 | |
| *** fhubik_afk is now known as fhubik | 09:45 | |
| *** alexpilotti has joined #openstack-infra | 09:45 | |
| *** masayukig_ has joined #openstack-infra | 09:45 | |
| *** bexelbie_ has joined #openstack-infra | 09:47 | |
| *** yolandamobile has quit IRC | 09:47 | |
| *** sergmelikyan has joined #openstack-infra | 09:49 | |
| *** yolandamobile has joined #openstack-infra | 09:49 | |
| *** salv-orlando has joined #openstack-infra | 09:51 | |
| *** yolandamobile has quit IRC | 09:51 | |
| *** yolandamobile has joined #openstack-infra | 09:52 | |
| *** laughterwym_ has joined #openstack-infra | 09:53 | |
| openstackgerrit | lifeless proposed openstack/requirements: Split project and requirement types out. https://review.openstack.org/195490 | 09:54 |
| *** yolandamobile has quit IRC | 09:54 | |
| openstackgerrit | lifeless proposed openstack/requirements: Move commands into cmds subdir. https://review.openstack.org/195491 | 09:54 |
| openstackgerrit | lifeless proposed openstack/requirements: Add edit-constraints command. https://review.openstack.org/195492 | 09:54 |
| lifeless | mordred: fungi: ^ | 09:55 |
| *** salv-orl_ has joined #openstack-infra | 09:56 | |
| *** laughterwym has quit IRC | 09:56 | |
| *** yolandamobile has joined #openstack-infra | 09:57 | |
| *** isviridov is now known as isviridov_away | 09:58 | |
| *** yolandamobile has quit IRC | 09:58 | |
| *** salv-orlando has quit IRC | 09:58 | |
| *** salv-orl_ has quit IRC | 09:59 | |
| *** salv-orlando has joined #openstack-infra | 09:59 | |
| *** mrmartin has joined #openstack-infra | 10:00 | |
| *** yolandamobile has joined #openstack-infra | 10:00 | |
| *** salv-orl_ has joined #openstack-infra | 10:01 | |
| *** yolandamobile has quit IRC | 10:01 | |
| *** yolandamobile has joined #openstack-infra | 10:02 | |
| *** salv-orlando has quit IRC | 10:03 | |
| *** yamada-h has joined #openstack-infra | 10:09 | |
| *** ParsectiX has quit IRC | 10:09 | |
| *** ParsectiX has joined #openstack-infra | 10:10 | |
| *** dizquierdo is now known as dizquierdo_afk | 10:11 | |
| *** ghostpl has quit IRC | 10:11 | |
| *** esker has joined #openstack-infra | 10:11 | |
| *** sarob has joined #openstack-infra | 10:12 | |
| *** yolandamobile has quit IRC | 10:12 | |
| *** hichihara has quit IRC | 10:12 | |
| *** yolandamobile has joined #openstack-infra | 10:13 | |
| *** Kennan2 has joined #openstack-infra | 10:14 | |
| *** Kennan has quit IRC | 10:14 | |
| *** dims has joined #openstack-infra | 10:14 | |
| *** BharatK has quit IRC | 10:15 | |
| electrical | Morning all | 10:16 |
| *** yolandamobile has quit IRC | 10:17 | |
| *** Kennan2 is now known as Kennan | 10:17 | |
| *** sarob has quit IRC | 10:17 | |
| *** husanu2 has joined #openstack-infra | 10:18 | |
| *** yolandamobile has joined #openstack-infra | 10:19 | |
| *** markvoelker has joined #openstack-infra | 10:21 | |
| *** yolandamobile has quit IRC | 10:21 | |
| *** mrmartin has quit IRC | 10:21 | |
| *** yolandamobile has joined #openstack-infra | 10:23 | |
| *** bdemers has quit IRC | 10:23 | |
| *** fhubik is now known as fhubik_afk | 10:23 | |
| *** husanu2 has quit IRC | 10:25 | |
| *** yolandamobile has quit IRC | 10:25 | |
| *** markvoelker has quit IRC | 10:25 | |
| *** fhubik_afk is now known as fhubik | 10:25 | |
| *** husanu2 has joined #openstack-infra | 10:26 | |
| *** abregman_|numnum is now known as abregman_ | 10:26 | |
| *** yolandamobile has joined #openstack-infra | 10:27 | |
| *** ihrachyshka has joined #openstack-infra | 10:27 | |
| openstackgerrit | Merged openstack-infra/yaml2ical: Make help text less OpenStack specific https://review.openstack.org/195086 | 10:28 |
| *** yolandamobile has quit IRC | 10:28 | |
| *** husanu2 has quit IRC | 10:28 | |
| *** esker has quit IRC | 10:29 | |
| *** yolandamobile has joined #openstack-infra | 10:29 | |
| *** husanu4 has joined #openstack-infra | 10:29 | |
| *** husanu4 has quit IRC | 10:31 | |
| *** yolandamobile has quit IRC | 10:31 | |
| *** jogo has quit IRC | 10:31 | |
| *** e0ne is now known as e0ne_ | 10:31 | |
| *** yolandamobile has joined #openstack-infra | 10:32 | |
| *** jasondot_ has joined #openstack-infra | 10:32 | |
| *** jasondotstar has quit IRC | 10:33 | |
| *** yolandamobile has quit IRC | 10:33 | |
| *** sergmelikyan has quit IRC | 10:33 | |
| *** jogo has joined #openstack-infra | 10:35 | |
| *** sergmelikyan has joined #openstack-infra | 10:36 | |
| *** e0ne_ has quit IRC | 10:36 | |
| *** jasondot_ is now known as jasondotstar | 10:37 | |
| *** sergmelikyan has quit IRC | 10:38 | |
| *** otter768 has joined #openstack-infra | 10:40 | |
| openstackgerrit | Adam Coldrick proposed openstack-infra/storyboard-webclient: Make result_set_pager show the correct numbers https://review.openstack.org/195511 | 10:42 |
| *** yamada-h_ has joined #openstack-infra | 10:43 | |
| *** KanagarajM has joined #openstack-infra | 10:44 | |
| svasheka | morning guys. I have a question. Who do I ping if I need some artifacts being copied to certain gate job (gate-horizon-dsvm-integration). Should I create ticket? | 10:44 |
| *** otter768 has quit IRC | 10:44 | |
| *** abregman_ has quit IRC | 10:45 | |
| *** abregman has joined #openstack-infra | 10:45 | |
| *** yamada-h has quit IRC | 10:47 | |
| *** sdake has quit IRC | 10:50 | |
| *** olaph has quit IRC | 10:52 | |
| *** esker has joined #openstack-infra | 10:54 | |
| *** e0ne has joined #openstack-infra | 10:55 | |
| AJaeger | svasheka: what do you want to do exactly? | 10:56 |
| svasheka | AJaeger, there's going to be failure screenshots made, would be grate to have them in the job for access on gates | 10:56 |
| svasheka | AJaeger, https://review.openstack.org/#/c/194646/ | 10:57 |
| dims | sdague: can you please bless https://review.openstack.org/#/c/195020/ ? need to switch off amqp1 job | 10:57 |
| AJaeger | svasheka: so, you want to create screenshots and copy them out after the test together with the log files? | 10:57 |
| svasheka | AJaeger, exactly | 10:57 |
| openstackgerrit | Brian Exelbierd proposed openstack-infra/yaml2ical: start_date now supported in schedules https://review.openstack.org/195262 | 10:58 |
| svasheka | AJaeger, and I'm not exactly aware of how it's done) | 10:58 |
| AJaeger | So, you need to change in the project-config repository the definition of the job - somewhere in jenkins/jobs/* | 10:58 |
| svasheka | AJaeger, but this is to be made only after my changes merged ? | 10:58 |
| *** sergmelikyan has joined #openstack-infra | 10:59 | |
| AJaeger | svasheka: you can create the change and mark it with "DEpends-On: " so that it does not merge before | 10:59 |
| svasheka | AJaeger, oh, yes. thanks | 10:59 |
| *** abregman is now known as abregman|brb | 10:59 | |
| AJaeger | svasheka: the job is defined in jenkins/jobs/horizon.yaml, you need to enhance the publishers | 11:00 |
| svasheka | AJaeger, thanks for the direction | 11:00 |
| *** fhubik is now known as fhubik_afk | 11:01 | |
| AJaeger | svasheka: publishers are defiend in macros.yaml | 11:01 |
| sdague | dims: done | 11:01 |
| *** dims has quit IRC | 11:02 | |
| *** mrmartin has joined #openstack-infra | 11:03 | |
| *** dims has joined #openstack-infra | 11:05 | |
| dims | sdague: thanks! | 11:06 |
| *** dims_ has joined #openstack-infra | 11:10 | |
| *** dims has quit IRC | 11:11 | |
| *** esker has quit IRC | 11:11 | |
| *** ghostpl has joined #openstack-infra | 11:11 | |
| *** AzherKhan has quit IRC | 11:12 | |
| *** yamada-h_ has quit IRC | 11:14 | |
| *** ZZelle-away has quit IRC | 11:14 | |
| *** salv-orl_ has quit IRC | 11:14 | |
| *** ZZelle-away has joined #openstack-infra | 11:15 | |
| *** e0ne is now known as e0ne_ | 11:15 | |
| *** ddieterly has joined #openstack-infra | 11:15 | |
| *** dims_ has quit IRC | 11:16 | |
| *** ghostpl has quit IRC | 11:17 | |
| *** david-lyle has quit IRC | 11:17 | |
| *** ghostpl has joined #openstack-infra | 11:19 | |
| *** fhubik_afk is now known as fhubik | 11:19 | |
| *** fhubik is now known as fhubik_afk | 11:20 | |
| *** ddieterly has quit IRC | 11:20 | |
| *** dims has joined #openstack-infra | 11:21 | |
| *** david-lyle has joined #openstack-infra | 11:21 | |
| openstackgerrit | Victor Stinner proposed openstack/requirements: Bump oslo.db to 1.12 https://review.openstack.org/195525 | 11:21 |
| *** yamamoto has quit IRC | 11:24 | |
| *** mrmartin has quit IRC | 11:24 | |
| *** hashar has quit IRC | 11:25 | |
| *** achanda has quit IRC | 11:25 | |
| *** e0ne_ has quit IRC | 11:26 | |
| openstackgerrit | Derek Higgins proposed openstack-infra/tripleo-ci: [WIP] Switch overcloud-f21-nonha to instack https://review.openstack.org/185151 | 11:27 |
| *** e0ne has joined #openstack-infra | 11:28 | |
| openstackgerrit | Kanagaraj Manickam proposed openstack-infra/project-config: Enables job for heat convergence functional test https://review.openstack.org/195531 | 11:31 |
| *** prad has joined #openstack-infra | 11:32 | |
| *** ParsectiX has quit IRC | 11:32 | |
| *** ociuhandu has joined #openstack-infra | 11:33 | |
| *** ericksonsantos has joined #openstack-infra | 11:33 | |
| openstackgerrit | Ihar Hrachyshka proposed openstack-infra/project-config: neutron: don't execute lbaas api tests for kilo and earlier https://review.openstack.org/195533 | 11:36 |
| *** markvoelker has joined #openstack-infra | 11:37 | |
| openstackgerrit | Kanagaraj Manickam proposed openstack-infra/project-config: Enables job for heat convergence functional test https://review.openstack.org/195531 | 11:37 |
| *** samueldmq has joined #openstack-infra | 11:37 | |
| *** esker has joined #openstack-infra | 11:37 | |
| *** bradjones has quit IRC | 11:38 | |
| *** EmilienM|off is now known as EmilienM | 11:38 | |
| *** ociuhandu_ has joined #openstack-infra | 11:39 | |
| *** ociuhandu has quit IRC | 11:40 | |
| *** ociuhandu_ is now known as ociuhandu | 11:40 | |
| *** bradjones has joined #openstack-infra | 11:41 | |
| *** bradjones has quit IRC | 11:41 | |
| *** bradjones has joined #openstack-infra | 11:41 | |
| *** weshay has quit IRC | 11:41 | |
| *** weshay has joined #openstack-infra | 11:41 | |
| *** markvoelker has quit IRC | 11:41 | |
| *** mwagner_afk has quit IRC | 11:46 | |
| openstackgerrit | Thierry Carrez proposed openstack-infra/release-tools: Fix adjust_blueprint.py milestone targeting fail https://review.openstack.org/195537 | 11:46 |
| *** salv-orlando has joined #openstack-infra | 11:49 | |
| AJaeger | fungi, good morning | 11:49 |
| *** arxcruz has quit IRC | 11:50 | |
| AJaeger | fungi, we still have full disks on the proposal (?) node - but only with horizon, see https://jenkins.openstack.org/job/horizon-upstream-translation-update/ . | 11:50 |
| *** jasondotstar has quit IRC | 11:50 | |
| fungi | AJaeger: yeah, i'm looking | 11:51 |
| openstackgerrit | Kanagaraj Manickam proposed openstack-infra/project-config: Enables job for heat convergence functional test https://review.openstack.org/195531 | 11:51 |
| AJaeger | fungi: but please drink your coffee first ;) | 11:52 |
| fungi | the job workspaces are in /home/jenkins which is on the 20gb root filesystem. that directory has 12gb of content | 11:52 |
| *** sshnaidm has quit IRC | 11:52 | |
| *** marcusvrn has quit IRC | 11:53 | |
| fungi | checking now to see if any particular job is taking the lion's share | 11:53 |
| *** esker has quit IRC | 11:54 | |
| AJaeger | AFAIR clarkb increased one of the disks a couple of weeks ago, perhaps a different workspace? At that point of time I think all jobs failed... | 11:54 |
| fungi | propose-requirements-updates is using 2.8gb and openstack-manuals-manuals-upstream-translation-update is using 2.0gb | 11:54 |
| fungi | all other job workspaces are <1gb in size | 11:54 |
| AJaeger | And horizon? | 11:54 |
| AJaeger | Why is only horizon getting a diskfull? The other jobs pass... | 11:55 |
| fungi | horizon-upstream-translation-update is currently using 0.4gb | 11:55 |
| AJaeger | Does it install a larger venv than the others? | 11:55 |
| fungi | maybe? we've got 237mb free in / right now | 11:55 |
| AJaeger | I have 130 MB in my horizon venv | 11:55 |
| AJaeger | And I see in the log file: "Command "tools/with_venv.sh pip install --upgrade -r /home/jenkins/workspace/horizon-upstream-translation-update/requirements.txt -r /home/jenkins/workspace/horizon-upstream-translation-update/test-requirements.txt" failed." | 11:56 |
| AJaeger | with "No space left on device" | 11:56 |
| fungi | well, because this is a long-lived worker, the workspaces on it accumulate cruft over time. the workspace for that job isn't removed and recreated | 11:56 |
| *** ParsectiX has joined #openstack-infra | 11:56 | |
| *** amotoki has quit IRC | 11:57 | |
| AJaeger | so, time for manual removal of all workspaces? | 11:57 |
| fungi | i can temporarily clear all the job workspaces and the agent will recreate them as new jobs run | 11:57 |
| AJaeger | Or what do you normally do ? | 11:57 |
| fungi | though i also need to rebuild this worker to run trusty anyway (because we'll need that for python 3 requirements constraints i think) | 11:58 |
| fungi | so when i do that in the coming days, i'll definitely make sure there's extra room for job workspaces | 11:58 |
| *** markvoelker has joined #openstack-infra | 11:58 | |
| AJaeger | thanks, fungi ! | 11:59 |
| sdague | fungi: should the long lived workers have a daily cleanup cronjob? | 12:00 |
| fungi | sdague: well, the workspace preservation is intended by jenkins as a performance measure | 12:00 |
| fungi | also i would hate to delete a workspace out from under a running job, so we'd need to put some thought into how to context lock it | 12:01 |
| fungi | for now, i've offlined that slave in jenkins and am blowing away all existing workspaces | 12:01 |
| *** KanagarajM_ has joined #openstack-infra | 12:01 | |
| fungi | okay, cleanup complete and the slave is online in jenkins again | 12:03 |
| fungi | that should hold us until i get the replacement up and running | 12:03 |
| *** abregman|brb is now known as abregman | 12:04 | |
| fungi | i also checked pypi.slave.o.o but it's got 7.7gb free so not in any danger for a while | 12:04 |
| *** fhubik_afk is now known as fhubik | 12:04 | |
| *** KanagarajM has quit IRC | 12:04 | |
| *** arxcruz has joined #openstack-infra | 12:05 | |
| AJaeger | great, thanks, fungi! | 12:06 |
| *** yamamoto has joined #openstack-infra | 12:06 | |
| fungi | there may be a job option we can use to clean out workspaces before or after each run, though i'm not finding it on a cursory search | 12:07 |
| AJaeger | fungi: https://wiki.jenkins-ci.org/display/JENKINS/Workspace+Cleanup+Plugin ? | 12:09 |
| fungi | oh, we probably don't want to go adding new jenkins plugins | 12:10 |
| fungi | perhaps a jjb macro would make more sense | 12:11 |
| *** marcusvrn has joined #openstack-infra | 12:12 | |
| *** pc_m has joined #openstack-infra | 12:12 | |
| *** hashar has joined #openstack-infra | 12:12 | |
| AJaeger | fungi: One that does "rm -rf .venv"? | 12:13 |
| AJaeger | As a publisher? | 12:14 |
| *** pc_m_ has joined #openstack-infra | 12:14 | |
| AJaeger | WE could make it intelligent and only cleanup once a day with a small script | 12:14 |
| *** KanagarajM_ has quit IRC | 12:15 | |
| fungi | well, it's not the virtualenv that's the issue in most cases, i don't think | 12:16 |
| *** ddieterly has joined #openstack-infra | 12:16 | |
| *** pc_m has quit IRC | 12:16 | |
| fungi | but that might help at least | 12:17 |
| *** john-davidge has joined #openstack-infra | 12:17 | |
| *** rfolco has joined #openstack-infra | 12:17 | |
| *** bknudson has joined #openstack-infra | 12:19 | |
| john-davidge | VMware NSX CI has been cosistently failing recently on neutron patches and linking to an empty logs folder | 12:19 |
| *** bswartz has quit IRC | 12:19 | |
| john-davidge | See https://review.openstack.org/#/c/194605/ and http://208.91.1.172/logs/neutron/194605/3/py27-1047/ for an example | 12:20 |
| john-davidge | can we get it disabled, please? | 12:20 |
| *** ddieterly has quit IRC | 12:20 | |
| *** esker has joined #openstack-infra | 12:21 | |
| dtantsur | fungi, hi! sorry for personal ping for review, but could you have a look at https://review.openstack.org/#/c/193941/ please? we'd really like to start real (=devstack) gate testing. Thanks! | 12:21 |
| *** fhubik is now known as fhubik_afk | 12:22 | |
| openstackgerrit | Thierry Carrez proposed openstack-infra/release-tools: adjust_blueprints: Only mention real modifications https://review.openstack.org/195550 | 12:22 |
| *** aysyanne has joined #openstack-infra | 12:22 | |
| *** EricGonczer_ has joined #openstack-infra | 12:25 | |
| *** fhubik_afk is now known as fhubik | 12:26 | |
| *** coolsvap is now known as coolsvap|away | 12:26 | |
| AJaeger | fungi, something as function like the following: http://paste.openstack.org/show/320926/ | 12:26 |
| *** akuznetsova has joined #openstack-infra | 12:27 | |
| *** btully has quit IRC | 12:28 | |
| *** btully has joined #openstack-infra | 12:28 | |
| *** dprince has joined #openstack-infra | 12:29 | |
| *** EricGonc_ has joined #openstack-infra | 12:29 | |
| *** zz_dimtruck is now known as dimtruck | 12:30 | |
| *** rossella_s has quit IRC | 12:30 | |
| *** mwagner_afk has joined #openstack-infra | 12:30 | |
| akuznetsova | hi there! I need some advice, in our project we need to install mysql-server-5.6 in our devstack-dsvm gate, what is the best way to do it? seems like using devstack-trusty won't solve our problem | 12:30 |
| *** rossella_s has joined #openstack-infra | 12:31 | |
| *** dims has quit IRC | 12:31 | |
| *** EricGonczer_ has quit IRC | 12:31 | |
| *** dims has joined #openstack-infra | 12:31 | |
| *** dtantsur is now known as dtantsur|brb | 12:32 | |
| *** EricGonc_ has quit IRC | 12:32 | |
| *** EricGonczer_ has joined #openstack-infra | 12:33 | |
| *** EricGonczer_ has quit IRC | 12:35 | |
| *** coolsvap|away is now known as coolsvap | 12:35 | |
| *** mase_home has joined #openstack-infra | 12:35 | |
| *** EricGonczer_ has joined #openstack-infra | 12:35 | |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config: Cleanup proposal slave workspaces once a day after builds https://review.openstack.org/195554 | 12:35 |
| AJaeger | fungi, sdague: here's a simple patch for the "daily cleanup" ^ | 12:35 |
| *** david-ly_ has joined #openstack-infra | 12:35 | |
| *** resker has joined #openstack-infra | 12:36 | |
| *** coolsvap is now known as coolsvap|away | 12:36 | |
| fungi | salv-orlando: are you aware of the situation with the vmware nsx ci? (see john-davidge's comment above) | 12:38 |
| *** esker has quit IRC | 12:38 | |
| john-davidge | I can pull up some more examples if you need them, but they’re pretty easy to find looking for patches with -1s on gerrit | 12:39 |
| *** david-lyle has quit IRC | 12:39 | |
| salv-orlando | fungi: yes I am aware but I have a pile of backlog to sort out before looking at that. I hoped somebody else will pick that up, but everyone is a bit busy at the moment. | 12:39 |
| salv-orlando | I can disable it by myself. | 12:39 |
| fungi | john-davidge: yep, just seeing if salv-orlando can get it disabled on his own | 12:39 |
| fungi | thanks salv-orlando! | 12:39 |
| salv-orlando | there's no reason to disturb you | 12:39 |
| john-davidge | great, thanks | 12:40 |
| *** hdd has joined #openstack-infra | 12:40 | |
| *** otter768 has joined #openstack-infra | 12:41 | |
| salv-orlando | fungi, john-davidge: you shall not see -1s anymore. | 12:41 |
| *** _nadya_ has joined #openstack-infra | 12:41 | |
| john-davidge | salv-orlando: Thank you. You have saved the day | 12:42 |
| fungi | dtantsur|brb: i've approved that, but left you an inline comment about eventually tightening up the overhead gap in the job timeouts | 12:42 |
| fungi | akuznetsova: what platforms provide mysql-server-5.6 by default? centos 7? fedora 22? | 12:43 |
| fungi | akuznetsova: it looks to me like ubuntu trusty has 5.6. what was the problem with it? | 12:44 |
| *** ghostpl has quit IRC | 12:44 | |
| fungi | akuznetsova: http://packages.ubuntu.com/trusty/mysql-server-5.6 | 12:44 |
| akuznetsova | fungi, yes, ubuntu-trusty supports it, but I thought that during devstack installation it run 'apt-get install mysql-server' and it will install 5.5 | 12:45 |
| *** otter768 has quit IRC | 12:46 | |
| *** jasondotstar has joined #openstack-infra | 12:46 | |
| *** kgiusti has joined #openstack-infra | 12:46 | |
| fungi | akuznetsova: oh, but you have a job that needs to install the "mysql-server-5.6" package instead of the "mysql-server" package | 12:46 |
| fungi | yes, i agree the mysql-server package is for 5.5 | 12:47 |
| *** sshnaidm has joined #openstack-infra | 12:47 | |
| akuznetsova | fungi, https://github.com/openstack-dev/devstack/blob/8cd5576a17aa06873bfb79d419a1dee312aa4324/lib/databases/mysql#L157 | 12:47 |
| fungi | akuznetsova: and you need to do it within the context of devstack (not a unit test job or something)? | 12:47 |
| *** bradjones has quit IRC | 12:47 | |
| akuznetsova | fungi, also, I thought that we can add something like 'apt-get install mysql-server-5.6' into the shell section in our config file, but again there will be problem during devstack installation cause we can't have two versions on mysql-server | 12:49 |
| fungi | akuznetsova: i recommend first getting some recommendations from the devstack developers in #openstack-qa, but i suspect you could replace the package with a plugin | 12:49 |
| wznoinsk | anteaya: do you use nagios to monitor 3rdparty CIs by any chance? | 12:49 |
| fungi | wznoinsk: nope | 12:50 |
| *** EricGonczer_ has quit IRC | 12:50 | |
| *** bradjones has joined #openstack-infra | 12:50 | |
| *** bradjones has quit IRC | 12:50 | |
| *** bradjones has joined #openstack-infra | 12:50 | |
| akuznetsova | fungi, yes, we have devstack-dsvm gate on every commit and we need to use it here | 12:50 |
| wznoinsk | so it's radar (resurrected) and patrick east's dashboard? | 12:50 |
| EmilienM | hello infra, can I have some feedback on https://review.openstack.org/#/c/194293/ ? Thanks a lot :-) | 12:50 |
| mordred | whats radar? | 12:50 |
| fungi | wznoinsk: and maybe driverlog? | 12:50 |
| *** husanu2 has joined #openstack-infra | 12:50 | |
| fungi | mordred: radar was the ci dashboard mikal/jhesketh did i think | 12:51 |
| mordred | ah | 12:51 |
| *** _nadya_ has quit IRC | 12:51 | |
| fungi | some time early last year? | 12:51 |
| *** ParsectiX has quit IRC | 12:51 | |
| *** _nadya_ has joined #openstack-infra | 12:52 | |
| akuznetsova | fungi, probably it is a crazy idea, but maybe we can mysql installation in devstack and just install it manually? | 12:52 |
| *** husanu2 has quit IRC | 12:52 | |
| akuznetsova | *disable | 12:52 |
| fungi | akuznetsova: that's the sort of thing you might do in a devstack plugin, i think. that's why i suggested consulting the devstack developers on options | 12:52 |
| *** rlandy has joined #openstack-infra | 12:53 | |
| rlandy | weshay: yes | 12:53 |
| *** claudiub|2 has joined #openstack-infra | 12:53 | |
| *** EricGonczer_ has joined #openstack-infra | 12:53 | |
| wznoinsk | fungi: I guess driverlog is only for intree drivers? | 12:54 |
| *** AzherKhan has joined #openstack-infra | 12:54 | |
| *** husanu1 has joined #openstack-infra | 12:54 | |
| fungi | wznoinsk: no idea. i don't really follow third-party ci closely | 12:54 |
| akuznetsova | fungi, thanks, I will ask devstack developers | 12:54 |
| sdague | fungi: oh hey, we've got > 50% of nodes in delete mode | 12:54 |
| fungi | sdague: i'll look into it | 12:54 |
| *** claudiub has quit IRC | 12:55 | |
| fungi | sdague: oh wow, i wonder if one of our providers is offline or something | 12:55 |
| *** resker has quit IRC | 12:55 | |
| *** mordredemacs has quit IRC | 12:55 | |
| *** esker has joined #openstack-infra | 12:55 | |
| wznoinsk | fungi: ok, I think it is by the looks of it's current config, anteaya (whenever you have a chance) would you know? | 12:55 |
| *** EricGonc_ has joined #openstack-infra | 12:56 | |
| *** husanu1 has quit IRC | 12:56 | |
| *** EricGonczer_ has quit IRC | 12:56 | |
| *** sshnaidm has quit IRC | 12:56 | |
| wznoinsk | < or krtaylor | 12:56 |
| *** sdake has joined #openstack-infra | 12:56 | |
| *** husanu3 has joined #openstack-infra | 12:56 | |
| fungi | sdague: huh, we've got ~250 deleting in hpcloud and ~50 in rax so maybe it's nodepool side rather than provider side | 12:57 |
| *** EricGonc_ has quit IRC | 12:58 | |
| *** EricGonczer_ has joined #openstack-infra | 12:58 | |
| openstackgerrit | Mateusz Matuszkowiak proposed openstack-infra/project-config: Added new SF repository For Fuel LMA Infrastructure Alerting plugin https://review.openstack.org/195562 | 12:58 |
| *** pc_m_ has quit IRC | 12:59 | |
| *** bswartz has joined #openstack-infra | 12:59 | |
| fungi | mordred: i'm getting a ERROR (Unauthorized): Unauthorized (HTTP 401) from hpcloud with our openstackjenkins2 credentials | 13:00 |
| *** coolsvap|away is now known as coolsvap | 13:00 | |
| yolanda | ah, there was an outage in hpcloud few time ago | 13:00 |
| yolanda | mordred, fungi | 13:00 |
| yolanda | should be solved now | 13:00 |
| fungi | yolanda: oh! fun thanks | 13:00 |
| yolanda | from our side, we had to renew auth token in nodepool | 13:00 |
| yolanda | notnownikki did it | 13:00 |
| *** husanu3 has quit IRC | 13:00 | |
| fungi | yolanda: renew auth token? i'm just manually trying to authenticate with novaclient and getting thus | 13:01 |
| *** julim has joined #openstack-infra | 13:01 | |
| fungi | this | 13:01 |
| *** sshnaidm has joined #openstack-infra | 13:01 | |
| *** hdd has quit IRC | 13:02 | |
| notnownikki | hi | 13:02 |
| *** husanu1 has joined #openstack-infra | 13:02 | |
| notnownikki | fungi first you have to find your auth token, run `nova --debug list` and you'll see it in one of the headers | 13:02 |
| *** EricGonczer_ has quit IRC | 13:02 | |
| notnownikki | fungi, then you have to delete it from keystone, i'll grab the incantation... | 13:02 |
| *** EricGonc_ has joined #openstack-infra | 13:02 | |
| *** husanu1 has quit IRC | 13:02 | |
| notnownikki | fungi, curl -i -X DELETE -H "X-Auth-Token: YOUR_TOKEN" https://region-b.geo-1.identity.hpcloudsvc.com:35357/v2.0/HP-IDM/v1.0/tokens/YOUR_TOKEN; | 13:03 |
| fungi | notnownikki: oh, fun. i guess these are cached for a very long time? i'm getting this while manually running novaclient somewhere that i haven't run it in days | 13:03 |
| *** husanux0 has joined #openstack-infra | 13:03 | |
| notnownikki | fungi, that'll force generation of a new token next auth and it should start working again | 13:03 |
| notnownikki | fungi, yes, cached for quite some time, i'm not sure of the length, just how to delete them when it goes wrong :) | 13:03 |
| yolanda | yep, i just got the same issue with nova, and i really hadn't run that in a week | 13:04 |
| *** e0ne is now known as e0ne_ | 13:04 | |
| fungi | yolanda: thanks. that's really not terribly user friendly :/ | 13:04 |
| fungi | notnownikki: thanks | 13:04 |
| fungi | i wonder how hpcloud is handling this across their customerbase | 13:04 |
| *** e0ne_ is now known as e0ne | 13:04 | |
| *** EricGonczer_ has joined #openstack-infra | 13:04 | |
| *** EricGonc_ has quit IRC | 13:05 | |
| rcarrillocruz | hah | 13:05 |
| *** bexelbie_ has quit IRC | 13:05 | |
| *** bexelbie has quit IRC | 13:05 | |
| yolanda | :) | 13:05 |
| rcarrillocruz | they'll leave the tokens to expire by themselves :D | 13:05 |
| openstackgerrit | Mateusz Matuszkowiak proposed openstack-infra/project-config: Added new SF repository For Fuel LMA Infrastructure Alerting plugin https://review.openstack.org/195562 | 13:06 |
| *** miqui has joined #openstack-infra | 13:06 | |
| fungi | notnownikki: does the YOUR_TOKEN in the url include the {SHA1} prefix, or just in the payload? | 13:06 |
| notnownikki | fungi, just the payload | 13:06 |
| fungi | and do i need to escape that ; at the end of the url? | 13:06 |
| notnownikki | fungi, just as it is | 13:07 |
| mordred | ok. I may have just exceed sanity for the day | 13:07 |
| *** dtantsur|brb is now known as dtantsur | 13:07 | |
| mordred | I just connected to my weechat with emacs | 13:07 |
| fungi | {"unauthorized":{"message":"Bad Credentials","code":401}} | 13:07 |
| mordred | so I'm now IRCing via my weechat bouncer in emacs | 13:08 |
| *** AzherKhan has quit IRC | 13:08 | |
| fungi | seems i've still got something not quite right there | 13:08 |
| mordred | that must mean something is wrong with my head somewhere | 13:08 |
| *** spzala has joined #openstack-infra | 13:08 | |
| *** afazekas has joined #openstack-infra | 13:08 | |
| *** rfolco has quit IRC | 13:08 | |
| *** radez is now known as radez_g0n3 | 13:08 | |
| notnownikki | fungi, ah, did you replace the https://region-b.geo-1.identity.hpcloudsvc.com:35357 part with the right auth url for you? I should have mentioned that :) | 13:08 |
| *** husanux0 has quit IRC | 13:08 | |
| *** ghostpl has joined #openstack-infra | 13:09 | |
| fungi | notnownikki: we already use https://region-b.geo-1.identity.hpcloudsvc.com:35357/v2.0 as our auth_url | 13:09 |
| *** EricGonczer_ has quit IRC | 13:10 | |
| mordred | fungi: oh my | 13:11 |
| mordred | notnownikki: next time you talk to them, please suggest that it would be easier if they would just delete all of the tokens cached cloud-side if the cached tokens are all invalid now | 13:11 |
| fungi | mordred: what, just now catching up with the situation where hpcloud has effectively taken their api offline for a few weeks while everyone's keystone auth token expire? | 13:11 |
| fungi | heh, yeah | 13:12 |
| notnownikki | mordred, yeah :( | 13:12 |
| fungi | i wonder what sort of crazy long expiration they have set. the default, if memory serves, is 1 hour | 13:12 |
| mordred | ok. so - nova list works just fine for my mordred account | 13:12 |
| notnownikki | fungi, so i'm not sure what to suggest, that worked for us | 13:12 |
| *** husanux2 has joined #openstack-infra | 13:13 | |
| fungi | mordred: are you able to reproduce the auth error i'm getting with openstackjenkins2 in region b? | 13:13 |
| *** husanux2 has quit IRC | 13:13 | |
| mordred | yes. I have now gotten that | 13:13 |
| mordred | k. so - I was able to run keystone catalog | 13:14 |
| *** husanux6 has joined #openstack-infra | 13:14 | |
| *** tlian has joined #openstack-infra | 13:15 | |
| *** ZZelle-away is now known as ZZelle | 13:15 | |
| openstackgerrit | Merged openstack-infra/project-config: Add Ironic Inspector simple integration test job https://review.openstack.org/193941 | 13:15 |
| *** shardy has quit IRC | 13:17 | |
| bknudson | with https://review.openstack.org/#/c/194801/ merged I thought I'd be able to `git push gerrit HEAD:refs/publish/feature/keystoneauth_integration/merge` , but it's saying I'm not allowed to upload merges | 13:17 |
| *** fallenpegasus has joined #openstack-infra | 13:18 | |
| notnownikki | fungi, did you get the DELETE call working? Might be worth doing the command without the ;, it might just have been luck of the shell that it worked for me | 13:18 |
| fungi | notnownikki: i tried it with and without the ; and also with an escaped ; | 13:18 |
| fungi | same result | 13:18 |
| notnownikki | :( | 13:18 |
| *** husanux6 has quit IRC | 13:18 | |
| fungi | well, actually escaping the ; it reported back that it couldn't find the url | 13:18 |
| notnownikki | i could ask for it to be removed in the hp support room | 13:19 |
| mordred | notnownikki: thanks! | 13:19 |
| notnownikki | fungi, if you pm me the token i'll see if i can get it taken care of for you | 13:19 |
| *** bexelbie_ has joined #openstack-infra | 13:19 | |
| *** peristeri has joined #openstack-infra | 13:20 | |
| *** husanux7 has joined #openstack-infra | 13:20 | |
| *** bexelbie has joined #openstack-infra | 13:20 | |
| *** e0ne is now known as e0ne_ | 13:20 | |
| fungi | notnownikki: sent. thanks! | 13:20 |
| *** ldnunes has joined #openstack-infra | 13:21 | |
| *** husanux7 has quit IRC | 13:21 | |
| bknudson | Maybe I did the acls wrong -- http://docs.openstack.org/infra/manual/drivers.html#merge-commits | 13:21 |
| bknudson | the docs say [access "refs/for/refs/*"] and pushMerge = group <projectname>-release | 13:21 |
| *** coolsvap is now known as coolsvap|away | 13:21 | |
| bknudson | but I was trying to give keystone-core auth to the feature branch only | 13:22 |
| *** sshnaidm has quit IRC | 13:22 | |
| *** e0ne_ is now known as e0ne | 13:22 | |
| *** shardy has joined #openstack-infra | 13:22 | |
| *** _jreslock has joined #openstack-infra | 13:23 | |
| *** aysyanne has quit IRC | 13:23 | |
| *** sshnaidm has joined #openstack-infra | 13:23 | |
| fungi | bknudson: oh, you want them to be able to propose merge commits from master to the feature branch, but not propose merge commits from the feature branch to master? | 13:24 |
| *** AzherKhan has joined #openstack-infra | 13:25 | |
| bknudson | fungi: that seems safest | 13:25 |
| *** ajo is now known as ajo|afk | 13:25 | |
| *** husanux2 has joined #openstack-infra | 13:25 | |
| *** yamamoto has quit IRC | 13:25 | |
| bknudson | gerrit doesn't know the change is from the feature branch when pushing a commit, does it? | 13:25 |
| bknudson | I don't want to give keystone-core merge-push to master... that's scary | 13:26 |
| fungi | bknudson: i don't think we have an example of that. if memory serves gerrit just knows that a merge commit has two parent commits. it doesn't really relate the second parent to a particular branch | 13:26 |
| bknudson | https://code.google.com/p/gerrit/source/browse/Documentation/access-control.txt?r=520d2972a0fa6a0ebded980bc176341139f857ca#596 | 13:26 |
| fungi | bknudson: well, being able to push merge commits to refs/for/refs/* just means they can propose a merge commit for review | 13:26 |
| bknudson | looks like I can use refs/for/refs/heads/BRANCH | 13:26 |
| fungi | but yeah, that sounds like it would work | 13:27 |
| *** salv-orl_ has joined #openstack-infra | 13:27 | |
| bknudson | [access "refs/heads/feature/*"] | 13:27 |
| *** ghostpl has quit IRC | 13:27 | |
| bknudson | is in gerrit/acls/openstack/python-keystoneclient.config | 13:27 |
| fungi | and sounds like you want [access "refs/for/refs/heads/feature/*"] | 13:27 |
| bknudson | does "refs/heads/*" get translated to "refs/for/refs/heads/*" | 13:28 |
| bknudson | ? | 13:28 |
| openstackgerrit | Thierry Carrez proposed openstack-infra/release-tools: Replace --dryrun with --target and --clean actions https://review.openstack.org/194689 | 13:28 |
| *** kir has quit IRC | 13:28 | |
| fungi | bknudson: refs/for/refs is a special tree gerrit uses for submitting changes for review | 13:29 |
| *** husanux2 has quit IRC | 13:29 | |
| bknudson | ok, I'll switch python-keystoneclient.config to refs/for. | 13:29 |
| fungi | bknudson: as i said, that will let them propose merge commits for review targeting the stated branch, but it won't allow you to limit which specific branch they're merging from | 13:30 |
| fungi | but probably good enough for what you want | 13:30 |
| bknudson | it still goes to review so isn't a ninja merge, right? | 13:30 |
| fungi | bknudson: right, someone still has to review and approve it and then it has to pass tests | 13:31 |
| *** salv-orlando has quit IRC | 13:31 | |
| *** mriedem has joined #openstack-infra | 13:31 | |
| AJaeger | bknudson,fungi: sorry for letting that review through - and thanks for the explanation, I know what to look out for next time | 13:32 |
| openstackgerrit | Brant Knudson proposed openstack-infra/project-config: Correct keystoneclient permissions https://review.openstack.org/195577 | 13:33 |
| *** maishsk_afk has joined #openstack-infra | 13:33 | |
| *** maishsk_afk is now known as maishsk | 13:34 | |
| *** alivigni has joined #openstack-infra | 13:34 | |
| bknudson | oops, doesn't pass gerrit | 13:34 |
| *** eharney has quit IRC | 13:35 | |
| openstackgerrit | Brant Knudson proposed openstack-infra/project-config: Correct keystoneclient permissions https://review.openstack.org/195577 | 13:35 |
| *** ajmiller has joined #openstack-infra | 13:35 | |
| *** yamamoto has joined #openstack-infra | 13:36 | |
| *** hdd has joined #openstack-infra | 13:36 | |
| *** tvanderwiel has joined #openstack-infra | 13:37 | |
| *** sshnaidm has quit IRC | 13:37 | |
| *** ghostpl has joined #openstack-infra | 13:37 | |
| *** mmedvede has joined #openstack-infra | 13:37 | |
| *** sshnaidm has joined #openstack-infra | 13:38 | |
| *** marcusvrn1 has joined #openstack-infra | 13:38 | |
| *** fhubik is now known as fhubik_afk | 13:38 | |
| fungi | bknudson: the section probably isn't sorted in alpha order | 13:38 |
| fungi | ahh, there you go | 13:39 |
| *** marcusvrn has quit IRC | 13:39 | |
| *** sergmelikyan has quit IRC | 13:39 | |
| *** dustins has joined #openstack-infra | 13:40 | |
| *** changbl_ has joined #openstack-infra | 13:40 | |
| *** wenlock has joined #openstack-infra | 13:42 | |
| *** sputnik13 has joined #openstack-infra | 13:42 | |
| *** shardy has quit IRC | 13:42 | |
| *** shardy has joined #openstack-infra | 13:44 | |
| *** julim has quit IRC | 13:44 | |
| *** crc32 has joined #openstack-infra | 13:44 | |
| *** hashar has quit IRC | 13:45 | |
| *** dboik has joined #openstack-infra | 13:46 | |
| *** _nadya_ has quit IRC | 13:46 | |
| *** sputnik13 has quit IRC | 13:47 | |
| *** ajmiller_ has joined #openstack-infra | 13:47 | |
| *** _nadya_ has joined #openstack-infra | 13:47 | |
| *** sputnik13 has joined #openstack-infra | 13:48 | |
| *** ajmiller_ has quit IRC | 13:48 | |
| *** sputnik13 has quit IRC | 13:48 | |
| *** jcoufal has quit IRC | 13:49 | |
| *** amitgandhinz has joined #openstack-infra | 13:49 | |
| *** ajmiller has quit IRC | 13:49 | |
| *** sputnik13 has joined #openstack-infra | 13:49 | |
| *** samueldmq has quit IRC | 13:51 | |
| *** _nadya_ has quit IRC | 13:51 | |
| *** ericksonsantos has quit IRC | 13:51 | |
| fungi | sdague: mordred: nodepool's able to authenticate to hpcloud again. not sure if the manual delete notnownikki walked me though did the trick but took a few minutes to actually process, or whether it just expired on its own | 13:52 |
| *** baoli has joined #openstack-infra | 13:52 | |
| sdague | gotcha, cool | 13:52 |
| *** AzherKhan has quit IRC | 13:52 | |
| *** AzherKhan has joined #openstack-infra | 13:53 | |
| notnownikki | fungi, we might be table to tell, if you run nova --debug list again and the auth token has changed, it was a delayed delete, if it hasn't changed, then it was a temp keystone failure of some kind | 13:54 |
| *** devvesa has joined #openstack-infra | 13:55 | |
| *** dizquierdo_afk is now known as dizquierdo | 13:56 | |
| * krtaylor reads scrollback | 13:56 | |
| *** baoli has quit IRC | 13:56 | |
| *** baoli has joined #openstack-infra | 13:57 | |
| krtaylor | wznoinsk, was your question about nagios service monitoring or third party CI monitoring dashboard? | 13:57 |
| *** ghostpl has quit IRC | 13:57 | |
| AJaeger | oh, somebody started the sdague approval bot ;) | 13:57 |
| fungi | notnownikki: yeah, new token for sure. however it still doesn't start with "HPAuth" so not sure what sort of crack the noc is smoking there | 13:57 |
| *** earlephilhower has quit IRC | 13:57 | |
| *** xyang1 has joined #openstack-infra | 13:58 | |
| sdague | AJaeger: yeh, I'm going through the already +2ed project-config changes to see what can be merged | 13:58 |
| *** eantysh9 has joined #openstack-infra | 13:58 | |
| fungi | notnownikki: and still doesn't tell us whether the token is new because the old one expired or because the delete worked | 13:58 |
| *** puranamr has joined #openstack-infra | 13:59 | |
| *** _diana_ has joined #openstack-infra | 14:00 | |
| mtreinish | sdague: pick me, pick me: https://review.openstack.org/194798 | 14:00 |
| *** eantyshev has quit IRC | 14:01 | |
| *** eharney has joined #openstack-infra | 14:01 | |
| AJaeger | sdague: really appreciated, thanks! | 14:02 |
| mordred | fungi: \o/ | 14:04 |
| notnownikki | fungi, so if we were still trying to use the old one and getting 401, but we couldn't delete it, it had most likely expired or invalidated but not been purged completely | 14:04 |
| *** julim has joined #openstack-infra | 14:04 | |
| fungi | notnownikki: yeah, that seems plausible | 14:04 |
| *** yamamoto has quit IRC | 14:05 | |
| *** eantysh9 has quit IRC | 14:05 | |
| *** eantysh9 has joined #openstack-infra | 14:05 | |
| *** marun has joined #openstack-infra | 14:06 | |
| *** markvan_ is now known as markvan | 14:06 | |
| *** stevemar has joined #openstack-infra | 14:06 | |
| *** Hal has joined #openstack-infra | 14:07 | |
| *** Hal is now known as Guest525 | 14:07 | |
| *** Guest27822 has quit IRC | 14:08 | |
| *** r-daneel has joined #openstack-infra | 14:09 | |
| *** pc_m has joined #openstack-infra | 14:09 | |
| *** k4n0 has quit IRC | 14:09 | |
| *** doug-fish has joined #openstack-infra | 14:10 | |
| *** ghostpl has joined #openstack-infra | 14:10 | |
| *** mrmartin has joined #openstack-infra | 14:10 | |
| *** devvesa has quit IRC | 14:11 | |
| *** e0ne is now known as e0ne_ | 14:13 | |
| jeblair | bknudson, fungi, AJaeger: that acl will let you propose merge commits to a feature branch (eg, bring it up to date with master), but what happens when the feature is complete? won't you want to merge the feature branch into master? | 14:13 |
| bknudson | jeblair: we will want to merge the feature branch back to master. | 14:14 |
| jeblair | bknudson, fungi, AJaeger: it seems like unecessary acl churn to have to enable it then. can we just allow -core to propose merge commits to any branch? | 14:14 |
| bknudson | that might be a different group, though? | 14:14 |
| fungi | presumably as the time for that nears, the drivers/milestone/release/whoever group will have a broader acl allowing that | 14:14 |
| fungi | i'm in favor of planning ahead and implementing that now as long as keystone knows who they want to be doing that part now | 14:15 |
| bknudson | maybe somebody already has pushMerge to master? | 14:15 |
| bknudson | I've got the file open so it's easy to add now | 14:15 |
| sdague | mriedem: gah, can we get a sqlalchemy-migrate release out there without the pbr cap? | 14:17 |
| mriedem | sdague: let me get back to you after the nova meeting | 14:18 |
| bknudson | I'll propose giving merge on all branches to keystone-release | 14:20 |
| *** devvesa has joined #openstack-infra | 14:22 | |
| *** yamamoto has joined #openstack-infra | 14:23 | |
| *** e0ne_ has quit IRC | 14:23 | |
| openstackgerrit | Brant Knudson proposed openstack-infra/project-config: Grant keystone-release pushMerge on all branches in client https://review.openstack.org/195607 | 14:24 |
| *** Zhongjun has joined #openstack-infra | 14:26 | |
| *** _nadya_ has joined #openstack-infra | 14:29 | |
| *** esker has quit IRC | 14:29 | |
| *** shardy has quit IRC | 14:29 | |
| *** esker has joined #openstack-infra | 14:30 | |
| *** fhubik_afk is now known as fhubik | 14:30 | |
| *** shardy has joined #openstack-infra | 14:30 | |
| *** yamahata has joined #openstack-infra | 14:30 | |
| *** marcusvrn has joined #openstack-infra | 14:31 | |
| *** fhubik is now known as fhubik_afk | 14:32 | |
| *** eharney has quit IRC | 14:33 | |
| *** samueldmq has joined #openstack-infra | 14:33 | |
| *** marcusvrn1 has quit IRC | 14:34 | |
| *** esker has quit IRC | 14:35 | |
| *** e0ne has joined #openstack-infra | 14:36 | |
| *** eharney has joined #openstack-infra | 14:40 | |
| sdake | mordred are you about | 14:40 |
| mordred | sdake: I am | 14:41 |
| sdake | hey I wanted to thank you for relicensing the ansible modules we need for olla in the short term | 14:41 |
| sdake | I now its a huge pita | 14:41 |
| sdake | we will follow whatever rules you want us to follow | 14:41 |
| *** otter768 has joined #openstack-infra | 14:41 | |
| *** yamamoto has quit IRC | 14:41 | |
| sdake | sam mentioned no changes internally - we are forcing that by making them RO int he repo and I have instructed teh core team to -2 any changes to those files | 14:41 |
| mordred | sdake: yah - that's the real key to making it workable | 14:42 |
| yolanda | hi fungi, so for logrotate fileremoval, i prefered to don't transform the entries | 14:42 |
| mordred | sdake: otherwise we are going to get into a world of hurt at some point | 14:42 |
| sdake | also in my mind this is just a short term solution to a legal issue | 14:42 |
| fungi | yolanda: i don't get it them | 14:42 |
| yolanda | because in that case, it shouldn't be cleaning up old jeepyb entries, but cleaning up the new ones | 14:42 |
| *** woodster_ has joined #openstack-infra | 14:42 | |
| mordred | yup. once they're released I think you have a much cleared consume story | 14:42 |
| fungi | yolanda: but instead you have it cleaning up log files themselves, not logrotate config files | 14:43 |
| *** marun has quit IRC | 14:43 | |
| sdake | ya - hopefully you understand my position, I cant pull gpl repos into asl code base | 14:43 |
| sdake | either copy or use some type of curl script | 14:43 |
| *** cindyo has joined #openstack-infra | 14:43 | |
| *** puranamr has quit IRC | 14:43 | |
| yolanda | fungi, my point to don't replace / with _, is that if you pass fileremoval /var/log/jeepyb/manage_projects.log | 14:43 |
| *** pc_m has quit IRC | 14:43 | |
| yolanda | it will be converting to _var_log_jeepyb_manage_projects.log | 14:43 |
| yolanda | then cleaning up same file logrotate created | 14:43 |
| sdake | it damages the COO | 14:43 |
| *** puranamr has joined #openstack-infra | 14:44 | |
| fungi | yolanda: but we don't want it to delete /var/log/jeepyb/manage_projects.log we want it to delete /etc/logrotate.d/var/log/jeepyb/manage_projects.log | 14:44 |
| yolanda | yep, that's it | 14:44 |
| sdake | but we *REALLY* want to use them rather then create some duplicate effort or hacky scripts to do the job | 14:44 |
| yolanda | i did that for your patch | 14:44 |
| yolanda | let me show | 14:44 |
| sdake | mordred I hate to apply more pressure :) but this is blocking us atm, so the faster you can get er done, i'd appreciate it :) | 14:44 |
| fungi | yolanda: you didn't. you're passing in the logfile names, not the logrotate config file names | 14:44 |
| fungi | yolanda: if you want to be able to pass in the logfile names and have logrotate::fileremoval be smart enough to know what config file names correspond to those and remove them, then the fileremoval change needs extra logic | 14:45 |
| yolanda | ah, so i did a mistake there,when refactoring | 14:45 |
| mordred | sdake: yah - I'll get you sorted out today | 14:46 |
| yolanda | i wanted to prepent /etc/logrotate.d/ on fileremoval.pp | 14:46 |
| yolanda | but not replace the / with _ | 14:46 |
| sdake | mordred that would rock thanks :) | 14:46 |
| fungi | yolanda: right, that's what i meant by "transform" the filenames | 14:46 |
| *** otter768 has quit IRC | 14:46 | |
| sdake | we are in releawe day today - so today is a dead day for us anyway | 14:46 |
| DuncanT | Hi. I'm working on some third party CI, and I'm finding that the fact I'm only allowed to return 'success' or 'failure' is very limiting - in particular, when I know the failure is in setup (unable to clone things, no nodes available, etc), I'd like to report something other than failure - I could not report, but that makes rechecks look bad | 14:46 |
| fungi | yolanda: as in transform the logfile name provided into a logrotate config file path | 14:46 |
| sdake | so its not blocking us today, its blocking us starting tomorrow :) | 14:46 |
| yolanda | fungi, ok, i'll add the /etc/logrotate.d, that was a mistake. But not the replacement part as it's on logrotate::file, because it will prevent to clean up buggy files | 14:47 |
| yolanda | does it make sense? | 14:47 |
| fungi | DuncanT: if you're using jenkins, you'll want to work with the jenkins devs to allow the use of additional result types. right now you only get those two out of it (other than some internally calculated ones it has like aborted and unstable) | 14:47 |
| dguerri | mordred: there is a bunch of reviews about shade waiting for you :> | 14:47 |
| DuncanT | A good number of the cinder CI failures are down to this failure mode. For people who can differentiate, it would be good if the did, but it currently isn't allowed due to infra policy | 14:47 |
| *** puranamr has quit IRC | 14:48 | |
| DuncanT | fungi: I'm not using jenkins... My only limitation in infra policy | 14:48 |
| dguerri | and Shrews ^^ of course! | 14:48 |
| fungi | DuncanT: i don't think infra has a policy on what your comment contains? we just provide guidelines to the projects which decide what they want you to provide | 14:48 |
| fungi | yolanda: yes, makes sense | 14:48 |
| yolanda | ok | 14:48 |
| DuncanT | fungi: Nope, there's definitely policy. I'll see if I can find the link | 14:49 |
| openstackgerrit | David Shrewsbury proposed openstack-infra/shade: Remove list_keypair_dicts method https://review.openstack.org/195616 | 14:49 |
| fungi | DuncanT: we have some tooling around certain comment formats which makes it easier to summarize them, if that's what you're thinking of | 14:49 |
| DuncanT | fungi: There's been statements, mostly by anteaya, that comments not matching the template simply aren't allowed and will have the account turned off | 14:50 |
| mriedem | fungi: jeblair: can you take a look at this? we have the corresponding change in nova so nova can control the cells job regex for tests - this is blocking us turning on the cells job for voting in nova change sin the gate | 14:50 |
| mriedem | https://review.openstack.org/#/c/194410/ | 14:51 |
| fungi | DuncanT: oh. well in that case i'll let her speak to that. it's not my position at any rate | 14:51 |
| DuncanT | fungi: Ok, thanks. I'll pop along to a 3rd party meeting I guess | 14:51 |
| fungi | DuncanT: we certainly want our own ci to have greater granulatity in reporting as well, and are currently somewhat limited by our use of jenkins in that regard | 14:51 |
| *** ghostpl has quit IRC | 14:52 | |
| *** MIDENN_ has quit IRC | 14:52 | |
| fungi | DuncanT: so i definitely wouldn't want to tell third-party ci operators not to do something we ourselves want to do. now as to whether we have the ability to support that conveniently today (in hideci.js and so on) is another matter | 14:52 |
| mordred | anteaya, DuncanT: I think we should probably amend that policy, if it's there, to be more like "reporting must be at least as good as the infra jenkins template" | 14:53 |
| mordred | and must also have the right hooks for things like hideci.js | 14:53 |
| mordred | but, golly, if they can exceed those standards, that seems awesome to me | 14:53 |
| openstackgerrit | Brant Knudson proposed openstack-infra/project-config: Grant keystone-release pushMerge on all branches in client https://review.openstack.org/195607 | 14:53 |
| anteaya | DuncanT: um, I don't recall saying comments not matching a template are not allowed and will be turned off | 14:53 |
| anteaya | DuncanT: I can't remember what I might have said that left that impression | 14:54 |
| anteaya | DuncanT: comments not matching the jenkins format don't appear in the results to the right of the list of reviewers | 14:54 |
| openstackgerrit | yolanda.robla proposed openstack-infra/puppet-logrotate: Add subclass to remove logrotate configurations https://review.openstack.org/192242 | 14:54 |
| anteaya | but that isn't required | 14:54 |
| *** stevemar has quit IRC | 14:55 | |
| anteaya | DuncanT: and also I have recently learned putting your recheck syntax in the comment it potentially dangerous | 14:55 |
| *** fhubik_afk is now known as fhubik | 14:55 | |
| *** maishsk has quit IRC | 14:56 | |
| *** mestery has joined #openstack-infra | 14:56 | |
| john-davidge | anteaya: recheck triggered by the mention of its own recheck syntax in the comment? | 14:56 |
| openstackgerrit | yolanda.robla proposed openstack-infra/puppet-user: Add a sane default value for key_id https://review.openstack.org/195618 | 14:56 |
| anteaya | john-davidge: that would the situation best avoided, yes | 14:56 |
| mordred | anteaya: ++ | 14:57 |
| sdague | DuncanT: is there example format up somewhere, honestly, I'd love tests to get in the habbit of reporting the stage that they failed in | 14:57 |
| anteaya | DuncanT: and actually working on having some way of reporting back a build failure as opposed to a test failure is a topic that came up at the neturon sprint | 14:57 |
| sdague | my big concern is things that didn't report success/failure | 14:57 |
| fungi | john-davidge: anteaya: though that's generally a case of a too-loose regular expression in configuration, and so can be triggered by an even looser still regex regardless of the comment content | 14:57 |
| sdague | in a standard way | 14:57 |
| anteaya | fungi: agreed | 14:57 |
| john-davidge | anteaya: That’s amazing! Is that what happened with the A10 Networks CI back in April? | 14:58 |
| *** mase_home has quit IRC | 14:58 | |
| sdague | DuncanT: I'd help with hideci.js changes to support it if needed | 14:58 |
| *** puranamr has joined #openstack-infra | 14:59 | |
| dougwig | john-davidge: yep, that was me. | 14:59 |
| openstackgerrit | Merged openstack-infra/release-tools: Optional include pypi package url link in release notes https://review.openstack.org/194850 | 14:59 |
| openstackgerrit | Merged openstack-infra/release-tools: Fix adjust_blueprint.py milestone targeting fail https://review.openstack.org/195537 | 14:59 |
| openstackgerrit | Merged openstack-infra/release-tools: adjust_blueprints: Only mention real modifications https://review.openstack.org/195550 | 14:59 |
| openstackgerrit | Merged openstack-infra/release-tools: Replace --dryrun with --target and --clean actions https://review.openstack.org/194689 | 14:59 |
| anteaya | what we discussed at neutron sprint yesterday is the worrying (for me) acceptance of running recheck as part of the workflow | 15:00 |
| *** fifieldt has joined #openstack-infra | 15:00 | |
| *** david-ly_ is now known as david-lyle | 15:01 | |
| anteaya | digging further I was told taht the source of this behaviour is the fact taht jenkins reports failure for both build and test failures | 15:01 |
| dougwig | anteaya: not just acceptance, but we've literally been trained that it's a waste of time to look at build results before two or three rechecks. | 15:01 |
| *** fifieldt has quit IRC | 15:01 | |
| john-davidge | anteaya: From a dev point of view, we do that because CI (particulary 3rd party) erroneously fails so often that most -1’s are assumed to be incorrect unless proven otherwise | 15:01 |
| anteaya | so the behaviour is to recheck automatically rather than look at logs since the likely hood is that is it a build failure | 15:01 |
| *** thedodd has joined #openstack-infra | 15:01 | |
| *** fifieldt has joined #openstack-infra | 15:01 | |
| anteaya | the hope is taht this behaviour can be broken if we identify and report build failures separately from test failures | 15:02 |
| dougwig | anteaya: i think you can refine that to automatically recheck if devstack fails to setup, and possibly not if it's later in the run. | 15:02 |
| john-davidge | anteaya: Reporting build failures separtely would be very helpful | 15:02 |
| fungi | anteaya: by "build failure" you mean something happens before or during devstack setup rather than a failure of one or more tempest tests? | 15:02 |
| *** armax has joined #openstack-infra | 15:03 | |
| fungi | because it's definitely possible for a proposed change to impact the ability to start services in devstack | 15:03 |
| sdague | right, so failure stage. We actually talked about this a bit in the ci-watch session as well from a debugging perspective. We have some standard phases. | 15:03 |
| anteaya | fungi: correct, that is what I am saying | 15:03 |
| dougwig | fungi: definitely, but it'll fail twice, and that is definitely not the norm. | 15:03 |
| fungi | more often than not, that's most of the legitimate failures we observe (the proposed change is so broken you can't even run devstack with it applied) | 15:03 |
| *** mpaolino has joined #openstack-infra | 15:04 | |
| *** mpaolino has quit IRC | 15:04 | |
| *** ayoung-ZZZzzz__ is now known as ayoung | 15:04 | |
| *** mtanino has joined #openstack-infra | 15:05 | |
| *** pfallenop has quit IRC | 15:05 | |
| sdague | fungi: though, if people are letting devstack clone git trees in 3rd party setups, because they aren't using zuul, the git network fails pop into the devstack phase | 15:05 |
| *** pfallenop has joined #openstack-infra | 15:05 | |
| *** reed has quit IRC | 15:05 | |
| sdague | that's one place where a devstack failure is clearly decoupled from the patch | 15:05 |
| dougwig | sdague: heh, you won't have that complaint soon :) : https://review.openstack.org/#/c/188626/ | 15:05 |
| fungi | fair enough. i meant the majority of devstack-based job failures in our upstream ci seem (to me) to usually happen when starting services under devstack. but i agree that may be biased by all the caching and mirroring we do | 15:06 |
| sdague | right | 15:06 |
| dougwig | sdague, fungi, lifeless: i'd appreciate your feedback on that patch, given that you guys definitely had opinions on how git cloning. | 15:06 |
| sdague | dougwig: neat! | 15:06 |
| *** hichihara has joined #openstack-infra | 15:06 | |
| dougwig | man, grammar fail. :) | 15:07 |
| *** unicell has joined #openstack-infra | 15:07 | |
| *** shardy has quit IRC | 15:08 | |
| *** shardy has joined #openstack-infra | 15:08 | |
| openstackgerrit | Mateusz Matuszkowiak proposed openstack-infra/project-config: Added new SF repository For Fuel LMA Infrastructure Alerting plugin https://review.openstack.org/195562 | 15:08 |
| *** med_` is now known as med_ | 15:09 | |
| *** flepied has quit IRC | 15:09 | |
| *** med_ is now known as med___ | 15:09 | |
| jeblair | mordred: can you please review https://review.openstack.org/194119 ? | 15:09 |
| mriedem | sdague: ok, so pbr capped in sqlalchemy-migrate huh, has anyone pushed a change to remove the cap since sqla-migrate isn't synced to g-r? | 15:09 |
| mordred | jeblair: yup! | 15:09 |
| Daviey | Hi, is zm03.openstack.org poorly? | 15:10 |
| *** ParsectiX has joined #openstack-infra | 15:10 | |
| mordred | jeblair: done | 15:10 |
| *** reed has joined #openstack-infra | 15:10 | |
| DuncanT | sdague: anteaya Ok, sounds like this is something we're agreeing on being a good thing. I'll start playing in the sandbox and build up a plan. Thanks | 15:10 |
| *** jasondotstar has quit IRC | 15:10 | |
| pabelanger | jeblair, docaedo. pleia2: I also have a elastic-recheck patchset up, if you want to review: https://review.openstack.org/#/c/195283/ | 15:10 |
| sdague | mriedem: I just did | 15:11 |
| mriedem | k | 15:11 |
| anteaya | DuncanT: sure, still curious how you got the impression infra had a comment template requirement, if you can find a link or recall | 15:11 |
| sdague | mriedem: https://review.openstack.org/#/c/195606/ | 15:11 |
| *** ghostpl has joined #openstack-infra | 15:11 | |
| Daviey | Anyone else seeing: 2015-06-25 14:15:08.382 | + git fetch http://zm03.openstack.org/p/openstack-dev/devstack refs/zuul/stable/juno/Zfb644759416341898b53cad0bd8255b5 | 15:11 |
| Daviey | 2015-06-25 14:15:08.621 | fatal: Couldn't find remote ref refs/zuul/stable/juno/Zfb644759416341898b53cad0bd8255b5 | 15:11 |
| sdague | anteaya: well I did beat up nova ci people that weren't posting success / fail | 15:11 |
| jeblair | Daviey: http://cacti.openstack.org/cacti/graph_view.php?action=tree&tree_id=1&leaf_id=134 | 15:11 |
| jeblair | Daviey: looks normal | 15:11 |
| sdague | because it was super confusing | 15:11 |
| anteaya | DuncanT: the only thing infra does require is a third party ci has to have a wikipage linked from the third party systems page and the comment has to contain a link to the ci wikipage | 15:12 |
| *** sabeen has quit IRC | 15:12 | |
| anteaya | sdague: ah okay well thanks? | 15:12 |
| *** puranamr has quit IRC | 15:12 | |
| fungi | Daviey: do you have a reason to believe that ref should exist? | 15:12 |
| anteaya | sdague: and having some sort of useful info in the comment is useful | 15:12 |
| *** pc_m has joined #openstack-infra | 15:13 | |
| Daviey | fungi: well, the check job thinks it should.. http://logs.openstack.org/90/194290/1/check/check-grenade-dsvm-partial-ncpu/6b10808/logs/devstack-gate-setup-workspace-new.txt.gz | 15:13 |
| *** shardy has quit IRC | 15:13 | |
| fungi | Daviey: the way devstack-gate works is by trying to fetch that ref, then seeing if it works or assuming there isn't one if it fails | 15:13 |
| sdague | anteaya: yes, it is | 15:13 |
| *** shardy has joined #openstack-infra | 15:13 | |
| sdague | I think DuncanT's idea is even more useful, especially if we can get it to be a defacto standard | 15:13 |
| anteaya | sdague: I have no argument with that | 15:14 |
| jeblair | what's DuncanT's idea? | 15:14 |
| anteaya | I not a fan of considering it a requirement | 15:14 |
| fungi | Daviey: if the check wasn't for a devstack patch to stable/juno then there probably won't be a zuul ref matching that (unless it's got a depends-on to a devstack change or is stacked in the gate pipeline with a devstack change ahead of it) | 15:14 |
| anteaya | more highly suggested | 15:14 |
| sdague | jeblair: also reporting some kind of "phase" of where the failure happened | 15:14 |
| *** yamamoto has joined #openstack-infra | 15:14 | |
| Daviey | fungi: ah | 15:14 |
| sdague | so at a glance you can figure out that it never made it to tests | 15:15 |
| *** mtanino_ has joined #openstack-infra | 15:15 | |
| DuncanT | jeblair: Reporting various stages of build failure as something other than 'failed', particularly straight infrastructure fails | 15:15 |
| fungi | Daviey: specifically, look for whether it successfully fetched zuul refs corresponding to the repo and branch of teh change being tested | 15:15 |
| *** pc_m_ has joined #openstack-infra | 15:15 | |
| jeblair | as a substitute for reading logs? | 15:15 |
| *** mtanino has quit IRC | 15:16 | |
| fungi | Daviey: unfortunately, git doesn't have a ref-exists feature, so we're stuck trying to fetch it and then filling that log with lots of git error messages | 15:16 |
| *** arxcruz has quit IRC | 15:16 | |
| DuncanT | jeblair: As an at-a-glance indication... reading the logs defacto doesn't happen much in cinder land, since the failures are 99% infrastructure (pip failed, node failed to come up, etc) | 15:17 |
| fungi | DuncanT: filtering out "infrastructure fails" (whatever that even means) from other sorts of problems is not necessarily straightforward, regardless of whether you know what "phase" it failed in | 15:17 |
| Daviey | fungi: ok, thanks! | 15:17 |
| anteaya | regardless of how much we want devs to read logs, they aren't reading them until the patch fails 2 or 3 times | 15:18 |
| *** ayoung is now known as ayoung-afk | 15:18 | |
| DuncanT | fungi: I can certainly tell on my install if a) I got a working node b) devstack cloned and installed it's dependencies c) devstack started d) tests failed | 15:18 |
| *** pc_m has quit IRC | 15:18 | |
| fungi | DuncanT: so ignoring teh logs if the job failed in the "setup phase" or whatever isn't necessarily a great idea | 15:18 |
| *** dannywilson has joined #openstack-infra | 15:18 | |
| john-davidge | On a similar subject, there’s a discussion on the dev mailer about whether we should reomve 3rd party ci voting | 15:19 |
| DuncanT | fungi: Currently people mostly ignore third party CI full stop | 15:19 |
| openstackgerrit | Paul Belanger proposed openstack-infra/system-config: Do not merge https://review.openstack.org/195288 | 15:19 |
| fungi | john-davidge: yes, i saw that. you started that thread, right? | 15:19 |
| jeblair | DuncanT: what about not reporting those failures then? | 15:19 |
| jeblair | DuncanT: (those failures == failures in a,b,c) | 15:20 |
| john-davidge | fungi: I did, it seemed like such a good idea when we were discussing it in vancouver but it never went anywhere | 15:20 |
| DuncanT | jeblair: I thought about that, but it makes rechecks messy | 15:20 |
| jeblair | how so? | 15:20 |
| *** dannywil_ has joined #openstack-infra | 15:20 | |
| jeblair | (the very first third-party system never reported a failure without human inspection because of this sort of thing) | 15:20 |
| *** dannywilson has quit IRC | 15:20 | |
| DuncanT | jeblair: I vote fail for whatever reason, somebody does a recheck, my fail is still sat there on the review | 15:21 |
| fungi | yeah, smokestack ran for some years with danprince personally inspecting each and every failure result | 15:21 |
| DuncanT | We've had months of continuous failures from some CIs :-( | 15:22 |
| jeblair | i don't think you should tolerate that :) | 15:22 |
| DuncanT | The ultimate response in one case was to propose a patch to remove the driver | 15:22 |
| fungi | sounds like you have a lot more than one case where you need to do that | 15:22 |
| DuncanT | jeblair: We've just in the last month got better about not tolerating it... most CI maintainers (eventually) responded to pokes | 15:23 |
| sdague | honestly, I'd personally find this pretty helpful for upstream CI as well, because when I'm trying to figure out patterns of what's going on, this would extract something that's taking a bunch of manual time and clicking through a lot of things | 15:23 |
| jeblair | anyway, don't get me wrong -- more information is great; i'm just not personally sure how i would expect someone to use that extra bit of information. it doesn't seem as cut and dry as we would like. | 15:23 |
| DuncanT | One didn't, the driver removal patch had one +2 last time I looked | 15:23 |
| dougwig | we should state up front that automatic rechecks aren't great, and we should be reading logs, and we should be fixing the failures... BUT, that ignores the reality that we're being trained to ignore failures and issue rechecks like a drunk monkey, simply because we're too busy to help chase things outside of our change, or don't feel like we can effect those | 15:23 |
| dougwig | things. the proposal to recheck on some failures is just a proposal to let computers do what humans are already doing (with added wasted time), until we get a handle on those issues. | 15:23 |
| openstackgerrit | melanie witt proposed openstack-infra/project-config: Make tempest-dsvm-cells job voting https://review.openstack.org/190894 | 15:24 |
| jeblair | wait, somoene is talking about automatic rechecks? | 15:24 |
| sdague | and, honestly, half the time I just have timetables for test jobs in my head, so I use the failure run time to have a good guess of where things failed without looking at the logs | 15:24 |
| sdague | like, 14s on a devstack job, that was a git clone fail for devstack-gate | 15:24 |
| john-davidge | dougwig: Taking 3rd party CI out of the voting could end a lot of the recheck problems | 15:25 |
| dougwig | jeblair: i was jumping to that conclusion for build setup failures that are most likely transient, yes. | 15:25 |
| DuncanT | sdague: I am spending more time trying to check 3rd party results for things, and finding several failure modes (pip failures, devstack git failures) that I'd like to have seen at a glance | 15:25 |
| krtaylor | DuncanT, one of our primary focus areas for Third Party CI WG is to improve trust in CI systems, but it is tough when CI systems come online that don't work in the first place, or have a team that supports them | 15:25 |
| john-davidge | I think if jenkins is the only CI giving -1s then they’ll be given more importance | 15:25 |
| openstackgerrit | Fabien Boucher proposed openstack-infra/puppet-openstackci: Add generic zuul manifests https://review.openstack.org/175970 | 15:25 |
| *** _diana_ has quit IRC | 15:25 | |
| sdague | DuncanT: yeh, definitely understandable | 15:26 |
| krtaylor | voting is not the problem, commenting is already ignored | 15:26 |
| DuncanT | Anyway, it doesn't sound like anybody hates the idea, so I'll work on it a bit and come back with an example | 15:26 |
| DuncanT | Work from there | 15:26 |
| dougwig | there's an issue with the failure rate of even the infra CI that is affecting dev behavior. | 15:26 |
| *** laughterwym_ has quit IRC | 15:27 | |
| jeblair | dougwig: i'm very strongly opposed to automatic rechecks. | 15:27 |
| krtaylor | ++ | 15:27 |
| *** DericHorn-HP has joined #openstack-infra | 15:27 | |
| dougwig | jeblair: i know, but they're already happen. it's just mechnical turk automation. | 15:27 |
| dougwig | happening, i mean. | 15:27 |
| *** mpavlase has joined #openstack-infra | 15:27 | |
| *** sergmelikyan has joined #openstack-infra | 15:28 | |
| *** mtanino has joined #openstack-infra | 15:28 | |
| *** _diana_ has joined #openstack-infra | 15:29 | |
| jeblair | dougwig: it's still a choice a human needs to make; automating rechecks would mean that the overall accuraccy continues to decrease until systems are running continuously to try to randomly post a success result | 15:29 |
| anteaya | dougwig: I think you are distracting from the point you made to me yesterday | 15:29 |
| jeblair | the underlying problem is unreliability in the systems themselves; that's what needs to be improved | 15:29 |
| jeblair | and if systems are unsatisfactory, we should not expose them to developers | 15:30 |
| anteaya | which is similar to DuncanT you would gain benefit from knowing what stage teh failure occured | 15:30 |
| jeblair | this is a choice any project can choose to make | 15:30 |
| openstackgerrit | Julia Kreger proposed openstack-infra/shade: WIP: Locking ironic API microversion https://review.openstack.org/195634 | 15:30 |
| *** _diana_ has quit IRC | 15:31 | |
| *** Swami has joined #openstack-infra | 15:31 | |
| *** mtanino_ has quit IRC | 15:31 | |
| sdague | mriedem: are you happy with my removing the sqla cap? | 15:31 |
| jeblair | and yeah, i'm not opposed to reporting failure phase, i'm just not sure what i would expect someone to do with that information | 15:31 |
| DuncanT | jeblair: I'm very much of the opinion that the difficulty people had getting CI to work at all says a great deal about what people's development priorities ought to be | 15:32 |
| mriedem | sdague: have to look again | 15:32 |
| sdague | because you see the comment before that line, which is *never* cap this, then it's capped | 15:32 |
| mriedem | sdague: schooling a guy in -nova atm | 15:32 |
| fungi | i'm not opposed to automatically restarting a job if it fails in a certain way, as long as that's 1. nor triggered by some bot adding comments to the review, and 2. the interim failures aren't reported to the change at all | 15:32 |
| DuncanT | jeblair: Openstack is not actually very robust at all... and I don't think many vendors realise that | 15:32 |
| mriedem | sdague: yeah, bknduson pointed that out | 15:32 |
| sdague | mriedem: I'm fine just deleting the cap | 15:32 |
| sdague | and leave the version uplift for later | 15:32 |
| jeblair | DuncanT: indeed :/ | 15:32 |
| *** sergmelikyan has quit IRC | 15:32 | |
| dougwig | anteaya: true, but i'm commenting on the use of that info. today, many devs will issue a recheck and not look at logs until then see a second or third failure. if we report build failures, we may get them to not recheck if it happened in the test phase, but the knee jerk rechecks on other failures will increase. | 15:32 |
| *** mpavlase has quit IRC | 15:33 | |
| mriedem | sdague: well, i figured it should just be SQLAlchemy>=0.9.7,<1.1.0 | 15:33 |
| mriedem | like in g-r | 15:33 |
| mriedem | regardless of what the comment says | 15:33 |
| DuncanT | I think if there's more info in the failure message, then it is easier to (mentally) extract patterns - I already know which cinder CIs I tend to ignore .v. the ones I check the logs for | 15:33 |
| fungi | dougwig: but if the failures can be classed so precisely as to be able to recheck them without looking at the logs, then perhaps have the ci automatically restart that job rather than reporting the result | 15:33 |
| dougwig | jeblair: right, i agree with everything you've said, but still find myself rechecking auomatically when i'm busy/stressed, because it pays off more often than not. | 15:34 |
| mriedem | sdague: b/c as we know, uncapped deps in libs can creep things that should be capped into gate runs and blow shit up | 15:34 |
| john-davidge | dougwig: I’m not sure it’s true that devs don’t check logs until the second recheck. For me it’s more that the failure is obviously unrelated to my patch, so I recheck. | 15:34 |
| anteaya | fungi: actually the usecase I think dougwig is talking about is devs not cis | 15:34 |
| DuncanT | fungi: That I think is stage 2... stage one is to classify them better | 15:34 |
| *** devvesa has quit IRC | 15:34 | |
| dougwig | fungi: totally 1000% agree, though some reporting of the failure probably needs to occur for maintainers to track issues/trends that need attention. | 15:34 |
| sdague | mriedem: we've had way more explodes in the gate since capping because things build incompatible states | 15:35 |
| sdague | libraries should not cap | 15:35 |
| sdague | applications are fine to | 15:35 |
| *** e0ne is now known as e0ne_ | 15:35 | |
| *** _diana_ has joined #openstack-infra | 15:36 | |
| mriedem | sdague: so when nova has capped sqla and requires sqla-migrate with uncapped sqla, how is that not a problem? | 15:36 |
| *** med___ is now known as med_ | 15:36 | |
| *** med_ has quit IRC | 15:36 | |
| *** med_ has joined #openstack-infra | 15:36 | |
| mriedem | won't sqla-migrate pull in latest sqla which nova is saying it doesn't want to use? | 15:36 |
| *** dizquierdo has quit IRC | 15:36 | |
| sdague | no | 15:36 |
| mriedem | i'm behind on the brave new world | 15:36 |
| sdague | it only changes the requirements when they mismatch | 15:37 |
| fungi | john-davidge: as far as not letting any third-party ci leave a verified +1/-1 vote on changes to your project, the list of accounts allowed to do that is under the control of your project-team. if you want to (collectively) make that an empty group, it's entirely in your power | 15:37 |
| *** e0ne_ is now known as e0ne | 15:37 | |
| sdague | pip install -e ./nova | 15:37 |
| openstackgerrit | Fabien Boucher proposed openstack-infra/system-config: Use openstackci module for zuul_merger zuul_prod and zuul_dev https://review.openstack.org/176010 | 15:38 |
| fungi | john-davidge: i would say that if you're not getting reliable enough results from a third-party ci on your project to trust in the accuracy of its vote, then that's a reason to take it out of the voting group. and if that means they all get taken out of that group for the same reason, so be it | 15:38 |
| *** ajmiller has joined #openstack-infra | 15:38 | |
| sdague | will do the right things, it will only trigger it if the library requires a range higher than nova required | 15:38 |
| sdague | because with caps in libraries now we're in this state where we need every library in the universe to push a release before we can use new software, because they are force downgrading stuff out from under us. | 15:39 |
| john-davidge | fungi: Great, thanks. I think this is the best way to move forward | 15:39 |
| krtaylor | fungi, and further, if they are not that reliable, and that decision is being made in the projects, maybe they shouldn't be commenting either | 15:39 |
| fungi | john-davidge: i gather you're talking about neutron. the group of accounts allowed to leave those votes on neutron changes is managed at https://review.openstack.org/#/admin/groups/510,members | 15:40 |
| *** mtanino has quit IRC | 15:40 | |
| *** annegentle has joined #openstack-infra | 15:40 | |
| *** hichihara has quit IRC | 15:41 | |
| fungi | john-davidge: and it's under the control of the members of https://review.openstack.org/#/admin/groups/150,members (currently mestery and dougwig) | 15:41 |
| *** bexelbie has quit IRC | 15:41 | |
| john-davidge | krtaylor: Personally I have no problem with them commenting. The multiple review emails can be easily filtered, and the results could be important to some. It’s the voting which can cause patches to go un-reviewed. | 15:41 |
| *** bexelbie has joined #openstack-infra | 15:41 | |
| mriedem | sdague: +W on ps2 | 15:42 |
| *** bexelbie has quit IRC | 15:42 | |
| sdague | mriedem: thanks | 15:42 |
| sdague | if we could get a release with that, it would be awesome | 15:42 |
| john-davidge | fungi: thanks, will ping kyle over in the neutron channel | 15:42 |
| mriedem | sdague: sure | 15:42 |
| krtaylor | john-davidge, I disagree, comments are then just ignored, always, if they cant be trusted | 15:43 |
| dougwig | jeblair: please don't get me wrong, the system is pretty damn good right now. the comment from DuncanT about slightly finer-grained failure status is an interesting step. | 15:43 |
| *** mkoderer_cloud has quit IRC | 15:44 | |
| sdague | mordred: so... I clearly am missing something with pbr with https://review.openstack.org/#/c/195292/ because when I try to use it in keystone here - https://review.openstack.org/#/c/195575/ I don't get any scripts installed | 15:44 |
| sdague | hints would be appreciated | 15:44 |
| sdague | I've got a manually setup environment locally to ensure every tree is at the right level | 15:44 |
| *** jasondotstar has joined #openstack-infra | 15:45 | |
| john-davidge | krtaylor: removing the comments though would stop anyone from being able to see the results at all, and for patches where the 3rd party results are relevant that would be a problem | 15:46 |
| john-davidge | krtaylor: Most of the time the comments are ignored, yes. But at least comments are easier to ignore than a -1 | 15:47 |
| *** marcusvrn1 has joined #openstack-infra | 15:47 | |
| *** marcusvrn has quit IRC | 15:47 | |
| *** dboik has quit IRC | 15:47 | |
| *** dboik has joined #openstack-infra | 15:48 | |
| dhellmann | sdague: you're looping over a string on line 255 | 15:49 |
| krtaylor | john-davidge, we are arguing different points, I am saying if the CI system is not working properly, it should not comment and should be removed until it proves otherwise | 15:49 |
| krtaylor | it should not continue to comment in hopes it shows something useful | 15:49 |
| sdague | dhellmann: doh, thank you | 15:50 |
| *** sdake_ has joined #openstack-infra | 15:50 | |
| mordred | sdague: I will now poke at that | 15:50 |
| dougwig | krtaylor: agreed. | 15:50 |
| krtaylor | john-davidge, my opinion is that comments should not be ignored, ever | 15:50 |
| john-davidge | krtaylor: Ah I see. I think I agree with you, but if i remember correctly infra no longer enforces whether a 3rd party ci is “working” | 15:50 |
| openstackgerrit | Sean Dague proposed openstack-dev/pbr: WIP: first sketch of wsgi_scripts support https://review.openstack.org/195292 | 15:50 |
| krtaylor | john-davidge, we are working to fix that with a CI monitoring dashboard | 15:51 |
| fungi | i always love it when i accidentally do that. ['y', 'o', 'u', ' ', 'f', 'a', 'i', 'l'] | 15:51 |
| dhellmann | sdague: while you're in there, you could make line 241 more obvious by adding an explicit list there instead of relying on the implicit tuple | 15:51 |
| sdague | dhellmann: yes, once I test that it works, I will | 15:51 |
| john-davidge | krtaylor: sounds like a nice addition | 15:51 |
| * mordred needs to make a docker container with pip stuff bootstrapped | 15:51 | |
| dhellmann | sdague: your update is still going to evaluate to a list because the () are superfluous. So you want ('wsgi_scripts',) or just remove the loop entirely | 15:51 |
| dhellmann | s/list/string | 15:52 |
| *** annegentle has quit IRC | 15:52 | |
| sdague | dhellmann: oh, right | 15:52 |
| dhellmann | sdague: it feels weird to loop over one thing, so I'd just drop that | 15:52 |
| sdague | dhellmann: you want the other tuple change in the same patch, or separate, as it's kind of a different concern | 15:52 |
| sdague | dhellmann: that's fair | 15:52 |
| dhellmann | sdague: I'm ok with it being in one patch since it's so small. | 15:52 |
| *** annegentle has joined #openstack-infra | 15:52 | |
| *** maishsk_afk has joined #openstack-infra | 15:53 | |
| *** ParsectiX has quit IRC | 15:53 | |
| *** sdake has quit IRC | 15:54 | |
| *** unicell has quit IRC | 15:54 | |
| *** arxcruz has joined #openstack-infra | 15:54 | |
| sdague | dhellmann: ok, so I'm going to do a gorpy loop to just figure out if the rest of this works. Then I'll extract some things to make it more sane before it goes un WIP | 15:54 |
| dhellmann | sdague: wfm | 15:54 |
| openstackgerrit | Sean Dague proposed openstack-dev/pbr: WIP: first sketch of wsgi_scripts support https://review.openstack.org/195292 | 15:55 |
| *** e0ne is now known as e0ne_ | 15:56 | |
| *** vnod has joined #openstack-infra | 15:57 | |
| *** sshnaidm has quit IRC | 15:57 | |
| openstackgerrit | Jeremy Stanley proposed openstack-infra/project-config: Fetch all heads and tags when caching source repos https://review.openstack.org/195650 | 15:57 |
| *** _diana_ has quit IRC | 15:57 | |
| fungi | gerrit's getting slow again. i'll see if our problem user has moved to a new ip address today | 15:58 |
| *** pc_m_ has quit IRC | 15:58 | |
| *** sshnaidm has joined #openstack-infra | 15:58 | |
| *** bexelbie_ has quit IRC | 15:58 | |
| *** pc_m has joined #openstack-infra | 15:58 | |
| *** Ala has quit IRC | 15:58 | |
| *** armax_ has joined #openstack-infra | 15:59 | |
| *** Longgeek has quit IRC | 15:59 | |
| jeblair | dhellmann: have a few mins to talk about the tagging spec? | 16:00 |
| *** Longgeek has joined #openstack-infra | 16:00 | |
| *** sergmelikyan has joined #openstack-infra | 16:00 | |
| jeblair | fungi: let me know if you need help | 16:00 |
| fungi | jeblair: at this point it's probably just cut-n-paste if it's the same situation we had earlier in the week | 16:01 |
| fungi | but will let you know | 16:01 |
| fungi | also need to skip out on some lunch errands momentarily | 16:01 |
| *** armax has quit IRC | 16:01 | |
| *** armax_ is now known as armax | 16:01 | |
| anteaya | fungi: happy skipping | 16:03 |
| fungi | this may be something different. over 1000 new connections (we're averaging a little over 8/sec) only 11% were from any single source ip address | 16:03 |
| *** Sukhdev has joined #openstack-infra | 16:03 | |
| *** yamamoto has quit IRC | 16:03 | |
| * anteaya pictures fungi follicking down the sidewalk | 16:03 | |
| mordred | sdague: it's the list thing | 16:03 |
| openstackgerrit | Sean M. Collins proposed openstack-infra/project-config: Graduate Linux Bridge job from experimental status https://review.openstack.org/195653 | 16:03 |
| mordred | sdague: you were looping over the characters in teh string | 16:03 |
| fungi | load average on gerrit is approaching 10 though | 16:04 |
| fungi | seems to maybe be dropping again now | 16:04 |
| *** sergmelikyan has quit IRC | 16:04 | |
| fungi | but the java process for gerrit was occupying basically 100% of all available vcpus | 16:05 |
| *** scheuran has quit IRC | 16:05 | |
| mordred | sdague: http://paste.openstack.org/show/321037/ works | 16:05 |
| sdague | yeh, my new thing works, I put it in a tupple | 16:06 |
| *** harlowja_still_a has quit IRC | 16:06 | |
| *** sabeen1 has joined #openstack-infra | 16:06 | |
| *** e0ne_ has quit IRC | 16:06 | |
| sdague | now I'm just trying to build apache rules to pair with it correctly | 16:06 |
| openstackgerrit | Sean M. Collins proposed openstack-infra/project-config: Graduate Linux Bridge job from experimental status https://review.openstack.org/195653 | 16:07 |
| sdague | so I can figure out how wonky pants this is all going to look like if we pull it all together | 16:07 |
| pleia2 | good morning | 16:08 |
| mordred | sdague: WONKY PANTS! | 16:09 |
| openstackgerrit | Matthew Treinish proposed openstack-infra/subunit2sql: Do not use oslo.db 1.12.0 https://review.openstack.org/195656 | 16:09 |
| anteaya | morning pleia2 | 16:10 |
| anteaya | apparently it is wonky pants day | 16:10 |
| anteaya | extra points for skipping | 16:10 |
| mordred | anteaya: wonky skipping? | 16:11 |
| *** sergmelikyan has joined #openstack-infra | 16:11 | |
| *** e0ne has joined #openstack-infra | 16:11 | |
| anteaya | mordred: not sure how to score that | 16:11 |
| anteaya | but sounds awesome | 16:11 |
| *** samueldmq has quit IRC | 16:12 | |
| *** spredzy is now known as spredzy|afk | 16:12 | |
| *** ParsectiX has joined #openstack-infra | 16:13 | |
| *** sergmelikyan has quit IRC | 16:16 | |
| dhellmann | if I create a stable branch for a library, does any part of the name matter other than the "stable/" at the front? for example, I may want to backport a fix to oslo.db and I have other unreleased changes that we shouldn't release today, so if I created a stable/1.12 branch and released 1.12.1 from it, would all of the release machinery work right? | 16:16 |
| fungi | dhellmann: i believe when we tried that last time, the main problem is that our integration testsuite wants to test against like-named branches | 16:17 |
| fungi | so you'll end up with your stable/1.12 branch cross-tested against master | 16:17 |
| *** armax has quit IRC | 16:17 | |
| *** maishsk_afk has quit IRC | 16:17 | |
| *** annegentle has quit IRC | 16:17 | |
| *** RichardRaseley has joined #openstack-infra | 16:17 | |
| fungi | since no other projects have a stable/1.12 branch to test that with | 16:17 |
| *** mpavone_ has quit IRC | 16:18 | |
| dhellmann | fungi: in this case I think that would be ok, since we're just exploring this as an option to avoid reverting some of the potentially breaking changes just to release a fix into master | 16:19 |
| dhellmann | fungi: but yes, that's the thing I couldn't remember the details about -- I knew names were special | 16:19 |
| *** geoffarnold has joined #openstack-infra | 16:19 | |
| fungi | dhellmann: then that sounds like what we did with pbr, though we did feature/nnn instead of stable/nnn (for other reasons i think?) | 16:19 |
| fungi | clarkb: jeblair: ^ do you recall the details from that? | 16:20 |
| dhellmann | fungi: we could go with feature/ too, I suppose. Maybe when there's less pressure, we can make bug/ work as a prefix for quick-fixes like this. | 16:20 |
| *** fhubik is now known as fhubik_afk | 16:20 | |
| *** DericHorn-HP has quit IRC | 16:20 | |
| dhellmann | where we aren't claiming long-term stable support, just trying to make a bug-fix release | 16:21 |
| jeblair | fungi: nak on details | 16:21 |
| *** maishsk_afk has joined #openstack-infra | 16:21 | |
| *** armax has joined #openstack-infra | 16:21 | |
| clarkb | fungi because feature/* is tested as if it were master | 16:22 |
| fungi | oh, right, we explicitly treat feature/* as master | 16:22 |
| jeblair | would that not be true for stable123? | 16:22 |
| jeblair | stable/123 | 16:22 |
| *** sarob has joined #openstack-infra | 16:22 | |
| crc32 | http://pastebin.com/TWdE6szX <-- something in devstack is mangling keystones retuirements and test-requirments.txt files after it clones them into /opt/stack/keystone but this is breaking my deployment because an erronous "python_version='2.7' is literally just getting shoved in test-requirments. Whats more is that when I do a git blame for this git shows these errored out line. How do I fix this or atleast stop devstack from injecting broken | 16:22 |
| *** geoffarnold has quit IRC | 16:23 | |
| *** afazekas has quit IRC | 16:23 | |
| clarkb | jeblair no, stable/123 is invalid and tests just fail | 16:23 |
| clarkb | because of the features.yaml file in d-g | 16:23 |
| fungi | crc32: that comes from devstack's explicit update from the openstack/requirements repo's global-requirements.txt | 16:23 |
| jeblair | oh, it's the test-matrix | 16:24 |
| fungi | crc32: in what way is that version marker erroneous? | 16:24 |
| *** geoffarnold has joined #openstack-infra | 16:24 | |
| fungi | crc32: are you running with a pip release prior to 6.0? | 16:24 |
| crc32 | pip 7.0.3 | 16:25 |
| *** esker has joined #openstack-infra | 16:25 | |
| *** esker has quit IRC | 16:25 | |
| jeblair | fungi, clarkb, dhellmann: i wonder if we should support 'stable/kilo/1.2.3' as an option | 16:25 |
| fungi | that's new enough it should handle markers in a requirements list just fine | 16:25 |
| *** sdake_ is now known as sdake | 16:25 | |
| *** esker has joined #openstack-infra | 16:25 | |
| jeblair | (for later; it sounds like feature/1.2.3 is the thing we can do right now) | 16:26 |
| crc32 | it added the line "python-ldap>=2.4;python_version=='2.7'" which breaks the install as shown in the pastbin | 16:26 |
| sdague | mordred: ok, this looks like it's doable, the apache config isn't too gross. But a follow up question, is pbr aware of "share" directories in any way? Because most of the packaging I've seen has been putting these files in /usr/share/keystone/wsgi/ as a standard model. And it would be awesome if pbr understood that as a place it could install to. | 16:26 |
| dhellmann | jeblair: in this particular case, we were trying to release a patch off of master, so it's not actually a stable release I'm just looking at how things work to see if there's a way to do what we want | 16:26 |
| fungi | jeblair: i'm not opposed to amending the test matrix for some additional patterns, noting that there won't be additional cross-testing between extended stable branches when there are some across multiple repos | 16:26 |
| jeblair | dhellmann: yep; i'm just spidering out to other ideas :) | 16:26 |
| dhellmann | jeblair: ok, cool | 16:26 |
| crc32 | like that ";" should be a '\n\ or something fungi | 16:26 |
| sdague | going to go get a bike ride now, will poke again later, and get a less gorpy pbr patch up | 16:27 |
| mordred | sdague: well, I'm not sure it's the right thing to do to put them in to shade if they are things you can just execute | 16:27 |
| anteaya | sdague: enjoy the ride | 16:27 |
| fungi | crc32: no, ; is the marker separator in a pip requirements list | 16:27 |
| dhellmann | jeblair, fungi : what would it take to make "bug/" work like "feature/"? if it's a lot of work, I can live with "feature/" for now, but if it's easy it would be nice to be a bit more descriptive | 16:27 |
| jeblair | dhellmann: sounds like a small patch to test-matrix.py in devstack-gate | 16:28 |
| mordred | sdague: also - I'm not sure pip can install things into share dirs ... dstufft ? it's bin and lib yeah? | 16:28 |
| dhellmann | jeblair: ok, I'll take a closer look after I get back from the store | 16:28 |
| fungi | crc32: the error in your paste looks like it might be coming from setuptools | 16:28 |
| crc32 | fungi "rror in setup command: 'tests_require' must be a string or list of strings containing valid project/version requirement specifiers; Expected ',' or end-of-list in python-ldap>=2.4;python_version=='2.7' at ;python_version=='2.7'" so this isn't the error that broke the deployment? | 16:28 |
| mordred | fungi: "error in setup command: 'tests_require' must be a string or list of strings containin" | 16:28 |
| fungi | crc32: are you running with an old setuptools release? | 16:28 |
| mordred | fungi: there was a bug for that that lifeless pushed a fix for | 16:28 |
| dstufft | mordred: not generally, not yet anyways. I mean there are ways but it'll break in common situations | 16:29 |
| jeblair | dhellmann: if you want to avoid doc builds for those branches, you may want to either adjust run-docs.sh in project config, or configure zuul not to run docs jobs on bug/ branches | 16:29 |
| mordred | dstufft: well, we don't want to do that | 16:29 |
| dhellmann | jeblair: noted | 16:29 |
| *** bswartz has quit IRC | 16:29 | |
| mordred | fungi, crc32: they're _not_ supported in tests_require yet - so update.py is _supposed_ to be stripping them for that last I understood | 16:29 |
| *** erlon has joined #openstack-infra | 16:29 | |
| *** markus_z has quit IRC | 16:29 | |
| fungi | mordred: oh, well in setup.py the marker separator is : instead of ; | 16:30 |
| fungi | which may be related to the error here | 16:30 |
| fungi | and i thought we were automatically translating those | 16:30 |
| *** rossella_s has quit IRC | 16:30 | |
| crc32 | so how do I get this to deploy? | 16:30 |
| jeblair | dhellmann: there may also be a potential change to project-requirements-change for completeness (in case requirements.txt is changed on bug/1.2.3) | 16:30 |
| *** rossella_s has joined #openstack-infra | 16:31 | |
| *** achanda has joined #openstack-infra | 16:31 | |
| jeblair | dhellmann: (that's also in project-config; 'git grep feature' in project-config and devstack-gate will probably catch everything) | 16:31 |
| fungi | crc32: is it bug 1468808 | 16:31 |
| openstack | bug 1468808 in devstack "stack.sh downgrades pbr" [Undecided,New] https://launchpad.net/bugs/1468808 | 16:31 |
| mordred | fungi: I also thought we were translating those - but also - something still sounds out of date in crc32's env, else we'd be seeing this _everywhere_ | 16:31 |
| *** mdenny has joined #openstack-infra | 16:31 | |
| fungi | i wonder if that's related to the recent request to remove the pbr cap from... was it sqlalchemy-migrate? | 16:31 |
| *** ghostpl has quit IRC | 16:31 | |
| fungi | anyway, i have to run some lunch errands. bbiab | 16:32 |
| *** alivigni has quit IRC | 16:32 | |
| *** cdent has quit IRC | 16:32 | |
| openstackgerrit | greghaynes proposed openstack-infra/nodepool: Trigger image builds using gearman https://review.openstack.org/195175 | 16:32 |
| openstackgerrit | greghaynes proposed openstack-infra/nodepool: Run geard for nodepool tests https://review.openstack.org/194928 | 16:32 |
| crc32 | I'm rm -rf /opt/stack to try to start over so I don't know what is out of date in my environ | 16:33 |
| *** sshnaidm has quit IRC | 16:33 | |
| *** sshnaidm has joined #openstack-infra | 16:33 | |
| ajmiller | crc32 I'm having the same problem, and just tried rerunning stack.sh after rm -rf /opt/stack, same result. | 16:33 |
| ajmiller | Also fetch/rebased devstack | 16:34 |
| *** Swami has quit IRC | 16:34 | |
| crc32 | I'm creating a vm to see if its broken in a clean environment as well. | 16:34 |
| *** radez_g0n3 is now known as radez | 16:38 | |
| *** ivar-lazzaro has joined #openstack-infra | 16:39 | |
| *** flepied has joined #openstack-infra | 16:40 | |
| *** derekh has quit IRC | 16:40 | |
| *** fhubik_afk is now known as fhubik | 16:40 | |
| *** yamahata has quit IRC | 16:40 | |
| *** otter768 has joined #openstack-infra | 16:42 | |
| *** unicell has joined #openstack-infra | 16:43 | |
| openstackgerrit | Arkadiy Kraminsky proposed openstack-infra/puppet-nodepool: Adds pip dependencies https://review.openstack.org/195675 | 16:43 |
| dtantsur | quick ques folks, ironic inspector devstack job is failing with "c-api didn't start" | 16:44 |
| dtantsur | is it some known problem? http://logs.openstack.org/31/174231/12/check/check-ironic-inspector-dsvm-nv/f365610/logs/devstacklog.txt.gz | 16:44 |
| *** shardy has quit IRC | 16:45 | |
| *** shardy has joined #openstack-infra | 16:45 | |
| *** ashleighfarnham has joined #openstack-infra | 16:45 | |
| *** dangers_away is now known as dangers | 16:47 | |
| *** otter768 has quit IRC | 16:47 | |
| *** maishsk_afk has quit IRC | 16:48 | |
| *** puranamr has joined #openstack-infra | 16:48 | |
| dtantsur | tbh I don't even see c-api enabled in http://logs.openstack.org/31/174231/12/check/check-ironic-inspector-dsvm-nv/f365610/logs/localrc.txt.gz Oo | 16:50 |
| dtantsur | why does it try to start? | 16:50 |
| openstackgerrit | Arkadiy Kraminsky proposed openstack-infra/puppet-zuul: add notification for existing pidfile https://review.openstack.org/194449 | 16:50 |
| crc32 | so is there a work around for https://bugs.launchpad.net/devstack/+bug/1468808 I mean do I just manually install pbr .6 or something? | 16:51 |
| openstack | Launchpad bug 1468808 in devstack "stack.sh downgrades pbr" [Undecided,Confirmed] | 16:51 |
| *** pelix has quit IRC | 16:52 | |
| *** rwsu has joined #openstack-infra | 16:52 | |
| *** Swami has joined #openstack-infra | 16:52 | |
| *** ramashri has joined #openstack-infra | 16:53 | |
| anteaya | dtantsur: I'm not sure which service is bringing in cinder api | 16:53 |
| anteaya | you have swift enabled so I would think that you wouldn't need cinder | 16:54 |
| *** maishsk_afk has joined #openstack-infra | 16:54 | |
| *** Krinkle_ is now known as Krinkle | 16:54 | |
| dtantsur | anteaya, I would say so too :) but how do I affect it? | 16:54 |
| anteaya | but I can try to dig through the devstack code to see what is bringing it in for that job | 16:54 |
| *** ghostpl has joined #openstack-infra | 16:54 | |
| greghaynes | mordred: Hey - how do we test our rax DIB images? Did we add an experimental job somewhere? | 16:54 |
| anteaya | uncertain at the moment, I have to start with the job definition and work backwards | 16:54 |
| * anteaya looks up files | 16:54 | |
| dtantsur | anteaya, thanks! | 16:55 |
| anteaya | dtantsur: you are welcome to play too | 16:55 |
| greghaynes | mordred: specifically, the glean fix that should be in the newly built images | 16:55 |
| *** bdemers has joined #openstack-infra | 16:55 | |
| *** jasondotstar has quit IRC | 16:55 | |
| dtantsur | anteaya, I remember last time I had to enable cinder services on my local devstack btw.. | 16:55 |
| dtantsur | i.e. not "enable", but "stop disabling" | 16:56 |
| openstackgerrit | Michael Krotscheck proposed openstack-infra/project-config: Added eslint-config-openstack https://review.openstack.org/194938 | 16:56 |
| SpamapS | shade patchbomb day yaaay :) | 16:56 |
| *** bdemers_ has joined #openstack-infra | 16:56 | |
| mordred | greghaynes: yeah - I think we add an experimental job that runs a devstack job on one of them | 16:56 |
| dtantsur | IIRC if I tried to disable cinder services, it still tried to start c-api and failed | 16:56 |
| anteaya | dtantsur: interesting | 16:57 |
| anteaya | dtantsur: sounds like that is related | 16:57 |
| greghaynes | mordred: hrm, having a hard time finding... it should be for ubuntu-trusty node type | 16:57 |
| *** ParsectiX has quit IRC | 16:57 | |
| SpamapS | mordred: would it be evil to suggest a shade REST API that deployers could add so non-python programs could have one way to talk to OpenStack? ;) | 16:57 |
| greghaynes | oh my | 16:58 |
| dtantsur | anteaya, here's what I ended up using: https://github.com/openstack/ironic-inspector/blob/master/CONTRIBUTING.rst#example-localconf (note that cinder services are not disabled) | 16:58 |
| greghaynes | if we need a rest api for our rest api's... | 16:58 |
| anteaya | dtantsur: looks like n-cpu might bring cinder with it by default http://logs.openstack.org/31/174231/12/check/check-ironic-inspector-dsvm-nv/f365610/logs/devstacklog.txt.gz#_2015-06-25_16_36_39_711 | 16:58 |
| mordred | SpamapS: I won't say I haven't thougt about that | 16:58 |
| mordred | SpamapS: I also won't say it's not evil | 16:58 |
| *** cindyo has quit IRC | 16:58 | |
| anteaya | dtantsur: how did this job get into your check queue if it doesn't work? | 16:59 |
| dtantsur | anteaya, hmm looks like "cinder" is enabled, while everything "c-*" is disabled.. | 16:59 |
| dtantsur | anteaya, sorry, I'm not sure I get you question. This is the first time this job ever run.. | 16:59 |
| anteaya | I don't see cinder or any c- services mentioned in your local.conf at all | 17:00 |
| SpamapS | greghaynes: http://cdn.meme.am/instances2/500x/449152.jpg | 17:00 |
| *** bdemers has quit IRC | 17:00 | |
| anteaya | dtantsur: then it should have started out in the experimental queue, not be in the check queue | 17:00 |
| *** bdemers_ has quit IRC | 17:00 | |
| *** AzherKhan has quit IRC | 17:00 | |
| greghaynes | SpamapS: I think you just named the project - xzibit | 17:00 |
| *** AzherKhan has joined #openstack-infra | 17:00 | |
| anteaya | dtantsur: the experimental queue is the place to figure out job build issues like this one | 17:00 |
| anteaya | dtantsur: anyway | 17:01 |
| SpamapS | mordred: we can write our first non-python client in rust. | 17:01 |
| anteaya | dtantsur: so it appears you need to experiment with explictly stating cinder is not to be run in your set-up | 17:01 |
| SpamapS | because zomg rust! | 17:01 |
| dtantsur | anteaya, not sure how my set up is special here. It looks a bug in devstack that cinder can be half-enabled.. | 17:02 |
| anteaya | dtantsur: it looks like tracking that down is your next step to getting your job to build | 17:03 |
| *** yamamoto has joined #openstack-infra | 17:03 | |
| *** dboik has quit IRC | 17:03 | |
| greghaynes | jeblair: fungi either of you know what the job we made is to test our rax dib images? | 17:03 |
| openstackgerrit | Merged openstack-infra/shade: Update keypair APIs to latest standards https://review.openstack.org/195318 | 17:04 |
| dtantsur | anteaya, year, I realize. ok thanks, will play with devstack locally again (tomorrow, as it's evening now). | 17:04 |
| *** mwagner_afk has quit IRC | 17:04 | |
| anteaya | dtantsur: okay, have a good evening | 17:04 |
| openstackgerrit | Merged openstack-infra/shade: Fix AttributeError in keystone functional tests https://review.openstack.org/195451 | 17:04 |
| *** jrniemijr has joined #openstack-infra | 17:04 | |
| *** ghostpl has quit IRC | 17:05 | |
| greghaynes | oh hey - http://nodepool.openstack.org/dib.ubuntu-trusty.log the ubuntu-trusty image build just failed | 17:05 |
| *** DericHorn-HP has joined #openstack-infra | 17:05 | |
| *** esker has quit IRC | 17:05 | |
| openstackgerrit | Merged openstack-infra/shade: Skip Swift functional tests if needed https://review.openstack.org/195452 | 17:05 |
| jeblair | greghaynes, fungi: looks like something with 'dibtest' in the job name | 17:05 |
| greghaynes | ImportError: No module named oslo_context - oslo rename? | 17:06 |
| greghaynes | in subunit2sql; | 17:06 |
| openstackgerrit | Kurt Taylor proposed openstack-infra/infra-specs: Host a CI monitoring dashboard service https://review.openstack.org/194437 | 17:06 |
| jeblair | greghaynes, fungi: except there are no jobs in layout.yaml with dibtest in their names | 17:06 |
| openstackgerrit | Merged openstack-infra/shade: Do not use environment for Swift unit tests https://review.openstack.org/195474 | 17:06 |
| greghaynes | jeblair: aha! | 17:07 |
| jeblair | mtreinish: is it possible to avoid using oslo in subunit2sql? | 17:07 |
| *** dtantsur is now known as dtantsur|afk | 17:07 | |
| greghaynes | jeblair: theres branch-designator: '-dibtest' | 17:07 |
| *** yamahata has joined #openstack-infra | 17:07 | |
| *** jasondotstar has joined #openstack-infra | 17:08 | |
| *** achanda has quit IRC | 17:08 | |
| *** patrickeast has joined #openstack-infra | 17:08 | |
| *** e0ne has quit IRC | 17:08 | |
| jeblair | greghaynes: yeah, i think the jobs exist, but i'm not sure they are specified to be triggered in zuul | 17:09 |
| *** yamamoto has quit IRC | 17:09 | |
| greghaynes | jeblair: awesome | 17:09 |
| jeblair | mtreinish: aha, that's even a TODO item, good :) | 17:09 |
| wznoinsk | krtaylor: I was wondering if you use nagios to monitor third party CI systems | 17:10 |
| anteaya | wznoinsk: you heard from fungi that infra does not monitor third party ci systems | 17:11 |
| mriedem | sdague: your migrate change just doesn't feel like every running through jenkins | 17:11 |
| mriedem | *ever | 17:11 |
| krtaylor | wznoinsk, yes, we do | 17:12 |
| *** sergmelikyan has joined #openstack-infra | 17:12 | |
| anteaya | wznoinsk: or use nagios | 17:12 |
| jeblair | mtreinish: subunit2sql has broken image builds, we need to do something about that asap. let me know when you have a minute. | 17:12 |
| krtaylor | wznoinsk, and there are some scripts for nagios in the third party ci tools repo | 17:12 |
| mtreinish | jeblair: yeah it's a TODO item, it was an early mistake | 17:12 |
| *** vnod has quit IRC | 17:12 | |
| mtreinish | but something that's become more and more difficult to remove over time | 17:12 |
| mtreinish | jeblair: the fix should be: https://review.openstack.org/195656 | 17:12 |
| mtreinish | I'll push a release once that lands | 17:13 |
| krtaylor | wznoinsk, https://github.com/stackforge/third-party-ci-tools | 17:13 |
| mtreinish | which should unblock the image builds | 17:13 |
| krtaylor | wznoinsk, see monitoring | 17:13 |
| krtaylor | wznoinsk, feel free to try them and patch them to make them better | 17:14 |
| jeblair | mtreinish: great, thanks; let me know when that's released | 17:14 |
| mtreinish | yep, will do | 17:14 |
| fungi | anteaya: yeah, openstack infra doesn't monitor third-party ci systems at all, nor use nagios to monitor anything at all, but some third-party ci operators may be monitoring the health of their own systems with nagios or something similar | 17:15 |
| krtaylor | wznoinsk, as anteaya said, upstream infra does not, but we (as in powerkvm ci) do, let me know if you have any questions on setting it up | 17:15 |
| krtaylor | fungi said it better :) | 17:16 |
| fungi | i almost called it netsaint. been around that block too many times i guess | 17:16 |
| anteaya | fungi: yes | 17:17 |
| krtaylor | fungi, ever though about using nagios for infra services? has that been discussed? I guess it could report problems here | 17:17 |
| fungi | krtaylor: it comes up about once every 3 months on average, i think | 17:18 |
| wznoinsk | krtaylor: I have a nagios script and was planning on more but I'll wait till the discussion about importance and -/+1 verified rights for 3rd party CIs will evolve on ML, I have a neutron CI, may still develop some more nagios checks for the compute ci... | 17:18 |
| krtaylor | wznoinsk, good to know | 17:18 |
| *** tonytan4ever has joined #openstack-infra | 17:18 | |
| fungi | krtaylor: the general counterargument is that we don't have any admins on "pager duty" and people tend to let us know pretty quickly when something is down/misbehaving | 17:19 |
| fungi | and if nobody's noticed, it's probably not urgent (yet) | 17:19 |
| *** ParsectiX has joined #openstack-infra | 17:19 | |
| pleia2 | trying to figure out script placement for a manual zanata script, can I get some infra brains weighing in? https://review.openstack.org/#/c/195421/ | 17:19 |
| wznoinsk | I've heard the emails are ignored too... been there, done that... it's a matter on keeping your monitoring/alerting system reliable | 17:19 |
| fungi | and we'd rather spend time fixing things so that they won't break, than reacting to constant alerts that may or may not be false positives | 17:19 |
| *** sdake_ has joined #openstack-infra | 17:20 | |
| *** yamahata has quit IRC | 17:20 | |
| *** yamahata has joined #openstack-infra | 17:20 | |
| krtaylor | fungi, understood | 17:21 |
| *** cindyo has joined #openstack-infra | 17:21 | |
| *** notnownikki has quit IRC | 17:21 | |
| *** sambetts has quit IRC | 17:21 | |
| *** sshnaidm has quit IRC | 17:21 | |
| *** sergmelikyan has quit IRC | 17:21 | |
| wznoinsk | in my opinion there's always a way to monitor something the reliable way... it's only in sometimes (rarely) the check oversizes the potential issue it would alert early about | 17:21 |
| *** sshnaidm has joined #openstack-infra | 17:22 | |
| fungi | where "fixing things so that they won't break" means redesigning to be more fault tolerant rather than simply being reactive | 17:22 |
| wznoinsk | fungi: I would agree if a quickly changing evnrionment | 17:23 |
| wznoinsk | s/if/in | 17:23 |
| fungi | ultimately a monitoring system is just one more thing to manage/tune/fix | 17:23 |
| fungi | so can be more of a time eater than a time saver | 17:23 |
| *** sdake has quit IRC | 17:24 | |
| wznoinsk | fungi: yes, but there's always issues that you would better have monitoring system to discover them early | 17:24 |
| fungi | we already have trending data we record so that we can use it to perform post-mortem analyses and make decisions based on that, which has usually been good enough for our needs | 17:24 |
| *** sambetts has joined #openstack-infra | 17:24 | |
| *** achanda has joined #openstack-infra | 17:25 | |
| wznoinsk | honestly, an issue with hanging stream-event is a perfect example of a simple (nagios) check notifications shouldn't be ignored on | 17:25 |
| *** shardy has quit IRC | 17:25 | |
| *** abregman has quit IRC | 17:25 | |
| *** bradjones has quit IRC | 17:26 | |
| *** shardy has joined #openstack-infra | 17:26 | |
| wznoinsk | when you have a good framework, checks are not that big to create | 17:26 |
| fungi | wznoinsk: we've not said we reject the notion of a monitoring system. it's just that so far nobody's been willing to put in the effort to design the deployment, sign up to receive the alerts, and perform the ongoing tuning to keep it in shape | 17:26 |
| wznoinsk | cause nobody likes to be hated ;-) | 17:27 |
| fungi | a few times people have indicated that they might be willing to take that on, but in the end they find they'd rather work on other things | 17:27 |
| *** _diana_ has joined #openstack-infra | 17:27 | |
| openstackgerrit | Kurt Taylor proposed openstack-infra/infra-specs: Host a CI monitoring dashboard service https://review.openstack.org/194437 | 17:28 |
| fungi | and this is a free software community where people decide what they want to spend their time on, rather than a company where people get duties assigned to them by a manager | 17:28 |
| *** bradjones has joined #openstack-infra | 17:28 | |
| *** bradjones has quit IRC | 17:28 | |
| *** bradjones has joined #openstack-infra | 17:28 | |
| *** ihrachyshka has quit IRC | 17:29 | |
| *** sdake_ is now known as sdake | 17:29 | |
| * mordred assigns duties | 17:31 | |
| mordred | people laugh at me | 17:31 |
| *** marun has joined #openstack-infra | 17:32 | |
| *** puranamr has quit IRC | 17:33 | |
| fungi | huh, infra manual changes aren't getting uploaded post merge. checking into it now | 17:33 |
| *** sergmelikyan has joined #openstack-infra | 17:34 | |
| *** jasondotstar has quit IRC | 17:34 | |
| *** yamamoto has joined #openstack-infra | 17:34 | |
| *** ihrachyshka has joined #openstack-infra | 17:34 | |
| *** marcusvrn1 has quit IRC | 17:34 | |
| *** sshnaidm has quit IRC | 17:35 | |
| fungi | http://logs.openstack.org/3d/3de11333d852eaf5189bbe4309567fbb3acbed13/post/infra-manual-infra-docs/574cea7/console.html.gz looks like it worked... | 17:35 |
| AJaeger | fungi: last update on "Fri 19 Jun 2015 09:27:29 PM CEST" for http://docs.openstack.org/infra/manual/core.html | 17:36 |
| *** sshnaidm has joined #openstack-infra | 17:36 | |
| AJaeger | fungi: isn't that the last change? | 17:36 |
| *** puranamr has joined #openstack-infra | 17:36 | |
| AJaeger | fungi: last merge at "Date: Fri Jun 19 19:00:50 2015 +0000" | 17:36 |
| fungi | AJaeger: it is, but i don't see the content the change added at http://docs.openstack.org/infra/manual/developers.html#peer-review | 17:36 |
| AJaeger | So, this looks like it worked fine. | 17:36 |
| anteaya | mordred: when was the last time you assigned me a duty? | 17:37 |
| * anteaya tries to remember | 17:37 | |
| *** ParsectiX has quit IRC | 17:37 | |
| AJaeger | fungi: Indeed, that is missing | 17:37 |
| fungi | AJaeger: note the change that merged adds list entry #7 and unless my brower caching has gone sideways i only see up to #6 in the published version | 17:37 |
| openstackgerrit | Arkadiy Kraminsky proposed openstack-infra/puppet-zuul: add notification for existing pidfile https://review.openstack.org/194449 | 17:38 |
| *** sambetts has quit IRC | 17:38 | |
| AJaeger | fungi: yes, confirmed. | 17:38 |
| AJaeger | fungi: Let's approve https://review.openstack.org/#/c/193309/ and see what happens ;) | 17:39 |
| fungi | AJaeger: though http://docs.openstack.org/infra/manual/_sources/developers.txt has it | 17:39 |
| *** sergmelikyan has quit IRC | 17:39 | |
| *** jasondotstar has joined #openstack-infra | 17:39 | |
| *** isviridov_away is now known as isviridov | 17:39 | |
| *** ddieterly has joined #openstack-infra | 17:39 | |
| *** sambetts has joined #openstack-infra | 17:40 | |
| *** mwagner_afk has joined #openstack-infra | 17:40 | |
| *** dboik has joined #openstack-infra | 17:40 | |
| *** puranamr_ has joined #openstack-infra | 17:40 | |
| fungi | AJaeger: and the docs-draft version that rendered for it during gating looks correct too | 17:40 |
| *** ihrachyshka has quit IRC | 17:41 | |
| openstackgerrit | Merged openstack-infra/shade: Add Neutron/Nova Floating IP delete (i.e. deallocate from project) https://review.openstack.org/183637 | 17:41 |
| openstackgerrit | Merged openstack-infra/shade: Fix available_floating_ip when using Nova network https://review.openstack.org/195476 | 17:41 |
| *** john-davidge has quit IRC | 17:41 | |
| *** samueldmq has joined #openstack-infra | 17:41 | |
| *** ParsectiX has joined #openstack-infra | 17:41 | |
| fungi | AJaeger: the rendered page is a couple changes behind actually, and also says "Last updated on Wed Jun 17 13:25:23 2015, commit f48e8b7." in the footer | 17:42 |
| fungi | i have a feeling we're not overwriting the html correctly/at all | 17:42 |
| *** AzherKhan has quit IRC | 17:43 | |
| jeblair | fungi: the last-modified-date corresponds with the upload | 17:43 |
| *** dguerri is now known as dguerri` | 17:43 | |
| *** puranamr has quit IRC | 17:43 | |
| fungi | jeblair: which last-modified-date? | 17:43 |
| fungi | jeblair: are you looking with an ftp client? | 17:44 |
| AJaeger | fungi: strange, http://logs.openstack.org/3d/3de11333d852eaf5189bbe4309567fbb3acbed13/post/infra-manual-infra-docs/574cea7/console.html.gz#_2015-06-19_19_28_16_541 uploads the file | 17:44 |
| jeblair | fungi: no, http. and both txt and html. Last-Modified | 17:44 |
| jeblair | Fri, 19 Jun 2015 19:27:30 GMT | 17:44 |
| jeblair | that ^ is for the .html | 17:45 |
| *** SumitNaiksatam has joined #openstack-infra | 17:45 | |
| *** AzherKhan has joined #openstack-infra | 17:45 | |
| fungi | interesting | 17:45 |
| AJaeger | fungi: I checked with ftp - changed on Fri 19th | 17:46 |
| fungi | so we're updating the file with an old copy? | 17:46 |
| AJaeger | but changed at "Fri, Jun 19 2015 15:27:00" | 17:46 |
| *** ramashri has quit IRC | 17:47 | |
| AJaeger | 4 hour difference looks strange to me | 17:47 |
| jeblair | AJaeger: do you have a timezone for that? | 17:48 |
| AJaeger | fungi: the change before https://review.openstack.org/#/c/192665/ was merged together with the other one | 17:48 |
| AJaeger | jeblair: might be | 17:48 |
| jeblair | oh, there could have been an upload race then | 17:48 |
| fungi | oh, i suppose it's possible we raced | 17:48 |
| fungi | yeah | 17:48 |
| AJaeger | that would explain it, indeed. | 17:48 |
| AJaeger | So, any reviewers for https://review.openstack.org/#/c/193309/ to fix the situation? ;) | 17:49 |
| fungi | though that doesn't completely explain the footer, unless it's dated for something other than when the job ran | 17:49 |
| *** afazekas has joined #openstack-infra | 17:49 | |
| *** fhubik has quit IRC | 17:50 | |
| fungi | AJaeger: lgtm. can't hurt to get some more data points on this | 17:50 |
| AJaeger | pleia2: are you around to chat about Zanata - or shall we tomorrow? | 17:50 |
| jeblair | AJaeger, fungi: oh look, the footer lists the commit, and it lists the _other_ commit | 17:50 |
| *** puranamr_ has quit IRC | 17:51 | |
| jeblair | pleia2: (thanks for that footer!) | 17:51 |
| AJaeger | jeblair: it does indeed. | 17:51 |
| jeblair | AJaeger, fungi: the date on the footer is the commit date for the specified commit | 17:51 |
| jeblair | AJaeger, fungi: (and that matches too) | 17:51 |
| fungi | jeblair: yeah, that checks out | 17:51 |
| *** puranamr has joined #openstack-infra | 17:51 | |
| fungi | so race explains this i think | 17:52 |
| AJaeger | so, indeed a race ;( | 17:52 |
| fungi | we have eventually consistent documentation :/ | 17:52 |
| AJaeger | ;) | 17:52 |
| *** marcusvrn has joined #openstack-infra | 17:53 | |
| *** ayoung-afk is now known as ayoung | 17:53 | |
| *** boris-42 has joined #openstack-infra | 17:54 | |
| fungi | looks like in a few minutes it will get in line behind the 9-hour post pipeline backlog | 17:54 |
| *** afazekas has quit IRC | 17:55 | |
| AJaeger | 9h post pipeline? Wow ;( | 17:56 |
| fungi | the hpcloud outage and subsequent token caching issue put a substantial hurt on us | 17:56 |
| fungi | i hope that now that things are picking up steam we'll catch back up fairly quickly | 17:57 |
| *** dimtruck is now known as zz_dimtruck | 17:57 | |
| fungi | zuul's at about a 2k job backlog at the moment | 17:57 |
| fungi | and looks like we're averaging ~600/hr | 17:58 |
| *** fallenpegasus has quit IRC | 17:58 | |
| fungi | so maybe 4 hours if everything holds and we don't get a ton of new changes dumped in | 17:58 |
| jeblair | fungi: how did the hpcloud thing get resolved? | 18:00 |
| *** ivar-lazzaro has quit IRC | 18:00 | |
| *** ramashri has joined #openstack-infra | 18:00 | |
| *** fitoduarte has joined #openstack-infra | 18:00 | |
| fungi | jeblair: unclear... it's possible my manual token delete worked but had a 10+ minute delay. it's also possible it just happened to expire on its own | 18:01 |
| *** hdd has quit IRC | 18:01 | |
| fungi | i don't _think_ my deletes worked, because they always returned a bad token error | 18:01 |
| *** hdd has joined #openstack-infra | 18:01 | |
| mordred | it's also possible that bunny rabbits danced around a fire and summoned cthulu | 18:01 |
| fungi | but this _is_ openstack, after all | 18:01 |
| jeblair | or mordred assigned duties | 18:02 |
| fungi | yes, cthulhu and the other great old ones may have smiled upon our unworthy souls | 18:02 |
| *** husanu91 has joined #openstack-infra | 18:02 | |
| * mordred assigned anteaya the duty of being a bunny rabbit dancing around the fire ... she declined | 18:02 | |
| *** dontalton has joined #openstack-infra | 18:02 | |
| jeblair | and laughed, i imagine | 18:02 |
| *** bswartz has joined #openstack-infra | 18:02 | |
| anteaya | I'll take that task actually | 18:03 |
| * anteaya dances at the sprint | 18:03 | |
| *** Sukhdev has quit IRC | 18:03 | |
| crc32 | I have a better question. How do I disable "setup_package_with_req_sync" since its breaking our deploys by manlging requirments.txt and test-requirements.txt in keystone? | 18:03 |
| AJaeger | where's the video? | 18:03 |
| anteaya | that was fun thanks | 18:03 |
| anteaya | and I am laughing too | 18:03 |
| AJaeger | ;) | 18:04 |
| anteaya | :) | 18:04 |
| fungi | crc32: i think there's an envvar you can pass with devstack to tell it not to update requirements lists | 18:04 |
| anteaya | so nibalizer looking at http://logs.openstack.org/46/195246/1/check/gate-infra-puppet-apply-centos6/e97bc19/console.html#_2015-06-25_17_23_10_558 | 18:04 |
| fungi | crc32: though then you'll get all the inconsistencies between requirements lists of various repos | 18:04 |
| anteaya | nibalizer: how do we feel about removing the puppet apply centos job? | 18:04 |
| anteaya | nibalizer: or at least making it non-voting | 18:05 |
| crc32 | fungi: Its better then ";python_version='2.8'' #KABOOM" | 18:05 |
| fungi | crc32: have you confirmed that the cause is an old pbr release? what pbr version is ending up in your env? | 18:05 |
| nibalizer | anteaya: do you know what our pass/fail rate is? | 18:05 |
| anteaya | nibalizer: no I do not | 18:05 |
| anteaya | I just know that I recheck when it fails and recommend others to do that same | 18:05 |
| *** gema has joined #openstack-infra | 18:05 | |
| mordred | yah. I'm concerned that this is failing for crc32 - because I would expect it to work since it works in the gate | 18:05 |
| *** jasondotstar has quit IRC | 18:05 | |
| crc32 | fungi no in my comment on the bug I confirm there is a bug breaking devstack deploys but I don't conclude its due tto downgrading pbr. | 18:05 |
| nibalizer | I think yolanda and pabelanger we're going to look at that | 18:05 |
| nibalizer | I'm a bit distracted at this exact moment | 18:06 |
| mordred | and I don't understand why it would be only broken for crc32 | 18:06 |
| gema | hi, can someone advice me on how to get the openstack meeting bot in a new channel? | 18:06 |
| *** enikanorov2 has quit IRC | 18:06 | |
| anteaya | nibalizer: understood | 18:06 |
| crc32 | morded its also broken for 3 others that I know of. | 18:06 |
| anteaya | yolanda and pabelanger do share your thoughts on the pass/fail rate of the puppet-apply centos job | 18:06 |
| fungi | crc32: the error you're getting should be fixed by http://git.openstack.org/cgit/openstack-dev/pbr/commit/?id=4b1503849c33cb9b6c0c0cb8b3322204d396249e which is in pbr 1.2.0 so that does sound consistent with something pinning the pbr version to <1.0 | 18:06 |
| anteaya | I'm interested in your findings | 18:06 |
| nibalizer | anteaya: I don't think i have the authority to say make it un-voting | 18:07 |
| anteaya | nibalizer: who does? | 18:07 |
| crc32 | https://bugs.launchpad.net/devstack/+bug/1468808 <-- should this be refiled since I don't believe its related to pbr being smacked own to 0.11.0 | 18:07 |
| openstack | Launchpad bug 1468808 in devstack "stack.sh downgrades pbr" [Undecided,Confirmed] | 18:07 |
| nibalizer | 'real' infra-cores | 18:07 |
| *** Sukhdev has joined #openstack-infra | 18:07 | |
| *** ildikov has quit IRC | 18:07 | |
| nibalizer | jeblair: clarkb jhesketh fungi mordred | 18:07 |
| fungi | crc32: also it's not being caused (any longer at least) by keystonemiddleware since latest keystonemiddleware doesn't insist on a lower pbr version... unless something else is capping the keystonemiddleware releases it will accept | 18:08 |
| nibalizer | really the centos test is much less important than the ubuntu test, since 90+% of our hosts are ubuntu | 18:08 |
| *** cindyo has quit IRC | 18:08 | |
| *** esker has joined #openstack-infra | 18:08 | |
| anteaya | nibalizer: yes, hence why I bring up the subject | 18:08 |
| nibalizer | the things that run centos are the git farm, so if we watch out for changes that affect centos we can probably scrape by without the centos test | 18:08 |
| sdake | is the gate misbehaving at the moment? | 18:08 |
| *** maishsk_afk has quit IRC | 18:09 | |
| *** zz_dimtruck is now known as dimtruck | 18:09 | |
| Clint | gema: http://docs.openstack.org/infra/system-config/irc.html | 18:09 |
| fungi | sdake: hpcloud apparently had a severe outage earlier, so we're playing catch-up with a sizable backlog now | 18:09 |
| crc32 | fungi: I know it is being cause by the addition of "lpython_version='1.7' so as a work around how do I kill the "setup_package_with_req_sync" behavior | 18:09 |
| sdake | fungi thans | 18:09 |
| sdake | fungi thanks | 18:09 |
| sdake | fungi my jobs are probably going to time out :( | 18:09 |
| *** maishsk_afk has joined #openstack-infra | 18:10 | |
| gema | Clint: thanks, I was thinking that maybe there is an instance out there that everybody uses | 18:10 |
| crc32 | as a LOL I chattr +i test-requirments.txt and requirments.txt and poof update.py complains about permisions. | 18:10 |
| *** liam__ has joined #openstack-infra | 18:11 | |
| fungi | crc32: yeah, i'm hunting through the devstack source right now to see if i can spot the toggle for that | 18:11 |
| Clint | gema: http://git.openstack.org/cgit/openstack-infra/system-config/tree/modules/openstack_project/manifests/eavesdrop.pp | 18:11 |
| *** liam__ has quit IRC | 18:11 | |
| anteaya | nibalizer: I'm fine with leaving them if the pass rate is high, but in my subjective experience it isn't really | 18:12 |
| anteaya | nibalizer: but I will await/find data | 18:12 |
| gema | Clint: thanks, so if I go through the steps on the system-config page openstack bot will appear in the channel | 18:12 |
| gema | right? | 18:12 |
| nibalizer | i"m okay with it going nonvoting, im in favor of it actually, but I'm not sure my authority reaches that far | 18:12 |
| clarkb | I am all for bumping up to centos7 it has just been low priority and will likely break shallow clones | 18:13 |
| *** puranamr has quit IRC | 18:13 | |
| clarkb | but I would expect that to make git hapoy as centos6 git is ancient | 18:13 |
| gema | Clint: oh, no, I should propose a change to that file to add the channel | 18:14 |
| gema | Clint: ta! | 18:14 |
| *** husanu91 has quit IRC | 18:14 | |
| anteaya | nibalizer: 53% failure rate in the last 48 hours | 18:14 |
| fungi | crc32: i thought there was an option to disable the requirements sync, but it looks like we only have a var for turning on soft updates so that it doesn't complain about additional entries missing from the global list. you would probably have to fork the setup_package_with_req_sync function or find a way to pin to an older openstack/requirements commit, unless there's an option i'm missing there | 18:14 |
| lifeless | crc32: fungi: mordred: tests_require' must be a string or list of strings -> old pbr | 18:14 |
| fungi | lifeless: yep, something is downgrading pbr in some peoples' ci systems under devstack | 18:15 |
| anteaya | nibalizer: http://tinyurl.com/pjhwvgz | 18:15 |
| lifeless | yes | 18:15 |
| lifeless | fungi: python-mistralclient is one such case | 18:15 |
| crc32 | I just did "export REQUIREMENTS_MODE="BUZZOFF" | 18:15 |
| lifeless | I think we fundamentally just need to chase the logs | 18:15 |
| crc32 | no joy | 18:15 |
| lifeless | crc32: projects that have merged the markers won't be affected either way by changing the requirements mode | 18:16 |
| *** puranamr has joined #openstack-infra | 18:16 | |
| lifeless | running with constraints should also fix this | 18:16 |
| fungi | lifeless: yeah, in this case we need to ask people to provide logs | 18:16 |
| crc32 | what is this marker your talking about? | 18:16 |
| crc32 | you mean the actual ";python_version='2.7'" is that a marker? Cause if so thats being added by devstack and not in the actual project. | 18:16 |
| lifeless | http://lists.openstack.org/pipermail/openstack-dev/2015-June/067823.html | 18:17 |
| *** fallenpegasus has joined #openstack-infra | 18:17 | |
| lifeless | crc32: its in some projects in git already | 18:17 |
| *** enikanorov2 has joined #openstack-infra | 18:17 | |
| fungi | crc32: it's being added because it's in http://git.openstack.org/cgit/openstack/requirements/tree/global-requirements.txt and projects will be adding those to their local requirements lists rsn | 18:17 |
| crc32 | ok so tjeres semicolon vs colon which is it? | 18:18 |
| lifeless | crc32: do you have a log ofthe failing run ? | 18:18 |
| crc32 | http://pastebin.com/TWdE6szX | 18:18 |
| crc32 | heres one | 18:18 |
| lifeless | crc32: depends on where it is; in setup.cfg or a text file | 18:18 |
| fungi | crc32: specifically we want to see all of the pip install output leading up to the error | 18:18 |
| crc32 | http://pastebin.com/TWdE6szX | 18:18 |
| lifeless | crc32: *all* the log. | 18:18 |
| lifeless | crc32: devstack installs pbr 1.2. Then something triggers a downgrade. Then you see this error. | 18:19 |
| fungi | crc32: are you using devstack-gate? i think we specifically need the setup logs so we can see what's downgrading pbr | 18:19 |
| wznoinsk | krtaylor: becaume my main focus is something else openstack I wouldn't be able to commit to driving any monitoring initiatives, I'm happy to be the other half if someone from infa/third party carers would decide to go with that and use Nagios for it | 18:19 |
| *** jasondotstar has joined #openstack-infra | 18:19 | |
| *** sergmelikyan has joined #openstack-infra | 18:20 | |
| crc32 | generating the failed log now. | 18:20 |
| wznoinsk | <- fungi too | 18:21 |
| ajmiller | BTW, I have exactly reproduced the failures crc32 is seeing | 18:21 |
| *** pblaho has quit IRC | 18:21 | |
| crc32 | I do a clone of devstack then run ./stack.sh | 18:21 |
| *** AzherKhan has quit IRC | 18:22 | |
| clarkb | fungi lifeless if this is an existing install pbr wont be upgraded right? | 18:23 |
| clarkb | could it just be old and stale? | 18:23 |
| crc32 | http://pastebin.com/h4ZQLah7 <-- jheres the failed stack run | 18:23 |
| dhellmann | jeblair: do you have a few minutes to chat about this release review spec? I think we're pretty close and I'd like to cut out a few rounds of back-and-forth through gerrit if we can. | 18:24 |
| wznoinsk | fungi: krtaylor having said that I could go and create a poc of a nagios system with 1-2 checks of your choice, to me the Nagios part is not the problem, the real questino is actually are there any fans of Nagios in infra that would take it and maintain it long run | 18:24 |
| fungi | wznoinsk: it's less a poc that we would need, it's the configuration management and automation around deploying it | 18:25 |
| *** sergmelikyan has quit IRC | 18:25 | |
| fungi | and also teh volunteers to subscribe to the information and fine-tune it | 18:25 |
| lifeless | clarkb: install_infra triggers pip install -U pbr | 18:25 |
| lifeless | clarkb: so pbr will indeed be upgraded | 18:25 |
| lifeless | my first attempt at solving this was to add a second pbr upgrade to the point where udpate.py is called | 18:26 |
| lifeless | but it was felt that that was too big a hammer | 18:26 |
| wznoinsk | I think your internal issues should be monitored first, then you could think of expanding... for start it would be strict infra team then - people that can actually do something to resolve the issue, i.e.: if you're after gerrit daily... checks for it would be alerting you (+1 more person for resilience) | 18:26 |
| fungi | lifeless: yeah, looks like it's keystonemiddleware perhaps, but i don't see why since keystonemiddleware's current release doesn't have the pbr!=0.7,<1.0,>=0.6 constraint mentioned | 18:27 |
| wznoinsk | fungi: but the version of keeystonemiddleware that's already installed (which also meeting the requirements <=>) may have? | 18:27 |
| openstackgerrit | Arkadiy Kraminsky proposed openstack-infra/puppet-nodepool: Adds pip dependencies https://review.openstack.org/195675 | 18:27 |
| fungi | wznoinsk: the people with that access are generally disinterested in receiving alerts about failures | 18:27 |
| *** julim has quit IRC | 18:28 | |
| fungi | wznoinsk: until someone else has confirmed things are actually broken | 18:28 |
| lifeless | crc32: that log is odd - it shows pbr 1.2 being installed, and then jumps with no more references to pbr, to 0.11 being installed | 18:28 |
| openstackgerrit | Merged openstack-infra/shade: Remove list_keypair_dicts method https://review.openstack.org/195616 | 18:28 |
| wznoinsk | fungi: I was there, 10+ years in infra tought me a bit, it's a matter of getting the alerts right and keep on tunning them | 18:28 |
| lifeless | crc32: are you using a pip proxy of some sort? bandersnatch mirror / devpi / pip-mirror ? | 18:28 |
| *** annegentle has joined #openstack-infra | 18:28 | |
| *** julim has joined #openstack-infra | 18:28 | |
| crc32 | no pip proxy | 18:28 |
| wznoinsk | fungi: agree, meaningful checks only | 18:29 |
| lifeless | ajmiller: ok; is it a shared env with crc32 or a separate case? If separate please get me your log | 18:29 |
| crc32 | crc@bitchslap:~/workspace/curr/devstack$ pip list | grep -E "(pip|pbr|setuptools)" | 18:29 |
| crc32 | pbr (0.11.0) | 18:29 |
| crc32 | pip (7.0.3) | 18:29 |
| crc32 | setuptools (18.0.1) | 18:29 |
| fungi | wznoinsk: yes, i spent more than 10 years as a slave to monitoring systems as well. i no longer work in a noc at a service provider and am glad for that | 18:29 |
| crc32 | I find it odd too cause after everything is down I'm at 0.11.0 | 18:29 |
| ajmiller | lifeless I'm on my own system | 18:29 |
| wznoinsk | if I'd take your stream-event as an example, a check to find stream-even (not) growing, queue growing or number of processes waiting would be the check you'd want | 18:30 |
| wznoinsk | fungi: if you're the receiver of the check you're creating you do it properly ;-) | 18:30 |
| lifeless | crc32: I think you might have a slightly stale devstack | 18:30 |
| lifeless | crc32: can you pull the latest master in please | 18:30 |
| crc32 | I actually keep recloning devstack | 18:31 |
| *** DericHorn-HP has quit IRC | 18:31 | |
| lifeless | crc32: ok, so please pastebin 'git show | head' | 18:31 |
| *** krtaylor has quit IRC | 18:31 | |
| fungi | lifeless: i'm wondering if today's devstack change to stop syncing requirements for projects not listed in openstack/requirements:projects.txt is the cause... | 18:31 |
| lifeless | crc32: because - that log showed the bug with pbr only being upgraded in the requirements venv | 18:31 |
| *** sshnaidm has quit IRC | 18:32 | |
| crc32 | what .vend I want to blow that directory away too. Where is it? | 18:32 |
| lifeless | crc32: which was fixed in 7ebe8e0751dab545091e0b114589087009cc4e22 | 18:32 |
| lifeless | crc32: its not your problem | 18:32 |
| *** amitgandhinz has quit IRC | 18:32 | |
| lifeless | crc32: please - the git show | 18:32 |
| lifeless | ajmiller: ok, so lets get your log as well | 18:33 |
| wznoinsk | fungi: re deployment youre right, but it would be less of a problem i fyou have deployment system in place already | 18:33 |
| lifeless | fungi: so this is a recent explosion of error reports? | 18:33 |
| ajmiller | lifeless I'm generating a fresh one now | 18:33 |
| fungi | lifeless: seems like in the past 4 hours? | 18:33 |
| lifeless | ajmiller: thanks | 18:33 |
| lifeless | fungi: ok so yeah, thats new | 18:33 |
| crc32 | crc@bitchslap:~/workspace/curr/devstack$ git show | head | 18:33 |
| crc32 | commit c782781a4ca5fc2671a6e5d188f68e99e7a656f4 | 18:33 |
| crc32 | Merge: 065fdf2 e123830 | 18:33 |
| crc32 | Author: Jenkins <jenkins@review.openstack.org> | 18:33 |
| crc32 | Date: Thu Jun 25 12:11:38 2015 +0000 | 18:33 |
| fungi | lifeless: it's possible that was a safety valve where we have devstack installing some projects which should be in projects.txt but aren't | 18:34 |
| pleia2 | AJaeger: sorry, had a call, still around? | 18:34 |
| *** sshnaidm has joined #openstack-infra | 18:35 | |
| lifeless | so, all the stuff I changed to do constraints mode | 18:35 |
| lifeless | deleted ? | 18:35 |
| lifeless | I am -so- confused | 18:35 |
| pleia2 | AJaeger: happy to just chat about translations plans tomorrow, I'll gather some thoughts today prior to | 18:36 |
| lifeless | ah 7af8a1b9b3180da54e2c9505228ad722db44ca27 | 18:36 |
| *** yamamoto has quit IRC | 18:36 | |
| crc32 | pbr is getting downgraded to 0.11.0 thought I don't get it either. | 18:36 |
| fungi | lifeless: https://review.openstack.org/195015 | 18:36 |
| *** ericksonsantos has joined #openstack-infra | 18:37 | |
| *** ericksonsantos has quit IRC | 18:37 | |
| crc32 | I can't find that hash "7af8a1b9b3180da54e2c9505228ad722db44ca27 | 18:37 |
| pleia2 | AJaeger: I'm going to toss some notes at the bottom of https://etherpad.openstack.org/p/zanata-install | 18:37 |
| openstackgerrit | Merged openstack-infra/infra-manual: Update gerrit ACL docs to mention library-release https://review.openstack.org/193309 | 18:37 |
| *** faraujo has joined #openstack-infra | 18:37 | |
| *** ericksonsantos has joined #openstack-infra | 18:38 | |
| fungi | crc32: it's in the openstack-dev/devstack repo | 18:38 |
| mordred | crc32: it's a commit in devstack | 18:38 |
| *** _nadya_ has quit IRC | 18:38 | |
| *** Zhongjun has quit IRC | 18:38 | |
| *** _nadya_ has joined #openstack-infra | 18:38 | |
| fungi | crc32: pondering whether that requirements enforcement change has caused some project which is now no longer getting its stale requirements list sync'd to downgrade pbr based on an old <1.0 cap | 18:39 |
| lifeless | crc32: can you confirm that your lib/infra looks like http://paste.openstack.org/show/321114/ ? | 18:39 |
| *** amitgandhinz has joined #openstack-infra | 18:39 | |
| mtreinish | anteaya: if you get a sec: https://review.openstack.org/#/c/194798/ should be a fast one-liner | 18:39 |
| *** sputnik13 has quit IRC | 18:40 | |
| ajmiller | lifeless crc32 http://paste.openstack.org/show/321116/ | 18:41 |
| ajmiller | lifeless I "rm -rf /opt/stack/*" just before this run | 18:42 |
| *** samueldmq has quit IRC | 18:43 | |
| lifeless | ajmiller: how did you generate the log file? | 18:43 |
| crc32 | that section of code looks the same yet | 18:43 |
| lifeless | ajmiller: I can't see any pip commands in there at all | 18:43 |
| *** ericksonsantos has quit IRC | 18:43 | |
| *** _nadya_ has quit IRC | 18:43 | |
| lifeless | fungi: that would be my assumption | 18:43 |
| ajmiller | Sorry, it looks truncated | 18:43 |
| *** otter768 has joined #openstack-infra | 18:43 | |
| *** amitgandhinz has quit IRC | 18:43 | |
| *** ayoung has quit IRC | 18:43 | |
| *** dontalton has quit IRC | 18:44 | |
| *** amitgandhinz has joined #openstack-infra | 18:45 | |
| *** annegentle has quit IRC | 18:45 | |
| crc32 | interesting. Attempting to stack really does bring pbr down to 0.11.0 | 18:45 |
| anteaya | mtreinish: done | 18:45 |
| ajmiller | lifeless this shoudl be better | 18:45 |
| ajmiller | http://pastebin.com/TP1j6GjV | 18:45 |
| mtreinish | anteaya: thanks | 18:45 |
| *** sputnik13 has joined #openstack-infra | 18:46 | |
| *** ZZelle_ has joined #openstack-infra | 18:46 | |
| lifeless | ajmiller: ok line 2067 successfully gets pbr 1.2 onto your system | 18:46 |
| *** jasondotstar has quit IRC | 18:47 | |
| lifeless | ajmiller: 2997 2015-06-25 18:34:16.684 | Collecting pbr!=0.7,<1.0,>=0.6 (from keystonemiddleware>=1.5.0) | 18:47 |
| lifeless | so ksm is downgrading | 18:47 |
| lifeless | which is from | 18:47 |
| lifeless | /usr/local/bin/pip install 'keystonemiddleware>=1.5.0' | 18:47 |
| fungi | the question is why. latest ksm releases don't have that | 18:47 |
| ajmiller | yes, and now: | 18:47 |
| ajmiller | ubuntu@decima-devstack:~/devstack$ pip freeze|grep pbr | 18:47 |
| ajmiller | pbr==0.11.0 | 18:47 |
| lifeless | 2015-06-25 18:34:16.675 | Requirement already satisfied (use --upgrade to upgrade): keystonemiddleware>=1.5.0 in /usr/local/lib/python2.7/dist-packages | 18:47 |
| *** pblaho has joined #openstack-infra | 18:47 | |
| lifeless | fungi: ^ | 18:47 |
| morganfainberg | fungi/lifeless: master? | 18:47 |
| anteaya | mtreinish: welcome | 18:47 |
| lifeless | morganfainberg: no | 18:47 |
| lifeless | there's an existing ksm | 18:47 |
| morganfainberg | stable/<something> | 18:47 |
| lifeless | no | 18:48 |
| morganfainberg | oh | 18:48 |
| morganfainberg | oh | 18:48 |
| morganfainberg | got it | 18:48 |
| morganfainberg | yeah | 18:48 |
| fungi | oh, it's not doing an install -U | 18:48 |
| lifeless | when we assert that its installed | 18:48 |
| *** ociuhandu has quit IRC | 18:48 | |
| morganfainberg | i see now | 18:48 |
| lifeless | pip downgrades to satsify | 18:48 |
| *** otter768 has quit IRC | 18:48 | |
| *** bhunter71 has joined #openstack-infra | 18:48 | |
| lifeless | crc32: ajmiller: short term fix: pip install -U keystonemiddleware | 18:48 |
| lifeless | rinse and repeat until nothing is triggering the downgrade | 18:48 |
| crc32 | do I do that systemwide? | 18:49 |
| lifeless | sudo -H pip install -U keystonemiddleware | 18:49 |
| fungi | crc32: yeah, it's finding it installed in /usr/local/lib/python2.7/dist-packages | 18:49 |
| fungi | so system-wide | 18:49 |
| fungi | not in a virtualenv | 18:49 |
| fungi | lifeless: `grep '^pbr.*<1' openstack/*/*requirements.txt` is depressing, btw | 18:50 |
| ajmiller | lifeless fungi did that, and am rerunning stack.sh, it has gotten past keystone, is now doing neutorn.... | 18:51 |
| *** julim has quit IRC | 18:52 | |
| openstackgerrit | Doug Hellmann proposed openstack-infra/infra-specs: Centralize Release Tagging https://review.openstack.org/191193 | 18:52 |
| *** rbrooker has joined #openstack-infra | 18:52 | |
| fungi | yeah, as someone who doesn't run and update a persistent devstack machine, it's unclear to me how this is meant to work. it's installing packages of these things but then not updating them | 18:53 |
| *** mrunge has quit IRC | 18:54 | |
| fungi | dtroyer: what's the mechanism by which updating a devstack deployment is supposed to pull in updated pypi dependencies? | 18:54 |
| crc32 | so what was the issue? I noticed this worked at work but not at home. | 18:54 |
| *** r-daneel has quit IRC | 18:54 | |
| *** r-daneel has joined #openstack-infra | 18:55 | |
| crc32 | look at that now theres now git diff show no changes in /opt/stack/keystone | 18:55 |
| fungi | crc32: it's possible you're now breaking on something else stale which also has an old pbr cap and needs updating, similar to keystonemiddleware | 18:55 |
| dtroyer | fungi: that happens when the pip install runs for each repo | 18:55 |
| dtroyer | in every stack.sh run | 18:55 |
| *** dguerri` is now known as dguerri | 18:55 | |
| fungi | dtroyer: but it's just trusting that the minimums are correct i guess, doesn't actually update them unless it has to? | 18:56 |
| *** ericksonsantos has joined #openstack-infra | 18:56 | |
| fungi | i suppose this could be a case of we should be stating a minimum keystonemiddleware version new enough to not have that cap | 18:56 |
| sdague | fungi: well, *we* did tell them to put it there | 18:57 |
| dtroyer | fungi: exactly. if the minimums are not correct, there is a problem in the requireemtns.txt | 18:57 |
| bhunter71 | can someone tell me what Zuul job configuration pushes this file; http://tarballs.openstack.org/python-troveclient/python-troveclient-master.tar.gz (more generally, the latest python client for each branch)? I don't see how it's up-to-date, yet I'm not seeing the post or periodic job configuration to do so. | 18:57 |
| crc32 | horizon failed to install | 18:57 |
| fungi | so that when something getting installed from source and depending on keystonemiddleware, it insists on a new enough version so as not to be broken | 18:57 |
| openstackgerrit | Anita Kuno proposed openstack-infra/project-config: Changes puppet-apply-centos job status https://review.openstack.org/195702 | 18:57 |
| *** RichardRaseley has quit IRC | 18:58 | |
| sdague | fungi: honestly, I kind of think we should probably just say that pbr can never break contract, and get rid of it's cap, because after the amount of effort it's going to take to clear <1, I can't ever see us able to clear <2 | 18:58 |
| dtroyer | right. that's the whole transient dep problem in a nutshell | 18:58 |
| crc32 | 2015-06-25 18:56:06.148 | ImportError: Could not import settings 'openstack_dashboard.settings' (Is it on sys.path? Is there an import error in the settings file?): cannot import name i18n | 18:58 |
| fungi | sdague: dtroyer: in the meantime, i guess anything in global-requirements.txt which may have had a pbr<1 cap at some point needs to have its minimum version updated to at least the next release after it removed that cap | 18:59 |
| anteaya | nibalizer: 195702 do share your thoughts when you are able | 18:59 |
| crc32 | ajmiller: Did you get stack to deploy? Now its borking out on horizon. | 18:59 |
| *** julim has joined #openstack-infra | 18:59 | |
| AJaeger | fungi: looking at https://review.openstack.org/#/c/194488/2/jenkins/jobs/infra.yaml - is an empty target ok for the scp publisher? No need to answer now, a review later would be nice... | 19:00 |
| *** dimtruck is now known as zz_dimtruck | 19:00 | |
| *** hashar has joined #openstack-infra | 19:01 | |
| mgagne | I'm about to merge https://review.openstack.org/#/c/189942/3 (bump pbr upper bound for python-jenkins) any objection? | 19:02 |
| *** ddieterly has quit IRC | 19:02 | |
| *** _diana_ has quit IRC | 19:02 | |
| *** julim has quit IRC | 19:02 | |
| fungi | AJaeger: not sure... i believe it ends up relative to the target directory configured in the server entry in jenkins regardless | 19:03 |
| *** julim has joined #openstack-infra | 19:03 | |
| *** ericksonsantos has quit IRC | 19:03 | |
| ajmiller | crc32, stepped away for a couple of minutes and found it dead, at horizon | 19:03 |
| *** ddieterly has joined #openstack-infra | 19:03 | |
| ajmiller | 2015-06-25 18:59:53.942 | ImportError: Could not import settings 'openstack_dashboard.settings' (Is it on sys.path? Is there an import error in the settings file?): No module named i18n | 19:04 |
| crc32 | same here. | 19:04 |
| crc32 | its dead at horizon. | 19:04 |
| asselin | ajmiller, saw that same error recently.... | 19:05 |
| *** EricGonczer_ has joined #openstack-infra | 19:05 | |
| crc32 | everything just feels so fragile | 19:05 |
| *** sshnaidm has quit IRC | 19:05 | |
| *** sshnaidm has joined #openstack-infra | 19:06 | |
| crc32 | on a new VM keystone installed fine. I just wish I know what directory I could just blow away to make all this work on broken systems. | 19:06 |
| *** jasondotstar has joined #openstack-infra | 19:07 | |
| fungi | crc32: we blow away the entire server between runs | 19:07 |
| crc32 | fungi: Yea thats what it feels liek. | 19:07 |
| crc32 | fungi: I'm just not used to the whole build a VM cause your dev environment just isn't stable enough between runs. :( | 19:08 |
| anteaya | so this job failed because /opt/stack/logs/status didn't get created: http://logs.openstack.org/13/195213/1/gate/gate-swift-dsvm-functional/e37a140/logs/devstack-gate-cleanup-host.txt | 19:08 |
| anteaya | that seems to me to be a swift log issue, does it not? | 19:08 |
| AJaeger | pleia2: I haven't found any documents to translate so far in Zanata | 19:09 |
| *** Svedrin has quit IRC | 19:09 | |
| pleia2 | AJaeger: yeah, drafting an email now to chat about our todo list | 19:09 |
| AJaeger | fungi: that's my expectation as well but I'm not sure either | 19:09 |
| AJaeger | pleia2: ok - I commented briefly on the etherpad | 19:09 |
| *** zz_dimtruck is now known as dimtruck | 19:09 | |
| anteaya | dougwig: you might like this patch: https://review.openstack.org/#/c/195702/ | 19:09 |
| pleia2 | AJaeger: great, thank you | 19:10 |
| AJaeger | anteaya: I love it ;) | 19:10 |
| fungi | anteaya: no, tests failed http://logs.openstack.org/13/195213/1/gate/gate-swift-dsvm-functional/e37a140/console.html#_2015-06-25_17_44_03_697 | 19:11 |
| *** achanda has quit IRC | 19:11 | |
| *** dprince has quit IRC | 19:11 | |
| *** Svedrin has joined #openstack-infra | 19:11 | |
| fungi | anteaya: i think the output from the swift functional testsuite just isn't particularly clear on that point | 19:12 |
| jeblair | dhellmann: heh, i tried to catch you earlier for the same thing; we seem to be missing each other | 19:12 |
| jeblair | dhellmann: around now though | 19:12 |
| dhellmann | jeblair: I posted a new draft. I'm not super keen on the filenames I've proposed, but let me know what you think. I also tried to capture the various iterations in the "alternatives" section. | 19:13 |
| jeblair | k, reading now | 19:13 |
| *** HeOS has quit IRC | 19:13 | |
| dougwig | anteaya: ty | 19:14 |
| pleia2 | AJaeger: which is the script used to send .po/pot files from git to transifex? | 19:16 |
| pleia2 | I'd been focused on the other end of things, lost track of imports of the po/pot files themselves | 19:17 |
| *** tim_o has quit IRC | 19:17 | |
| anteaya | dougwig: thanks | 19:18 |
| anteaya | AJaeger: thanks | 19:18 |
| anteaya | fungi: ah thank you, yes I missed that | 19:18 |
| anteaya | fungi: I'm looking at where you see a test failure in the logs and still can't see how I would interpret failure from those lines | 19:19 |
| anteaya | is the absence of ERROR at the end of that line the indication? | 19:20 |
| *** sergmelikyan has joined #openstack-infra | 19:21 | |
| mordred | anteaya: "return 137" | 19:21 |
| *** sshnaidm has quit IRC | 19:21 | |
| anteaya | yeah sc68cal and I are looking at that | 19:21 |
| anteaya | what does that mean? | 19:22 |
| mordred | anteaya: is most likely communicating a non-zero return code, which is failure | 19:22 |
| *** sshnaidm has joined #openstack-infra | 19:22 | |
| anteaya | I should know that, I guess? | 19:22 |
| * anteaya didn't know that | 19:22 | |
| *** ildikov has joined #openstack-infra | 19:22 | |
| anteaya | mordred: thanks | 19:22 |
| * sc68cal breaks out his POSIX guide for ERRNO | 19:22 | |
| mordred | anteaya: well, should or should not - now you know - on linux, non-zero return code == error | 19:22 |
| mordred | anteaya: unless you are puppet | 19:22 |
| *** sergmelikyan has quit IRC | 19:22 | |
| *** ayoung has joined #openstack-infra | 19:22 | |
| mordred | anteaya: in which case 2 is also success | 19:22 |
| anteaya | thank you | 19:23 |
| sc68cal | lol puppet... | 19:23 |
| mordred | anteaya: as you might imagine, that's not very pleasing to people who expect puppet to behave the way all other programs do | 19:23 |
| openstackgerrit | Sukhdev Kapur proposed openstack-infra/project-config: Move networking-arista from stackforge/ to openstack/ https://review.openstack.org/195341 | 19:23 |
| anteaya | mordred: I'm gathering the not very pleasing aspect from you, yes | 19:23 |
| anteaya | not much imagination required on my part | 19:23 |
| sc68cal | ok, so we had a test failure, but that job doesn't have a testr results file for us to see the details of the failure | 19:23 |
| mordred | anteaya: on the other hand, salt returns 0 for success for _All_ invocations, including errors | 19:23 |
| anteaya | ha ha ha | 19:24 |
| mordred | anteaya: which is also displeasing, but in a different way | 19:24 |
| lifeless | fungi: :( | 19:24 |
| anteaya | let's be different | 19:24 |
| mordred | anteaya: yay! | 19:24 |
| anteaya | yeah different is more important than useful | 19:24 |
| pabelanger | anteaya, nibalizer I should have more info tonight about the failing job | 19:24 |
| lifeless | dtroyer: so the minimums aren't strictly wrong | 19:24 |
| mordred | anteaya: actually, honestly, when people ask "why ansible" for infra, a LARGE part of it had to do with return codes from the programs :) | 19:24 |
| anteaya | pabelanger: can we make it non-voting in the meantime | 19:24 |
| lifeless | dtroyer: but the transitive deps are hugely problematic | 19:24 |
| anteaya | pabelanger: it is still on check | 19:24 |
| pabelanger | anteaya, well, there is an elastic-rechecking too | 19:24 |
| anteaya | https://review.openstack.org/#/c/195702/ | 19:25 |
| pabelanger | anteaya, if you are going to do anything with the job, maybe juts move it to experimental | 19:25 |
| anteaya | pabelanger: if that is what you perfer I can do that | 19:25 |
| * anteaya respins | 19:25 | |
| pabelanger | Ya, that works for me | 19:25 |
| pabelanger | Something is slowing down the OS, not just git is affected | 19:25 |
| jeblair | dhellmann: new rev lgtm; actually looks really good to me, i like the amount of thought/discussion it has captured. i left one more thought inline -- i feel like the latest-vs-full-history choice is more of a toss-up than you describe, but as such, i'm not actually prepared to disagree with your preference :) | 19:25 |
| jeblair | dhellmann: you may want to consider replying explicitly to ttx's comment | 19:26 |
| pabelanger | everything is taking longer. EG: compile puppet catalogs | 19:26 |
| lifeless | crc32: ajmiller: fungi: worth trying exporting PIP_UPRADE=True before stack.sh | 19:26 |
| dhellmann | jeblair: for the history thing, I'm thinking of the script that is going to see "file X has changed" and then have to go through all of the tags to ensure they all exist vs. looking at one tag | 19:27 |
| lifeless | fungi: have those reviews for the periodic jbo been A'd ? | 19:27 |
| openstackgerrit | Joshua Harlow proposed openstack/requirements: Add new oslo futurist library to requirements https://review.openstack.org/195711 | 19:27 |
| dhellmann | jeblair: I'm also looking at this as the *requests* for tags, and not a history of the project (that lives in the source tree) | 19:27 |
| dhellmann | jeblair: maybe you have a different use case in mind that I'm not seeing? | 19:27 |
| jeblair | dhellmann: yeah, i think that will be a series of extremely fast git operations | 19:27 |
| dtroyer | lifeless: it's a mess, and I am forever thankful that you (and others) are taking on sorting this as I've long ago gotten over wanting to since the only high-percentage solution is widely dis-favored in these circles | 19:28 |
| dhellmann | jeblair: I suppose so, but I was hoping to build on the release script I have today and that isn't very fast because it also does a bunch of launchpad work | 19:28 |
| ajmiller | lifeless OK, I'll give it a go | 19:28 |
| dtroyer | lifeless: also, we had a devstack patch that wanted to add -U to all pip invocations, I forget exactly why we shot it down as I've vacationed since then… ;) | 19:29 |
| dhellmann | jeblair: in particular, I'm thinking about how often we have to re-run that whole script, and the fact that I don't want to wait for it to process all of those releases if launchpad gives me an error on the last one | 19:29 |
| lifeless | dtroyer: yeah, so constraints will tackle that, very close now | 19:29 |
| jeblair | dhellmann: i don't really have another use case; until we develop more (we never have scope creep, right?) there's probably not something to suggest one way or the other | 19:30 |
| lifeless | dtroyer: https://review.openstack.org/#/c/194886/ | 19:30 |
| jeblair | dhellmann: i have been thinking about git more than launchpad | 19:30 |
| dhellmann | jeblair: yeah, git and gerrit are the easy reliable part of this process, so I tend not to consider those a big deal | 19:30 |
| dtroyer | lifeless: coolness…anticipate Japanese $BEVERAGE in your future… | 19:30 |
| jeblair | dhellmann: it might be worth having the script do that first since it's easier and faster, and then perform launchpad operations based on the results of that part | 19:30 |
| dhellmann | jeblair: it does | 19:31 |
| *** jrniemijr has quit IRC | 19:31 | |
| *** jamespage_ has joined #openstack-infra | 19:31 | |
| dhellmann | jeblair: the problem is, if the tag is there but launchpad fails, we still have to figure out if the launchpad work needs to be done | 19:31 |
| jeblair | yeah | 19:31 |
| jeblair | dhellmann: i do think that regardless, it should be idempotent, and not have to figure out which files changed; that's tricky to do reliably and complicates things if we need to re run a job for some reason. | 19:31 |
| dhellmann | jeblair: for today, i would really just like to start with a way for folks to submit requests for releases other than emailing someone off of the -dev list where it can get lost | 19:32 |
| jeblair | dhellmann: the nice thing about automating this is that eventually you don't have to wait for it to complete :) | 19:32 |
| fungi | lifeless: all but the last one have been approved. https://review.openstack.org/195360 still needs reviews | 19:32 |
| *** jamespage_ has quit IRC | 19:32 | |
| openstackgerrit | Paul Belanger proposed openstack-infra/system-config: Do not merge https://review.openstack.org/195288 | 19:32 |
| jeblair | dhellmann: i take your point about the lp slowness though, and that does seem to favor limiting the number of things the script examines. so i'm cool with the current rev. | 19:33 |
| dhellmann | jeblair: yeah, I'm still not entirely sure how the CI system is going to be able to do this, so I'm thinking about automating it up to the point of actually running the script to create the tags | 19:33 |
| fungi | interesting. has diskimage-builder been in the integrated change queue for a while? or did gate-tempest-dsvm-ironic-pxe_ssh somehow recently drag it in? | 19:34 |
| pabelanger | anteaya: FYI https://review.openstack.org/#/c/195283/ if you want to help get that merged, it should take _some_ pressure off people | 19:34 |
| jeblair | dhellmann: sounds like a plan | 19:35 |
| openstackgerrit | Anita Kuno proposed openstack-infra/project-config: Changes puppet-apply-centos job status https://review.openstack.org/195702 | 19:35 |
| *** ramashri has quit IRC | 19:35 | |
| *** haypo has joined #openstack-infra | 19:36 | |
| anteaya | pabelanger: +1 is the best I can offer on that repo, and you have it, thank you | 19:36 |
| jeblair | anteaya, pabelanger: why move it to experimental instead of non-voting? | 19:38 |
| haypo | hi. gate-infra-puppet-apply-centos6 is failing on openstack-infra/project-config. it's maybe related to "getsebool httpd_enable_cgi". ex: http://logs.openstack.org/28/194128/2/gate/gate-infra-puppet-apply-centos6/489f6e9/console.html | 19:38 |
| haypo | is someone aware of the issue? | 19:38 |
| jeblair | if the goal is "watch closely for git.o.o related changes, but otherwise ignore" non-voting will make that easier... | 19:38 |
| ajmiller | lifeless crc32 setting " export PIP_UPGRADE=True" allowed it to fail in a new and exciting way: | 19:38 |
| pleia2 | haypo: thanks we're in the middle of working on it :) | 19:38 |
| anteaya | jeblair: I don't care where it is, just trying to make pabelanger happy | 19:38 |
| ajmiller | + /usr/local/bin/glance-manage db_sync | 19:39 |
| ajmiller | failed with | 19:39 |
| anteaya | jeblair: I'm move it whereever as long is it isn't voting | 19:39 |
| haypo | pleia2: any clue? | 19:39 |
| ajmiller | ImportError: cannot import name deploy | 19:39 |
| pleia2 | anteaya: I think non-voting makes most sense | 19:39 |
| pleia2 | haypo: ongoing :) | 19:39 |
| anteaya | pabelanger: jeblair and pleia2 seem to be leaning toward non-voting | 19:39 |
| anteaya | pabelanger: what say you? | 19:40 |
| *** linkedinyou has joined #openstack-infra | 19:40 | |
| pleia2 | anteaya, pabelanger: if it "fixes itself" i'd like to see in the non-voting jobs passing | 19:40 |
| pleia2 | with experimental we won't see them | 19:40 |
| anteaya | makes sense to me | 19:40 |
| anteaya | conversation is happening at another table I will return | 19:41 |
| * anteaya moves to the other side of the room | 19:41 | |
| *** ajo|afk is now known as ajo | 19:41 | |
| *** shardy has quit IRC | 19:42 | |
| *** shardy has joined #openstack-infra | 19:43 | |
| AJaeger | pleia2: jenkins/scripts/upstream_translation_* are the scripts for pushing from the post job to transifex | 19:44 |
| pleia2 | AJaeger: thanks | 19:44 |
| *** julim has quit IRC | 19:44 | |
| *** _diana_ has joined #openstack-infra | 19:46 | |
| lifeless | mordred: pleia2: https://review.openstack.org/195360 please | 19:48 |
| lifeless | fungi: speaking of depressing, so many things are, blinkers on :) | 19:48 |
| lifeless | ajmiller: no idea about that one | 19:49 |
| lifeless | ajmiller: perhaps try again now without PIP_UPGRADE set ? | 19:50 |
| *** DericHorn-HP has joined #openstack-infra | 19:50 | |
| *** jtomasek has quit IRC | 19:51 | |
| *** esker has quit IRC | 19:54 | |
| *** Krinkle has left #openstack-infra | 19:54 | |
| *** sshnaidm has quit IRC | 19:54 | |
| *** _diana_ has quit IRC | 19:55 | |
| *** ayoung has quit IRC | 19:55 | |
| *** sshnaidm has joined #openstack-infra | 19:55 | |
| jogo | somethign in infra just broke multinode | 19:56 |
| jogo | both nodes have the same hostname (ubuntu) | 19:57 |
| jogo | and nova expects unique hostnames | 19:57 |
| ajmiller | lifeless -- just tried that, same result | 19:57 |
| dhellmann | jeblair: do you want a spec for the "bugs/" branch work, or should I just submit the patches needed? | 19:57 |
| jeblair | dhellmann: no spec needed i think | 19:57 |
| jogo | before http://logs.openstack.org/25/194325/3/check/check-tempest-dsvm-multinode-full/4135c54/logs/screen-n-cpu.txt.gz#_2015-06-23_22_08_28_833 | 19:57 |
| dhellmann | jeblair: ok | 19:58 |
| jogo | after: V | 19:58 |
| jogo | http://logs.openstack.org/67/190267/17/check/check-tempest-dsvm-multinode-full/ad8669c/logs/screen-n-cpu.txt.gz#_2015-06-25_17_40_21_558 | 19:58 |
| *** cindyo has joined #openstack-infra | 19:58 | |
| jogo | this may be a HP vs RAX cloud issue? | 19:58 |
| jogo | clarkb fungi jeblair: ^ | 19:59 |
| *** jtomasek has joined #openstack-infra | 19:59 | |
| lifeless | ajmiller: :( | 20:01 |
| lifeless | ajmiller: so its dying in horizon now ? | 20:01 |
| *** mtanino has joined #openstack-infra | 20:01 | |
| ajmiller | Its trying to create the glance database | 20:02 |
| lifeless | fungi: jenkins is sad on https://review.openstack.org/#/c/195360/ | 20:02 |
| mordred | jogo: we should be setting hostnames on all of our node as part of node setup | 20:02 |
| mordred | jogo: because clouds do it wrong | 20:02 |
| lifeless | fungi: oh, I see you know :) | 20:02 |
| openstackgerrit | Doug Hellmann proposed openstack-infra/devstack-gate: Treat bug/ and feature/ branches the same way https://review.openstack.org/195724 | 20:03 |
| jogo | mordred: if you set the hostnames on all nodes that would be wonderful | 20:03 |
| jogo | mordred: but the logs point to you aren't | 20:03 |
| mordred | jogo: I mean, we're supposed to be doing that | 20:03 |
| mordred | jogo: that doesn't mean we ARE | 20:03 |
| jogo | mordred: yeah you used to at least | 20:03 |
| jogo | :) | 20:03 |
| jogo | anyway that is blocking all the multinode stuff | 20:03 |
| mordred | awesome! | 20:03 |
| jogo | and making nova very confused | 20:03 |
| mordred | well, nova makes me confused a lot - so we're even ;) | 20:03 |
| jogo | because both compute nodes listen on the same thing so you can't talk to a specific one | 20:04 |
| mordred | jogo: now probably isn't the time to ask why, given openstack's otherwise pervassive obsession with adding uuid's to something, compute nodes care about their host's hostname? | 20:04 |
| jeblair | mordred: i think i noticed the other day that dib nodes had "ubuntu" as the hostname | 20:04 |
| mordred | jeblair: ah! | 20:04 |
| clarkb | glwan | 20:05 |
| mordred | jeblair: I betcha I know why | 20:05 |
| *** yamamoto has joined #openstack-infra | 20:05 | |
| clarkb | it doesnt set hostbames | 20:05 |
| mordred | jeblair: and by "I betcha" I'd say "I absolutely know why" | 20:05 |
| jogo | mordred: that is a really good question, we could just generate a UUID | 20:05 |
| greghaynes | mordred: oh? | 20:05 |
| jogo | dansmith: ^ mordred had a good idea -- why not just generate UUIDs for nova to use ? | 20:06 |
| mordred | wait - doesn't cloud-init set hostnames? we aren't running multi-node on our new not-yet-tested nodes are we? | 20:06 |
| greghaynes | mordred: yea, so is this in HP or in rax? | 20:06 |
| mordred | greghaynes: HP | 20:06 |
| dansmith | jogo: mordred you mean as the hostname? | 20:06 |
| clarkb | mordred we run them on devstack-trusty which in HP may be glean | 20:06 |
| mordred | clarkb: aha! | 20:06 |
| greghaynes | which will be glean not cloud-iknit | 20:06 |
| greghaynes | yea | 20:06 |
| greghaynes | and we just release glean... | 20:06 |
| clarkb | I would have to check nodepool ad I have been afk | 20:06 |
| *** ParsectiX has quit IRC | 20:07 | |
| mordred | then yes. that's us | 20:07 |
| mordred | greghaynes: well, glean has never set a hostname | 20:07 |
| jogo | dansmith: instead of using hostname for the unique identifier for rabbit | 20:07 |
| dansmith | jogo: just setting it to hostname=$HOSTNAME-$NODENUMBER would be fine too | 20:07 |
| fungi | mordred: even with bare-.* nodes, i discovered this. we work around it in the git-review functional tests by overriding the default git e-mail address because git falls over trying to make one up on hpcloud workers due to the hostname | 20:07 |
| *** Sukhdev has quit IRC | 20:07 | |
| greghaynes | mordred: yea, so thats good | 20:07 |
| mordred | do we want it to? | 20:07 |
| mordred | or do we want somethign else to? | 20:07 |
| dansmith | jogo: that's a whole thing, because instance.host *has* to be unique when it's on one host or the other | 20:07 |
| greghaynes | mordred: well, I dont get what was setting hostname before? | 20:07 |
| mordred | greghaynes: cloud-init | 20:07 |
| clarkb | cloud init was | 20:07 |
| dansmith | jogo: else evacuation breaks, message routing breaks, shared storage checks break, etc | 20:07 |
| jogo | dansmith: true, I guess it is just a default config really | 20:07 |
| greghaynes | we switched to glean in HP a while ago? | 20:07 |
| mordred | greghaynes: we may just have gotten lucky and not run a multi-node test in hp | 20:08 |
| mordred | this will only show up as an issue on multi-node | 20:08 |
| clarkb | or it failed and no one debugged | 20:08 |
| mordred | yah | 20:08 |
| jogo | all failures are in HP it looks like | 20:08 |
| jogo | http://logstash.openstack.org/#eyJzZWFyY2giOiIgYnVpbGRfbmFtZTpcImNoZWNrLXRlbXBlc3QtZHN2bS1uZXV0cm9uLW11bHRpbm9kZS1zbW9rZVwiIEFORCBtZXNzYWdlOlwiRmluaXNoZWRcIiBBTkQgTk9UIG1lc3NhZ2U6XCJsb2FkaW5nXCIiLCJmaWVsZHMiOltdLCJvZmZzZXQiOjAsInRpbWVmcmFtZSI6IjE3MjgwMCIsImdyYXBobW9kZSI6ImNvdW50IiwidGltZSI6eyJ1c2VyX2ludGVydmFsIjowfSwibW9kZSI6InRyZW5kIiwiYW5hbHl6ZV9maWVsZCI6ImJ1aWxkX25vZGUiLCJzdGFtcCI6MTQzNTI1OTE4NTU2 | 20:08 |
| greghaynes | ok, so maybe we should add hostname to the very small list of glean features | 20:08 |
| jogo | OX0= | 20:08 |
| mordred | so I don't thnik we can make assumptions about when it broke - but I do thinkn we can categorically say that we expect it to not work | 20:09 |
| mordred | greghaynes: I'm not sure we should | 20:09 |
| mordred | greghaynes: I could also make the case that it should be the ready-script's job | 20:09 |
| mordred | I'm not sure we sholdn't | 20:09 |
| *** krtaylor has joined #openstack-infra | 20:09 | |
| mordred | just opening it up for opinions | 20:09 |
| mordred | (both are simple to do) | 20:09 |
| clarkb | its set from metadata gleans job is to act on that data | 20:09 |
| clarkb | imo it should | 20:09 |
| mordred | jeblair, fungi: ^^ opinions? | 20:09 |
| mordred | clarkb: I can totally buy that argument | 20:10 |
| jogo | mordred: so can I actually blame you for this one :) | 20:10 |
| greghaynes | Yea, I think if its something we need to do based on the config-drive info then we make it a glean thing | 20:10 |
| mordred | jogo: yes | 20:10 |
| *** yamamoto has quit IRC | 20:10 | |
| jogo | mordred: \o/ one last time | 20:10 |
| greghaynes | mordred: in rax we get a hostname from config-drive right? | 20:10 |
| mordred | doublechecking | 20:10 |
| fungi | seems like glean would be the logical choice, as it's setting other network details | 20:11 |
| mordred | but it IS part of the nova metadata | 20:11 |
| jeblair | ooooh | 20:11 |
| mordred | jogo: is the hostname in nova metadata the nova munged hostname, or the real one? | 20:11 |
| jeblair | so the thing that actually broke is that both nodes were named 'ubuntu' and the multinode setup added that to /etc/hosts | 20:11 |
| mordred | I think hostname is munged, and name is non-munged, yeah? | 20:11 |
| *** tonytan4ever has quit IRC | 20:11 | |
| jogo | mordred: it depends on admin user or non admin too | 20:11 |
| mordred | jogo: for normal non-admin users | 20:12 |
| dansmith | jeblair: no, the thing that actually broke is nova uses the hostname to coordinate...everything | 20:12 |
| dansmith | jeblair: so it assumed the instances were on both nodes simultaneously, so we'd ask node A to boot it and then node B to attach a volume | 20:12 |
| greghaynes | I also think its hilarious that two days after saying hopefully no new glean features.... | 20:12 |
| jeblair | okay, so probably the first thing to do is agree on whether we think test nodes should have unique names at all | 20:12 |
| clarkb | dansmirh thats a bad idea ib general | 20:12 |
| clarkb | but ok :) | 20:12 |
| dansmith | clarkb: it's been baked into nova since day one, AFAIK | 20:13 |
| clarkb | nice | 20:13 |
| mordred | jeblair: we launch them with unique names to the nova name command already, right? | 20:13 |
| dansmith | clarkb: we resolve it in very few places, but it's a convention used by many subsystems | 20:13 |
| greghaynes | jeblair: sounds like we dont have much of a choice ^ | 20:13 |
| *** Longgeek has quit IRC | 20:13 | |
| jeblair | mordred: yes | 20:13 |
| *** hashar has quit IRC | 20:14 | |
| mordred | jeblair: well, I will say I've wanted for LONG TIME for "nova boot foo.bar.com" to result in the hostname of the vm being foo.bar.com | 20:14 |
| dansmith | clarkb: the funny thing is, right now if you change your hostname on a box, it will delete all its instances and disks | 20:14 |
| jeblair | mordred: we could either ignore or change that | 20:14 |
| mordred | jeblair: so I'm happy to make this work for us | 20:14 |
| mordred | broadly | 20:14 |
| *** kgiusti has quit IRC | 20:14 | |
| dansmith | clarkb: and it will proudly tell you it did so | 20:14 |
| dansmith | clarkb: even if said change was by accident :D | 20:14 |
| jeblair | mordred: yeah | 20:14 |
| lifeless | question - https://pypi.python.org/pypi/futurist has no sdist published but there is a wheel | 20:14 |
| lifeless | whats the right way to debug that ? | 20:14 |
| openstackgerrit | Doug Hellmann proposed openstack-infra/project-config: Ignore doc publishing jobs in bug branches https://review.openstack.org/195725 | 20:15 |
| jeblair | if we decide we should have unique hostnames, i sort of think glean should do it | 20:15 |
| * jeblair waits to see if greghaynes says "NO" | 20:15 | |
| dhellmann | jeblair, fungi : I have *no* idea if ^^ is right, so please look at that one carefully | 20:15 |
| greghaynes | oh, is this a test? ;) | 20:15 |
| fungi | lifeless: i'm going to see if i can find the release pipeline logs from the sha of that tag | 20:16 |
| greghaynes | I reluctantly agree that glean should do this | 20:16 |
| *** mjturek1 has left #openstack-infra | 20:16 | |
| *** tonytan4ever has joined #openstack-infra | 20:16 | |
| jeblair | greghaynes wins kobayashi maru | 20:16 |
| lifeless | fungi: ah - http://logs.openstack.org/f2/f2f606203eed2fc7e5ebdbc3100ea2810e471514/ | 20:17 |
| *** ramashri has joined #openstack-infra | 20:17 | |
| fungi | lifeless: http://logs.openstack.org/f2/f2f606203eed2fc7e5ebdbc3100ea2810e471514/release/futurist-pypi-both-upload/d2bd7bd/console.html | 20:17 |
| fungi | yeah, that | 20:17 |
| fungi | HTTPError: 503 Server Error: backend read error | 20:17 |
| fungi | so the reason is "pypi fell over" | 20:18 |
| *** SumitNaiksatam has quit IRC | 20:18 | |
| *** dahoo has joined #openstack-infra | 20:18 | |
| jeblair | dhellmann: that looks right to me | 20:18 |
| lifeless | fungi: huh | 20:18 |
| *** SumitNaiksatam has joined #openstack-infra | 20:18 | |
| greghaynes | mordred: so are you hacking this into glean or should I poke at it? | 20:18 |
| fungi | lifeless: basically twine seems to have successfully uploaded the file, but pypi threw back a failure | 20:18 |
| *** dahoo is now known as jerryz | 20:18 | |
| fungi | lifeless: so the job aborted before it got around to uploading the sdist | 20:19 |
| lifeless | fungi: why do we call twine separately for each artifact? | 20:19 |
| mordred | greghaynes: I'm verifying which thing goes into config drive in which form | 20:19 |
| openstackgerrit | Marton Kiss proposed openstack-infra/groups: Resolve date_ical missing library issue https://review.openstack.org/195728 | 20:19 |
| fungi | lifeless: good question. does it support multiple artifacts? i thought its second optional positional parameter was for a detach signature | 20:19 |
| mordred | greghaynes: if you wanna hack in actually setting the hostname while I find which hostname we want | 20:19 |
| fungi | er, detached | 20:20 |
| *** sshnaidm has quit IRC | 20:20 | |
| lifeless | fungi: IMBW :) | 20:20 |
| *** sshnaidm has joined #openstack-infra | 20:20 | |
| lifeless | anyhoo, dstufft ^^^ | 20:20 |
| dstufft | what'd I do now | 20:20 |
| lifeless | dstufft: http://logs.openstack.org/f2/f2f606203eed2fc7e5ebdbc3100ea2810e471514/release/futurist-pypi-both-upload/d2bd7bd/console.html | 20:21 |
| fungi | lifeless: this is at least the second time i've seen twine get a "backend read error" response when the upload was successful | 20:21 |
| lifeless | dstufft: HTTPError: 503 Server Error: backend read error | 20:21 |
| lifeless | dstufft: the new openstack project 'futurist' cut a release | 20:21 |
| fungi | though it's been a month or two | 20:21 |
| lifeless | dstufft: and the wheel uploaded but not the sdist | 20:21 |
| lifeless | fungi: more importantly, why are we uploading wheels before sdists? | 20:21 |
| dstufft | wheels upload first in twine | 20:21 |
| lifeless | fungi: if we're going to bail on failure, I'd really rather the source be uploaded.... | 20:21 |
| dstufft | well, if you specify multiple at once | 20:22 |
| dstufft | and if you upload wheels first dependencies will be listed because sdists are bad | 20:22 |
| fungi | dstufft: oh, you _can_ do multiple in one call? | 20:22 |
| lifeless | dstufft: we specified just the one | 20:22 |
| fungi | then yes we should do that | 20:22 |
| dstufft | fungi: sure | 20:22 |
| dstufft | twine upload foo1 foo2 foo3 | 20:22 |
| jogo | mordred: thanks for fixing the hostname stuff | 20:22 |
| dstufft | or twine upload dist/* | 20:22 |
| dstufft | or w/e | 20:22 |
| lifeless | dstufft: we don't re-queue or retry on failure here | 20:22 |
| mordred | greghaynes: ok. it's name from meta_data.json we want - not hostname | 20:22 |
| lifeless | dstufft: so uploaded wheels first is really bad with our current infrastructure | 20:22 |
| dstufft | 503 on upload is a known issue because of timeouts | 20:22 |
| greghaynes | mordred: splendid | 20:23 |
| mordred | greghaynes: because hostname is the massively useless ${hostname_i_requested}.novalocal | 20:23 |
| fungi | lifeless: well, in this case it would have failed on retry because pypi wouldn't have allowed us to replace it either (since it uploaded successfully and claimed to have failed) | 20:23 |
| lifeless | fungi: right | 20:23 |
| lifeless | fungi: so I'm thinking we need twine to try much harder before whinging | 20:23 |
| lifeless | fungi: so our pipeline can at least have a chance at DTRT | 20:23 |
| mordred | greghaynes: because a) let's completely hamstring the user's ability to make and express a choice and b) while doing that, let's set a value to something we _know_ will ALWAYS be broken just to prove a point that someone somewhere had a theory that hostanmes shouldn't matter | 20:24 |
| *** HeOS has joined #openstack-infra | 20:24 | |
| *** sdake_ has joined #openstack-infra | 20:24 | |
| *** mrmartin has quit IRC | 20:24 | |
| mordred | greghaynes: I have to run for a bit - you have everything you need? | 20:25 |
| greghaynes | mordred: yep, I think so | 20:25 |
| *** tlian2 has joined #openstack-infra | 20:28 | |
| *** sdake has quit IRC | 20:28 | |
| fungi | lifeless: okay, going back through the twine usage details i see i was confused. it seems to base detached signature identification on filename matching (so expects additional files with the same names but a .asc suffix) | 20:28 |
| fungi | anyway, unrelated to the current conundrum... | 20:29 |
| *** sdake_ is now known as sdake | 20:29 | |
| *** tlian has quit IRC | 20:29 | |
| fungi | i guess we can consider calling twine once rather than twice, though not yet sure if it continues to upload additional files if it gets a failure for one of them | 20:30 |
| fungi | dstufft: ^ any idea on that behavior, or should i go diving in the source? | 20:30 |
| dstufft | it won't continue | 20:30 |
| *** rossella_s has quit IRC | 20:30 | |
| openstackgerrit | James E. Blair proposed openstack-infra/infra-specs: WIP: Add Zuul v3 spec. https://review.openstack.org/164371 | 20:30 |
| *** rossella_s has joined #openstack-infra | 20:31 | |
| openstackgerrit | Anita Kuno proposed openstack-infra/project-config: Changes puppet-apply-centos job status https://review.openstack.org/195702 | 20:31 |
| fungi | okay, so without adding some retry/poll feature in twine, probably won't help solve this particular issue | 20:31 |
| fungi | lifeless: i'll get the tarball uploaded for futurist for now | 20:32 |
| *** BobBall has quit IRC | 20:32 | |
| *** BobBall has joined #openstack-infra | 20:34 | |
| *** marcusvrn has quit IRC | 20:34 | |
| *** BobBall is now known as Guest54958 | 20:34 | |
| openstackgerrit | Anita Kuno proposed openstack-infra/project-config: Changes puppet-apply-centos job status https://review.openstack.org/195702 | 20:35 |
| *** jtomasek has quit IRC | 20:36 | |
| nibalizer | anteaya: +1 from me | 20:37 |
| fungi | lifeless: it's present at https://pypi.python.org/pypi/futurist now and should be on our mirrors in a few minutes | 20:37 |
| *** _diana_ has joined #openstack-infra | 20:37 | |
| *** GheRivero has quit IRC | 20:39 | |
| *** jasondotstar has quit IRC | 20:39 | |
| *** jaypipes has quit IRC | 20:39 | |
| *** afazekas has joined #openstack-infra | 20:41 | |
| openstackgerrit | Merged openstack/diskimage-builder: Updated from global requirements https://review.openstack.org/192296 | 20:41 |
| openstackgerrit | Merged openstack/diskimage-builder: Wait longer for root device to become available https://review.openstack.org/192768 | 20:43 |
| *** GheRivero has joined #openstack-infra | 20:44 | |
| *** otter768 has joined #openstack-infra | 20:44 | |
| openstackgerrit | James E. Blair proposed openstack-infra/infra-specs: Add Zuul v3 spec. https://review.openstack.org/164371 | 20:44 |
| jeblair | it lost its WIP tag ^ (intentionally) | 20:46 |
| *** sarob_ has joined #openstack-infra | 20:46 | |
| *** puranamr_ has joined #openstack-infra | 20:46 | |
| *** julim has joined #openstack-infra | 20:47 | |
| fungi | yay progress! | 20:47 |
| lifeless | harlowja: ^^^ | 20:47 |
| fungi | it's all grown up now | 20:47 |
| anteaya | nibalizer Clint thank you | 20:47 |
| lifeless | are we systemically at capacity or something now - seeing 4+Hour queue times on zuul's status page | 20:48 |
| *** sarob has quit IRC | 20:49 | |
| fungi | lifeless: hpcloud had a massive outage earlier | 20:49 |
| *** otter768 has quit IRC | 20:49 | |
| fungi | lifeless: we've been catching up all day | 20:49 |
| *** puranamr has quit IRC | 20:49 | |
| lifeless | ack | 20:49 |
| * lifeless shakes fist | 20:49 | |
| fungi | lifeless: when they came back up, everyone's cached keystone tokens were invalid | 20:50 |
| *** whoops has joined #openstack-infra | 20:50 | |
| *** linkedinyou has quit IRC | 20:50 | |
| fungi | so that made the outage considerably longer than it should have been | 20:50 |
| *** dboik has quit IRC | 20:52 | |
| lifeless | fungi: because nothing handles that correctly today? | 20:53 |
| fungi | lifeless: well, the tokens being handed back by keystone auth requests were not authorized | 20:53 |
| fungi | lifeless: the workaround is apparently to call the keystone api and ask it to delete that token | 20:54 |
| lifeless | keystone gave back bad tokens... iz bug! | 20:54 |
| harlowja | lifeless ya, thx, now just need that universal wheel fixings | 20:54 |
| fungi | harlowja: universal = 1 in the [wheel] section of your setup.cfg | 20:55 |
| harlowja | fungi ya | 20:55 |
| *** amuller has joined #openstack-infra | 20:55 | |
| *** RichardRaseley has joined #openstack-infra | 20:55 | |
| harlowja | fungi https://review.openstack.org/#/c/195714/ :) | 20:55 |
| fungi | it's like magic pixie dust | 20:55 |
| amuller | sdague: I've been observing http://status.openstack.org/zuul/, specifically the openstack-infra gate queue | 20:55 |
| amuller | it seems like its reseting the tests every time it merges a patch | 20:56 |
| amuller | so that the 5th patch in the queue is running its tests 5 times, for example | 20:56 |
| *** ayoung has joined #openstack-infra | 20:56 | |
| amuller | I imagine this is not expected behavior? | 20:56 |
| fungi | amuller: it restarts jobs on a change of a change ahead of it fails a job | 20:56 |
| lifeless | we should put that in cookiecutter | 20:57 |
| amuller | it's reseting in case of success right now | 20:57 |
| amuller | look right now, it's about to merge a patch | 20:57 |
| anteaya | I was distracted from asking for gerrit account id 14208 to be re-enabled: http://paste.openstack.org/show/321166/ by having to pile onto the keep-jogo nova spec: https://review.openstack.org/#/c/194774/ | 20:57 |
| fungi | amuller: nope. that job is about to time out at 30 minutes | 20:57 |
| openstackgerrit | Merged openstack-infra/project-config: Add puppet-manager-core to puppet-monasca https://review.openstack.org/194263 | 20:57 |
| fungi | oh! it succeeded | 20:57 |
| amuller | it's about to merge 194263, look what happens to the patch after it | 20:57 |
| fungi | in theory 193906,2 should finish its one running job | 20:58 |
| fungi | hopefully in the next 4 minutes to avoid the timeout on that job | 20:58 |
| *** sshnaidm has quit IRC | 20:59 | |
| *** sshnaidm has joined #openstack-infra | 20:59 | |
| fungi | the log on the centos6 job for 193906,2 is not looking hopeful though. it's been stalled for 15 minutes now | 21:00 |
| fungi | yeah, it merged 194263 and 193906 is still running its remaining job. about to time out in a minute though | 21:01 |
| amuller | fungi: Of course now that you're looking its not doing that =D | 21:01 |
| *** ivar-lazzaro has joined #openstack-infra | 21:01 | |
| fungi | then 193156,2 and all the changes after it will get all their tests rerun | 21:01 |
| fungi | to make sure that 193906,2 wasn't somehow influencing their results | 21:01 |
| *** rm_work is now known as rm_work|away | 21:02 | |
| *** dannywilson has joined #openstack-infra | 21:02 | |
| openstackgerrit | Merged openstack-infra/project-config: Add non-voting Neutron fullstack check job https://review.openstack.org/193906 | 21:02 |
| *** ivar-lazzaro has quit IRC | 21:02 | |
| openstackgerrit | Merged openstack-infra/project-config: Make yaql py34 job voting https://review.openstack.org/193156 | 21:02 |
| fungi | it passed! | 21:02 |
| *** patrickeast_ has joined #openstack-infra | 21:03 | |
| *** ivar-lazzaro has joined #openstack-infra | 21:03 | |
| amuller | fungi: I thought that when a patch merged it reset the tests for the patches after it, but maybe it was because a test timed out | 21:03 |
| amuller | it's possible but I don't think so | 21:03 |
| *** annegentle has joined #openstack-infra | 21:03 | |
| fungi | amuller: yeah, so aside from the fact that we have a terribly performing centos6 job that's often hanging arbitrarily long periods of time and often getting aborted because it exceeds the configured timeout, things seem to be operating as expected there | 21:03 |
| *** cindyo has quit IRC | 21:04 | |
| *** isviridov is now known as isviridov_away | 21:04 | |
| amuller | fungi: ack, thanks for your time. I'll try to take peeks over the next few days otherwise this incident will bring me one step closer to insanity | 21:04 |
| fungi | amuller: and we've got a change in the queue which will switch that job to nonvoting in check and stop running it in the gate while we try to work out the git bug we're encountering there | 21:04 |
| amuller | uh huh | 21:04 |
| *** RichardRaseley has quit IRC | 21:05 | |
| *** prad has quit IRC | 21:05 | |
| *** patrickeast has quit IRC | 21:05 | |
| *** patrickeast_ is now known as patrickeast | 21:05 | |
| *** dannywil_ has quit IRC | 21:05 | |
| *** e0ne has joined #openstack-infra | 21:05 | |
| jeblair | i'm going to self-approve https://review.openstack.org/193838 so that we can re-enable puppet on git.o.o; i don't think it was contentious | 21:05 |
| *** linkedinyou has joined #openstack-infra | 21:06 | |
| fungi | oh, i somehow missed that one existed | 21:06 |
| fungi | lgtm | 21:06 |
| *** rm_work|away is now known as rm_work | 21:06 | |
| *** sshnaidm has quit IRC | 21:08 | |
| *** DericHorn-HP has quit IRC | 21:08 | |
| *** linkedinyou has quit IRC | 21:08 | |
| *** sshnaidm has joined #openstack-infra | 21:09 | |
| *** DericHorn-HP has joined #openstack-infra | 21:11 | |
| *** ddieterly has quit IRC | 21:11 | |
| *** chlong has quit IRC | 21:11 | |
| anteaya | can someone kindly re-enable gerriit account id 14208: http://paste.openstack.org/show/321166/ | 21:13 |
| *** ldnunes has quit IRC | 21:13 | |
| fungi | anteaya: i can do it, but might not be kindly | 21:14 |
| fungi | done | 21:14 |
| fungi | hopefully that wasn't _too_ unkind | 21:14 |
| *** linkedinyou has joined #openstack-infra | 21:14 | |
| *** RichardRaseley has joined #openstack-infra | 21:15 | |
| *** sshnaidm has quit IRC | 21:16 | |
| *** sshnaidm has joined #openstack-infra | 21:16 | |
| anteaya | fungi: I felt it to be kind | 21:17 |
| anteaya | you typed kindly | 21:17 |
| anteaya | and thank you | 21:17 |
| openstackgerrit | David Lyle proposed openstack-infra/project-config: don't run horizon-npm-run-* on kilo https://review.openstack.org/195745 | 21:17 |
| fungi | anteaya: you're welcome. i used the softest keyboard on my desk, just in case | 21:17 |
| jeblair | fungi: better you than me then | 21:18 |
| openstackgerrit | David Lyle proposed openstack-infra/project-config: don't run horizon-npm-run-* on kilo https://review.openstack.org/195745 | 21:19 |
| anteaya | fungi: :) | 21:19 |
| anteaya | jeblair: hehehe | 21:19 |
| openstackgerrit | greghaynes proposed openstack-infra/glean: Support setting hostname https://review.openstack.org/195748 | 21:20 |
| *** soren has quit IRC | 21:20 | |
| greghaynes | mordred: jeblair jogo ^ | 21:20 |
| *** Longgeek has joined #openstack-infra | 21:21 | |
| SpamapS | weird, f2 isn't working for me in gertty | 21:21 |
| fungi | jeblair: yeah, these cherry mx blue mechanical switches are only _barely_ quieter than a model m | 21:21 |
| fungi | i've added rubber cushion rungs under the keycaps to help deaden them a little and reduce the keystroke depth | 21:22 |
| anteaya | but how is the kindness rating? | 21:22 |
| *** faraujo has quit IRC | 21:22 | |
| openstackgerrit | greghaynes proposed openstack-infra/glean: Support setting hostname https://review.openstack.org/195748 | 21:22 |
| jeblair | fungi: reduce the depth? you're killing me! | 21:22 |
| lifeless | depends how fast the keyboard is swung | 21:22 |
| fungi | lifeless: wins | 21:22 |
| *** dboik has joined #openstack-infra | 21:22 | |
| anteaya | hehe | 21:22 |
| *** RichardRaseley has quit IRC | 21:23 | |
| greghaynes | fungi: Is it loud enough to confuse your typing with fireworks going off? | 21:23 |
| greghaynes | because that could be an effective way to get back at the tourists... | 21:23 |
| lifeless | sorry for going there. Naughty lifeless. | 21:23 |
| anteaya | not unless every keystroke sounds like a gunshot | 21:23 |
| *** chlong has joined #openstack-infra | 21:23 | |
| anteaya | actually I have heard quieter guns | 21:24 |
| *** dboik_ has joined #openstack-infra | 21:24 | |
| fungi | greghaynes: definitely not. though i do have an _actual_ model m i can get out if i want that | 21:24 |
| fungi | also makes a much better bludgeon than this fancy modern thing with all the backlights | 21:24 |
| fungi | i could take out a tourist or two easily with that | 21:25 |
| *** sshnaidm has quit IRC | 21:26 | |
| jeblair | fungi: i find the thing i like most about the model m is that the keys travel so far after the spring buckles that by the time they hit bottom, it's just the barest tap (when i'm typing correctly, of course). | 21:26 |
| *** asahlin is now known as asahlin_away | 21:26 | |
| *** rlandy has quit IRC | 21:27 | |
| *** dboik has quit IRC | 21:27 | |
| *** sshnaidm has joined #openstack-infra | 21:27 | |
| fungi | jeblair: yep, i don't really let these strike bottom anyway. i've got it where i tend to release as soon as the spring clicks | 21:27 |
| jeblair | (unless, of course, i'm typing about openstack api return values) | 21:27 |
| *** eharney has quit IRC | 21:28 | |
| fungi | but this way if it does strike all the way down, it's a rubber contact so it doesn't make terrible clacking/ringing sounds | 21:28 |
| *** dguerri is now known as dguerri` | 21:28 | |
| fungi | and helps mute the spring click noise at the same time | 21:28 |
| jeblair | ah interesting side effect | 21:28 |
| *** cindyo has joined #openstack-infra | 21:29 | |
| anteaya | notmyname: do you know anything about swift-dsvm logs and what return 137 might mean? https://review.openstack.org/#/c/195213/ | 21:29 |
| anteaya | notmyname: so far sc68cal and myself are at a loss as to what the issue is, and we are hitting it consistently | 21:30 |
| openstackgerrit | James E. Blair proposed openstack-infra/infra-specs: Fix puppet-apply spec typos https://review.openstack.org/189499 | 21:30 |
| *** marun has quit IRC | 21:33 | |
| openstackgerrit | James E. Blair proposed openstack-infra/devstack-gate: Remove re-exec functionality https://review.openstack.org/189316 | 21:33 |
| openstackgerrit | James E. Blair proposed openstack-infra/system-config: Change review->change in zuul table https://review.openstack.org/169934 | 21:35 |
| krotscheck | Hey, horizon's got kilo branch builds failing because I was being stupid. Can I get two cores to go look at this please? https://review.openstack.org/#/c/195745 | 21:36 |
| krotscheck | I promise it's tiny | 21:37 |
| krotscheck | fungi: Thanks! | 21:37 |
| sc68cal | notmyname: anteaya: http://logs.openstack.org/13/195213/1/gate/gate-swift-dsvm-functional/e37a140/logs/devstack-gate-post_test_hook.txt.gz#_2015-06-25_17_44_03_682 | 21:38 |
| *** mtanino has quit IRC | 21:38 | |
| *** hogepodge has quit IRC | 21:38 | |
| *** _diana_ has quit IRC | 21:39 | |
| *** e0ne is now known as e0ne_ | 21:39 | |
| *** shardy has quit IRC | 21:40 | |
| *** Longgeek has quit IRC | 21:40 | |
| *** _diana_ has joined #openstack-infra | 21:41 | |
| *** e0ne_ is now known as e0ne | 21:41 | |
| *** mriedem has quit IRC | 21:44 | |
| *** tonytan4ever has quit IRC | 21:45 | |
| dims | anyone else seeing this? "Don't know how to generate uuid strings on this platform" | 21:45 |
| anteaya | dims: seeing it where? | 21:45 |
| sc68cal | Looks like the check-swift-dsvm-functional test is failing quite a bit - http://tinyurl.com/nsd9mn8 | 21:46 |
| dims | anteaya: just saw it in a handful of jobs | 21:46 |
| anteaya | dims: such as? | 21:46 |
| fungi | dims: seems to be an error generated by apache proton | 21:46 |
| dims | http://logs.openstack.org/07/194507/3/check/gate-oslo.messaging-pep8/aa202ca/console.html | 21:46 |
| dims | fungi: y | 21:46 |
| anteaya | http://logs.openstack.org/07/194507/3/check/gate-oslo.messaging-pep8/aa202ca/console.html#_2015-06-25_21_26_51_554 | 21:47 |
| dims | will ping them | 21:47 |
| *** krtaylor has quit IRC | 21:47 | |
| *** amitgandhinz has quit IRC | 21:48 | |
| *** peristeri has left #openstack-infra | 21:48 | |
| *** julim has quit IRC | 21:51 | |
| jeblair | fungi, mordred, greghaynes: i think all of our rax instances are ending up in one region | 21:51 |
| jeblair | SpamapS, Shrews: ^ | 21:51 |
| *** tlian2 has quit IRC | 21:53 | |
| openstackgerrit | Davanum Srinivas (dims) proposed openstack-infra/elastic-recheck: Add query for proton library build failure https://review.openstack.org/195758 | 21:53 |
| jeblair | when i nova list dfw, i see instances with 'ord' and 'iad' in their names; when i list 'ord' and 'iad', i see only 'ord' and 'iad' respectively. | 21:54 |
| *** faraujo has joined #openstack-infra | 21:54 | |
| SpamapS | jeblair: forgive me for being a little disconnected, but do we use clouds.yaml yet? | 21:54 |
| jeblair | SpamapS: i believe so | 21:55 |
| fungi | jeblair: can you narrow that down? does it look like things using dib are only getting booted in dfw? | 21:55 |
| fungi | and the only stuff you're seeing in iad/ord are from snapshots? | 21:55 |
| jeblair | it's starting to look more like maybe we have a bunch of old nodes from wrong regions in dfw... | 21:55 |
| *** timothyb89 has quit IRC | 21:55 | |
| fungi | (bare-.* mostly?) | 21:55 |
| sc68cal | anteaya: notmyname: sorry - bad query - here's a better one http://tinyurl.com/create.php | 21:55 |
| fungi | oh, ick | 21:55 |
| jeblair | so this may not be an ongoing problem, but perhaps the remnants of an earlier error? | 21:55 |
| sc68cal | anteaya: notmyname: sorry - bad query - here's a better one http://tinyurl.com/o82bg8p | 21:56 |
| jeblair | okay, yeah, it looks like all of the "misfiled" instances are from the same time period, approx 2015-05-29T15:28:14Z | 21:57 |
| jeblair | i imagine something went wrong with our config then, but is probably corrected now | 21:57 |
| jeblair | i'll just delete them | 21:57 |
| jeblair | that will get us 72 instances back | 21:58 |
| fungi | aha, a month old | 21:58 |
| *** annegentle has quit IRC | 21:58 | |
| fungi | and yeah, because of the behavior of alien-list i was normally discounting nodes named something other than the region in question | 21:58 |
| jeblair | fungi: yeah, i think that only applies to hp tho | 21:59 |
| fungi | so wouldn't have spotted those in my alien mopups | 21:59 |
| jeblair | off they go | 22:00 |
| *** chlong has quit IRC | 22:00 | |
| lifeless | righto, now back to fixing check.*requirements | 22:00 |
| jeblair | and nodepool is already happier | 22:00 |
| *** dannywilson has quit IRC | 22:01 | |
| *** dannywilson has joined #openstack-infra | 22:02 | |
| *** dannywilson has quit IRC | 22:02 | |
| *** dannywilson has joined #openstack-infra | 22:03 | |
| fungi | that 'splains the spiky envelope on the test nodes graph | 22:03 |
| *** cindyo has quit IRC | 22:03 | |
| *** spzala has quit IRC | 22:03 | |
| *** Rockyg has joined #openstack-infra | 22:04 | |
| dougwig | lifeless: regarding your feedback to put "neutron" into requirements.txt of neutron's child repos, i've got that review up, and it works fine, but the review to get it into global requirements bombs expectedly because it's not in pypi: https://review.openstack.org/#/c/195392/ | 22:04 |
| jeblair | fungi: well, at least 25% of it | 22:04 |
| dougwig | lifeless: and the review in the actual repo: https://review.openstack.org/#/c/191161/ | 22:04 |
| lifeless | dougwig: ah yes. gnah. | 22:05 |
| dougwig | lifeless: the git clones are gone as of a few hours ago, so i'm not going to stress the requirements file, unless you feel strongly that i need to chase that part. | 22:07 |
| lifeless | dougwig: are the neutron child repos in projects.txt? | 22:07 |
| greghaynes | jeblair: ah, thats from right around when we switch to os-client-config and I think we had a bug where we werent setting the region name in rax | 22:08 |
| dougwig | some of them (neutron-*aas), some of the vendor plugins are not. | 22:08 |
| dougwig | lifeless: ^^ | 22:08 |
| lifeless | so see #openstack-qa | 22:08 |
| jeblair | greghaynes: okay cool | 22:08 |
| *** amuller has left #openstack-infra | 22:08 | |
| lifeless | I think there's a possible needle-hole at the moment | 22:08 |
| *** rm_work is now known as rm_work|away | 22:09 | |
| lifeless | dougwig: but git clones gone++. The rest, I suspect wait :) | 22:09 |
| dougwig | lifeless: ok, good, it's almost beer o'clock here at the mid-cycle. | 22:11 |
| *** _jreslock has quit IRC | 22:11 | |
| *** hichihara has joined #openstack-infra | 22:11 | |
| *** timothyb89 has joined #openstack-infra | 22:13 | |
| *** doug-fish has left #openstack-infra | 22:14 | |
| *** tvanderwiel has quit IRC | 22:16 | |
| *** _diana_ has quit IRC | 22:16 | |
| *** faraujo has quit IRC | 22:17 | |
| openstackgerrit | Merged openstack-infra/project-config: Remove inherit-from from designate-dashboard ACL https://review.openstack.org/193247 | 22:19 |
| krotscheck | I've got a +1 from the ironic PTL on adding https://review.openstack.org/#/c/194424/ , any takers/ | 22:19 |
| krotscheck | ? | 22:19 |
| krotscheck | Similarly, I have a +1 from the QA PTL (mtreinish) on this one https://review.openstack.org/#/c/194938/ | 22:19 |
| krotscheck | :D | 22:19 |
| openstackgerrit | Merged openstack-infra/subunit2sql: Do not use oslo.db 1.12.0 https://review.openstack.org/195656 | 22:23 |
| greghaynes | jeblair: we should be good to respin images | 22:23 |
| greghaynes | (subunit2sql fix landed) | 22:24 |
| openstackgerrit | Michael Krotscheck proposed openstack-infra/project-config: don't run horizon-npm-run-* on kilo https://review.openstack.org/195745 | 22:24 |
| greghaynes | oh, do we have to wait on release? | 22:24 |
| krotscheck | jeblair, fungi: Something about that patch failed in the centos puppet apply. I recalled hitting that yesterday, just did a rebase to see if it'll fix it. ^^ | 22:25 |
| jeblair | krotscheck: that job is flaky and is on its way to being made non-voting until we figure out a cause or move to centos7 | 22:25 |
| jeblair | krotscheck: re+3 | 22:25 |
| cody-somerville | What is responsible for pumping data into graphite re: ELK cluster queue and what not? | 22:25 |
| *** e0ne has quit IRC | 22:26 | |
| krotscheck | jeblair: ++. Jobs should not be flaky. Pastries should be flaky. Jobs are not Pastries. | 22:26 |
| * jeblair is hungry | 22:26 | |
| jeblair | cody-somerville: http://docs.openstack.org/infra/system-config/logstash.html | 22:26 |
| jeblair | cody-somerville: er | 22:26 |
| jeblair | cody-somerville: well, that's responsible for moving data into logstash.... | 22:27 |
| jeblair | cody-somerville: which graphite data are you interested in? | 22:27 |
| cody-somerville | jeblair: logstash job queue that shows up on http://status.openstack.org/zuul/ | 22:27 |
| nibalizer | jeblair: I'm +1 on puppet-apply-centos going nonvoting | 22:27 |
| jeblair | ah that | 22:27 |
| nibalizer | until its fixed, sounds like yolanda and pabelanger are banging on a solution | 22:27 |
| jeblair | cody-somerville: it's geard running on logstash.o.o | 22:28 |
| *** jerryz has quit IRC | 22:28 | |
| cody-somerville | jeblair: Ah, right. That makes sense. Thanks. | 22:28 |
| *** krtaylor has joined #openstack-infra | 22:30 | |
| *** dims has quit IRC | 22:30 | |
| openstackgerrit | Merged openstack-infra/subunit2sql: Add support to subunit2sql for storing attachments https://review.openstack.org/186152 | 22:30 |
| ianw | sorry, there's way too much scrollback -- do we know all the image builds are failing "ImportError: No module named oslo_context" | 22:30 |
| jeblair | ianw: yes, fix just merged and release made | 22:31 |
| jeblair | good timing :) | 22:31 |
| ianw | jeblair: thanks, that was much quicker than me trying to dig around :) | 22:31 |
| mtreinish | jeblair: well I haven't pushed the release yet | 22:32 |
| mtreinish | kinda preoccupied with the qa meeting | 22:32 |
| openstackgerrit | Merged openstack-infra/infra-specs: Fix puppet-apply spec typos https://review.openstack.org/189499 | 22:32 |
| *** dontalton has joined #openstack-infra | 22:32 | |
| *** hichihara has quit IRC | 22:34 | |
| *** _diana_ has joined #openstack-infra | 22:34 | |
| *** Guest525 has quit IRC | 22:35 | |
| *** rm_work|away is now known as rm_work | 22:36 | |
| *** ddieterly has joined #openstack-infra | 22:37 | |
| *** Longgeek has joined #openstack-infra | 22:37 | |
| *** ddieterly has quit IRC | 22:38 | |
| *** dims_ has joined #openstack-infra | 22:38 | |
| *** bhunter71 has quit IRC | 22:39 | |
| notmyname | sc68cal: anteaya: I just got off a plane and am catching up. still things you need me to look at? | 22:40 |
| jeblair | mtreinish: okay, would be good to push that today so that we get new images tomorrow | 22:41 |
| jeblair | ianw: ^ | 22:41 |
| mtreinish | jeblair: yes I'm jumping back and forth between the meeting and writing the tag notes | 22:42 |
| *** Longgeek has quit IRC | 22:42 | |
| openstackgerrit | Paul Belanger proposed openstack-infra/system-config: Do not merge https://review.openstack.org/195288 | 22:43 |
| *** armax has quit IRC | 22:45 | |
| *** dustins_ has joined #openstack-infra | 22:45 | |
| *** otter768 has joined #openstack-infra | 22:45 | |
| *** DericHorn-HP has quit IRC | 22:45 | |
| *** otherwiseguy has quit IRC | 22:45 | |
| *** dontalton has quit IRC | 22:45 | |
| *** alivigni has joined #openstack-infra | 22:46 | |
| *** dhellmann has quit IRC | 22:46 | |
| *** dhellmann has joined #openstack-infra | 22:46 | |
| *** _diana_ has quit IRC | 22:47 | |
| *** baoli has quit IRC | 22:47 | |
| *** EricGonczer_ has quit IRC | 22:47 | |
| *** dustins has quit IRC | 22:47 | |
| *** baoli has joined #openstack-infra | 22:48 | |
| *** otherwiseguy has joined #openstack-infra | 22:48 | |
| *** _diana_ has joined #openstack-infra | 22:50 | |
| *** otter768 has quit IRC | 22:50 | |
| *** armax has joined #openstack-infra | 22:51 | |
| *** rm_work is now known as rm_work|away | 22:52 | |
| mtreinish | jeblair, greghaynes: ok release pushed to gerrit, waiting on the tarball and pypi upload | 22:54 |
| jhesketh | Morning | 22:55 |
| *** dangers is now known as dangers_away | 22:56 | |
| *** dimtruck is now known as zz_dimtruck | 22:57 | |
| *** hogepodge has joined #openstack-infra | 22:58 | |
| *** amitgandhinz has joined #openstack-infra | 22:59 | |
| *** annegentle has joined #openstack-infra | 22:59 | |
| *** camunoz_gone is now known as camunoz | 22:59 | |
| *** sdake_ has joined #openstack-infra | 23:00 | |
| *** hdd has quit IRC | 23:00 | |
| *** hdd has joined #openstack-infra | 23:01 | |
| *** mriedem has joined #openstack-infra | 23:02 | |
| mattoliverau | anteaya: looks like debian/patches/CVE-2013-1752-httplib-2.patch has put in a MAX_HEADERS limit of 100, and this looks to be triggered in some func tests causing the container server to be error limited. clayg is looking at how to monkey patch this blocker now. | 23:03 |
| *** amitgandhinz has quit IRC | 23:03 | |
| *** sdake has quit IRC | 23:04 | |
| mtreinish | jeblair, greghaynes: wow, I didn't realize things were that backed up. It might be a bit before zuul gets to the release jobs | 23:04 |
| greghaynes | mtreinish: haha, yea, there is quite a queue today | 23:04 |
| jeblair | mtreinish: i don't see anything in the release pipeline | 23:05 |
| *** annegentle has quit IRC | 23:05 | |
| mtreinish | jeblair: I don't think zuul has gotten to it yet | 23:05 |
| jeblair | oh you reckon it's in the event queue | 23:06 |
| mtreinish | yeah | 23:06 |
| *** mriedem has quit IRC | 23:06 | |
| *** mriedem has joined #openstack-infra | 23:06 | |
| mtreinish | jeblair: heh, it's only been ~10 mins. http://paste.openstack.org/show/321206/ | 23:07 |
| *** sdake has joined #openstack-infra | 23:07 | |
| *** mestery has quit IRC | 23:07 | |
| *** fallenpegasus has quit IRC | 23:09 | |
| *** sdake_ has quit IRC | 23:11 | |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul: Stop setting build descriptions https://review.openstack.org/195785 | 23:11 |
| jeblair | ...related to that ^ | 23:12 |
| cody-somerville | jeblair: What are build descriptions exactly? | 23:12 |
| jeblair | cody-somerville: the html text in jenkins | 23:12 |
| *** ZZelle_ has quit IRC | 23:13 | |
| *** _diana_ has quit IRC | 23:13 | |
| jeblair | cody-somerville: all of the text between the title and "No changes" on https://jenkins01.openstack.org/job/gate-swift-docs/4670/ | 23:13 |
| cody-somerville | jeblair: Oh. I find that really really useful. | 23:14 |
| jeblair | cody-somerville: cool, what do you use it for? | 23:14 |
| *** dustins_ has quit IRC | 23:14 | |
| cody-somerville | jeblair: If I'm debugging an issue with a job I might jump between jobs for that same change to see if it is systemic issue. | 23:15 |
| cody-somerville | jeblair: also like easy link to the change in gerrit | 23:15 |
| sc68cal | notmyname: yeah I think there's a lot of failures on that job - seeing them affecting multiple patches, not just mine | 23:15 |
| cody-somerville | jeblair: Don't care about the result of the other jobs those I suppose it is nice - I certainly wouldn't mind if it only happened at the end or something. | 23:15 |
| jeblair | cody-somerville: okay, can you -1 that and describe what you use it for? we may be able to reduce it a bit and get the performance benefit and still help you | 23:16 |
| *** larainema has quit IRC | 23:16 | |
| *** thedodd has quit IRC | 23:16 | |
| cody-somerville | kk | 23:16 |
| jeblair | (cause yeah, right now it's _everything concievable_ in real time :) | 23:16 |
| notmyname | sc68cal: I saw someone in -swift say that there was a CVE patch to httplib that limited the number of headers to something way under what we test. might be releated | 23:16 |
| sc68cal | notmyname: gotcha. The tough part is the job doesn't have a testr_results.html or whatever to show more about why the tests failed | 23:17 |
| lifeless | cool, https://review.openstack.org/#/c/194886/ is nearly green | 23:18 |
| *** whoops has quit IRC | 23:20 | |
| *** Sukhdev has joined #openstack-infra | 23:21 | |
| openstackgerrit | fumihiko kakuma proposed openstack-infra/puppet-jenkins: Wait jenkins by '--wait' option at jenkins-jobs command https://review.openstack.org/182868 | 23:21 |
| *** _diana_ has joined #openstack-infra | 23:22 | |
| *** rbrooker has quit IRC | 23:22 | |
| *** sabeen1 has quit IRC | 23:23 | |
| *** armax has quit IRC | 23:24 | |
| *** markvoelker has quit IRC | 23:24 | |
| *** otter768 has joined #openstack-infra | 23:26 | |
| jeblair | mtreinish, ianw: https://pypi.python.org/pypi/subunit2sql/0.6.0 | 23:26 |
| *** xarses has joined #openstack-infra | 23:26 | |
| xarses | So I'm likely doing something wrong here. I want to re-trigger https://review.openstack.org/#/c/179645/ but it doesn't seem to respond to the command. Is there something I'm missing here? | 23:28 |
| jeblair | xarses: see http://status.openstack.org/zuul/ it's in the queue | 23:30 |
| xarses | jeblair: :facepalm: yep, its there now, took longer to show up than I expected | 23:32 |
| *** rlandy has joined #openstack-infra | 23:32 | |
| xarses | the refresh also appears to mess with ^f | 23:32 |
| *** waynr has quit IRC | 23:33 | |
| *** cindyo has joined #openstack-infra | 23:33 | |
| jeblair | yeah, the refresh is pretty simple and just replaces everything | 23:33 |
| tonyb | If possible can a couple of people look at: https://review.openstack.org/#/c/194488/ and https://review.openstack.org/#/c/194504/ | 23:33 |
| *** waynr has joined #openstack-infra | 23:33 | |
| tonyb | fungi: your input on the first one would be great (as you created the publisher in the first place) | 23:33 |
| openstackgerrit | fumihiko kakuma proposed openstack-infra/puppet-jenkins: Wait jenkins by '--wait' option at jenkins-jobs command https://review.openstack.org/182868 | 23:34 |
| openstackgerrit | Khai Do proposed stackforge/python-jenkins: Support scriptText api to execute groovy scripts on the server https://review.openstack.org/194468 | 23:35 |
| openstackgerrit | Clint 'SpamapS' Byrum proposed openstack-infra/system-config: Initial effort on a single-node cloud manifest https://review.openstack.org/195276 | 23:36 |
| *** hichihara has joined #openstack-infra | 23:36 | |
| *** sputnik13 has quit IRC | 23:39 | |
| mtreinish | jeblair: great, now the image build should pass | 23:40 |
| SpamapS | crinkle: ^ updates | 23:41 |
| SpamapS | greghaynes: ^ | 23:41 |
| SpamapS | pabelanger: ^ | 23:41 |
| mtreinish | oh, there was a new migration script to add a table in the release, but I don't think it should cause any issues | 23:41 |
| crinkle | SpamapS: \o/ | 23:41 |
| SpamapS | need to EOD now, but I have ironic API at least responding w/ keystone auth | 23:41 |
| SpamapS | will plod forward on glance+nova+neutron tomorrow and then the fun part: enrolling hardware | 23:41 |
| mtreinish | fungi, jeblair: oh awesome the puppet to do the migration actually worked automagically | 23:42 |
| mtreinish | because it was jsut adding a new table it didn't hit the ansible timeout like the last one :) | 23:42 |
| SpamapS | and then I'm in Vegas for the weekend so I _may_ either come back as a zombie or disappear into the desert next week. ;) | 23:42 |
| *** ddieterly has joined #openstack-infra | 23:43 | |
| *** hdd has quit IRC | 23:43 | |
| openstackgerrit | Steve Leon proposed openstack-infra/shade: Adding SSL arguments to glance client https://review.openstack.org/195793 | 23:44 |
| *** juggler_ is now known as juggler | 23:44 | |
| *** esker has joined #openstack-infra | 23:46 | |
| *** esker has quit IRC | 23:46 | |
| *** pc_m has quit IRC | 23:46 | |
| *** esker has joined #openstack-infra | 23:47 | |
| *** ramashri has quit IRC | 23:47 | |
| openstackgerrit | Merged openstack-infra/project-config: Upload other assets with irc ical publish https://review.openstack.org/185213 | 23:48 |
| *** chlong has joined #openstack-infra | 23:48 | |
| *** pc_m has joined #openstack-infra | 23:49 | |
| *** otter768 has quit IRC | 23:49 | |
| *** Swami has quit IRC | 23:49 | |
| *** erlon has quit IRC | 23:51 | |
| *** hichihara has quit IRC | 23:51 | |
| *** hichihara has joined #openstack-infra | 23:51 | |
| *** pc_m has quit IRC | 23:51 | |
| *** faraujo has joined #openstack-infra | 23:51 | |
| *** leseb has quit IRC | 23:52 | |
| *** puranamr_ has quit IRC | 23:53 | |
| *** laughterwym has joined #openstack-infra | 23:53 | |
| *** sdake has quit IRC | 23:54 | |
| *** leseb has joined #openstack-infra | 23:55 | |
| *** sdake has joined #openstack-infra | 23:55 | |
| anteaya | notmyname mattoliverau thank you! | 23:55 |
| *** camunoz is now known as camunoz_mtg | 23:56 | |
| *** Sukhdev has quit IRC | 23:56 | |
| anteaya | I'm off for dinner now at the neutron mid-cycle, will be back looking at this again tomorrow | 23:56 |
| anteaya | thank you | 23:56 |
| *** esker has quit IRC | 23:57 | |
| *** Rockyg has quit IRC | 23:57 | |
| openstackgerrit | Khai Do proposed openstack-infra/jenkins-job-builder: Allow JJB to set connection timeout to jenkins server https://review.openstack.org/151472 | 23:58 |
| *** rlandy has quit IRC | 23:59 | |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!
