| *** olaph1 has joined #openstack-infra | 00:01 | |
| *** olaph has quit IRC | 00:02 | |
| *** tosky has quit IRC | 00:04 | |
| *** danpawlik has quit IRC | 00:07 | |
| *** edmondsw has joined #openstack-infra | 00:09 | |
| openstackgerrit | Ian Wienand proposed openstack-infra/puppet-askbot master: systemd support https://review.openstack.org/559601 | 00:13 |
|---|---|---|
| openstackgerrit | Ian Wienand proposed openstack-infra/puppet-askbot master: Basic rspec test https://review.openstack.org/558991 | 00:13 |
| *** edmondsw has quit IRC | 00:14 | |
| *** odyssey4me has quit IRC | 00:23 | |
| *** odyssey4me has joined #openstack-infra | 00:23 | |
| *** danpawlik has joined #openstack-infra | 00:38 | |
| openstackgerrit | Ian Wienand proposed openstack-infra/puppet-askbot master: systemd support https://review.openstack.org/559601 | 00:43 |
| openstackgerrit | Ian Wienand proposed openstack-infra/puppet-askbot master: Basic rspec test https://review.openstack.org/558991 | 00:43 |
| *** danpawlik has quit IRC | 00:47 | |
| *** danpawlik has joined #openstack-infra | 00:48 | |
| *** shu-mutow-AWAY is now known as shu-mutow | 00:50 | |
| *** danpawlik has quit IRC | 00:57 | |
| ianw | Creating new volume 536870950 on replication site afs02.dfw.openstack.org: Failed to create the ro volume: : Input/output error | 00:59 |
| ianw | this does not seem good | 00:59 |
| *** hemna_ has joined #openstack-infra | 01:01 | |
| *** hemna_ has quit IRC | 01:05 | |
| *** dhill_ has quit IRC | 01:06 | |
| ianw | "This is a recovery of previously failed release | 01:08 |
| ianw | " might be the key here | 01:08 |
| *** dhill_ has joined #openstack-infra | 01:11 | |
| *** bobh has quit IRC | 01:13 | |
| *** salv-orlando has quit IRC | 01:16 | |
| *** salv-orlando has joined #openstack-infra | 01:16 | |
| *** bobh has joined #openstack-infra | 01:19 | |
| *** jamesmcarthur has joined #openstack-infra | 01:20 | |
| *** dhill_ has quit IRC | 01:23 | |
| *** dhill_ has joined #openstack-infra | 01:23 | |
| ianw | /dev/mapper/main-vicepa 3.0T 3.0T 8.1M 100% /vicepa | 01:24 |
| ianw | and i would say that's the problem ... | 01:24 |
| *** bobh has quit IRC | 01:24 | |
| *** hongbin has joined #openstack-infra | 01:27 | |
| *** danpawlik has joined #openstack-infra | 01:28 | |
| *** bobh has joined #openstack-infra | 01:33 | |
| *** danpawlik has quit IRC | 01:37 | |
| *** bobh has quit IRC | 01:38 | |
| *** bobh has joined #openstack-infra | 01:43 | |
| *** bobh has quit IRC | 01:48 | |
| *** tonyb has joined #openstack-infra | 01:50 | |
| *** larainema has joined #openstack-infra | 01:52 | |
| *** jamesmcarthur has quit IRC | 01:57 | |
| *** edmondsw has joined #openstack-infra | 01:57 | |
| *** alex_xu has joined #openstack-infra | 01:58 | |
| *** stakeda has quit IRC | 02:01 | |
| *** edmondsw has quit IRC | 02:02 | |
| *** bobh has joined #openstack-infra | 02:02 | |
| *** stakeda has joined #openstack-infra | 02:04 | |
| *** olaph1 has quit IRC | 02:05 | |
| *** olaph has joined #openstack-infra | 02:05 | |
| *** bobh has quit IRC | 02:07 | |
| *** danpawlik has joined #openstack-infra | 02:08 | |
| *** slaweq has joined #openstack-infra | 02:10 | |
| *** ramishra has joined #openstack-infra | 02:14 | |
| *** slaweq has quit IRC | 02:15 | |
| auristor | Free space on server afs02.dfw.openstack.org:7005 partition /vicepa: 8216 K blocks out of total 3170520488 | 02:15 |
| auristor | openafs doesn't track free space particularly well and running out of space can result in data corruption when writing. I suggestion monitoring free space and not filling a partition above 90% | 02:17 |
| ianw | auristor: yeah, i'm adding a volume now | 02:18 |
| *** bobh has joined #openstack-infra | 02:18 | |
| auristor | **** Could not attach volume 536870983 **** | 02:20 |
| auristor | **** Could not attach volume 536870950 **** | 02:20 |
| *** danpawlik has quit IRC | 02:22 | |
| *** jamesmcarthur has joined #openstack-infra | 02:22 | |
| *** bobh has quit IRC | 02:23 | |
| auristor | vos zap both of those volumes | 02:23 |
| auristor | they are partial clones of mirror.debian.readonly and mirror.ubuntu.readonly on afs02.dfw.openstack.org | 02:24 |
| *** jamesmcarthur has quit IRC | 02:25 | |
| *** jamesmcarthur has joined #openstack-infra | 02:25 | |
| *** markvoelker has quit IRC | 02:28 | |
| *** markvoelker has joined #openstack-infra | 02:29 | |
| *** bobh has joined #openstack-infra | 02:32 | |
| *** dayou has quit IRC | 02:33 | |
| *** markvoelker has quit IRC | 02:34 | |
| *** mikal_ has quit IRC | 02:36 | |
| *** mikal has joined #openstack-infra | 02:36 | |
| *** bobh has quit IRC | 02:36 | |
| ianw | auristor: hmmm ... http://paste.openstack.org/show/718692/ | 02:37 |
| ianw | this is my first time with vos zap | 02:37 |
| *** dayou has joined #openstack-infra | 02:39 | |
| *** gagehugo has quit IRC | 02:40 | |
| auristor | bos salvage afs02.dfw.openstack.org a 536870950 -forceDAFS | 02:40 |
| auristor | its possible that might fail as well due to the lack of free space | 02:43 |
| *** bobh has joined #openstack-infra | 02:43 | |
| *** VW has joined #openstack-infra | 02:45 | |
| ianw | auristor: ok cool, it seems the salvage worked. i can examine it again now | 02:46 |
| auristor | when the new volume is added to the server and mounted as vicepb you will need to restart the dafs bnode. | 02:47 |
| auristor | bos restart afs02.dfw.openstack.org dafs | 02:47 |
| ianw | auristor: oh, it's all on lvm, so i've just grown /vicepa | 02:47 |
| auristor | ok. you should still restart the fileserver so it sees the additional sapce | 02:48 |
| *** liujiong has joined #openstack-infra | 02:48 | |
| *** gagehugo has joined #openstack-infra | 02:48 | |
| *** bobh has quit IRC | 02:48 | |
| ianw | cool; i'll just reboot the host, that will clear up everything | 02:49 |
| auristor | when its back you will need to re-release mirror.debian and mirror.ubuntu | 02:50 |
| *** caphrim007 has quit IRC | 02:51 | |
| ianw | ok, it's "Deleting extant RO_DONTUSE site on afs02.dfw.openstack.org..." but going | 02:52 |
| *** danpawlik has joined #openstack-infra | 02:53 | |
| *** bobh has joined #openstack-infra | 02:53 | |
| auristor | its going to take a while. | 02:54 |
| *** caphrim007 has joined #openstack-infra | 02:58 | |
| *** bobh has quit IRC | 02:58 | |
| ianw | cool, i've got the locks to stop the cron jobs conflicting | 03:01 |
| *** caphrim007 has quit IRC | 03:02 | |
| *** danpawlik has quit IRC | 03:02 | |
| *** danpawlik has joined #openstack-infra | 03:03 | |
| *** bobh has joined #openstack-infra | 03:03 | |
| ianw | it must have been adding arm64 to the debian mirror that did this. i checked space on afs01, but now i think about it, i probably didn't on afs02 | 03:05 |
| *** jamesmcarthur has quit IRC | 03:06 | |
| *** imacdonn has joined #openstack-infra | 03:08 | |
| *** bobh has quit IRC | 03:08 | |
| *** danpawlik has quit IRC | 03:12 | |
| *** danpawlik has joined #openstack-infra | 03:13 | |
| *** VW_ has joined #openstack-infra | 03:15 | |
| *** bobh has joined #openstack-infra | 03:15 | |
| *** VW has quit IRC | 03:18 | |
| *** sree has joined #openstack-infra | 03:19 | |
| *** bobh has quit IRC | 03:20 | |
| *** danpawlik has quit IRC | 03:22 | |
| *** spiffxp has quit IRC | 03:28 | |
| *** vipul has quit IRC | 03:29 | |
| *** Kevin_Zheng has joined #openstack-infra | 03:30 | |
| *** spiffxp has joined #openstack-infra | 03:36 | |
| *** bobh has joined #openstack-infra | 03:37 | |
| *** jamesmcarthur has joined #openstack-infra | 03:37 | |
| *** vipul has joined #openstack-infra | 03:41 | |
| *** bobh has quit IRC | 03:42 | |
| *** bobh has joined #openstack-infra | 03:42 | |
| openstackgerrit | Ian Wienand proposed openstack-infra/puppet-askbot master: systemd support https://review.openstack.org/559601 | 03:45 |
| openstackgerrit | Ian Wienand proposed openstack-infra/puppet-askbot master: Basic rspec test https://review.openstack.org/558991 | 03:45 |
| *** edmondsw has joined #openstack-infra | 03:45 | |
| *** annp has joined #openstack-infra | 03:46 | |
| *** bobh has quit IRC | 03:47 | |
| *** edmondsw has quit IRC | 03:50 | |
| *** danpawlik has joined #openstack-infra | 03:53 | |
| *** bobh has joined #openstack-infra | 03:53 | |
| *** vipul has quit IRC | 03:53 | |
| *** janki has joined #openstack-infra | 03:53 | |
| *** vipul has joined #openstack-infra | 03:54 | |
| openstackgerrit | Merged openstack-infra/irc-meetings master: Update requirements meeting time https://review.openstack.org/557453 | 03:56 |
| *** bobh has quit IRC | 03:58 | |
| *** zhangfei has joined #openstack-infra | 04:00 | |
| *** bobh has joined #openstack-infra | 04:02 | |
| *** danpawlik has quit IRC | 04:02 | |
| *** VW_ has quit IRC | 04:02 | |
| *** toshii has joined #openstack-infra | 04:03 | |
| toshii | has the job queue been reset recently? I see a +W'd patch doesn't appear in zuul.openstack.org | 04:04 |
| *** hongbin has quit IRC | 04:05 | |
| *** bobh has quit IRC | 04:07 | |
| *** jamesmcarthur has quit IRC | 04:09 | |
| *** shu-mutow has quit IRC | 04:09 | |
| dmsimard | toshii: When that happens, it's maybe because there is a depends-on patch or a parent patch that hasn't landed | 04:09 |
| dmsimard | (or the depends-on patch landed and you need to do a recheck or +W again if the changes aren't queued together) | 04:10 |
| toshii | dmsimard, I looked over depends-on line. thanks | 04:10 |
| *** shu-mutow has joined #openstack-infra | 04:10 | |
| openstackgerrit | Ian Wienand proposed openstack-infra/puppet-askbot master: systemd support https://review.openstack.org/559601 | 04:13 |
| openstackgerrit | Ian Wienand proposed openstack-infra/puppet-askbot master: Basic rspec test https://review.openstack.org/558991 | 04:13 |
| *** udesale has joined #openstack-infra | 04:17 | |
| *** psachin has joined #openstack-infra | 04:19 | |
| *** links has joined #openstack-infra | 04:25 | |
| *** bobh has joined #openstack-infra | 04:25 | |
| *** bobh has quit IRC | 04:30 | |
| *** andreas_s has joined #openstack-infra | 04:30 | |
| *** jchhatbar has joined #openstack-infra | 04:30 | |
| *** bobh has joined #openstack-infra | 04:32 | |
| *** janki has quit IRC | 04:33 | |
| *** danpawlik has joined #openstack-infra | 04:33 | |
| *** andreas_s has quit IRC | 04:35 | |
| *** bobh has quit IRC | 04:36 | |
| *** danpawlik has quit IRC | 04:42 | |
| *** dhajare has joined #openstack-infra | 04:44 | |
| *** bobh has joined #openstack-infra | 04:45 | |
| openstackgerrit | Ian Wienand proposed openstack-infra/puppet-askbot master: systemd support https://review.openstack.org/559601 | 04:46 |
| openstackgerrit | Ian Wienand proposed openstack-infra/puppet-askbot master: Basic rspec test https://review.openstack.org/558991 | 04:46 |
| *** jamesmcarthur has joined #openstack-infra | 04:47 | |
| *** bobh has quit IRC | 04:50 | |
| *** dhajare has quit IRC | 04:52 | |
| *** dhajare has joined #openstack-infra | 04:53 | |
| openstackgerrit | Keiichi Hikita proposed openstack-infra/project-config master: Follow-up change for adding slogging project https://review.openstack.org/559465 | 04:55 |
| *** bobh has joined #openstack-infra | 04:55 | |
| *** bobh has quit IRC | 05:00 | |
| openstackgerrit | Ian Wienand proposed openstack-infra/puppet-askbot master: rspec test https://review.openstack.org/558991 | 05:01 |
| openstackgerrit | Ian Wienand proposed openstack-infra/puppet-askbot master: fix python deps https://review.openstack.org/559622 | 05:01 |
| openstackgerrit | Ian Wienand proposed openstack-infra/puppet-askbot master: ensure ordering https://review.openstack.org/559623 | 05:01 |
| openstackgerrit | Ian Wienand proposed openstack-infra/puppet-askbot master: Use earlier six https://review.openstack.org/559624 | 05:01 |
| *** hemna_ has joined #openstack-infra | 05:03 | |
| *** hemna_ has quit IRC | 05:08 | |
| *** danpawlik has joined #openstack-infra | 05:13 | |
| *** bobh has joined #openstack-infra | 05:17 | |
| openstackgerrit | Ian Wienand proposed openstack-infra/project-config master: Ensure /etc/resolv.conf is a regular file in final image https://review.openstack.org/557842 | 05:18 |
| *** pgadiya has joined #openstack-infra | 05:21 | |
| *** danpawlik has quit IRC | 05:22 | |
| *** bobh has quit IRC | 05:22 | |
| *** danpawlik has joined #openstack-infra | 05:23 | |
| *** vivsoni has joined #openstack-infra | 05:26 | |
| openstackgerrit | Ian Wienand proposed openstack-infra/puppet-askbot master: systemd support https://review.openstack.org/559601 | 05:26 |
| openstackgerrit | Ian Wienand proposed openstack-infra/puppet-askbot master: fix python deps https://review.openstack.org/559622 | 05:26 |
| openstackgerrit | Ian Wienand proposed openstack-infra/puppet-askbot master: ensure ordering https://review.openstack.org/559623 | 05:26 |
| openstackgerrit | Ian Wienand proposed openstack-infra/puppet-askbot master: Use earlier six https://review.openstack.org/559624 | 05:26 |
| openstackgerrit | Ian Wienand proposed openstack-infra/puppet-askbot master: rspec test https://review.openstack.org/558991 | 05:26 |
| *** udesale_ has joined #openstack-infra | 05:30 | |
| *** olaph1 has joined #openstack-infra | 05:31 | |
| *** olaph has quit IRC | 05:32 | |
| *** udesale has quit IRC | 05:33 | |
| *** edmondsw has joined #openstack-infra | 05:33 | |
| *** bobh has joined #openstack-infra | 05:38 | |
| *** edmondsw has quit IRC | 05:38 | |
| *** bobh has quit IRC | 05:42 | |
| *** zhurong has joined #openstack-infra | 05:44 | |
| *** bobh has joined #openstack-infra | 05:45 | |
| *** jtomasek has joined #openstack-infra | 05:48 | |
| *** bobh has quit IRC | 05:50 | |
| *** danpawlik has quit IRC | 05:52 | |
| *** bobh has joined #openstack-infra | 05:53 | |
| openstackgerrit | Ian Wienand proposed openstack-infra/puppet-askbot master: systemd support https://review.openstack.org/559601 | 05:53 |
| openstackgerrit | Ian Wienand proposed openstack-infra/puppet-askbot master: fix python deps https://review.openstack.org/559622 | 05:53 |
| openstackgerrit | Ian Wienand proposed openstack-infra/puppet-askbot master: ensure ordering https://review.openstack.org/559623 | 05:53 |
| openstackgerrit | Ian Wienand proposed openstack-infra/puppet-askbot master: Use earlier six https://review.openstack.org/559624 | 05:53 |
| openstackgerrit | Ian Wienand proposed openstack-infra/puppet-askbot master: rspec test https://review.openstack.org/558991 | 05:53 |
| *** bobh has quit IRC | 05:57 | |
| openstackgerrit | Joshua Hesketh proposed openstack-infra/system-config master: Make links in CI results optional https://review.openstack.org/559634 | 05:58 |
| *** pcichy has joined #openstack-infra | 05:59 | |
| openstackgerrit | Joshua Hesketh proposed openstack-infra/system-config master: Add gerrit commentlink for noop job in zuulv3 https://review.openstack.org/559006 | 05:59 |
| *** jbadiapa has joined #openstack-infra | 06:00 | |
| *** berndbausch has joined #openstack-infra | 06:03 | |
| *** bobh has joined #openstack-infra | 06:03 | |
| *** vipul has quit IRC | 06:07 | |
| *** spiffxp has quit IRC | 06:07 | |
| *** bobh has quit IRC | 06:08 | |
| *** spiffxp has joined #openstack-infra | 06:10 | |
| *** AJaeger has quit IRC | 06:10 | |
| *** slaweq has joined #openstack-infra | 06:12 | |
| *** d0ugal has quit IRC | 06:12 | |
| berndbausch | ianw: I tried following the instructions for 3rd party CI setup. First failure because when I follow them unchanged, they install an antediluvian version of nodepool. Second failure because the current version of nodepool seems to assume Python3, but the instructions seem to use Python2.7 only. My goal is to learn about Infra so that I can contribute, be it documentation or code. I am not sure how to continue now. | 06:13 |
| berndbausch | ianw: Would it make sense to find the "right" version of nodepool? | 06:13 |
| berndbausch | ianw: Which matches the instructions? | 06:13 |
| *** bobh has joined #openstack-infra | 06:13 | |
| *** vipul has joined #openstack-infra | 06:14 | |
| berndbausch | ianw: Or figuring out how to make the instructions work with the current nodepool version (much more work, I guess, and might require more skills than I currently have,but perhaps the right approach)? | 06:14 |
| ianw | berndbausch: yeah, so as you're seeing i think you've probably jumped in on this between the nodepool/zuul v2->v3 transition, and we're not quite there yet with all this :) | 06:15 |
| berndbausch | third option would be abandoning this fight and look for something else to cut my teeth on :) | 06:15 |
| *** snapiri- has quit IRC | 06:16 | |
| *** snapiri has joined #openstack-infra | 06:16 | |
| berndbausch | ianw: Is there an area where I can learn without getting caught in this transition? | 06:16 |
| *** slaweq has quit IRC | 06:17 | |
| *** bobh has quit IRC | 06:17 | |
| *** lpetrut has joined #openstack-infra | 06:18 | |
| ianw | berndbausch: well, yes ... your goal was mostly to set it up to see how it worked, rather than needing to report ci results on a particular project? | 06:18 |
| berndbausch | ianw: Exactly. I am just a hobbyist. | 06:19 |
| berndbausch | ianw: And, as a side effect, understand better how Infra is set up. | 06:20 |
| *** bobh has joined #openstack-infra | 06:22 | |
| *** danpawlik has joined #openstack-infra | 06:23 | |
| ianw | ok, i don't want to turn you off working on the 3rd party ci stuff, it is certainly valuable to the community to have some focus on that | 06:25 |
| ianw | berndbausch: have you looked at https://git.openstack.org/cgit/openstack-infra/system-config/tree/manifests/site.pp#n1166 ? | 06:26 |
| *** bobh has quit IRC | 06:27 | |
| *** namnh has joined #openstack-infra | 06:29 | |
| *** AJaeger has joined #openstack-infra | 06:31 | |
| *** armaan has joined #openstack-infra | 06:32 | |
| berndbausch | ianw: That's the nodepool server declaration. How can I use it? (I could help by removing the hpcloud credentials:) | 06:32 |
| berndbausch | ianw: Well before you answer, perhaps I should spend a few minutes studying it. | 06:33 |
| *** andreas_s has joined #openstack-infra | 06:33 | |
| ianw | berndbausch: right, i'm sure there's a review out there to clean up some of the credentials unmerged | 06:34 |
| ianw | but, i'm not sure if the 3rd party ci instructions would be referencing setting up a builder at all? | 06:34 |
| *** pcaruana has joined #openstack-infra | 06:35 | |
| berndbausch | ianw: Looks like something I can investigate. Leave me with this before I bug you again. | 06:35 |
| ianw | berndbausch: you starting from http://git.openstack.org/cgit/openstack-infra/puppet-openstackci/tree/manifests/single_node_ci.pp#n272 ? | 06:36 |
| ianw | oh, well that just bails for zuulv3 | 06:37 |
| ianw | http://git.openstack.org/cgit/openstack-infra/puppet-openstackci/tree/manifests/single_node_ci.pp#n312 | 06:37 |
| *** bobh has joined #openstack-infra | 06:38 | |
| berndbausch | ianw: some of the fog is disappearing, thanks! | 06:39 |
| *** armaan has quit IRC | 06:41 | |
| *** armaan has joined #openstack-infra | 06:41 | |
| berndbausch | ianw: So it won't work with current nodepool or current anything without a major rewrite. Definitely not my cup of tea for now. I will look for the latest pre-Python3 nodepool version. | 06:41 |
| *** andreas_s has quit IRC | 06:42 | |
| *** oanson has quit IRC | 06:42 | |
| *** bobh has quit IRC | 06:43 | |
| *** oanson has joined #openstack-infra | 06:44 | |
| berndbausch | ianw: You have given me enough to occupy me for a few hours. Have a great evening. | 06:44 |
| *** ethfci has quit IRC | 06:45 | |
| *** namnh has quit IRC | 06:46 | |
| *** udesale__ has joined #openstack-infra | 06:46 | |
| *** namnh has joined #openstack-infra | 06:46 | |
| *** udesale_ has quit IRC | 06:49 | |
| *** ralonsoh has joined #openstack-infra | 06:54 | |
| *** bobh has joined #openstack-infra | 06:55 | |
| *** dhajare has quit IRC | 06:56 | |
| *** slaweq has joined #openstack-infra | 06:56 | |
| AJaeger | config-core, could you review https://review.openstack.org/#/c/559415/ to fix docs publishing to developer.o.o, please? | 06:57 |
| *** hashar has joined #openstack-infra | 06:58 | |
| *** neiloy has joined #openstack-infra | 06:58 | |
| *** bobh has quit IRC | 07:00 | |
| *** bobh has joined #openstack-infra | 07:06 | |
| *** florianf has joined #openstack-infra | 07:08 | |
| *** agopi_ has quit IRC | 07:09 | |
| *** bobh has quit IRC | 07:11 | |
| *** zhurong has quit IRC | 07:13 | |
| openstackgerrit | Merged openstack-infra/project-config master: add missing afs sync jobs for developer.o.o and docs.o.o https://review.openstack.org/559415 | 07:13 |
| *** spiffxp_ has joined #openstack-infra | 07:13 | |
| *** tesseract has joined #openstack-infra | 07:14 | |
| *** namnh has quit IRC | 07:14 | |
| *** namnh has joined #openstack-infra | 07:14 | |
| *** spiffxp has quit IRC | 07:15 | |
| *** vipul has quit IRC | 07:16 | |
| *** rcernin has quit IRC | 07:16 | |
| *** spiffxp_ has quit IRC | 07:17 | |
| *** vipul has joined #openstack-infra | 07:17 | |
| *** spiffxp has joined #openstack-infra | 07:19 | |
| *** Hal has joined #openstack-infra | 07:21 | |
| *** edmondsw has joined #openstack-infra | 07:21 | |
| ianw | berndbausch: i've had some more thoughts; i wonder what people like corvus are thinking of the long term future of this 3rd party ci type instructions | 07:24 |
| *** lpetrut has quit IRC | 07:25 | |
| ianw | berndbausch: i'm sure you've seen https://zuul-ci.org/ which is/has extracted zuul from being a wholly openstack thing to where it is now | 07:25 |
| *** andreas_s has joined #openstack-infra | 07:26 | |
| ianw | where puppet modules to deploy this live is probably a very good area for discussion and i'm sure some of the developers have thoughts | 07:26 |
| *** edmondsw has quit IRC | 07:26 | |
| *** bobh has joined #openstack-infra | 07:27 | |
| openstackgerrit | Ian Wienand proposed openstack-infra/puppet-askbot master: systemd support https://review.openstack.org/559601 | 07:28 |
| openstackgerrit | Ian Wienand proposed openstack-infra/puppet-askbot master: fix python deps https://review.openstack.org/559622 | 07:28 |
| openstackgerrit | Ian Wienand proposed openstack-infra/puppet-askbot master: ensure ordering https://review.openstack.org/559623 | 07:28 |
| openstackgerrit | Ian Wienand proposed openstack-infra/puppet-askbot master: Use earlier six https://review.openstack.org/559624 | 07:28 |
| openstackgerrit | Ian Wienand proposed openstack-infra/puppet-askbot master: rspec test https://review.openstack.org/558991 | 07:28 |
| *** zoli is now known as zoli|wfh | 07:29 | |
| *** zoli|wfh is now known as zoli | 07:29 | |
| ianw | ok, the fixed ubuntu volume is released, moving onto debian | 07:29 |
| *** bobh has quit IRC | 07:31 | |
| openstackgerrit | Ian Wienand proposed openstack-infra/project-config master: Ensure /etc/resolv.conf is a regular file in final image https://review.openstack.org/557842 | 07:32 |
| *** d0ugal has joined #openstack-infra | 07:32 | |
| *** d0ugal has quit IRC | 07:32 | |
| *** d0ugal has joined #openstack-infra | 07:32 | |
| *** bobh has joined #openstack-infra | 07:33 | |
| *** abyz has joined #openstack-infra | 07:33 | |
| *** jistr is now known as jistr|dentist | 07:34 | |
| *** tosky has joined #openstack-infra | 07:37 | |
| *** amoralej|off is now known as amoralej | 07:37 | |
| *** bobh has quit IRC | 07:37 | |
| berndbausch | ianw: I know that the Zuul project has a life outside of OpenStack, but wasn't aware that it could replace the entire CI. Or can it? | 07:39 |
| *** dhajare has joined #openstack-infra | 07:40 | |
| *** electrofelix has joined #openstack-infra | 07:41 | |
| *** katkapilatova has joined #openstack-infra | 07:47 | |
| *** rossella_s has joined #openstack-infra | 07:49 | |
| *** jpena|off is now known as jpena | 07:50 | |
| *** katkapilatova has quit IRC | 07:51 | |
| *** lpetrut has joined #openstack-infra | 07:51 | |
| *** bobh has joined #openstack-infra | 07:52 | |
| *** lucas-afk is now known as lucasagomes | 07:56 | |
| *** bobh has quit IRC | 07:57 | |
| *** lpetrut has quit IRC | 08:02 | |
| *** rossella_s has quit IRC | 08:03 | |
| *** bobh has joined #openstack-infra | 08:03 | |
| *** shardy has joined #openstack-infra | 08:05 | |
| *** shardy has quit IRC | 08:05 | |
| *** jpich has joined #openstack-infra | 08:05 | |
| *** rossella_s has joined #openstack-infra | 08:05 | |
| *** shardy has joined #openstack-infra | 08:06 | |
| openstackgerrit | Ian Wienand proposed openstack-infra/puppet-askbot master: systemd support https://review.openstack.org/559601 | 08:07 |
| openstackgerrit | Ian Wienand proposed openstack-infra/puppet-askbot master: fix python deps https://review.openstack.org/559622 | 08:07 |
| openstackgerrit | Ian Wienand proposed openstack-infra/puppet-askbot master: ensure ordering https://review.openstack.org/559623 | 08:07 |
| openstackgerrit | Ian Wienand proposed openstack-infra/puppet-askbot master: Use earlier six https://review.openstack.org/559624 | 08:07 |
| openstackgerrit | Ian Wienand proposed openstack-infra/puppet-askbot master: rspec test https://review.openstack.org/558991 | 08:07 |
| *** bobh has quit IRC | 08:08 | |
| *** pcichy has quit IRC | 08:10 | |
| *** pcichy has joined #openstack-infra | 08:10 | |
| AJaeger | berndbausch: Zuul can for sure - that's how we use it. | 08:12 |
| *** witek has joined #openstack-infra | 08:13 | |
| *** rossella_s has quit IRC | 08:13 | |
| witek | hello, could someone please have a look at https://review.openstack.org/555237 ? | 08:14 |
| *** rossella_s has joined #openstack-infra | 08:15 | |
| *** dsariel has joined #openstack-infra | 08:17 | |
| *** bobh has joined #openstack-infra | 08:18 | |
| *** danpawlik has quit IRC | 08:19 | |
| *** dbecker has joined #openstack-infra | 08:20 | |
| *** bobh has quit IRC | 08:23 | |
| openstackgerrit | Simon Westphahl proposed openstack-infra/zuul master: Allow using remote refs to find commits for change https://review.openstack.org/544964 | 08:25 |
| *** s-shiono has quit IRC | 08:31 | |
| *** derekh has joined #openstack-infra | 08:32 | |
| *** shardy has quit IRC | 08:36 | |
| *** shardy has joined #openstack-infra | 08:37 | |
| ianw | berndbausch: well for third party ci, i imagine the long term story will be about how to plug zuul into openstack; more so than "here's how to replicate openstack infra" | 08:40 |
| *** rossella_s has quit IRC | 08:42 | |
| *** rossella_s has joined #openstack-infra | 08:42 | |
| *** berndbausch has quit IRC | 08:43 | |
| *** danpawlik has joined #openstack-infra | 08:44 | |
| *** efoley has joined #openstack-infra | 08:47 | |
| *** bobh has joined #openstack-infra | 08:51 | |
| openstackgerrit | Ian Wienand proposed openstack-infra/puppet-askbot master: fix python deps https://review.openstack.org/559622 | 08:52 |
| openstackgerrit | Ian Wienand proposed openstack-infra/puppet-askbot master: ensure ordering https://review.openstack.org/559623 | 08:52 |
| openstackgerrit | Ian Wienand proposed openstack-infra/puppet-askbot master: Use earlier six https://review.openstack.org/559624 | 08:52 |
| openstackgerrit | Ian Wienand proposed openstack-infra/puppet-askbot master: rspec test https://review.openstack.org/558991 | 08:52 |
| openstackgerrit | Witold Bedyk proposed openstack-infra/project-config master: Set openstack-tox-cover as voting for monasca-ui https://review.openstack.org/553368 | 08:52 |
| *** danpawlik has quit IRC | 08:54 | |
| *** bobh has quit IRC | 08:55 | |
| *** sambetts|afk is now known as sambetts | 08:55 | |
| *** toshii has quit IRC | 08:58 | |
| frickler | infra-root: openstackrecheck starting sending regular "Console logs not available" messages at about 04:00z, maybe related to the AFS issue? | 08:59 |
| frickler | also http://status.openstack.org/elastic-recheck/ has "Data Last Updated: Mon Apr 09 2018 01:30:01 GMT+0000 (UTC)" | 08:59 |
| * frickler will have a meeting now before being able to do more looking | 09:00 | |
| *** HeOS has joined #openstack-infra | 09:00 | |
| *** jogo has quit IRC | 09:01 | |
| openstackgerrit | Bharat Kunwar proposed openstack/diskimage-builder master: Fix yaml dependency issue with dracut-regenerate for python3 https://review.openstack.org/558497 | 09:01 |
| *** bobh has joined #openstack-infra | 09:01 | |
| *** jistr|dentist is now known as jistr | 09:02 | |
| *** alex_xu has quit IRC | 09:03 | |
| *** rossella_s has quit IRC | 09:03 | |
| openstackgerrit | Ghanshyam Mann proposed openstack-infra/project-config master: Add reno job for valence https://review.openstack.org/559669 | 09:03 |
| *** hemna_ has joined #openstack-infra | 09:05 | |
| *** rossella_s has joined #openstack-infra | 09:05 | |
| *** bobh has quit IRC | 09:06 | |
| *** alex_xu has joined #openstack-infra | 09:08 | |
| *** hemna_ has quit IRC | 09:09 | |
| *** edmondsw has joined #openstack-infra | 09:10 | |
| *** gfidente has joined #openstack-infra | 09:13 | |
| *** gfidente has joined #openstack-infra | 09:13 | |
| *** edmondsw has quit IRC | 09:14 | |
| ianw | frickler: hmm, not sure related to afs as that was only r/o mirrors on afs02 | 09:15 |
| openstackgerrit | Simon Westphahl proposed openstack-infra/zuul master: Allow using remote refs to find commits for change https://review.openstack.org/544964 | 09:16 |
| *** bobh has joined #openstack-infra | 09:17 | |
| *** jogo has joined #openstack-infra | 09:20 | |
| *** bobh has quit IRC | 09:22 | |
| *** kbaikov[m] has joined #openstack-infra | 09:24 | |
| *** armaan has quit IRC | 09:24 | |
| *** pblaho has joined #openstack-infra | 09:28 | |
| *** shu-mutow is now known as shu-mutow-AWAY | 09:29 | |
| *** sshnaidm|off is now known as sshnaidm | 09:30 | |
| *** armaan has joined #openstack-infra | 09:30 | |
| *** bobh_ has joined #openstack-infra | 09:30 | |
| *** liujiong has quit IRC | 09:33 | |
| *** gyankum has joined #openstack-infra | 09:34 | |
| frickler | logstash apache error log is full of this "[proxy_http:error] [pid 18932:tid 139826787497728] [client 185.27.182.2:34122] AH01114: HTTP: failed to make connection to backend: elasticsearch03.openstack.org" | 09:35 |
| *** bobh_ has quit IRC | 09:35 | |
| *** armaan has quit IRC | 09:35 | |
| *** andreas_s has quit IRC | 09:40 | |
| frickler | Active: failed (Result: signal) since Mon 2018-04-09 02:00:07 UTC; 7h ago | 09:41 |
| *** lpetrut has joined #openstack-infra | 09:42 | |
| openstackgerrit | Ghanshyam Mann proposed openstack-infra/project-config master: Add reno, doc build job for valence https://review.openstack.org/559669 | 09:42 |
| frickler | #status log the elasticsearch.service on es03.o.o was down since 02:00z, restarted now | 09:43 |
| *** bobh has joined #openstack-infra | 09:43 | |
| openstackstatus | frickler: finished logging | 09:43 |
| *** jamesmcarthur has quit IRC | 09:45 | |
| *** vabada has joined #openstack-infra | 09:48 | |
| *** bobh has quit IRC | 09:48 | |
| openstackgerrit | Jens Harbott (frickler) proposed openstack-infra/elastic-recheck master: Update query for bug 1298006 https://review.openstack.org/559681 | 09:53 |
| openstack | bug 1298006 in OpenStack-Gate "network sometimes unreachable accessing our git server" [Medium,Triaged] https://launchpad.net/bugs/1298006 | 09:53 |
| frickler | infra-root: ^^ had 386 hits for that in the last 7d, but mostly last week, seems much better currently. still an issue to watch out for IMO | 09:53 |
| *** jchhatbar has quit IRC | 09:55 | |
| *** bobh has joined #openstack-infra | 09:56 | |
| *** pblaho has quit IRC | 09:56 | |
| *** pblaho has joined #openstack-infra | 09:57 | |
| *** namnh has quit IRC | 09:58 | |
| *** namnh has joined #openstack-infra | 09:58 | |
| *** bobh has quit IRC | 10:01 | |
| *** bobh has joined #openstack-infra | 10:04 | |
| *** efoley_ has joined #openstack-infra | 10:07 | |
| *** namnh has quit IRC | 10:08 | |
| *** stakeda has quit IRC | 10:08 | |
| *** bobh has quit IRC | 10:09 | |
| *** efoley has quit IRC | 10:10 | |
| *** garyk has quit IRC | 10:11 | |
| openstackgerrit | Ghanshyam Mann proposed openstack-infra/project-config master: Add reno, doc build job for valence https://review.openstack.org/559669 | 10:12 |
| *** bobh has joined #openstack-infra | 10:13 | |
| *** garyk has joined #openstack-infra | 10:14 | |
| *** andreas_s has joined #openstack-infra | 10:16 | |
| *** bobh has quit IRC | 10:18 | |
| *** rossella_s has quit IRC | 10:23 | |
| *** rossella_s has joined #openstack-infra | 10:24 | |
| ianw | pabelanger: do you remeber fiddling with ask.o.o settings.py file at any point? because the file on the live server isn't the same as the one in puppet | 10:26 |
| *** gyankum has quit IRC | 10:27 | |
| *** bobh has joined #openstack-infra | 10:28 | |
| *** rossella_s has quit IRC | 10:29 | |
| *** bobh has quit IRC | 10:31 | |
| *** bobh has joined #openstack-infra | 10:31 | |
| *** slaweq has quit IRC | 10:32 | |
| openstackgerrit | Ian Wienand proposed openstack-infra/puppet-askbot master: systemd support https://review.openstack.org/559601 | 10:32 |
| openstackgerrit | Ian Wienand proposed openstack-infra/puppet-askbot master: fix python deps https://review.openstack.org/559622 | 10:32 |
| openstackgerrit | Ian Wienand proposed openstack-infra/puppet-askbot master: ensure ordering https://review.openstack.org/559623 | 10:32 |
| openstackgerrit | Ian Wienand proposed openstack-infra/puppet-askbot master: Use earlier six https://review.openstack.org/559624 | 10:32 |
| openstackgerrit | Ian Wienand proposed openstack-infra/puppet-askbot master: rspec test https://review.openstack.org/558991 | 10:32 |
| openstackgerrit | Ian Wienand proposed openstack-infra/puppet-askbot master: update settings https://review.openstack.org/559694 | 10:32 |
| *** rossella_s has joined #openstack-infra | 10:33 | |
| *** pblaho has quit IRC | 10:33 | |
| *** pblaho has joined #openstack-infra | 10:33 | |
| *** nicolasbock has joined #openstack-infra | 10:34 | |
| *** slaweq has joined #openstack-infra | 10:35 | |
| *** bobh has quit IRC | 10:36 | |
| ianw | #status log afs02.dfw.o.o ran out of space in /vicepa. added +1tb volume (bringing in-line with afs01). two volumes appear to have become corrupt due to out-of-disk errors; ubuntu & debian. recovery involves a full release. ubuntu is done, debian is currently going; i have the cron lock | 10:38 |
| openstackstatus | ianw: finished logging | 10:38 |
| ianw | #thanks many thanks to auristor for helping with AFS recovery after a full disk issue. detailed knowledge pinpointing issues saved much time and effort | 10:40 |
| openstackstatus | ianw: Added your thanks to Thanks page (https://wiki.openstack.org/wiki/Thanks) | 10:40 |
| *** lpetrut has quit IRC | 10:40 | |
| *** sree has quit IRC | 10:43 | |
| *** bobh has joined #openstack-infra | 10:44 | |
| *** bobh has quit IRC | 10:49 | |
| *** cshastri has joined #openstack-infra | 10:49 | |
| openstackgerrit | Merged openstack-infra/project-config master: Add reno, doc build job for valence https://review.openstack.org/559669 | 10:51 |
| openstackgerrit | Merged openstack-infra/project-config master: Set openstack-tox-cover as voting for monasca-ui https://review.openstack.org/553368 | 10:51 |
| *** lpetrut has joined #openstack-infra | 10:52 | |
| *** bobh has joined #openstack-infra | 10:53 | |
| *** aviau has quit IRC | 10:53 | |
| *** aviau has joined #openstack-infra | 10:54 | |
| openstackgerrit | Jens Harbott (frickler) proposed openstack-infra/elastic-recheck master: Update query for bug 1298006 https://review.openstack.org/559681 | 10:56 |
| openstack | bug 1298006 in OpenStack-Gate "network sometimes unreachable accessing our git server" [Medium,Triaged] https://launchpad.net/bugs/1298006 | 10:56 |
| *** bobh has quit IRC | 10:57 | |
| *** edmondsw has joined #openstack-infra | 10:58 | |
| openstackgerrit | Ian Wienand proposed openstack-infra/puppet-askbot master: systemd support https://review.openstack.org/559601 | 11:01 |
| openstackgerrit | Ian Wienand proposed openstack-infra/puppet-askbot master: fix python deps https://review.openstack.org/559622 | 11:01 |
| openstackgerrit | Ian Wienand proposed openstack-infra/puppet-askbot master: ensure ordering https://review.openstack.org/559623 | 11:01 |
| openstackgerrit | Ian Wienand proposed openstack-infra/puppet-askbot master: Use earlier six https://review.openstack.org/559624 | 11:01 |
| openstackgerrit | Ian Wienand proposed openstack-infra/puppet-askbot master: rspec test https://review.openstack.org/558991 | 11:01 |
| openstackgerrit | Ian Wienand proposed openstack-infra/puppet-askbot master: fix captca version https://review.openstack.org/559704 | 11:01 |
| *** zhangfei has quit IRC | 11:01 | |
| *** edmondsw has quit IRC | 11:03 | |
| *** bobh has joined #openstack-infra | 11:03 | |
| *** lpetrut has quit IRC | 11:04 | |
| *** lucasagomes is now known as lucas-hungry | 11:05 | |
| *** armaan has joined #openstack-infra | 11:05 | |
| ianw | #status log ask-staging.o.o is pointing to a new xenial-based server. old server is at ask-staging-old.o.o for now | 11:06 |
| openstackstatus | ianw: finished logging | 11:06 |
| *** larainema has quit IRC | 11:07 | |
| *** bobh has quit IRC | 11:08 | |
| *** cshastri has quit IRC | 11:09 | |
| frickler | ianw: should something be running at the new server already? getting connection refused | 11:09 |
| ianw | frickler: no, not yet | 11:09 |
| frickler | ianw: ah, o.k., then I was too fast with trying to look at it ;) | 11:10 |
| ianw | also the debian volume is released now too, so mirrors should all be ok now | 11:12 |
| frickler | ianw: great, thanks for cleaning that up | 11:13 |
| *** Hal has quit IRC | 11:14 | |
| *** rosmaita has joined #openstack-infra | 11:15 | |
| *** efoley_ has quit IRC | 11:15 | |
| *** xinliang has quit IRC | 11:16 | |
| *** bobh has joined #openstack-infra | 11:19 | |
| *** udesale__ has quit IRC | 11:20 | |
| ianw | yeah, i *just* got the apply test to work on xenial, which means it at least installs, runs all the django syncdb/migrate etc and has celeryd started | 11:21 |
| ianw | the stack needs cleaning and lint changes etc, but at least it will be likely to do something useful | 11:22 |
| *** katkapilatova has joined #openstack-infra | 11:22 | |
| ianw | http://logs.openstack.org/91/558991/37/check/legacy-puppet-beaker-rspec-infra/3b0a64a/ | 11:22 |
| *** pgadiya has quit IRC | 11:22 | |
| *** efoley has joined #openstack-infra | 11:23 | |
| *** bobh has quit IRC | 11:23 | |
| *** pblaho has quit IRC | 11:24 | |
| *** zoli is now known as zoli|lunch | 11:25 | |
| *** dhill_ has quit IRC | 11:26 | |
| *** xinliang has joined #openstack-infra | 11:27 | |
| *** xinliang has quit IRC | 11:27 | |
| *** xinliang has joined #openstack-infra | 11:27 | |
| frickler | infra-root: seems that https://review.openstack.org/543403 broke showing existing autoholds http://paste.openstack.org/show/718726/ | 11:28 |
| *** bobh has joined #openstack-infra | 11:33 | |
| *** ldnunes has joined #openstack-infra | 11:39 | |
| *** bobh has quit IRC | 11:39 | |
| *** jpena is now known as jpena|lunch | 11:42 | |
| *** bobh has joined #openstack-infra | 11:42 | |
| *** pbourke_ has quit IRC | 11:45 | |
| *** pbourke_ has joined #openstack-infra | 11:45 | |
| *** gyankum has joined #openstack-infra | 11:45 | |
| *** pblaho has joined #openstack-infra | 11:46 | |
| *** bobh has quit IRC | 11:47 | |
| *** dprince has joined #openstack-infra | 11:47 | |
| *** rfolco has joined #openstack-infra | 11:50 | |
| *** rfolco is now known as rfolco|rover | 11:50 | |
| *** psachin has quit IRC | 11:53 | |
| *** bobh has joined #openstack-infra | 11:53 | |
| *** florianf has quit IRC | 11:55 | |
| *** e0ne has joined #openstack-infra | 11:56 | |
| *** lucas-hungry is now known as lucasagomes | 11:56 | |
| *** bobh has quit IRC | 11:59 | |
| *** edmondsw has joined #openstack-infra | 12:04 | |
| *** gyankum has quit IRC | 12:04 | |
| *** bobh has joined #openstack-infra | 12:06 | |
| *** bobh has quit IRC | 12:07 | |
| *** florianf has joined #openstack-infra | 12:10 | |
| auristor | ianw: there are no active transactions in progress but mirror.debian is still LOCKED for a release operation. Confirm that no release operations are still running and "vos unlock mirror.debian". | 12:11 |
| *** hwoarang has joined #openstack-infra | 12:12 | |
| *** psachin has joined #openstack-infra | 12:13 | |
| *** sree_ has joined #openstack-infra | 12:14 | |
| *** rlandy has joined #openstack-infra | 12:14 | |
| *** sree_ is now known as Guest87354 | 12:14 | |
| *** Guest87354 has quit IRC | 12:19 | |
| openstackgerrit | Sam Betts proposed openstack-infra/project-config master: Pin networking-cisco build sphinx docs to stable/queens https://review.openstack.org/553816 | 12:19 |
| ianw | auristor: interesting, thanks; i ran that unlock. the "vos release" did complete successfully however? | 12:22 |
| *** efoley has quit IRC | 12:23 | |
| auristor | You will have to look at any logs you have. | 12:23 |
| *** zhangfei has joined #openstack-infra | 12:25 | |
| auristor | AuriStorFS wouldn't have required cleanup after the out of space error and it would have informed the vos operator of the out of space condition. The OpenAFS volume transaction processes are fragile and opaque. | 12:25 |
| ianw | -rw------- 1 root root 31M May 13 2015 coresalsrv.dafs | 12:27 |
| ianw | -rw------- 1 root root 31M Apr 1 2015 corevol.dafs | 12:27 |
| ianw | oh i bet that's those core files | 12:27 |
| auristor | those are old | 12:28 |
| *** udesale has joined #openstack-infra | 12:28 | |
| *** e0ne has quit IRC | 12:31 | |
| *** e0ne has joined #openstack-infra | 12:31 | |
| *** udesale has quit IRC | 12:31 | |
| *** udesale has joined #openstack-infra | 12:32 | |
| *** jpena|lunch is now known as jpena | 12:32 | |
| *** nicolasbock has quit IRC | 12:33 | |
| *** efoley has joined #openstack-infra | 12:34 | |
| *** armaan has quit IRC | 12:36 | |
| *** efoley has quit IRC | 12:36 | |
| *** efoley has joined #openstack-infra | 12:36 | |
| *** e0ne has quit IRC | 12:36 | |
| *** armaan has joined #openstack-infra | 12:36 | |
| *** Hal has joined #openstack-infra | 12:37 | |
| *** nicolasbock has joined #openstack-infra | 12:37 | |
| auristor | ianw: do afsdb01 and afsdb02 need to have IPv6 enabled on the hosts? | 12:38 |
| *** odyssey4me has quit IRC | 12:38 | |
| *** odyssey4me has joined #openstack-infra | 12:38 | |
| auristor | The existence of IPv6 DNS SRV records for afsdb01 and afsdb02 cause delays for the AuriStorFS tooling because OpenAFS doesn't support IPv6. | 12:39 |
| *** dave-mccowan has joined #openstack-infra | 12:40 | |
| *** dpawlik has joined #openstack-infra | 12:43 | |
| *** dave-mcc_ has joined #openstack-infra | 12:45 | |
| *** dave-mccowan has quit IRC | 12:45 | |
| *** pcaruana has quit IRC | 12:46 | |
| *** gouthamr has joined #openstack-infra | 12:49 | |
| *** jamesmcarthur has joined #openstack-infra | 12:51 | |
| *** jamesmcarthur has quit IRC | 12:51 | |
| *** jamesmca_ has joined #openstack-infra | 12:51 | |
| *** gouthamr has quit IRC | 12:52 | |
| openstackgerrit | Stephen Finucane proposed openstack-infra/zuul-jobs master: Default warning-is-error to True for non-legacy Sphinx projects https://review.openstack.org/559348 | 12:52 |
| *** Goneri has joined #openstack-infra | 12:52 | |
| *** kgiusti has joined #openstack-infra | 12:54 | |
| *** jcoufal has joined #openstack-infra | 12:57 | |
| *** hwoarang has quit IRC | 12:57 | |
| *** hwoarang has joined #openstack-infra | 12:57 | |
| *** rkukura has joined #openstack-infra | 12:58 | |
| *** gouthamr has joined #openstack-infra | 13:02 | |
| *** sree has joined #openstack-infra | 13:02 | |
| *** mriedem has joined #openstack-infra | 13:03 | |
| *** psachin has quit IRC | 13:04 | |
| *** zoli|lunch is now known as zoli|wfh | 13:10 | |
| *** zoli|wfh is now known as zoli | 13:10 | |
| *** pcaruana has joined #openstack-infra | 13:11 | |
| *** udesale has quit IRC | 13:12 | |
| *** udesale has joined #openstack-infra | 13:13 | |
| *** jamesmca_ has quit IRC | 13:13 | |
| *** HeOS has quit IRC | 13:14 | |
| *** gouthamr has quit IRC | 13:15 | |
| *** gouthamr has joined #openstack-infra | 13:17 | |
| *** Goneri has quit IRC | 13:21 | |
| *** janki has joined #openstack-infra | 13:22 | |
| *** neiloy has quit IRC | 13:23 | |
| *** zhangfei has quit IRC | 13:26 | |
| *** gouthamr has quit IRC | 13:26 | |
| *** lbragstad has joined #openstack-infra | 13:27 | |
| *** rpioso|afk is now known as rpioso | 13:27 | |
| *** jamesmcarthur has joined #openstack-infra | 13:29 | |
| *** david-lyle has quit IRC | 13:31 | |
| *** markvoelker has joined #openstack-infra | 13:34 | |
| *** hwoarang has quit IRC | 13:35 | |
| *** VW has joined #openstack-infra | 13:35 | |
| *** markvoelker_ has joined #openstack-infra | 13:36 | |
| *** markvoelker has quit IRC | 13:39 | |
| *** esberglu has joined #openstack-infra | 13:39 | |
| dmsimard | infra-root: There's a critical update for etherpad available: http://blog.etherpad.org/2018/04/07/important-release-1-6-4/ | 13:41 |
| dmsimard | I'll look at ours and assess the situation. | 13:41 |
| *** sree has quit IRC | 13:42 | |
| *** sree has joined #openstack-infra | 13:43 | |
| *** sree has quit IRC | 13:47 | |
| *** jchhatbar has joined #openstack-infra | 13:50 | |
| pabelanger | ianw: I do not | 13:51 |
| *** janki has quit IRC | 13:52 | |
| dmsimard | Moving to incident for etherpad.o.o | 13:53 |
| *** r-daneel has joined #openstack-infra | 13:54 | |
| *** felipemonteiro has joined #openstack-infra | 14:00 | |
| *** anteaya has joined #openstack-infra | 14:00 | |
| *** abyz has quit IRC | 14:01 | |
| *** markvoelker_ has quit IRC | 14:03 | |
| *** dsariel has quit IRC | 14:05 | |
| *** bnemec has joined #openstack-infra | 14:05 | |
| openstackgerrit | Merged openstack-infra/project-config master: Remove leftover stable/queens pre-release ACL https://review.openstack.org/557733 | 14:06 |
| openstackgerrit | Tobias Henkel proposed openstack-infra/zuul master: Fix missing semaphore release on zk error https://review.openstack.org/559745 | 14:06 |
| AJaeger | infra-root, we have a couple of changes for infra-manual - could I get a second review for these, please? | 14:07 |
| *** eharney has joined #openstack-infra | 14:08 | |
| *** felipemonteiro_ has joined #openstack-infra | 14:09 | |
| *** tosky has quit IRC | 14:10 | |
| *** eernst has joined #openstack-infra | 14:11 | |
| openstackgerrit | Merged openstack-infra/project-config master: Adds Cyborg to StoryBoard https://review.openstack.org/558327 | 14:12 |
| *** felipemonteiro has quit IRC | 14:12 | |
| *** bobh has joined #openstack-infra | 14:13 | |
| *** tosky has joined #openstack-infra | 14:14 | |
| *** larainema has joined #openstack-infra | 14:14 | |
| *** r-daneel_ has joined #openstack-infra | 14:16 | |
| *** r-daneel has quit IRC | 14:18 | |
| *** r-daneel_ is now known as r-daneel | 14:18 | |
| *** dpawlik has quit IRC | 14:19 | |
| openstackgerrit | Merged openstack-infra/infra-manual master: Reorder cookiecutter template section https://review.openstack.org/549902 | 14:19 |
| *** krtaylor has quit IRC | 14:19 | |
| openstackgerrit | David Moreau Simard proposed openstack-infra/system-config master: Add etherpad group to system-config's site.pp https://review.openstack.org/559748 | 14:21 |
| dmsimard | infra-root: ^ for etherpad update | 14:21 |
| AJaeger | corvus: thanks! | 14:22 |
| *** jroll has quit IRC | 14:22 | |
| *** jroll has joined #openstack-infra | 14:23 | |
| openstackgerrit | Merged openstack-infra/zuul master: Remove openstack-infra reference https://review.openstack.org/559586 | 14:23 |
| corvus | how about we restart zuul now to pick up the role checkout changes we recently merged? | 14:27 |
| corvus | they'll require a full executor+scheduler restart | 14:27 |
| *** r-daneel has quit IRC | 14:27 | |
| *** armaan has quit IRC | 14:28 | |
| mnaser | infra-root: simple 2 second review https://review.openstack.org/#/c/559249/ :) | 14:29 |
| *** gouthamr has joined #openstack-infra | 14:29 | |
| dmsimard | mnaser: +3 | 14:30 |
| openstackgerrit | Merged openstack-infra/infra-manual master: Corrects reported errors in the sandbox documentation https://review.openstack.org/559180 | 14:30 |
| mnaser | dmsimard: merci | 14:30 |
| corvus | dmsimard: can you handle the redirect? | 14:30 |
| dmsimard | corvus: yep, after etherpad | 14:31 |
| corvus | cool, thx | 14:31 |
| *** gouthamr has quit IRC | 14:31 | |
| pabelanger | mnaser: dmsimard: did we check to see if there wan't any meetings on-going | 14:33 |
| mnaser | i don't know :x /me is infra-core only | 14:33 |
| openstackgerrit | Merged openstack-infra/project-config master: Add python35 job to neutron-classifier. https://review.openstack.org/558329 | 14:33 |
| dmsimard | pabelanger: I -W'd, good call | 14:33 |
| pabelanger | mnaser: dmsimard: yah, if we merge meetbot changes while meetings are running, we'll mess up the topic of the channel. So best to do those after hours. | 14:34 |
| *** r-daneel has joined #openstack-infra | 14:35 | |
| dmsimard | pabelanger: how do you even check if there are any ongoing meetings ? | 14:36 |
| openstackgerrit | David Shrewsbury proposed openstack-infra/zuul master: Fix streaming decoding boundaries https://review.openstack.org/559326 | 14:37 |
| corvus | dmsimard: i look at the calendar | 14:40 |
| pabelanger | dmsimard: I look at eavesdrop.o.o at meetings times | 14:43 |
| pabelanger | clarkb: https://review.openstack.org/557842 is ready, /etc/resolv.conf.ORIG change for images. When you have time | 14:43 |
| *** HeOS has joined #openstack-infra | 14:47 | |
| *** dklyle has joined #openstack-infra | 14:51 | |
| *** Goneri has joined #openstack-infra | 14:54 | |
| *** pblaho has quit IRC | 14:55 | |
| *** zhangfei has joined #openstack-infra | 14:56 | |
| *** caphrim007 has joined #openstack-infra | 15:03 | |
| *** zoli is now known as zoli|afk | 15:03 | |
| *** armaan has joined #openstack-infra | 15:03 | |
| openstackgerrit | Tobias Henkel proposed openstack-infra/zuul master: Fix missing semaphore release on zk error https://review.openstack.org/559745 | 15:05 |
| clarkb | pabelanger: I'll look after brealfast | 15:05 |
| *** ramishra has quit IRC | 15:09 | |
| *** gouthamr has joined #openstack-infra | 15:13 | |
| *** VW_ has joined #openstack-infra | 15:14 | |
| *** VW has quit IRC | 15:14 | |
| *** e0ne has joined #openstack-infra | 15:14 | |
| *** VW_ is now known as VW | 15:14 | |
| *** hwoarang has joined #openstack-infra | 15:14 | |
| *** dayou has quit IRC | 15:16 | |
| pabelanger | wfm | 15:16 |
| *** sree has joined #openstack-infra | 15:19 | |
| *** felipemonteiro_ has quit IRC | 15:19 | |
| *** xinliang has quit IRC | 15:19 | |
| *** felipemonteiro_ has joined #openstack-infra | 15:19 | |
| *** xinliang has joined #openstack-infra | 15:21 | |
| *** xinliang has quit IRC | 15:21 | |
| *** xinliang has joined #openstack-infra | 15:21 | |
| *** zhangfei has quit IRC | 15:23 | |
| corvus | i've drafted an email about the zuul change: https://etherpad.openstack.org/p/W6oDo3OOR1 | 15:23 |
| corvus | but i'm going to hold off on restarting zuul while the etherpad work is continuing | 15:23 |
| openstackgerrit | Jeremy Stanley proposed openstack-infra/system-config master: Use nodejs 6.x on etherpad-dev.o.o https://review.openstack.org/559767 | 15:24 |
| *** olaph has joined #openstack-infra | 15:24 | |
| *** olaph1 has quit IRC | 15:24 | |
| openstackgerrit | Chris Dent proposed openstack-infra/project-config master: Add entries for #openstack-placement IRC channel https://review.openstack.org/559768 | 15:24 |
| *** dsariel has joined #openstack-infra | 15:24 | |
| *** armaan_ has joined #openstack-infra | 15:25 | |
| *** armaan has quit IRC | 15:25 | |
| *** slaweq has quit IRC | 15:26 | |
| *** slaweq has joined #openstack-infra | 15:26 | |
| *** rpioso is now known as rpioso|eat | 15:27 | |
| *** jmorgan1 has joined #openstack-infra | 15:28 | |
| *** andreas_s has quit IRC | 15:28 | |
| frickler | corvus: +1 on the draft. did you see my earlier note about autohold-list being broken? | 15:29 |
| *** krenczewski has joined #openstack-infra | 15:29 | |
| *** andreas_s has joined #openstack-infra | 15:29 | |
| corvus | frickler: thanks; i did not see the earlier note | 15:29 |
| openstackgerrit | Chris Dent proposed openstack-infra/system-config master: Add statusbot and meetbot to #openstack-placement https://review.openstack.org/559769 | 15:29 |
| frickler | corvus: just do a "zuul autohold-list" on zuul01 to see the error | 15:30 |
| corvus | frickler: whoops | 15:30 |
| *** slaweq has quit IRC | 15:30 | |
| *** krenczewski has quit IRC | 15:31 | |
| frickler | seems we need grenade for zuul ;) | 15:31 |
| corvus | frickler: oh, is it a version mismatch? | 15:32 |
| frickler | corvus: no, I assume it is applying https://review.openstack.org/543403 with existing autohold entries that lack the new attribute. but a grenade-like test should have could that | 15:32 |
| frickler | s/could/caught | 15:33 |
| corvus | frickler: the autohold list is entirely in memory | 15:33 |
| corvus | so there's no schema upgrade path | 15:33 |
| frickler | corvus: oh, is it? then it is indeed mismatch between client and running version only | 15:34 |
| *** krenczewski has joined #openstack-infra | 15:34 | |
| *** germs has joined #openstack-infra | 15:34 | |
| *** germs has quit IRC | 15:34 | |
| *** germs has joined #openstack-infra | 15:34 | |
| corvus | ok, so that's a problem which will be fixed shortly, as soon as the etherpad changes merge. but also, we'll drop the autohold entries. | 15:34 |
| openstackgerrit | Merged openstack-infra/puppet-etherpad_lite master: Don't install npm dependencies in /var/log https://review.openstack.org/528625 | 15:34 |
| *** germs has quit IRC | 15:35 | |
| frickler | corvus: o.k., so that should be no significant issue, then. ftr, this is the last state I saw last week http://paste.openstack.org/show/718752/ | 15:35 |
| *** germs has joined #openstack-infra | 15:36 | |
| *** germs has quit IRC | 15:36 | |
| *** germs has joined #openstack-infra | 15:36 | |
| *** links has quit IRC | 15:37 | |
| *** andreas_s has quit IRC | 15:38 | |
| *** andreas_s has joined #openstack-infra | 15:38 | |
| openstackgerrit | Merged openstack-infra/openstack-zuul-jobs master: Run xstatic version checker in project repo https://review.openstack.org/559373 | 15:38 |
| Shrews | i looked a little at that autohold thing. i mean, we _could_ fix it, but once we restart the scheduler to pick up the fix, it would no longer be necessary b/c the list would disappear | 15:39 |
| frickler | Shrews: ya, I assumed that the list would be persisted across a restart. if that isn't the case, we only have a temporary issue that should be safe to be ignored | 15:41 |
| *** jpena is now known as jpena|brb | 15:41 | |
| anteaya | was staring at a beautiful fox on my lawn, had to share | 15:41 |
| *** jmorgan1 has quit IRC | 15:42 | |
| *** andreas_s has quit IRC | 15:43 | |
| corvus | eventually i assume we'll put the list in zk. but not for a while yet. :) | 15:43 |
| Shrews | it is a good reminder for us though to consider the current "data schema" (whether in memory or zk) when a change is submitted for it | 15:44 |
| *** electrofelix has quit IRC | 15:44 | |
| *** gyee has joined #openstack-infra | 15:45 | |
| Shrews | that's probably a hole in our testing, now that i think about it | 15:46 |
| *** timothyb89_ is now known as timothyb89 | 15:46 | |
| *** andreas_s has joined #openstack-infra | 15:48 | |
| *** neiloy has joined #openstack-infra | 15:48 | |
| corvus | Shrews: we're not in a position to support running different components on different versions. | 15:52 |
| *** dsariel has quit IRC | 15:52 | |
| *** nicolasbock has quit IRC | 15:52 | |
| corvus | we're doing something that i would not recommend other folks do: installing zuul without restarting it. | 15:52 |
| Shrews | corvus: yeah, i know. but would be nice to have some guarantee that if we, say, add a new field to a znode, or data format of an existing field, that we have good coverage of whatever uses that znode. | 15:54 |
| *** agopi_ has joined #openstack-infra | 15:54 | |
| Shrews | i think we do a "pretty good" job of that, but maybe something to keep in mind | 15:55 |
| *** sree has quit IRC | 15:56 | |
| corvus | Shrews: yeah, we have no mechanism for zk schema changes right now, so we can't really change a field until we develop one. adding new fields is easy though, and our usual practice of adding new tests for new things should cover that :) | 15:56 |
| *** gouthamr has quit IRC | 15:56 | |
| *** sree has joined #openstack-infra | 15:56 | |
| *** agopi_ has quit IRC | 15:57 | |
| *** agopi_ has joined #openstack-infra | 15:57 | |
| Shrews | corvus: it's the assumption of that new field always being there (not taking account of existing zk data) that usually trips us up (pretty much the same as the autohold thing today) | 15:58 |
| *** dayou has joined #openstack-infra | 16:00 | |
| *** jmorgan1 has joined #openstack-infra | 16:01 | |
| *** sree has quit IRC | 16:01 | |
| *** dpawlik has joined #openstack-infra | 16:03 | |
| *** dpawlik has quit IRC | 16:04 | |
| *** jcoufal has quit IRC | 16:04 | |
| *** dpawlik has joined #openstack-infra | 16:04 | |
| *** lpetrut has joined #openstack-infra | 16:05 | |
| *** andreas_s has quit IRC | 16:06 | |
| *** andreas_s has joined #openstack-infra | 16:07 | |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config master: Fix some docs publishing https://review.openstack.org/559777 | 16:09 |
| *** hashar has quit IRC | 16:10 | |
| *** jchhatbar has quit IRC | 16:10 | |
| smcginnis | corvus: Just checked and there are no release jobs in gate or any post queues. So clear on my end whenever you need to do anything. | 16:11 |
| *** lpetrut has quit IRC | 16:11 | |
| *** lpetrut has joined #openstack-infra | 16:12 | |
| corvus | smcginnis: great, thanks! i'm waiting for some emergency infra changes to land, then will restart shortly. i'll let you know when it's all-clear. | 16:12 |
| openstackgerrit | Merged openstack-infra/system-config master: Update puppet-nodejs to 2.3.0 https://review.openstack.org/526978 | 16:12 |
| smcginnis | corvus: Sounds good. | 16:12 |
| *** dpawlik_ has joined #openstack-infra | 16:12 | |
| *** dpawlik has quit IRC | 16:12 | |
| *** dpawlik_ has quit IRC | 16:13 | |
| *** dpawlik has joined #openstack-infra | 16:13 | |
| *** Hal has quit IRC | 16:15 | |
| AJaeger | config-core, could you review https://review.openstack.org/559777 to fix an omission in a recent change to fix publishing for developer.o.o again, please? | 16:15 |
| *** neiloy has quit IRC | 16:16 | |
| openstackgerrit | Merged openstack-infra/system-config master: Use nodejs 6.x on etherpad-dev.o.o https://review.openstack.org/559767 | 16:16 |
| *** zoli|afk is now known as zoli|wfh | 16:17 | |
| *** zoli|wfh is now known as zoli | 16:17 | |
| *** felipemonteiro_ has quit IRC | 16:17 | |
| *** dpawlik_ has joined #openstack-infra | 16:18 | |
| *** lucasagomes is now known as lucas-afk | 16:19 | |
| *** andreas_s has quit IRC | 16:20 | |
| *** lpetrut has quit IRC | 16:20 | |
| *** dpawlik has quit IRC | 16:21 | |
| *** dpawlik has joined #openstack-infra | 16:22 | |
| *** dpawlik_ has quit IRC | 16:22 | |
| *** _alastor_ is now known as Guest83262 | 16:23 | |
| dmsimard | ttx: there's a security warning when browsing forumtopics.openstack.org on firefox. Doesn't reproduce on chrome though: https://i.imgur.com/3Ssz7Xm.png | 16:23 |
| *** jpena|brb is now known as jpena | 16:24 | |
| dmsimard | It happens after logging into the Ubuntu one SSO | 16:24 |
| *** eernst has quit IRC | 16:25 | |
| *** andreas_s has joined #openstack-infra | 16:26 | |
| *** sambetts is now known as sambetts|afk | 16:26 | |
| *** dpawlik_ has joined #openstack-infra | 16:27 | |
| *** dpawlik_ has quit IRC | 16:27 | |
| *** dpawlik_ has joined #openstack-infra | 16:27 | |
| *** dpawlik has quit IRC | 16:27 | |
| *** germs has quit IRC | 16:28 | |
| *** germs has joined #openstack-infra | 16:28 | |
| *** germs has quit IRC | 16:28 | |
| *** germs has joined #openstack-infra | 16:28 | |
| *** germs has quit IRC | 16:28 | |
| *** germs has joined #openstack-infra | 16:28 | |
| *** germs has quit IRC | 16:28 | |
| *** germs has joined #openstack-infra | 16:28 | |
| ttx | dmsimard: yeah, that's because forumtopics runs on HTTP | 16:28 |
| clarkb | ttx: ^ I want to say that is beacuse formtopics is http but the redirect from openid login goes from https to http | 16:29 |
| ttx | patches welcome :) | 16:29 |
| fungi | corvus: based on conversation in #openstack-infra-incident i don't think we need to worry about postponing the zuul scheduler restart further? | 16:29 |
| corvus | fungi: cool, i'll begin that process shortly | 16:30 |
| corvus | i'm going to try to use more ansible playbooks to do it this time | 16:30 |
| *** germs has quit IRC | 16:31 | |
| *** hongbin has joined #openstack-infra | 16:32 | |
| *** jpich has quit IRC | 16:33 | |
| *** germs has joined #openstack-infra | 16:33 | |
| *** germs has quit IRC | 16:33 | |
| *** germs has joined #openstack-infra | 16:33 | |
| *** lpetrut has joined #openstack-infra | 16:33 | |
| openstackgerrit | Merged openstack-infra/nodepool master: Add a backoff for failed builds https://review.openstack.org/558686 | 16:34 |
| *** wolverineav has joined #openstack-infra | 16:34 | |
| *** germs has quit IRC | 16:35 | |
| *** andreas_s has quit IRC | 16:35 | |
| openstackgerrit | Merged openstack-infra/project-config master: Ensure /etc/resolv.conf is a regular file in final image https://review.openstack.org/557842 | 16:36 |
| *** yamamoto has joined #openstack-infra | 16:37 | |
| *** yamamoto has quit IRC | 16:38 | |
| *** trown is now known as trown|lunch | 16:38 | |
| corvus | restarting zuul now | 16:39 |
| *** dhill_ has joined #openstack-infra | 16:39 | |
| *** andreas_s has joined #openstack-infra | 16:40 | |
| jlvillal | gerritbot review request: https://review.openstack.org/#/c/545469/ Some cleanup/refactoring and adding unit tests. Has one +2 Thank you. | 16:41 |
| dansmith | corvus: should zuul still be rejecting status connections? | 16:42 |
| *** hashar has joined #openstack-infra | 16:42 | |
| *** SumitNaiksatam has quit IRC | 16:42 | |
| corvus | dansmith: yes, it's still down while i'm waiting for the executors to all stop | 16:42 |
| dansmith | okay | 16:42 |
| corvus | (i could probably bring it back up at this point, but i don't want to test it now; i'll think about doing that as an improvement later) | 16:43 |
| corvus | status notice zuul was restarted to update to the latest code; please recheck any changes uploaded within the past 10 minutes | 16:44 |
| corvus | how's that look ^ ? | 16:44 |
| corvus | (for once i start it) | 16:44 |
| * fungi heads out for a lunchtime appointment, but will return in an hour-ish | 16:44 | |
| clarkb | corvus: lgtm | 16:45 |
| corvus | okay, all stopped. starting now | 16:45 |
| *** udesale has quit IRC | 16:46 | |
| *** mikal has quit IRC | 16:46 | |
| openstackgerrit | Raoul Scarazzini proposed openstack-infra/project-config master: Create tripleo-ha-utils project (gerrit) https://review.openstack.org/555832 | 16:46 |
| *** mikal has joined #openstack-infra | 16:47 | |
| *** dhill_ has quit IRC | 16:48 | |
| *** dhill_ has joined #openstack-infra | 16:48 | |
| *** tesseract has quit IRC | 16:48 | |
| *** eernst has joined #openstack-infra | 16:49 | |
| *** armaan_ has quit IRC | 16:49 | |
| *** andreas_s has quit IRC | 16:49 | |
| *** armaan has joined #openstack-infra | 16:50 | |
| corvus | #status notice zuul was restarted to update to the latest code; please recheck any changes uploaded within the past 10 minutes | 16:50 |
| openstackstatus | corvus: sending notice | 16:50 |
| corvus | re-enqueing now | 16:50 |
| corvus | dansmith: should be up | 16:50 |
| dansmith | corvus: aye.. oh and the queues will have to re-start? | 16:51 |
| *** jgwentworth is now known as melwitt | 16:51 | |
| corvus | dansmith: yeah, i'm re-adding the changes that were in there from before, but their jobs will need to start over | 16:51 |
| -openstackstatus- NOTICE: zuul was restarted to update to the latest code; please recheck any changes uploaded within the past 10 minutes | 16:51 | |
| pabelanger | mnaser: it looks like we might have leaked volumes again in vexxhost. Do you have some time today to see why that would be? I think we'd need to dive into openstack logs on cloud side. | 16:51 |
| mnaser | pabelanger: i'm noticing that not just on your side but across other tenants | 16:52 |
| mnaser | it seems to consistently happen across multiple tenants so i've been digging | 16:52 |
| pabelanger | mnaser: ack | 16:52 |
| corvus | infra-root, config-core: note there is a behavior change in current zuul -- i'm about to send this email to the -dev list: https://etherpad.openstack.org/p/W6oDo3OOR1 | 16:53 |
| corvus | infra-root, config-core: please keep an eye out for any errors that could be related to that | 16:53 |
| *** dpawlik_ has quit IRC | 16:53 | |
| openstackstatus | corvus: finished sending notice | 16:53 |
| *** krtaylor has joined #openstack-infra | 16:53 | |
| AJaeger | corvus: thanks for heads-up | 16:54 |
| *** armaan has quit IRC | 16:54 | |
| *** andreas_s has joined #openstack-infra | 16:54 | |
| openstackgerrit | Tobias Henkel proposed openstack-infra/zuul master: Refactor load sensors into drivers https://review.openstack.org/549275 | 16:55 |
| dmsimard | corvus: should we add a link to the override-checkout docs ? | 16:56 |
| corvus | dmsimard: probably but i already sent it | 16:56 |
| dmsimard | corvus: ok, I'll reply | 16:57 |
| pabelanger | clarkb: ianw: I think we can also bring online debian-stretch today: https://review.openstack.org/557902/ I have time to monitor builds | 16:57 |
| corvus | dmsimard: thx | 16:57 |
| *** florianf has quit IRC | 16:58 | |
| openstackgerrit | Tobias Henkel proposed openstack-infra/zuul master: Refactor load sensors into drivers https://review.openstack.org/549275 | 16:58 |
| *** dhajare has quit IRC | 16:58 | |
| openstackgerrit | Tobias Henkel proposed openstack-infra/zuul master: Refactor load sensors into drivers https://review.openstack.org/549275 | 16:59 |
| *** derekh has quit IRC | 16:59 | |
| AJaeger | config-core, could you review https://review.openstack.org/559777 to fix an omission in a recent change to fix publishing for developer.o.o again, please? | 17:01 |
| clarkb | AJaeger: looking | 17:01 |
| AJaeger | thanks, clarkb | 17:01 |
| *** hemna_ has joined #openstack-infra | 17:02 | |
| *** ralonsoh has quit IRC | 17:02 | |
| *** dhill__ has joined #openstack-infra | 17:02 | |
| *** dhill_ has quit IRC | 17:03 | |
| *** david-lyle has joined #openstack-infra | 17:04 | |
| openstackgerrit | Tobias Henkel proposed openstack-infra/zuul master: Refactor load sensors into drivers https://review.openstack.org/549275 | 17:04 |
| *** germs has joined #openstack-infra | 17:06 | |
| *** germs has quit IRC | 17:06 | |
| *** germs has joined #openstack-infra | 17:06 | |
| *** dklyle has quit IRC | 17:07 | |
| clarkb | pabelanger: I've +2' | 17:08 |
| clarkb | er | 17:08 |
| clarkb | +2'd the stretch change with one minor observation that maybe we clean up in a followup? | 17:08 |
| clarkb | pabelanger: I'll let you decide as the person that has volunteered to babysit it | 17:08 |
| pabelanger | yah, noticed that too. will do a followup | 17:09 |
| *** andreas_s has quit IRC | 17:09 | |
| pabelanger | will +A now, and monitor | 17:09 |
| *** germs has quit IRC | 17:09 | |
| *** germs has joined #openstack-infra | 17:10 | |
| *** germs has quit IRC | 17:10 | |
| *** germs has joined #openstack-infra | 17:10 | |
| *** germs has quit IRC | 17:12 | |
| *** SumitNaiksatam has joined #openstack-infra | 17:14 | |
| openstackgerrit | Raoul Scarazzini proposed openstack-infra/project-config master: Create tripleo-ha-utils project (gerrit) https://review.openstack.org/555832 | 17:14 |
| *** yamamoto has joined #openstack-infra | 17:16 | |
| *** jpena is now known as jpena|away | 17:18 | |
| *** felipemonteiro has joined #openstack-infra | 17:24 | |
| *** bhavik1 has joined #openstack-infra | 17:25 | |
| *** zoli is now known as zoli|gone | 17:27 | |
| openstackgerrit | Tobias Henkel proposed openstack-infra/zuul master: Update Github3.py to 1.0.3 https://review.openstack.org/559798 | 17:27 |
| *** gouthamr has joined #openstack-infra | 17:27 | |
| *** zoli|gone is now known as zoli | 17:27 | |
| *** andreas_s has joined #openstack-infra | 17:28 | |
| *** pbourke_ has quit IRC | 17:30 | |
| *** wolverineav has quit IRC | 17:30 | |
| openstackgerrit | Merged openstack-infra/project-config master: Add debian stretch nodes to arm64/amd64 clouds https://review.openstack.org/557902 | 17:31 |
| *** wolverineav has joined #openstack-infra | 17:31 | |
| *** andreas_s has quit IRC | 17:32 | |
| *** r-daneel has quit IRC | 17:33 | |
| *** bhavik1 has quit IRC | 17:33 | |
| *** gouthamr has quit IRC | 17:35 | |
| *** wolverineav has quit IRC | 17:35 | |
| openstackgerrit | Paul Belanger proposed openstack-infra/project-config master: Clean up diskimages for debian-(jessie|stretch) https://review.openstack.org/559802 | 17:36 |
| pabelanger | clarkb: ianw: cleans up diskimages for debian | 17:36 |
| pabelanger | ^ | 17:36 |
| *** trown|lunch is now known as trown | 17:37 | |
| openstackgerrit | Tobias Henkel proposed openstack-infra/zuul master: WIP: Add cgroup support to ram sensor https://review.openstack.org/549506 | 17:40 |
| *** felipemonteiro_ has joined #openstack-infra | 17:42 | |
| *** felipemonteiro has quit IRC | 17:46 | |
| *** rpioso|eat is now known as rpioso | 17:46 | |
| openstackgerrit | Raoul Scarazzini proposed openstack-infra/project-config master: Create tripleo-ha-utils project (gerrit) https://review.openstack.org/555832 | 17:46 |
| rasca | AJaeger, hey hi, I added the file, but what do you mean that the line it's the default? I just used the same format as all the other projects | 17:48 |
| *** eharney has quit IRC | 17:50 | |
| *** jcoufal has joined #openstack-infra | 17:50 | |
| *** wolverineav has joined #openstack-infra | 17:51 | |
| *** wolverineav has joined #openstack-infra | 17:51 | |
| *** wolverineav has quit IRC | 17:51 | |
| *** wolverineav has joined #openstack-infra | 17:51 | |
| *** dklyle has joined #openstack-infra | 17:52 | |
| *** david-lyle has quit IRC | 17:53 | |
| *** eharney has joined #openstack-infra | 17:56 | |
| *** wolverineav has quit IRC | 17:56 | |
| *** wolverineav has joined #openstack-infra | 17:56 | |
| *** yamamoto has quit IRC | 17:57 | |
| *** jamesmcarthur has quit IRC | 17:58 | |
| *** jamesmcarthur has joined #openstack-infra | 17:59 | |
| openstackgerrit | Merged openstack-infra/puppet-etherpad_lite master: Xenial: add a service file https://review.openstack.org/528626 | 17:59 |
| clarkb | AJaeger: is there some context for that? | 18:00 |
| clarkb | er rasca ^ | 18:00 |
| openstackgerrit | Merged openstack-infra/puppet-etherpad_lite master: Update default version of nodejs https://review.openstack.org/528156 | 18:00 |
| clarkb | fungi: ^ | 18:00 |
| rasca | clarkb, what do you mean with context? | 18:00 |
| *** andreas_s has joined #openstack-infra | 18:00 | |
| clarkb | rasca: for your question to AJaeger what is the context for that? is there a change review maybe? | 18:01 |
| fungi | clarkb: thanks! i just got back in the door | 18:01 |
| clarkb | fungi: let me know if I can help | 18:01 |
| rasca | clarkb, yep is the one right before in the chat https://review.openstack.org/555832 many thanks! | 18:01 |
| *** tpsilva has joined #openstack-infra | 18:01 | |
| clarkb | rasca: the default is to use a config file named after the project itself. So only necessary to list it if using a non default config file name | 18:03 |
| fungi | clarkb: that change was only really useful to get the rspec testing to work, i think. we override it in system-config anyway | 18:03 |
| clarkb | fungi: ah | 18:03 |
| *** amoralej is now known as amoralej|off | 18:03 | |
| fungi | but it was nice that we had it providing some indication the module works with nodejs 6.x | 18:03 |
| clarkb | rasca: some projects use shared files with a common name for example | 18:03 |
| rasca | clarkb, ok so I should be enough with the file itself and not the line, correct? | 18:03 |
| clarkb | rasca: ya | 18:03 |
| rasca | clarkb, oh now it's clear, many thanks | 18:04 |
| rasca | I'll correct immediately | 18:04 |
| *** jamesmca_ has joined #openstack-infra | 18:04 | |
| openstackgerrit | Raoul Scarazzini proposed openstack-infra/project-config master: Create tripleo-ha-utils project (gerrit) https://review.openstack.org/555832 | 18:05 |
| rasca | clarkb, done ^^^ thanks | 18:05 |
| *** jamesmcarthur has quit IRC | 18:08 | |
| *** CrayZee has quit IRC | 18:12 | |
| *** andreas_s has quit IRC | 18:15 | |
| *** lpetrut has quit IRC | 18:22 | |
| *** jpena|away is now known as jpena|off | 18:27 | |
| *** slaweq has joined #openstack-infra | 18:28 | |
| fungi | corvus: since you write several urwid-based tools, thought you might be interested in https://pypi.org/project/panwid/ (see https://github.com/tonycpsu/panwid for an actual description) which extends it with several new types of widgets | 18:29 |
| fungi | no idea if any are useful, it just came to my attention a few moments ago | 18:30 |
| corvus | fungi: nice! boartty actually needs "dropdown menu widget with autocomplete support" | 18:33 |
| corvus | i thought i was going to have to write it :) | 18:33 |
| *** efoley has quit IRC | 18:33 | |
| fungi | it's being packaged in debian for, of all things, a tool which enables viewing of live baseball video streams https://github.com/tonycpsu/mlbstreamer | 18:35 |
| fungi | somebody really is joining their loves of baseball and terminal applications | 18:36 |
| corvus | it's gpl, with no version specifier, so the 'any later version' applies. so if i did use it, boartty would be an effectively-gpl work. | 18:38 |
| fungi | mmm, i didn't think to look at the license | 18:41 |
| fungi | interesting that they picked a different license (gpl) than urwid (lgpl) | 18:42 |
| corvus | yeah, it's curious | 18:43 |
| corvus | (just to be clear, i don't consider this to be a blocker :) | 18:44 |
| fungi | and the trove classifier in https://github.com/tonycpsu/panwid/blob/master/setup.py#L18 just says it's gplv2 | 18:44 |
| fungi | out of my own curiosity, i opened https://github.com/tonycpsu/panwid/issues/7 about the different license choice | 18:51 |
| *** Goneri has quit IRC | 18:52 | |
| clarkb | does lgpl allow derivative works to be gpl? | 18:54 |
| *** gfidente has quit IRC | 18:54 | |
| *** dpawlik has joined #openstack-infra | 18:54 | |
| mtreinish | smcginnis: do you remember the pointer you linked to a few weeks ago about a similar py3 issue as the stestr one we've been banging our head against a wall on? | 18:56 |
| mtreinish | I can't remember what it was, just that I said I'd look at it | 18:56 |
| smcginnis | mtreinish: I don't have that tab open anymore, but IIRC, mordred pointed me at it in the -qa channel. | 18:57 |
| mtreinish | hmm, ok I'll dig up the logs | 18:57 |
| smcginnis | mtreinish: I can try to scan back through eavesdrop on that channel and see if I can find anything. | 18:57 |
| smcginnis | mtreinish: I'll let you know if I find anything. | 18:57 |
| corvus | clarkb: yes; in fact in v3 lgpl is implemented as an additional permission so it's very clear it can be removed. | 18:57 |
| mtreinish | smcginnis: I think it was this patch: https://review.openstack.org/#/c/555388/3 | 18:58 |
| smcginnis | mtreinish: Yep, that looks familiar. I thought there was another github PR that was related, but can't recall now. | 18:59 |
| smcginnis | mtreinish: But maybe this review is all that's really needed. | 18:59 |
| *** VW_ has joined #openstack-infra | 19:00 | |
| smcginnis | mtreinish: Found the logs: http://eavesdrop.openstack.org/irclogs/%23openstack-qa/%23openstack-qa.2018-03-22.log.html#t2018-03-22T17:50:33 | 19:01 |
| portdirect | hey - Im seeing some issues related to required projects | 19:01 |
| portdirect | eg: https://review.openstack.org/#/c/559199/ | 19:01 |
| portdirect | openstack-helm-dev-deploy-ceph-radosgw-ovs openstack-helm-dev-deploy-ceph-radosgw-ovs : ERROR Unable to find playbook /var/lib/zuul/builds/40ffa71788d84296b70d9ffbdbefb5aa/untrusted/project_0/git.openstack.org/openstack/openstack-helm/../openstack-helm-infra/tools/gate/playbooks/osh-infra-upgrade-host.yaml | 19:02 |
| portdirect | has the behavior changed here recently, or am i missing something? | 19:02 |
| *** VW_ has quit IRC | 19:02 | |
| *** VW_ has joined #openstack-infra | 19:02 | |
| dmsimard | portdirect: corvus sent an email to openstack-dev about a behavior change, not sure if that's what you're seeing though. | 19:03 |
| *** VW has quit IRC | 19:04 | |
| clarkb | portdirect: I want to say that is intentional. Playbooks must be run out of the current project but roles can be shared | 19:05 |
| clarkb | it is possible that the recent changes actually made that the reality | 19:05 |
| *** dpawlik has quit IRC | 19:05 | |
| *** VW_ has quit IRC | 19:06 | |
| *** xarses has quit IRC | 19:06 | |
| *** VW has joined #openstack-infra | 19:06 | |
| *** xarses has joined #openstack-infra | 19:08 | |
| *** xarses has quit IRC | 19:08 | |
| *** xarses has joined #openstack-infra | 19:09 | |
| clarkb | oh ya new code could break the ../ since its per project now | 19:10 |
| clarkb | /untrusted/project_0/git.openstack.org/openstack/openstack-helm/../openstack-helm-infra/ won't work beacuse its /untrusted/project_1/git.openstack.org/openstack/openstack-helm-infra now (note the project_X changes per project iirc) | 19:11 |
| clarkb | portdirect: dmsimard ^ so ya latest change would've done that (also I don't know if project_X is stable I think its intentionally done in a way to not be an external facing interface) | 19:11 |
| AJaeger | pre-run: ../openstack-helm-infra/ -wo | 19:12 |
| AJaeger | clarkb: http://git.openstack.org/cgit/openstack/openstack-helm/tree/.zuul.yaml#n141 ^ | 19:13 |
| dmsimard | I never actually realized that was a thing | 19:13 |
| fungi | ianw: since you've been hacking on the etherpad nodejs upfit recently-ish... do you happen to know what we need to do to make sure npm gets installed by puppet when not using system packages? | 19:15 |
| clarkb | I dont think it was an intentional thing | 19:15 |
| AJaeger | portdirect: that is some very unportable code ;( | 19:15 |
| clarkb | playbooks are supposed ti be per project and roles and jobs the method of sharing aiui | 19:15 |
| dmsimard | clarkb: that was my understranding as well | 19:15 |
| mtreinish | smcginnis: do you have a link the cinder bug about the stestr failures handy? | 19:15 |
| dmsimard | understanding* | 19:15 |
| mtreinish | smcginnis: we probably should open up an stestr bug about that anyway so we can track it there too | 19:16 |
| AJaeger | portdirect: move the jobs to openstack-helm-infra instead of the hack you do | 19:16 |
| portdirect | AJaeger: i dont quite follow | 19:17 |
| smcginnis | mtreinish: I can't remember now if I opened a Cinder bug about that. But I can definitely open an stestr one and reference some of the Cinder and Glance patches related to it. | 19:17 |
| portdirect | would it not be the other way round here? | 19:17 |
| portdirect | but thanks guys - I'll try to resolve asap | 19:18 |
| *** yamamoto has joined #openstack-infra | 19:18 | |
| *** claudiub has quit IRC | 19:18 | |
| *** claudiub has joined #openstack-infra | 19:19 | |
| mtreinish | smcginnis: fwiw, I pushed: https://github.com/mtreinish/stestr/commit/e2cfe2da2564b2e6a312a5920e3ee21080573e20 to see if that is it (I think I got all the stream read calls, but I might have missed one) | 19:20 |
| smcginnis | mtreinish: Nice! I still have a glance patch rotting out there. I can pull this down and test it out with that to see if there's any difference. | 19:25 |
| mtreinish | smcginnis: fwiw, I'm not expecting anything with it. (as everything we've tried hasn't fixed anything) But it was easy enough to hack together to see | 19:26 |
| smcginnis | mrhillsman: Worth a shot. | 19:29 |
| mrhillsman | reading | 19:30 |
| openstackgerrit | Paul Belanger proposed openstack-infra/project-config master: Clean up diskimages for debian-(jessie|stretch) https://review.openstack.org/559802 | 19:31 |
| pabelanger | clarkb: ianw: ^updated to add missing label on nl01.o.o for debian-stretch too | 19:31 |
| mrhillsman | oh smcginnis maybe that was for matt? | 19:31 |
| mtreinish | smcginnis, mrhillsman: yeah I think probably tab complete issue :) | 19:32 |
| mrhillsman | ++ | 19:32 |
| *** yamamoto has quit IRC | 19:33 | |
| *** eernst has quit IRC | 19:33 | |
| *** armaan has joined #openstack-infra | 19:34 | |
| mtreinish | clarkb, dmsimard, pabelanger, ianw, fungi: if you get a sec can you take a look at: https://review.openstack.org/#/q/status:open+topic:even-more-firehose | 19:35 |
| mtreinish | it should be fairly straightforward, just adding mqtt events to statusbot | 19:36 |
| *** dsariel has joined #openstack-infra | 19:37 | |
| dmsimard | fungi: wow that autocomplete dropdown widget | 19:38 |
| smcginnis | Sorry mrhillsman, that was indeed a tab completion issue. :) | 19:39 |
| dmsimard | sortable tables too o_O | 19:39 |
| smcginnis | mrhillsman: Not that I don't value your opinion on things too. ;) | 19:39 |
| *** eernst has joined #openstack-infra | 19:40 | |
| mtreinish | mrhillsman: although if you have any thoughts on that stestr bug, that would be awesome. Because smcginnis and I have been staring at it for months without much success :/ | 19:41 |
| *** mriedem has quit IRC | 19:41 | |
| *** xarses_ has joined #openstack-infra | 19:41 | |
| *** xarses_ has quit IRC | 19:42 | |
| smcginnis | :) | 19:43 |
| *** xarses_ has joined #openstack-infra | 19:44 | |
| *** jklare has quit IRC | 19:44 | |
| *** mriedem has joined #openstack-infra | 19:45 | |
| *** yamahata has quit IRC | 19:47 | |
| *** Roamer` has quit IRC | 19:48 | |
| *** jklare has joined #openstack-infra | 19:48 | |
| *** Roamer` has joined #openstack-infra | 19:48 | |
| *** jtomasek has quit IRC | 19:49 | |
| *** katkapilatova has quit IRC | 19:59 | |
| *** VW has quit IRC | 19:59 | |
| *** VW has joined #openstack-infra | 20:00 | |
| *** VW has quit IRC | 20:01 | |
| *** VW has joined #openstack-infra | 20:02 | |
| *** eernst has quit IRC | 20:03 | |
| *** pcichy has quit IRC | 20:04 | |
| openstackgerrit | Matthew Treinish proposed openstack-infra/system-config master: Enable mqtt statusbot events on eavesdrop.o.o https://review.openstack.org/547685 | 20:04 |
| clarkb | fungi: (/Stage[main]/Etherpad_lite/Exec[install_etherpad_dependencies]/returns) Please install npm ( https://npmjs.org ) | 20:07 |
| clarkb | fungi: I'm guessing something didn't install correctly on the nodejs update? | 20:07 |
| openstackgerrit | Slawek Kaplonski proposed openstack-infra/project-config master: [Grafana] Add openstack-tox-lower-constraints to neutron dashboard https://review.openstack.org/559162 | 20:08 |
| *** yamamoto has joined #openstack-infra | 20:09 | |
| fungi | clarkb: yeah, i was asking ianw if he knew what we need to do to tell the node module to also install npm now that they're split apart | 20:10 |
| fungi | but got sidetracked with meetings. about to pick it back up again and dig in the docs for that module | 20:11 |
| clarkb | fungi: I thought it came in the same package? | 20:11 |
| clarkb | but I also don't see it attempting to install nodejs and the version of nodejs prsent on the server is 0.10 | 20:11 |
| fungi | oh, was it that they merged it rather than split it? if so, maybe we have a race condition there | 20:11 |
| fungi | yeah, possible it's giving up before trying to upgrade nodejs | 20:12 |
| clarkb | I believe distro packaging has them split but upstream packaging has them in a single package | 20:12 |
| fungi | may need a before or a requires or some such | 20:12 |
| clarkb | the ordering looks correct to me but I'm just quickly eyeballing it | 20:12 |
| *** jamesmca_ has quit IRC | 20:13 | |
| fungi | i'll need to dig back through the log and see what it did and didn't do | 20:15 |
| *** HeOS has quit IRC | 20:16 | |
| *** HeOS has joined #openstack-infra | 20:17 | |
| *** yamamoto has quit IRC | 20:18 | |
| *** eernst has joined #openstack-infra | 20:18 | |
| clarkb | fungi: the apt sources list is updated for the nodesource 6.x package series | 20:18 |
| openstackgerrit | Kendall Nelson proposed openstack-infra/storyboard master: Reorganize Docs https://review.openstack.org/559827 | 20:18 |
| *** yamamoto has joined #openstack-infra | 20:19 | |
| *** rkukura has quit IRC | 20:20 | |
| clarkb | fungi: 1644UTC is where the log is interesting in syslog | 20:20 |
| *** yamamoto has quit IRC | 20:22 | |
| *** olaph1 has joined #openstack-infra | 20:22 | |
| *** olaph has quit IRC | 20:22 | |
| clarkb | fungi: it looks like it ran and removed nodejs debug and npm (or at least ensured they weren't present) but no corresponding please install happened | 20:26 |
| *** dprince has quit IRC | 20:27 | |
| clarkb | oh I think maybe beacuse nodejs the package is already installed from distro and up to date? | 20:27 |
| clarkb | I think this may just be a failed case of updating | 20:27 |
| *** esberglu has quit IRC | 20:27 | |
| *** esberglu has joined #openstack-infra | 20:28 | |
| prometheanfire | ianw: tag? :D | 20:29 |
| *** esberglu_ has joined #openstack-infra | 20:30 | |
| clarkb | fungi: are you ok if I do an apt-get update? | 20:31 |
| *** esbergl__ has joined #openstack-infra | 20:31 | |
| clarkb | fungi: currently apt seems to only see the distro version which seems wrong given everything else which makes me wonder if update is just not happy | 20:31 |
| clarkb | fungi: or just tell me to get out of the way :P | 20:32 |
| *** esbergl__ is now known as esberglu__ | 20:32 | |
| mordred | clarkb, fungi: just looking at scrollback ... i've learned many things about javascript over the last few months - anything I can be of assistance with wrt etherpad? | 20:32 |
| *** esberglu has quit IRC | 20:33 | |
| clarkb | mordred: right now I don't think so its mostly an apt issue I think | 20:33 |
| clarkb | mordred: maybe alsoa puppet issue, but if we get new node installed and its still unhappy we'll probably bug you | 20:33 |
| mordred | kk | 20:33 |
| mordred | clarkb: are we using ubuntu provided node packages or the additional node repo? | 20:34 |
| * mordred can go look ... | 20:34 | |
| fungi | clarkb: feel free, i'm trying to pay attention in a meeting and catch up on some household chores before i get back to it | 20:34 |
| *** esberglu_ has quit IRC | 20:34 | |
| fungi | clarkb: i wonder if that node puppet module doesn't do package manglement sanely | 20:35 |
| clarkb | mordred: we were using the ubuntu system provided packages but then did an update to pull from nodesource | 20:35 |
| clarkb | mordred: and I think the puppet moduke can't transition them | 20:35 |
| clarkb | fungi: ya that | 20:35 |
| mordred | ah - fun | 20:35 |
| clarkb | ok I just did an update and a sudo apt-get install nodejs and it seemed to do the things | 20:36 |
| clarkb | I think the problem is that apt saw the package as installed so puppet wasn't going t oexplicitly do a new install | 20:36 |
| clarkb | which would bring in a new version | 20:36 |
| *** salv-orlando has quit IRC | 20:36 | |
| clarkb | chances are that our daily upgrades would've brought in the new version for us then things would work | 20:36 |
| clarkb | I have a working npm now | 20:36 |
| *** pcaruana has quit IRC | 20:36 | |
| *** salv-orlando has joined #openstack-infra | 20:36 | |
| clarkb | and nodejs reports the expected version. Lets see if next puppet run does the right things | 20:36 |
| fungi | excellent, so maybe we just need to... yeah that | 20:36 |
| *** esberglu__ is now known as esberglu | 20:37 | |
| mordred | clarkb: fwiw, I don't see the nodesource repo on etherpad.o.o | 20:38 |
| clarkb | mordred: correct this is etherpad-dev currently so far | 20:38 |
| mordred | ah. well good then :) | 20:38 |
| fungi | mordred: triggered by 559767 | 20:42 |
| clarkb | mordred: fungi it appears to still be broken but in a new way | 20:45 |
| clarkb | I think npm may be broken and assume user's home is /home/$USER | 20:45 |
| clarkb | thats the sort of amazing bug that would be expected here I think | 20:46 |
| mordred | clarkb: hrm | 20:46 |
| clarkb | npm ERR! Error: EACCES: permission denied, mkdir '/home/eplite' | 20:46 |
| corvus | isn't that what we expected? | 20:46 |
| corvus | there was a hardcode of /home/$USER at some point | 20:46 |
| corvus | if that's not where our eplite user's homedir ended up, maybe the problem is the env var we set in puppet? | 20:47 |
| corvus | (the notaminusone from that review) | 20:47 |
| mordred | ~eplite is /var/log/eplite | 20:47 |
| fungi | that was one of the recent patches to puppet-etherpad_lite we approved a little bit ago | 20:47 |
| fungi | checking to see if it didn't merge | 20:47 |
| corvus | oh, er, is the issue that isn't upgradable? | 20:47 |
| corvus | like, we're stuck with the old homedir on old servers, but would be correct on new ones? | 20:48 |
| corvus | https://review.openstack.org/528625 is change in question | 20:48 |
| *** Goneri has joined #openstack-infra | 20:48 | |
| openstackgerrit | Brian Rosmaita proposed openstack-infra/openstack-zuul-jobs master: Restrict legacy-tempest-dsvm-neutron-src job https://review.openstack.org/559839 | 20:48 |
| corvus | so maybe puppet user module doesn't handle moving the homedir for us. i can't blame it. :) | 20:49 |
| clarkb | corvus: not sure I have the full hsitroy there but the server definitely hasn't had that user's homedir moved | 20:49 |
| fungi | oh, fair, it may not update the existing account | 20:49 |
| clarkb | so maybe this is a manual cleanup? | 20:49 |
| fungi | yeah, we probably need to chfn | 20:50 |
| fungi | or whatever method of adjusting the existing account | 20:50 |
| *** trown is now known as trown|outtypewww | 20:50 | |
| mordred | edit /etc/passwd :) | 20:51 |
| mordred | eplite:x:999:1001::/var/log/eplite:/usr/sbin/nologin | 20:51 |
| fungi | vipw! ;) | 20:51 |
| fungi | yeah | 20:51 |
| mordred | corvus: I agree with the theory that th euser module doesn't move the homedir | 20:51 |
| mordred | I also agree that I don't blame it for not doing that | 20:52 |
| mordred | wow. etherpad-dev.o.o also has: | 20:52 |
| fungi | this is starting to look more and more like a maintenance window (or fresh rebuild) of etherpad.o.o when we're ready to upgrade it | 20:52 |
| mordred | colord:x:109:116:colord colour management daemon,,,:/var/lib/colord:/bin/false | 20:53 |
| clarkb | where was the homedir change? | 20:53 |
| * clarkb looks at git log -p | 20:53 | |
| fungi | clarkb: https://review.openstack.org/559839 | 20:53 |
| fungi | er, no | 20:53 |
| fungi | https://review.openstack.org/528625 | 20:53 |
| *** olaph has joined #openstack-infra | 20:53 | |
| clarkb | aha found it | 20:53 |
| fungi | corvus pasted it earlier | 20:53 |
| clarkb | oh missed that thanks | 20:54 |
| clarkb | so ya maybe for etherpad dev its a rebuild | 20:54 |
| clarkb | then we followup and do similarwith prod server? | 20:54 |
| *** olaph1 has quit IRC | 20:54 | |
| *** yamamoto has joined #openstack-infra | 20:54 | |
| fungi | well, i'm happy to manually vipw on -dev so we can iterate more quickly and ascertain what other problems we're facing | 20:54 |
| clarkb | fungi: up to you :P surgery like that always seems to end up in problems when I attempt it | 20:55 |
| clarkb | but its the dev server so not much to lose if you are willing | 20:55 |
| fungi | i've already blown away the etherpad install itself a couple times on this server earlier today | 20:55 |
| *** rfolco|rover is now known as rfolco|off | 20:55 | |
| clarkb | you'll also likely need to make the homedir | 20:56 |
| fungi | yep, done and done | 20:56 |
| fungi | vipw, mkdir ~eplite, chown eplite:eplite ~eplite | 20:57 |
| *** yamamoto has quit IRC | 20:58 | |
| *** r-daneel has joined #openstack-infra | 21:00 | |
| *** slaweq has quit IRC | 21:01 | |
| srwilkers | quick question re: zuulv3. if i want to include roles from another project, whats the correct syntax for including that role? would it be `- zuul: openstack/<project_name>`, so long as that project had a roles/ directory at the project root? | 21:01 |
| *** slaweq has joined #openstack-infra | 21:02 | |
| *** harlowja has joined #openstack-infra | 21:03 | |
| corvus | srwilkers: yep, that should do it. | 21:03 |
| corvus | srwilkers: if you haven't seen it, the docs have all the gory details: https://zuul-ci.org/docs/zuul/user/config.html#attr-job.roles let me know if you have any questions | 21:04 |
| *** jamesmcarthur has joined #openstack-infra | 21:06 | |
| ianw | auristor: re your ipv6 question on the afsdb hosts ... i guess they don't *have* to have ipv6 enabled, but most every server we have does | 21:06 |
| *** slaweq has quit IRC | 21:06 | |
| ianw | fungi: i don't remember anything in particular about installing npm explicitly, but it's been a while ... i have a bunch of changes out around that | 21:07 |
| *** jamesmcarthur has quit IRC | 21:09 | |
| *** jamesmcarthur has joined #openstack-infra | 21:10 | |
| fungi | ianw: apparently switching from system to 6.x for the nodejs_version doesn't upgrade the packages. probably having to do with switching where the package was obtained from | 21:10 |
| *** VW_ has joined #openstack-infra | 21:10 | |
| fungi | it seems to have set up the nodesource package repo, but didn't actually replace the existing package | 21:12 |
| *** eharney has quit IRC | 21:12 | |
| *** rkukura has joined #openstack-infra | 21:12 | |
| fungi | until clarkb manually did an apt-get install/upgrade/something | 21:12 |
| clarkb | I did apt-get update and apt-get install nodejs | 21:12 |
| clarkb | I think the puppetry saw the package as already installed as it wasn't checkign the exact version | 21:12 |
| *** dpawlik has joined #openstack-infra | 21:14 | |
| ianw | is this affecting production? | 21:14 |
| *** VW has quit IRC | 21:14 | |
| *** r-daneel has quit IRC | 21:15 | |
| clarkb | no, production we haven't attempted to udpate yet because it isn't affected by the security issue from what we can tell | 21:15 |
| *** VW_ has quit IRC | 21:15 | |
| *** ldnunes has quit IRC | 21:15 | |
| *** r-daneel has joined #openstack-infra | 21:16 | |
| ianw | oh, ok catching up. is this like a security bug with a fancy name type bug, or just a regular bug | 21:17 |
| ianw | oh got it http://blog.etherpad.org/2018/04/07/important-release-1-6-4/ | 21:17 |
| clarkb | ianw: just a regular one | 21:17 |
| ianw | "in recent versions" which i presume covers ours | 21:18 |
| *** dpawlik has quit IRC | 21:18 | |
| ianw | oh, the points have all the details, fair enough | 21:19 |
| fungi | there's relevant discussion in #openstack-infra-incident | 21:19 |
| fungi | in short, etherpad-dev.o.o was probably (at least possibly) impacted by all three bullets in that advisory | 21:19 |
| fungi | etherpad.o.o (prod) is only impacted by the third bullet | 21:19 |
| fungi | and we don't see that one as a huge hole in our typical etherpad use cases | 21:20 |
| *** gouthamr has joined #openstack-infra | 21:20 | |
| fungi | so with etherpad-dev.o.o at least offline, we're safe there while we work out what a good course of action should be for getting it upgraded, and we may as well roll a prod upgrade into this while we're at it | 21:21 |
| *** edmondsw has quit IRC | 21:22 | |
| *** panda is now known as panda|off | 21:22 | |
| *** edmondsw has joined #openstack-infra | 21:24 | |
| clarkb | my yard was just poisoned by the state of oregon and I think I'm supposed to water it in now. (yay invasive pests). So I'm going to afk for a bit to do that now and then I've got an appoitment at 5pm I need to leave for about an hour ahead of time. TLDR i may be afk for a bit starting now | 21:24 |
| fungi | and the homedir adjustment seems to have been the final missing piece? https://etherpad-dev.openstack.org/ is up as of 21:15z (start time of the node process) | 21:24 |
| ianw | fungi: cool, i guess you've seen about how far i got with the xenial update. from memory the rspec test gets everything installed, but it didn't respond on ports | 21:24 |
| fungi | so i guess everyone should feel free to bang on that and see if it seems at all broken still | 21:25 |
| clarkb | hrm apparently rain is good enoguh and it is supposed to rain tomorrow. Maybe I can just ignore the yard for half a day | 21:25 |
| pabelanger | FYI: https://review.openstack.org/559802/ is a clean up for debian-stretch on nodepool.o.o | 21:26 |
| pabelanger | fixed an exception with nodepool client on nl01.o.o | 21:27 |
| pabelanger | fixes* | 21:27 |
| fungi | i have created https://etherpad-dev.openstack.org/p/testpad | 21:27 |
| fungi | so far, so good | 21:27 |
| fungi | looks like this https://etherpad-dev.openstack.org/p/ZIOvayInPk is a pad i created as a test a couple years ago after an upgrade, according to my irc logs. it's still there | 21:28 |
| clarkb | nice rain is sufficient | 21:29 |
| *** edmondsw has quit IRC | 21:29 | |
| fungi | https://etherpad-dev.openstack.org/p/woo-trusty-woo and https://etherpad-dev.openstack.org/p/things and https://etherpad-dev.openstack.org/p/testpad_lkhegl and https://etherpad-dev.openstack.org/p/clarkb-test2 are some other old tests | 21:29 |
| clarkb | fungi: clarkb-test should be an old one too | 21:30 |
| fungi | hrm, yeah https://etherpad-dev.openstack.org/p/clarkb-test is loading pad content for a while now. chat loaded | 21:32 |
| clarkb | fungi: ya it seems unworking for me | 21:33 |
| fungi | oh, i think that was the point of that pad | 21:33 |
| clarkb | that could be | 21:33 |
| fungi | it's expected to be broken, based on the chat discussion around it | 21:33 |
| *** hashar has quit IRC | 21:37 | |
| * fungi finds it hard to believe mriedem has never seen an ietf rfc | 21:38 | |
| clarkb | ya I think as long as the others are working it should be fine | 21:38 |
| *** HeOS has quit IRC | 21:41 | |
| mriedem | request for cake? | 21:42 |
| mriedem | round funny cat | 21:42 |
| auristor | ianw: kafs and auristorfs clients are ipv6 capable and they will try to contact the openstack.org vlserver and ptserver using ipv6 first (as required by IETF Happy Eyeballs/Fast Fallback). This results in delays because the OpenAFS services are not IPv6 capable but DNS is advertising IPv6 addresses for them. | 21:43 |
| *** dbecker has quit IRC | 21:47 | |
| *** dbecker has joined #openstack-infra | 21:48 | |
| fungi | sounds like we should drop the v6 addresses from dns for our afs01.dfw.o.o, afs02.dfw.o.o and afs01.ord.o.o servers | 21:49 |
| auristor | or create a separate set of A only records that are named in the DNS SRV records | 21:49 |
| fungi | infra-root: any objections to removing those aaaa resource records? if no, i'm happy to take care if it | 21:50 |
| ianw | no, i mean they're not doing anything other than afs, and causes problems for !openafs clients, so i think it all adds up | 21:52 |
| corvus | fungi: no objection; probably worth a note in afs.rst | 21:52 |
| fungi | yeah, i'll draft the documentation update now | 21:52 |
| fungi | great idea | 21:53 |
| *** xarses_ has quit IRC | 21:55 | |
| *** rlandy has quit IRC | 21:55 | |
| openstackgerrit | Jeremy Stanley proposed openstack-infra/system-config master: Note missing AAAA records for AFS servers https://review.openstack.org/559851 | 21:56 |
| fungi | corvus: ^ | 21:56 |
| jhesketh | Morning | 21:57 |
| openstackgerrit | Clark Boylan proposed openstack-infra/zuul master: Make db queries asynchronous in zuul-web https://review.openstack.org/559852 | 21:59 |
| fungi | #status log removed AAAA RRs for afs01.dfw.o.o, afs02.dfw.o.o and afs01.ord.o.o per https://review.openstack.org/559851 | 22:00 |
| openstackstatus | fungi: finished logging | 22:00 |
| corvus | it's zuul meeting time in #openstack-meeting-alt | 22:00 |
| fungi | there's always room for zuul! | 22:00 |
| *** xarses_ has joined #openstack-infra | 22:03 | |
| *** bobh has quit IRC | 22:03 | |
| *** andreww has joined #openstack-infra | 22:04 | |
| auristor | fungi: thank you | 22:06 |
| fungi | auristor: thanks for bringing it to our attention! | 22:06 |
| *** jamesmcarthur has quit IRC | 22:06 | |
| auristor | it was annoying me last night when trying to help ianw | 22:07 |
| *** xarses_ has quit IRC | 22:07 | |
| fungi | i saw a bunch of kafs bits merge for linux 4.16, so hoping we can have much better options in the near future | 22:08 |
| *** tosky has quit IRC | 22:08 | |
| auristor | every release cycle we improve kafs | 22:08 |
| clarkb | hrm I wonder if I could get it working on tumbleweed via that | 22:09 |
| * fungi swears he doesn't see lkml as liesure reading ;) | 22:09 | |
| fungi | er, leisure | 22:09 |
| *** jamesmcarthur has joined #openstack-infra | 22:09 | |
| *** ianychoi has quit IRC | 22:10 | |
| auristor | for 4.17 there will be @sys path substitution, improved storing of data written via mmap, directory parsing optimizations, and some rx related timer fixes. | 22:10 |
| fungi | ooh! | 22:10 |
| auristor | and a ton of fcache improvements as a side effect of kafs work | 22:10 |
| clarkb | possibly related, is there any work being done to address the window size problems that openafs has/had? | 22:11 |
| clarkb | its really painful for long distance connections | 22:11 |
| auristor | for 4.18 the focus is on namespace management for kafs mounts | 22:11 |
| auristor | not in openafs | 22:11 |
| *** ianychoi has joined #openstack-infra | 22:11 | |
| auristor | if openstack.org would accept the offer of an AuriStorFS license I could provide the community a vast number of improvements. This morning we had the world's first production transfer of a 6TB volume and in a couple of days the 50TB volume transfer will complete | 22:13 |
| *** rcernin has joined #openstack-infra | 22:14 | |
| auristor | The OpenAFS Rx window size is stuck at 30 packets because anything larger than that results in worse performance | 22:14 |
| clarkb | auristor: even across large rtt distances? | 22:15 |
| *** r-daneel has quit IRC | 22:15 | |
| fungi | it's a tempting offer but we make a point of sticking to free/libre open source software to run our services | 22:15 |
| *** r-daneel has joined #openstack-infra | 22:16 | |
| *** tpsilva has quit IRC | 22:18 | |
| auristor | I understand the philosophy. Sadly, I had to abandon running OpenAFS and move to a co-op model for access to our source code in order to ensure that the developers working on the project could be paid. | 22:18 |
| fungi | i can totally understand that. it's a tough call | 22:19 |
| *** wolverineav has quit IRC | 22:19 | |
| *** wolverineav has joined #openstack-infra | 22:19 | |
| *** jamesmcarthur has quit IRC | 22:20 | |
| auristor | The call was the AuriStor team continuing to do what it loves and work towards the completion of the AFS Vision described in the 1980s papers or close up shop and get a real job | 22:20 |
| auristor | clarkb: the auristor rx stack supports larger window sizes. Its lockless and isn't tied to a single thread performing all network operations ("the rx listener thread in OpenAFS"). This permits AuriStorFS clients and servers to process packets on multiple cores in parallel without contention. | 22:24 |
| *** rpioso is now known as rpioso|afk | 22:24 | |
| clarkb | ah and I'm guessing that contention is why larger packet windows lead to degredation | 22:24 |
| clarkb | granted as rtt goes up that should reduce contention because eventually you're just faster than the time between acks | 22:25 |
| auristor | clarkb: We also manage the call packets such that we can lookup and modify packets in O(1) instead of O(n^2) | 22:25 |
| *** jcoufal has quit IRC | 22:26 | |
| auristor | Where the maximum pain impacts OpenAFS Rx is when packets are dropped or received out of order because then the queue length grows and packet insertion, retransmission, etc becomes increasingly more expensive. | 22:27 |
| *** esberglu has quit IRC | 22:27 | |
| *** esberglu has joined #openstack-infra | 22:28 | |
| auristor | If there was only a single call you would be right but when one long distance call is mixed with hundreds or thousands of other calls the pain from reordered packet impacts all of the calls in flight by reducing the packet processing rate of the single listener thread | 22:28 |
| *** esberglu_ has joined #openstack-infra | 22:29 | |
| auristor | It is not unusual for AuriStor's customers to deploy fileservers that server a half million rx connections at a time from tens of thousands of clients deployed across mixed cloud and internal data centers. | 22:29 |
| *** olaph1 has joined #openstack-infra | 22:30 | |
| *** olaph has quit IRC | 22:30 | |
| openstackgerrit | Merged openstack-infra/system-config master: Note missing AAAA records for AFS servers https://review.openstack.org/559851 | 22:31 |
| clarkb | auristor: interesting | 22:31 |
| *** r-daneel has quit IRC | 22:32 | |
| clarkb | our use case tends to be more access through specific "gateways" rather than native access | 22:32 |
| clarkb | simply because we are sticking afs behind existing tooling like pip | 22:32 |
| *** r-daneel has joined #openstack-infra | 22:32 | |
| *** esberglu has quit IRC | 22:33 | |
| auristor | I understand. the openstack use case leverages the afs client's caching to create dynamically allocated and self regulating mirrors | 22:33 |
| *** esberglu_ has quit IRC | 22:34 | |
| auristor | The hot content ends up replicated to more web servers and the unloved content stays on the afs fileservers. | 22:34 |
| auristor | the openstack use case also has no security requirements since the data is all freely available. however, if it did AuriStorFS' volume and fileserver security policies would be the perfect fit. | 22:35 |
| *** felipemonteiro_ has quit IRC | 22:36 | |
| auristor | I will continue to stick around. It is my hope that openstack.org deploys kafs and helps to push the various Linux distributions to include it in future kernel builds. | 22:36 |
| *** mriedem is now known as mriedem_afk | 22:38 | |
| clarkb | auristor: I definitely think it is a useful tool to have in linuxes | 22:38 |
| ianw | auristor: i do think that if we want to load-balance in, or otherwise setup one of our mirrors using kafs instead of openafs for testing purposes, we could | 22:38 |
| auristor | ianw: that would be very helpful because I do not believe that the OpenAFS client has a long term future on Linux. There are too many barriers to out of tree filesystems with non-GPL source trees. To be a first class filesystem on Linux the support has to be in-tree. | 22:42 |
| *** gouthamr has quit IRC | 22:43 | |
| *** gouthamr has joined #openstack-infra | 22:44 | |
| fungi | truer words were never spoken | 22:45 |
| *** hongbin has quit IRC | 22:45 | |
| fungi | to be any kind of first-class kernel module on the linux kernel it needs to be *in* the linux kernel | 22:46 |
| *** olaph has joined #openstack-infra | 22:48 | |
| *** olaph1 has quit IRC | 22:48 | |
| *** dave-mcc_ has quit IRC | 22:49 | |
| *** esberglu has joined #openstack-infra | 22:50 | |
| *** Goneri has quit IRC | 22:57 | |
| *** yamahata has joined #openstack-infra | 23:01 | |
| *** slaweq has joined #openstack-infra | 23:02 | |
| *** patriciadomin has quit IRC | 23:04 | |
| *** e0ne has quit IRC | 23:04 | |
| *** patriciadomin has joined #openstack-infra | 23:05 | |
| *** andreww has quit IRC | 23:06 | |
| *** r-daneel has quit IRC | 23:07 | |
| *** slaweq has quit IRC | 23:07 | |
| openstackgerrit | James E. Blair proposed openstack-infra/irc-meetings master: End regularly scheduled Zuul meetings https://review.openstack.org/559861 | 23:08 |
| openstackgerrit | Merged openstack-infra/zuul master: Fix streaming decoding boundaries https://review.openstack.org/559326 | 23:11 |
| corvus | i'm almost positive i've asked and gotten an answer on this before, so i'm sorry -- but is there anything blocking our use of kafs right now? maybe something about our kerberos/afs token setup? | 23:12 |
| *** caphrim007_ has joined #openstack-infra | 23:14 | |
| *** caphrim00_ has joined #openstack-infra | 23:16 | |
| ianw | corvus: "Kauth, Kerberos 4 and Kerberos 5 security; plain, partial and full encryption. " ... that covers it? | 23:16 |
| *** lbragstad has quit IRC | 23:17 | |
| *** caphrim007_ has quit IRC | 23:18 | |
| *** caphrim007 has quit IRC | 23:18 | |
| *** caphrim00_ has quit IRC | 23:20 | |
| *** yamamoto has joined #openstack-infra | 23:22 | |
| *** gouthamr has quit IRC | 23:23 | |
| *** bobh has joined #openstack-infra | 23:23 | |
| *** yamamoto has quit IRC | 23:26 | |
| *** SumitNaiksatam has quit IRC | 23:33 | |
| *** lbragstad has joined #openstack-infra | 23:36 | |
| *** lbragstad has quit IRC | 23:47 | |
| auristor | kafs supports the same range of token formats and security as openafs | 23:54 |
| auristor | It does not yet have the RXGK security that AuriStorFS supports. That will most likely be for 4.18. | 23:55 |
| auristor | There is a kaklog tool that obtains tokens. The AuriStorFS and kafs kernel modules can co-exist and the AuriStorFS aklog will populate tokens for both. OpenAFS and kafs do not co-exist. | 23:56 |
| ianw | pabelanger: oh, i realised what's going on with ask settings; we deploy the production settings from system-config, thus the version in puppet-askbot has become out of sync. sorry, i saw a file in there with .pabelanger on it and thought you might have look :) | 23:57 |
| *** yamamoto has joined #openstack-infra | 23:59 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!
