| *** rh-jelabarre has joined #openstack-infra | 00:05 | |
| *** mattw4 has quit IRC | 00:06 | |
| *** jamesmcarthur has quit IRC | 00:06 | |
| *** xenos76 has quit IRC | 00:07 | |
| *** derekh has quit IRC | 00:08 | |
| *** armax has quit IRC | 00:08 | |
| *** derekh has joined #openstack-infra | 00:08 | |
| *** gyee has quit IRC | 00:10 | |
| clarkb | aspiers: I do think if we wanted to experiment with early run jobs again we should define metrics that we can measure and actually measure them | 00:11 |
|---|---|---|
| aspiers | Makes sense | 00:11 |
| clarkb | things like doe spatchset count go up and if so did we end up spending more resources and wall time waiting | 00:11 |
| *** dchen has quit IRC | 00:11 | |
| *** dchen has joined #openstack-infra | 00:12 | |
| clarkb | (also we should encourage people to run tox locally to catch simple problems that can be caught locally) | 00:15 |
| clarkb | that reduces RTT | 00:15 |
| *** bobh has joined #openstack-infra | 00:23 | |
| *** guoqiao has joined #openstack-infra | 00:25 | |
| *** bobh has quit IRC | 00:30 | |
| *** rosmaita has quit IRC | 00:38 | |
| *** jamesmcarthur has joined #openstack-infra | 00:40 | |
| *** ricolin has joined #openstack-infra | 00:43 | |
| *** rosmaita has joined #openstack-infra | 00:51 | |
| *** guoqiao has quit IRC | 00:51 | |
| *** markvoelker has joined #openstack-infra | 00:51 | |
| *** markvoelker has quit IRC | 00:56 | |
| *** jamesmcarthur has quit IRC | 01:06 | |
| *** jamesmcarthur has joined #openstack-infra | 01:06 | |
| *** slaweq has joined #openstack-infra | 01:11 | |
| *** mattw4 has joined #openstack-infra | 01:11 | |
| *** slaweq has quit IRC | 01:16 | |
| *** jamesmcarthur has quit IRC | 01:21 | |
| *** spsurya has joined #openstack-infra | 01:28 | |
| *** bobh has joined #openstack-infra | 01:33 | |
| *** bobh has quit IRC | 01:34 | |
| ianw | Couldn't download package libss2 (ver 1.44.1-1 arch amd64) at http://mirror.regionone.fortnebula.opendev.org/ubuntu/pool/main/e/e2fsprogs/libss2_1.44.1-1_amd64.deb ... weird | 01:41 |
| tkajinam | Hi. I'm not sure this is the right place to ask it, but I see some trouble when I tried to login to gerrit | 01:53 |
| tkajinam | When I click login, I see the page with error "Provider is not supported, or was incorrectly entered." I remember that I saw login page for UbuntuOne before. Is there any announced change about it? | 01:54 |
| clarkb | oh interesting could that be fallout of the restart? | 01:55 |
| clarkb | https://login.ubuntu.com/+openid is a 503 | 01:57 |
| clarkb | I wonder if that is the problem | 01:57 |
| clarkb | tkajinam: I think that is the underlying issue | 01:57 |
| tkajinam | clarkb, thanks for the info. That make much sense! | 01:57 |
| tkajinam | I tried with chrome with clearing cache but still see the error. So this is definitely server-side problem. | 01:58 |
| clarkb | yup a 503 typically indicates a server side problem | 01:59 |
| clarkb | tkajinam: can you try it now? | 01:59 |
| clarkb | I think it is working now | 02:00 |
| tkajinam | clarkb, yes! seems that it is eventually coming back | 02:01 |
| tkajinam | now I logged in to gerrit :-) | 02:01 |
| *** jamesmcarthur has joined #openstack-infra | 02:05 | |
| *** dchen has quit IRC | 02:10 | |
| *** hongbin has joined #openstack-infra | 02:13 | |
| *** roman_g has quit IRC | 02:16 | |
| *** jamesmcarthur has quit IRC | 02:18 | |
| *** jamesmcarthur has joined #openstack-infra | 02:20 | |
| *** dchen has joined #openstack-infra | 02:22 | |
| *** jamesmcarthur has quit IRC | 02:26 | |
| *** jamesmcarthur_ has joined #openstack-infra | 02:26 | |
| *** derekh has quit IRC | 02:29 | |
| *** derekh has joined #openstack-infra | 02:29 | |
| *** jamesmcarthur_ has quit IRC | 02:31 | |
| *** jamesmcarthur has joined #openstack-infra | 02:34 | |
| *** dave-mccowan has quit IRC | 02:38 | |
| *** tinwood has quit IRC | 02:49 | |
| *** tinwood has joined #openstack-infra | 02:50 | |
| *** dchen has quit IRC | 02:50 | |
| *** rh-jelabarre has quit IRC | 02:55 | |
| *** larainema has joined #openstack-infra | 03:01 | |
| *** slaweq has joined #openstack-infra | 03:11 | |
| *** slaweq has quit IRC | 03:16 | |
| prometheanfire | I'm guessing we need rechecks? | 03:17 |
| *** dchen has joined #openstack-infra | 03:25 | |
| *** dchen has quit IRC | 03:25 | |
| *** slaweq has joined #openstack-infra | 03:32 | |
| *** jamesmcarthur has quit IRC | 03:34 | |
| *** jamesmcarthur has joined #openstack-infra | 03:37 | |
| *** dchen has joined #openstack-infra | 03:37 | |
| *** rfolco has quit IRC | 03:38 | |
| *** jamesmcarthur has quit IRC | 03:42 | |
| *** dchen has joined #openstack-infra | 03:52 | |
| *** nicolasbock has quit IRC | 03:58 | |
| *** dchen has joined #openstack-infra | 04:01 | |
| *** derekh has quit IRC | 04:02 | |
| *** derekh has joined #openstack-infra | 04:03 | |
| *** ramishra has joined #openstack-infra | 04:04 | |
| *** jamesmcarthur has joined #openstack-infra | 04:05 | |
| *** igordc has joined #openstack-infra | 04:10 | |
| *** jamesmcarthur has quit IRC | 04:12 | |
| *** dchen has joined #openstack-infra | 04:17 | |
| *** dchen has quit IRC | 04:17 | |
| *** dchen has joined #openstack-infra | 04:21 | |
| *** diga has joined #openstack-infra | 04:22 | |
| *** igordc has quit IRC | 04:23 | |
| *** jaosorior has quit IRC | 04:34 | |
| *** jaosorior has joined #openstack-infra | 04:34 | |
| *** hongbin has quit IRC | 04:35 | |
| *** dchen has joined #openstack-infra | 04:40 | |
| *** jamesmcarthur has joined #openstack-infra | 05:09 | |
| *** jamesmcarthur has quit IRC | 05:13 | |
| *** zhurong has quit IRC | 05:15 | |
| *** markvoelker has joined #openstack-infra | 05:15 | |
| *** markvoelker has quit IRC | 05:20 | |
| *** pcaruana has joined #openstack-infra | 05:25 | |
| openstackgerrit | Ian Wienand proposed openstack/diskimage-builder master: Uninstall linux-firmware and linux-firmware-whence https://review.opendev.org/675547 | 05:36 |
| *** jamesmcarthur has joined #openstack-infra | 05:49 | |
| *** jaosorior has quit IRC | 05:49 | |
| *** jaosorior has joined #openstack-infra | 05:51 | |
| *** udesale has joined #openstack-infra | 05:54 | |
| *** jamesmcarthur has quit IRC | 05:54 | |
| *** mattw4 has quit IRC | 06:00 | |
| *** raukadah is now known as chandankumar | 06:03 | |
| *** markvoelker has joined #openstack-infra | 06:06 | |
| *** markvoelker has quit IRC | 06:10 | |
| *** dchen has quit IRC | 06:13 | |
| *** dchen has joined #openstack-infra | 06:14 | |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/project-config master: Normalize projects.yaml https://review.opendev.org/680581 | 06:15 |
| *** lpetrut has joined #openstack-infra | 06:16 | |
| *** pgaxatte has joined #openstack-infra | 06:22 | |
| *** xinranwang has joined #openstack-infra | 06:24 | |
| *** markvoelker has joined #openstack-infra | 06:36 | |
| *** kopecmartin|off is now known as kopecmartin | 06:36 | |
| *** dchen has joined #openstack-infra | 06:38 | |
| *** dychen has joined #openstack-infra | 06:41 | |
| *** markvoelker has quit IRC | 06:41 | |
| openstackgerrit | Merged openstack/diskimage-builder master: Rename openSUSE 15.1 testing to 15 https://review.opendev.org/679367 | 06:41 |
| *** dchen has quit IRC | 06:43 | |
| *** dychen has quit IRC | 06:46 | |
| *** dchen has joined #openstack-infra | 06:46 | |
| *** virendra-sharma has joined #openstack-infra | 06:48 | |
| *** pkopec has joined #openstack-infra | 06:50 | |
| *** jamesmcarthur has joined #openstack-infra | 06:51 | |
| *** trident has quit IRC | 06:53 | |
| *** dchen has quit IRC | 06:55 | |
| *** jamesmcarthur has quit IRC | 06:55 | |
| openstackgerrit | Ian Wienand proposed opendev/system-config master: rsync-mirrors: add flag to pause mirroring https://review.opendev.org/680586 | 06:57 |
| *** dchen has joined #openstack-infra | 06:59 | |
| openstackgerrit | Merged openstack/project-config master: Normalize projects.yaml https://review.opendev.org/680581 | 07:01 |
| *** kjackal has joined #openstack-infra | 07:01 | |
| *** trident has joined #openstack-infra | 07:04 | |
| *** dchen has quit IRC | 07:05 | |
| *** markvoelker has joined #openstack-infra | 07:06 | |
| *** dchen has joined #openstack-infra | 07:06 | |
| *** threestrands has quit IRC | 07:10 | |
| *** rcernin|brb has quit IRC | 07:10 | |
| *** tesseract has joined #openstack-infra | 07:11 | |
| *** markvoelker has quit IRC | 07:11 | |
| *** ccamacho has joined #openstack-infra | 07:13 | |
| *** zhurong has joined #openstack-infra | 07:16 | |
| *** markvoelker has joined #openstack-infra | 07:16 | |
| *** apetrich has joined #openstack-infra | 07:18 | |
| *** markvoelker has quit IRC | 07:20 | |
| *** jamesmcarthur has joined #openstack-infra | 07:22 | |
| *** pkopec_ has joined #openstack-infra | 07:22 | |
| *** pkopec_ has quit IRC | 07:23 | |
| *** pkopec has quit IRC | 07:24 | |
| *** pkopec has joined #openstack-infra | 07:25 | |
| openstackgerrit | Ian Wienand proposed openstack/diskimage-builder master: Add fedora-30 testing to gate https://review.opendev.org/680531 | 07:32 |
| *** gfidente has joined #openstack-infra | 07:35 | |
| *** diablo_rojo__ has joined #openstack-infra | 07:35 | |
| ianw | clarkb: i have the lock for mirror.fedora and am doing a release on it. when i have it back in a consistent state, i will enable file auditing and run manually again so we can see why it is taking so long | 07:37 |
| *** diablo_rojo has quit IRC | 07:39 | |
| *** jpena|off is now known as jpena | 07:41 | |
| *** jamesmcarthur has quit IRC | 07:42 | |
| *** zhurong has quit IRC | 07:43 | |
| *** ralonsoh has joined #openstack-infra | 07:45 | |
| *** pkopec has quit IRC | 08:04 | |
| *** pkopec has joined #openstack-infra | 08:04 | |
| *** pkopec has quit IRC | 08:05 | |
| *** dchen has quit IRC | 08:07 | |
| *** xenos76 has joined #openstack-infra | 08:07 | |
| *** pcaruana has quit IRC | 08:08 | |
| *** pkopec has joined #openstack-infra | 08:13 | |
| *** ociuhandu has joined #openstack-infra | 08:13 | |
| *** pcaruana has joined #openstack-infra | 08:13 | |
| *** ociuhandu has quit IRC | 08:18 | |
| *** e0ne has joined #openstack-infra | 08:24 | |
| *** tkajinam has quit IRC | 08:32 | |
| *** pkopec has quit IRC | 08:40 | |
| *** trident has quit IRC | 08:40 | |
| openstackgerrit | Merged openstack/diskimage-builder master: Uninstall linux-firmware and linux-firmware-whence https://review.opendev.org/675547 | 08:43 |
| *** xinranwang has quit IRC | 08:45 | |
| *** pkopec has joined #openstack-infra | 08:46 | |
| *** tosky has joined #openstack-infra | 08:49 | |
| *** trident has joined #openstack-infra | 08:51 | |
| *** zigo has quit IRC | 08:57 | |
| *** zigo has joined #openstack-infra | 09:04 | |
| *** slaweq has quit IRC | 09:06 | |
| *** rfolco has joined #openstack-infra | 09:07 | |
| *** markvoelker has joined #openstack-infra | 09:10 | |
| *** roman_g has joined #openstack-infra | 09:12 | |
| *** markvoelker has quit IRC | 09:20 | |
| *** kjackal has quit IRC | 09:32 | |
| *** prometheanfire has quit IRC | 09:34 | |
| *** prometheanfire has joined #openstack-infra | 09:35 | |
| *** slaweq has joined #openstack-infra | 09:36 | |
| *** dtantsur|afk is now known as dtantsur | 09:37 | |
| openstackgerrit | Merged openstack/diskimage-builder master: Allow configurable gzip binary name https://review.opendev.org/669225 | 09:47 |
| *** kjackal has joined #openstack-infra | 09:48 | |
| openstackgerrit | Merged openstack/diskimage-builder master: Fixed use of flake8 https://review.opendev.org/677059 | 10:01 |
| *** ricolin has quit IRC | 10:15 | |
| *** jcoufal has joined #openstack-infra | 10:15 | |
| *** ociuhandu has joined #openstack-infra | 10:15 | |
| *** ociuhandu has quit IRC | 10:19 | |
| openstackgerrit | Merged openstack/diskimage-builder master: Do not delete cracklib from /usr/share https://review.opendev.org/676983 | 10:25 |
| *** ociuhandu has joined #openstack-infra | 10:29 | |
| *** ociuhandu has quit IRC | 10:30 | |
| *** pgaxatte has quit IRC | 10:33 | |
| *** jcoufal has quit IRC | 10:34 | |
| noonedeadpunk | Hi everyone. Have you merged anything regarding ara report generation? Since today we have non-working ara reports (I mean they are in sqlite format), while yestarday everything was fine. So we rely on ara_report_type for log collection type here https://opendev.org/openstack/openstack-ansible/src/branch/master/zuul.d/playbooks/post.yml#L31 | 10:44 |
| noonedeadpunk | And I see that default is still html https://opendev.org/zuul/zuul-jobs/src/branch/master/roles/ara-report/defaults/main.yaml#L15 | 10:45 |
| noonedeadpunk | Just chandankumar pointed me to http://lists.openstack.org/pipermail/openstack-discuss/2019-September/009148.html | 10:46 |
| frickler | noonedeadpunk: https://review.opendev.org/#/c/680501/2/zuul.d/jobs.yaml | 10:47 |
| noonedeadpunk | ok, so we need to define it ourselves now, right? | 10:48 |
| *** roman_g has quit IRC | 10:49 | |
| noonedeadpunk | yeah, ok | 10:49 |
| frickler | noonedeadpunk: I think so. it also seems that corvus' claim that our change doesn't affect nested ara reports was incomplete | 10:49 |
| noonedeadpunk | yep, but you didn't know that we're relying on provided var inside zuul | 10:50 |
| noonedeadpunk | thanks for pointing me to the patch | 10:54 |
| *** ociuhandu has joined #openstack-infra | 10:55 | |
| *** iurygregory has joined #openstack-infra | 11:02 | |
| aspiers | Hi folks, is it possible that some check jobs got stuck due to the recent Gerrit hiccup? | 11:17 |
| aspiers | Suggestion from #openstack-nova is that newer jobs are getting processed before some which have been in the queue for ~20 hours | 11:18 |
| *** jcoufal has joined #openstack-infra | 11:24 | |
| aspiers | I'm not convinced that's the case, but at least it seems that check is very backlogged, while gate is fine | 11:25 |
| aspiers | I guess still just side-effects of yesterday's issue | 11:26 |
| frickler | aspiers: iiuc we try to fairly share capacities between projects, causing nova and other high-volume projects to suffer most from the current shortage of test-resources | 11:26 |
| aspiers | frickler: yeah, I guess it makes sense to dedicate some resources to the gate too | 11:27 |
| *** roman_g has joined #openstack-infra | 11:27 | |
| *** jpena is now known as jpena|lunch | 11:36 | |
| *** nicolasbock has joined #openstack-infra | 11:38 | |
| *** rh-jelabarre has joined #openstack-infra | 11:46 | |
| *** bobh has joined #openstack-infra | 11:46 | |
| *** bobh has quit IRC | 11:51 | |
| *** johnsom has quit IRC | 11:53 | |
| *** lsell has quit IRC | 11:54 | |
| *** lsell has joined #openstack-infra | 11:54 | |
| *** jpena|lunch has quit IRC | 11:54 | |
| *** johnsom has joined #openstack-infra | 11:55 | |
| *** bnemec has quit IRC | 11:55 | |
| *** jpena has joined #openstack-infra | 11:56 | |
| *** bnemec has joined #openstack-infra | 11:56 | |
| *** pgaxatte has joined #openstack-infra | 11:58 | |
| *** rcernin|brb has joined #openstack-infra | 12:03 | |
| *** ociuhandu has quit IRC | 12:14 | |
| *** ociuhandu has joined #openstack-infra | 12:15 | |
| *** larainema has quit IRC | 12:17 | |
| *** ociuhandu has quit IRC | 12:19 | |
| *** virendra-sharma has quit IRC | 12:20 | |
| *** ociuhandu has joined #openstack-infra | 12:21 | |
| *** AdamMork has joined #openstack-infra | 12:23 | |
| AdamMork | hello! I try to build container requirements. And got ERROR: Could not satisfy constraints for 'tap-as-a-service': installation from path or url cannot be constrained to a version | 12:24 |
| *** EmilienM is now known as EvilienM | 12:24 | |
| *** spsurya has quit IRC | 12:26 | |
| fungi | AdamMork: that's generally a conflict with trying to install a python package from source when it's also included in a constraints list you're applying | 12:28 |
| *** rcernin|brb has quit IRC | 12:30 | |
| *** dave-mccowan has joined #openstack-infra | 12:30 | |
| *** dave-mccowan has quit IRC | 12:31 | |
| fungi | infra-root: https://exim.org/static/doc/security/CVE-2019-15846.txt is what we need to be on the lookout for. debian's already got updated packages up but i don't see it mentioned yet at https://lists.ubuntu.com/archives/ubuntu-security-announce/2019-September/thread.html nor https://usn.ubuntu.com/ | 12:31 |
| *** dave-mccowan has joined #openstack-infra | 12:32 | |
| AdamMork | fungi, i'm trying to deploy taas , and now i try to build docker container. https://docs.openstack.org/openstack-helm/latest/install/plugins/deploy-tap-as-a-service-neutron-plugin.html | 12:33 |
| fungi | AdamMork: it's possible you've found the wrong irc channel | 12:34 |
| fungi | this is where we coordinate building and maintaining services that the openstack community relies on for communication and development activities | 12:35 |
| AdamMork | i got a gerrit refspec and try: docker build https://opendev.org/openstack/loci.git \ | 12:35 |
| AdamMork | yes, possible | 12:35 |
| AdamMork | sorry | 12:35 |
| fungi | AdamMork: there's a list of openstack irc channels at https://wiki.openstack.org/wiki/IRC | 12:36 |
| AdamMork | ok, sorry | 12:36 |
| fungi | AdamMork: https://governance.openstack.org/tc/reference/projects/loci.html suggests there's an #openstack-loci channel too, though it's not mentioned on the wiki | 12:37 |
| fungi | you might try asking there | 12:37 |
| AdamMork | thx | 12:37 |
| *** dave-mccowan has quit IRC | 12:38 | |
| *** markvoelker has joined #openstack-infra | 12:45 | |
| *** mriedem has joined #openstack-infra | 12:46 | |
| *** Goneri has joined #openstack-infra | 12:51 | |
| AdamMork | fungi i | 12:55 |
| AdamMork | about taas. | 12:56 |
| corvus | frickler, noonedeadpunk: we can add the var back to the base job if that would be most appropriate | 13:03 |
| noonedeadpunk | corvus: I'm not sure what's better... I've placed the following patch to cover this as for now https://review.opendev.org/#/c/680656/ | 13:06 |
| *** jamesmcarthur has joined #openstack-infra | 13:07 | |
| openstackgerrit | James E. Blair proposed opendev/base-jobs master: Add ara vars back to base https://review.opendev.org/680683 | 13:10 |
| corvus | noonedeadpunk, infra-root: as i think about it more, i do think that ^ is the better solution | 13:10 |
| corvus | noonedeadpunk: your change is still fine of course; we can merge both | 13:13 |
| *** apetrich has quit IRC | 13:15 | |
| noonedeadpunk | not usre that it's needed, onlyu if to cover our back in case you decide to drop these vars all in all | 13:15 |
| *** pkopec has quit IRC | 13:17 | |
| *** apetrich has joined #openstack-infra | 13:18 | |
| *** eharney has joined #openstack-infra | 13:18 | |
| openstackgerrit | James E. Blair proposed opendev/system-config master: Mitigate exim vulnerability https://review.opendev.org/680685 | 13:21 |
| corvus | infra-root: ^ | 13:22 |
| fungi | corvus: thanks! i was hoping ubuntu would have had patched packages for us already | 13:24 |
| corvus | me too :( | 13:25 |
| *** yamamoto has quit IRC | 13:26 | |
| *** yamamoto has joined #openstack-infra | 13:27 | |
| *** jcoufal_ has joined #openstack-infra | 13:37 | |
| fungi | i mean, i woke up and my personal debian servers had already upgraded exim and restarted it | 13:39 |
| corvus | yeah, and this isn't the first time this has happened. | 13:39 |
| *** jcoufal has quit IRC | 13:39 | |
| openstackgerrit | Merged opendev/base-jobs master: Add ara vars back to base https://review.opendev.org/680683 | 13:40 |
| openstackgerrit | Albin Vass proposed zuul/zuul-jobs master: Clean non-bare remote repos https://review.opendev.org/680689 | 13:42 |
| *** gfidente has quit IRC | 13:46 | |
| *** ociuhandu has quit IRC | 13:47 | |
| *** pgaxatte has quit IRC | 13:50 | |
| *** udesale has quit IRC | 13:56 | |
| *** udesale has joined #openstack-infra | 13:57 | |
| *** ociuhandu has joined #openstack-infra | 14:06 | |
| openstackgerrit | James E. Blair proposed opendev/base-jobs master: Test switch to prepare-workspace-git role https://review.opendev.org/680695 | 14:07 |
| openstackgerrit | James E. Blair proposed opendev/base-jobs master: Switch to prepare-workspace-git role https://review.opendev.org/680696 | 14:07 |
| *** jamesmcarthur has quit IRC | 14:07 | |
| *** markvoelker has quit IRC | 14:07 | |
| aspiers | clarkb: here's another way of addressing the problem of already-failed jobs running when devs don't need them to carry on: allow jobs to be manually cancelled by the review owner and cores | 14:08 |
| aspiers | clarkb: but as you're usually 19 steps ahead of me, I guess you've already thought of this too :) | 14:08 |
| openstackgerrit | James E. Blair proposed zuul/zuul-jobs master: Update the base-roles test to use prepare-workspace-git https://review.opendev.org/680703 | 14:15 |
| *** jamesmcarthur has joined #openstack-infra | 14:15 | |
| openstackgerrit | James E. Blair proposed zuul/zuul-jobs master: Clean non-bare remote repos https://review.opendev.org/680689 | 14:16 |
| openstackgerrit | James E. Blair proposed opendev/base-jobs master: Remove use-cached-repos role https://review.opendev.org/680705 | 14:17 |
| *** pgaxatte has joined #openstack-infra | 14:23 | |
| *** ociuhandu has quit IRC | 14:23 | |
| clarkb | aspiers: if you push a new patchset the existing jobs are cancelled. There is work to add authenticated web management to zuul but I dont think it is tied to gerrit auth/users. Not sure if that would work for this | 14:26 |
| aspiers | clarkb: Gotcha. Yeah I noticed that - I'd prefer not to tweak whitespace in a commit message just to cancel a job, but I guess it's a workaround for now. | 14:27 |
| *** auristor has quit IRC | 14:28 | |
| clarkb | well I think it would be fix issues in the change caught by tests preferrably | 14:30 |
| clarkb | if tests are passing no reason to evict | 14:31 |
| openstackgerrit | Tristan Cacqueray proposed zuul/zuul-base-jobs master: Add support for kubectl connection https://review.opendev.org/680711 | 14:31 |
| *** bdodd_ is now known as bdodd | 14:31 | |
| openstackgerrit | Tristan Cacqueray proposed zuul/zuul-jobs master: Add prepare-workspace-openshift role https://review.opendev.org/631402 | 14:33 |
| openstackgerrit | Tristan Cacqueray proposed zuul/zuul-jobs master: Add remove-zuul-sshkey https://review.opendev.org/680712 | 14:33 |
| openstackgerrit | Tristan Cacqueray proposed zuul/zuul-base-jobs master: Add support for kubectl connection https://review.opendev.org/680711 | 14:33 |
| *** jaosorior has quit IRC | 14:38 | |
| *** auristor has joined #openstack-infra | 14:39 | |
| aspiers | clarkb: I'm talking about failures which are unrelated to the change under test. But yeah, I know we should try to minimise those :) | 14:41 |
| *** mattw4 has joined #openstack-infra | 14:42 | |
| *** michael-beaver has joined #openstack-infra | 14:49 | |
| *** mattw4 has quit IRC | 14:49 | |
| *** mattw4 has joined #openstack-infra | 14:49 | |
| *** diablo_rojo__ has quit IRC | 14:50 | |
| *** markvoelker has joined #openstack-infra | 14:51 | |
| openstackgerrit | Matt McEuen proposed openstack/project-config master: Add per-project Airship groups https://review.opendev.org/680717 | 14:54 |
| *** ociuhandu has joined #openstack-infra | 14:54 | |
| *** chandankumar is now known as raukadah | 14:54 | |
| *** markvoelker has quit IRC | 14:55 | |
| *** mattw4 has quit IRC | 14:57 | |
| *** ociuhandu has quit IRC | 14:59 | |
| fungi | need to disappear to run some errands and grab lunch, but will be back as soon as i can | 15:00 |
| *** yamamoto has quit IRC | 15:01 | |
| *** yamamoto has joined #openstack-infra | 15:02 | |
| *** yamamoto has quit IRC | 15:02 | |
| *** yamamoto has joined #openstack-infra | 15:03 | |
| *** lpetrut has quit IRC | 15:03 | |
| sean-k-mooney | quick question | 15:04 |
| sean-k-mooney | if i have a multinode job | 15:04 |
| sean-k-mooney | and there is space for 1 node | 15:04 |
| sean-k-mooney | will zuul wait for the second | 15:04 |
| *** bnemec is now known as beekneemech | 15:04 | |
| sean-k-mooney | or allocate 1 node then start waiting for the secodn | 15:05 |
| clarkb | zuul will wait for nodepool to return with a completed node request. Node requests are handled in aggregate together as providers are able to fulfill them | 15:05 |
| sean-k-mooney | ok and is there a timeout for how long it will wait | 15:06 |
| clarkb | if we are blocking due to quota no (it waits for jobs currently running to release quota back to the pool then it will take it) | 15:06 |
| sean-k-mooney | looking at http://zuul.openstack.org/status for 679656 in the experimental queu | 15:06 |
| clarkb | it does timeout waiting for nodes to successfully build once nova boot has been issued and will try up to 3 times per instances per region | 15:07 |
| *** ociuhandu has joined #openstack-infra | 15:07 | |
| *** yamamoto has quit IRC | 15:07 | |
| sean-k-mooney | right | 15:07 |
| sean-k-mooney | im tring to deterim is the node_failure still a quota issue or not | 15:07 |
| clarkb | node failures should never be a quota issue. Node failures mean I tried 3 times to boot this instance in every cloud region that provides the label and all of them failed | 15:08 |
| clarkb | or the cloud label does not exist | 15:08 |
| clarkb | s/cloud/ | 15:08 |
| sean-k-mooney | we were getting quota issue yesterday for this | 15:09 |
| clarkb | I think that is only possible if the quota we have doesn't reflect what nova can boot | 15:09 |
| clarkb | because nodepool will honor quota | 15:09 |
| clarkb | so if the quota values are higher than what nova can actually boot that may cause nova boots to fail due to no valid host found or whatever that error is | 15:10 |
| sean-k-mooney | http://paste.openstack.org/show/771452/ | 15:10 |
| sean-k-mooney | that was form Shrews yesterday | 15:10 |
| sean-k-mooney | i think donnyd remove the quota entirly to prevent this and only rely on the node pool limits | 15:11 |
| clarkb | for quota exceeded errors I think that only happens if nova gets out of sync with its quota | 15:12 |
| clarkb | (which we have seen happen in some clouds) | 15:12 |
| clarkb | because nodepool will track what it is using otherwise | 15:12 |
| Shrews | sean-k-mooney: clarkb: i don't think that particular node_failure seen in the experimental queue is due to quota. The node request would not be fulfilled if nodepool failed to deliver the requested node. I'm confused what that error actually means. | 15:12 |
| sean-k-mooney | the job was working before i change the lables but the lables do exisits in nodepool and use the same image and flavor as the old lable. | 15:15 |
| *** gfidente has joined #openstack-infra | 15:15 | |
| sean-k-mooney | it has not worked since https://github.com/openstack/project-config/commit/747a1283e82ff60e2d652b085909b78477c4f8a7 | 15:15 |
| openstackgerrit | James E. Blair proposed zuul/zuul master: Record handler tasks in json job output https://review.opendev.org/680726 | 15:16 |
| *** snecklifter has joined #openstack-infra | 15:16 | |
| clarkb | sean-k-mooney: can you show us the job def too? | 15:16 |
| sean-k-mooney | sure its teh numa one i was talking to you able earlier in the week. | 15:17 |
| sean-k-mooney | https://review.opendev.org/#/c/679656/10 | 15:17 |
| sean-k-mooney | patchset 7 worked fine with the old lables. | 15:17 |
| fungi | infra-root: https://usn.ubuntu.com/4124-1/ is up now | 15:19 |
| sean-k-mooney | clarkb: ill test teh job at the weekend when the gate is less bussy to make sure its still correct | 15:21 |
| clarkb | sean-k-mooney: both labels exist at https://zuul.openstack.org/labels and the specified flavor for those labels also exists. We probably need to check the launcher logs | 15:21 |
| sean-k-mooney | i assume those are not public/in logstash | 15:22 |
| *** kaiokmo has quit IRC | 15:22 | |
| *** snecklifter has quit IRC | 15:22 | |
| clarkb | nodepool.exceptions.ConnectionTimeoutException: Timeout waiting for connection to 2001:470:e045:8000:f816:3eff:fe44:fdf0 on port 22 | 15:23 |
| clarkb | sean-k-mooney: they are not | 15:23 |
| *** snecklifter has joined #openstack-infra | 15:23 | |
| sean-k-mooney | clarkb: ye i assumed so | 15:23 |
| sean-k-mooney | ok so timeout sshing in | 15:23 |
| clarkb | sean-k-mooney: a previous attempt hit openstack.exceptions.SDKException: Error in creating the server (no further information available) | 15:24 |
| sean-k-mooney | that the nodepool verification thing where it connect to a newly build node an validate its active before assingin it to zuul right? | 15:24 |
| clarkb | er wait no that no further info error was for a different node id | 15:25 |
| clarkb | sean-k-mooney: yes | 15:25 |
| clarkb | in this case the three attempts all appear to be ssh failures | 15:26 |
| sean-k-mooney | ok thats intersting since donnyd was able to launch them manually. | 15:26 |
| clarkb | sean-k-mooney: the flavor changed compared to your earlier tests right? | 15:26 |
| sean-k-mooney | yes but it shoudl be the same flavor effectivly | 15:26 |
| donnyd | yes, its exactly the same | 15:26 |
| donnyd | i haven' | 15:26 |
| clarkb | de9b933d-3c26-45ed-9530-ccc9d11ac98d is a node uuid from 2019-09-06 14:11:33,108 UTC ish if that helps debug cloud side | 15:27 |
| clarkb | the timeout is 5 minutes | 15:28 |
| clarkb | perhaps that isn't always long enough depending on load? | 15:28 |
| sean-k-mooney | that is 5 mins after the vm is active right? | 15:28 |
| sean-k-mooney | e.g. there is a seperate time out for the connectivity after its active vs build timeout or am i thinking of tempest | 15:29 |
| clarkb | https://zuul-ci.org/docs/nodepool/configuration.html#attr-providers.[openstack].boot-timeout and https://zuul-ci.org/docs/nodepool/configuration.html#attr-providers.[openstack].launch-timeout are both in play here | 15:29 |
| clarkb | 5 minutes is the total timeout from telling nova to boot to giving up | 15:29 |
| clarkb | 2 minutes is how long we get from active to ssh working | 15:29 |
| Shrews | clarkb: Oh, you know, i thought zuul would abandon all other jobs in the buildset if one of the jobs failed to get its nodes. That apparently is a wrong assumption because nodepool did fail to supply that node. Sorry for the wrong info | 15:30 |
| Shrews | luckily clarkb is sharper than i am today | 15:32 |
| sean-k-mooney | Shrews: it will fail the job if all node requested by that job can be fufilled but ya the other jobs will continue | 15:33 |
| Shrews | sean-k-mooney: yeah, i have NO idea why i was making that assumption | 15:33 |
| Shrews | can i blame friday? | 15:33 |
| sean-k-mooney | and if the job that did not get nodes is marked as non voting the over all set will still pass or fail based on the other jobs | 15:34 |
| clarkb | sean-k-mooney: correct | 15:35 |
| sean-k-mooney | the dnm change that follow that up only run 1 job so i can test wtih those instaead to not waste gate time. | 15:35 |
| pabelanger | for opendev zuul user in github, did we enable 2FA there too? IIRC, admins have it today, right? | 15:35 |
| *** Diabelko has quit IRC | 15:36 | |
| *** Diabelko has joined #openstack-infra | 15:36 | |
| *** jbadiapa has quit IRC | 15:37 | |
| *** trident has quit IRC | 15:37 | |
| clarkb | pabelanger: do you mean the opendev admin account or the opendev zuul application (I want to say the application is configured via the admin account and itself doesn't have an account?) | 15:37 |
| *** jbadiapa has joined #openstack-infra | 15:37 | |
| clarkb | the admin account has 2fa configured | 15:37 |
| pabelanger | clarkb: ya, I don't think the opendev-zuul service account supports 2fa, but some user has the server app attached to their account. I wanted to see if that had 2fa enabled (github user) | 15:38 |
| pabelanger | okay | 15:38 |
| *** yamamoto has joined #openstack-infra | 15:38 | |
| clarkb | still not sure I understand. I didn't think there was an account tied to zuul instead it is an "application" which is something accounts configure (but maybe I've totally misunderstood how you do that in github) | 15:39 |
| pabelanger | yah, the application needs to live in a user account | 15:39 |
| pabelanger | it looks like that is https://github.com/opendevorg | 15:41 |
| pabelanger | which, if I understand, you are saying has 2fa enabled? | 15:41 |
| clarkb | pabelanger: sort of. That is an org not a user. The user that manages that org has 2fa enabled | 15:43 |
| *** snecklifter has quit IRC | 15:43 | |
| sean-k-mooney | donnyd: do you think its worth trying to raise the timeout? i think that requires a nodepool restart which is kind of a pain but if the issue still happens when the load is less it might be worth a try. | 15:43 |
| clarkb | sean-k-mooney: it does not require a nodepool retart | 15:43 |
| *** yamamoto has quit IRC | 15:43 | |
| pabelanger | clarkb: got it, so the org has the service account | 15:43 |
| pabelanger | thanks! | 15:43 |
| clarkb | sean-k-mooney: nodepool will reload its config on its next pass through its loop and use the new timeout as soon as that is done | 15:43 |
| sean-k-mooney | clarkb: oh that never used to work for me | 15:44 |
| sean-k-mooney | although maybe that is becasue it was becuase when i tried it i had errors | 15:44 |
| clarkb | sean-k-mooney: if you change the clouds.yaml I think a restart may be required | 15:44 |
| *** dtantsur is now known as dtantsur|afk | 15:44 | |
| clarkb | if you change nodepool's own yaml config it should find it | 15:44 |
| donnyd | Sorry I was on a call | 15:45 |
| sean-k-mooney | what about zuul | 15:45 |
| sean-k-mooney | it gets most of its config form git | 15:45 |
| donnyd | when I try to provision the instance using that flavor I get 100% success | 15:45 |
| sean-k-mooney | does it reload its primay config | 15:45 |
| donnyd | It shouldn't work any different that any other flavor | 15:46 |
| *** trident has joined #openstack-infra | 15:46 | |
| *** virendra-sharma has joined #openstack-infra | 15:46 | |
| donnyd | I think we should move it to its own dedicated space | 15:46 |
| donnyd | like we had it before, so there is no contention | 15:46 |
| sean-k-mooney | donnyd: the issue is is ssh connectivy working | 15:46 |
| clarkb | sean-k-mooney: for configs in project repos that is all dynamically loaded as changes merge. For config that sits on disk like its tenant config you have to SIGHUP or run zuul-schedule reload-configuration (did I get that command correct? soemthing like that) | 15:46 |
| sean-k-mooney | clarkb: ah it was the tenant config i was thinking of | 15:47 |
| donnyd | that is really interesting, because at any other point in time I have 60ish instances that work fine on the same network | 15:47 |
| clarkb | donnyd: changing the nodepool config to the extra pools as before shouldn't change the ssh timeout behavior (that is all about the instances ability to boot in the cloud itself) | 15:47 |
| sean-k-mooney | ok good to know sighup works | 15:47 |
| donnyd | increasing the ssh timeout shouldn't hurt anything | 15:47 |
| sean-k-mooney | donnyd: did you try sshing into the instace you booted | 15:48 |
| donnyd | It just doesn't really make a lot of sense because the only thing different from this label from the other is the flavor | 15:48 |
| donnyd | sean-k-mooney: no, but i will right now | 15:48 |
| sean-k-mooney | ya and the flavor should be identical. they ones you showed me on past bin were anyway | 15:49 |
| *** mattw4 has joined #openstack-infra | 15:51 | |
| donnyd | https://www.irccloud.com/pastebin/FWxMEIqc/ | 15:51 |
| *** tesseract has quit IRC | 15:52 | |
| sean-k-mooney | yep they look the same to me | 15:52 |
| sean-k-mooney | the first flavor was the one that was fine wand im now usein gboth of the others | 15:53 |
| sean-k-mooney | mainly to reduce memory | 15:53 |
| sean-k-mooney | i was hoping to eventrually swap to the non expanded memory version | 15:53 |
| *** gyee has joined #openstack-infra | 15:55 | |
| donnyd | https://www.irccloud.com/pastebin/L56tUY9W/ | 15:55 |
| clarkb | I concur those flavors in the paste appear to be the same | 15:55 |
| donnyd | and also ssh works fine in my test nodes | 15:55 |
| clarkb | let me boot a test instance and try to hit it from our end (I suppose it could be networking trouble on the internets too?) | 15:56 |
| sean-k-mooney | yep. may it is taking more then 2 mins for ssh to come up. it really should not | 15:56 |
| sean-k-mooney | but maybe | 15:57 |
| openstackgerrit | Merged zuul/zuul-website master: Add Zuul FAQ page https://review.opendev.org/679670 | 15:57 |
| clarkb | I am able to ssh in from ovh and from the launcher that manages fn instances | 15:59 |
| clarkb | could it simply have been a scheduling/noisy neighbor fluke on that cloud? | 16:00 |
| clarkb | sean-k-mooney: we might just want to try again and gather more data | 16:00 |
| sean-k-mooney | i can recheck one of the later patches to not was as many gate resouces | 16:00 |
| sean-k-mooney | but this has failed 4-5 times now since i made the change | 16:01 |
| clarkb | sean-k-mooney: one trick for creating new jobs like this is to remove all the other jobs in the config | 16:01 |
| *** AdamMork has quit IRC | 16:01 | |
| sean-k-mooney | clarkb: yep | 16:01 |
| sean-k-mooney | i had done that and then i added them back because it was working | 16:01 |
| clarkb | you might also try the other label again and see if we can narrow it down to that label | 16:02 |
| sean-k-mooney | ya i can do both. ill go create test patches | 16:02 |
| sean-k-mooney | ok i addded https://review.opendev.org/#/c/680738/ and https://review.opendev.org/#/c/680739/ | 16:09 |
| sean-k-mooney | one will just run the job with the new lable and the the second runs it with the old lable | 16:10 |
| sean-k-mooney | i also move it to check and disable the other jobs | 16:10 |
| *** mriedem is now known as mriedem_lunch | 16:11 | |
| sean-k-mooney | those might take w while to start since they are in check btu we will see what happens. | 16:11 |
| sean-k-mooney | anyway thanks for looking into this. | 16:12 |
| openstackgerrit | Clark Boylan proposed zuul/zuul-website master: Add faq entry about kubernetes and docker images https://review.opendev.org/680742 | 16:14 |
| clarkb | sean-k-mooney: let us know what you find from those changes and we'll go from there | 16:15 |
| sean-k-mooney | if one works and the other does not i will be very confused | 16:16 |
| *** armax has joined #openstack-infra | 16:19 | |
| donnyd | sean-k-mooney: me 2 | 16:25 |
| *** mlavalle has joined #openstack-infra | 16:26 | |
| mlavalle | The Neutron team has decided to remove some feature branches from the Neutron repo. I am reading https://docs.openstack.org/project-team-guide/other-branches.html and couldn't find a process to delete feature branches. Sould I just send a message to the ML requesting the deletion of these branches? | 16:28 |
| clarkb | mlavalle: the release team typically handles brnach management which may be why we don't have that documented there | 16:29 |
| clarkb | mlavalle: I think you need to decide if the state should be preserved and if so make tags for them (similar to stable branch retirements) then ya we can remove the brnaches for you, email request is nice papertrail | 16:30 |
| *** pgaxatte has quit IRC | 16:31 | |
| *** sneharai42 has joined #openstack-infra | 16:32 | |
| *** sneharai42 has quit IRC | 16:32 | |
| *** kopecmartin is now known as kopecmartin|off | 16:36 | |
| *** pcaruana has quit IRC | 16:37 | |
| *** e0ne has quit IRC | 16:39 | |
| *** Sneha has joined #openstack-infra | 16:40 | |
| *** derekh has quit IRC | 16:42 | |
| *** vesper11 has quit IRC | 16:48 | |
| *** jpena is now known as jpena|off | 16:48 | |
| *** vesper11 has joined #openstack-infra | 16:48 | |
| *** gfidente has quit IRC | 16:50 | |
| *** pcaruana has joined #openstack-infra | 16:51 | |
| *** Sneha has quit IRC | 16:54 | |
| *** gyee has quit IRC | 16:56 | |
| *** gyee has joined #openstack-infra | 16:56 | |
| *** tosky has quit IRC | 16:58 | |
| *** kjackal has quit IRC | 16:59 | |
| *** ociuhandu has quit IRC | 17:03 | |
| *** markvoelker has joined #openstack-infra | 17:25 | |
| *** yamamoto has joined #openstack-infra | 17:27 | |
| *** udesale has quit IRC | 17:27 | |
| *** ramishra has quit IRC | 17:31 | |
| *** yamamoto has quit IRC | 17:32 | |
| *** mriedem_lunch is now known as mriedem | 17:38 | |
| corvus | clarkb: on afsdb01 the command "vos examine -localauth project.starlingx.readonly" is not returning | 17:41 |
| corvus | without the .readonly it's fine | 17:41 |
| *** bdodd_ has joined #openstack-infra | 17:41 | |
| clarkb | so the cron is hanging at that point? | 17:41 |
| corvus | this is probably preventing starlingx, zuul, and opendev docs volumes from releasing, since our periodic release script goes through them in series | 17:42 |
| corvus | yeah | 17:42 |
| corvus | i think it eventually times out and bails, because it does still run | 17:42 |
| clarkb | listvldb doesn't show anything that looks wrong for project.starlingx to me | 17:42 |
| *** bdodd has quit IRC | 17:42 | |
| *** jamesmcarthur has quit IRC | 17:45 | |
| clarkb | do we want to try that command on another host? | 17:45 |
| corvus | yeah i'll try on 02 | 17:45 |
| clarkb | I know ianw was planning to add auditing, perhaps that could be related? | 17:45 |
| corvus | also hangs | 17:45 |
| clarkb | ok that script runs vos examine on the RW and RO to compare the last updated times to know if it needs to release | 17:49 |
| *** jcoufal_ has quit IRC | 17:49 | |
| clarkb | (catching up with why that is run) | 17:49 |
| corvus | the docs say that vos examine reports the vldb and the volume header. so it's basically a combo of 'vos listvldb' and 'vos listvol'. | 17:49 |
| corvus | listvldb works as you noted | 17:49 |
| corvus | this hangs: vos listvol -server afs02.dfw.openstack.org | 17:49 |
| corvus | (the same on 01 works) | 17:49 |
| corvus | so that seems to suggest something along the lines of "volume header info on afs02 is bad" | 17:50 |
| corvus | maybe has something to do with "ssh afs02.dfw.openstack.org" not working | 17:50 |
| clarkb | oh heh | 17:51 |
| *** bdodd_ is now known as bdodd | 17:51 | |
| corvus | http://cacti.openstack.org/cacti/graph.php?action=view&local_graph_id=6397&rra_id=all | 17:52 |
| corvus | something's gone screwy there | 17:52 |
| clarkb | could that be related to the auditing? | 17:52 |
| clarkb | the last message from ianw says he was waiting on the fedora vos release to complete before enabling auditing | 17:53 |
| clarkb | (in scrollback of this channel) | 17:53 |
| corvus | and http://cacti.openstack.org/cacti/graph.php?action=view&local_graph_id=6396&rra_id=all | 17:53 |
| corvus | badness seemed to start around 22:00 utc | 17:54 |
| clarkb | ianw's message is from 0700 UTC today quite a bit after 2200UTC | 17:54 |
| corvus | yeah, doesn't seem to line up | 17:55 |
| clarkb | no messages from rax about that host | 17:55 |
| clarkb | (so don't think it is external) | 17:55 |
| corvus | could still maybe be an iscsi sad | 17:55 |
| corvus | i'm impressed snmp is still working | 17:56 |
| *** markvoelker has quit IRC | 17:57 | |
| corvus | i know i ask this every time, and i'm sorry, i just can't seem to remember after all the changes over the years -- what's the console log situation in rax? | 17:57 |
| clarkb | corvus: you log into the web ui and pull up the dashboard for the server instance. Then there is a management console link that loads up in your browser | 17:57 |
| clarkb | (you cannot use nova's console log api) | 17:57 |
| corvus | ok i'll do that now | 17:57 |
| *** e0ne has joined #openstack-infra | 17:58 | |
| corvus | https://imgur.com/a/6RMB8YP | 18:00 |
| corvus | clarkb: reboot now? | 18:01 |
| clarkb | I want to say we've seen that as a fallout of live migration | 18:01 |
| clarkb | so ya I think we reboot | 18:01 |
| corvus | doing so now | 18:02 |
| corvus | several orphaned inodes | 18:02 |
| corvus | it answers ssh now | 18:02 |
| *** markvoelker has joined #openstack-infra | 18:03 | |
| *** virendra-sharma has quit IRC | 18:04 | |
| clarkb | corvus: do we want to clean up the processes that were running for websites releaess and let the next pass try again? | 18:04 |
| *** yamamoto has joined #openstack-infra | 18:04 | |
| corvus | clarkb: yeah, is there one or more of those still hanging around? | 18:04 |
| corvus | nothing is running on 02 right now | 18:05 |
| clarkb | looks like there are actually none now | 18:05 |
| corvus | Fri Sep 6 18:02:27 2019: bosserver: Something is wrong (-1) with the bos configuration file /etc/openafs/BosConfig; aborting | 18:05 |
| corvus | that file is identical to 01 | 18:05 |
| clarkb | I confirm that | 18:06 |
| corvus | bosserver on 01 has not been restarted since the BosConfig file was updated on august 2 | 18:09 |
| clarkb | permissions and filesize are also the same (so doubt weird unicode snuck in) | 18:09 |
| clarkb | ah | 18:09 |
| corvus | the old file (preserved in ~) has arguments on one line, the new file has split them | 18:09 |
| *** yamamoto has quit IRC | 18:09 | |
| clarkb | https://docs.openstack.org/infra/system-config/afs.html#creating-a-volume documents the parameters we want | 18:10 |
| corvus | it's formatted like that, with the newlines | 18:10 |
| corvus | i think the solution here is to concatenate those lines in the bosserver file and restart; agree? | 18:11 |
| clarkb | system-config/tools/afs-server-restart.sh does it with a bash var | 18:12 |
| clarkb | could it be that bash var's literal newlines were ending up in the file? | 18:12 |
| clarkb | the docs we have don'tseem to have changed | 18:12 |
| clarkb | the docs were updated february 2018 | 18:12 |
| *** markvoelker has quit IRC | 18:12 | |
| corvus | that seems like a plausible theory | 18:13 |
| corvus | clarkb: i've edited the file in-place on afs02 if you want to take a look | 18:13 |
| clarkb | looking | 18:14 |
| clarkb | corvus: lgtm | 18:14 |
| corvus | (afaik, this is what bos create does, it just writes that file) | 18:15 |
| *** e0ne has quit IRC | 18:15 | |
| *** markvoelker has joined #openstack-infra | 18:15 | |
| corvus | clarkb: shall i restart now? | 18:15 |
| clarkb | ya things seem to line up pretty identically. I think restart sounds good | 18:15 |
| clarkb | I see the 4 processes now | 18:17 |
| corvus | vos listvol says: Total volumes onLine 27 ; Total volumes offLine 0 ; Total busy 0 | 18:17 |
| corvus | and vos examine on project.starlingx.readonly works | 18:18 |
| corvus | nothing in the salvager log | 18:18 |
| corvus | that's everything i know to check | 18:18 |
| clarkb | k I guess we wait for the next project.zuul vos release in ~2 minutes? | 18:18 |
| corvus | ++ | 18:18 |
| openstackgerrit | James E. Blair proposed opendev/system-config master: Remove newlines in afs server params variable https://review.opendev.org/680754 | 18:20 |
| clarkb | https://zuul-ci.org/docs/faq.html success | 18:21 |
| corvus | if that change gets hit with some kind of linter thing about the line being too long i'm going to lose it | 18:21 |
| corvus | clarkb: \o/ | 18:21 |
| clarkb | we should fix the BosConfig on 01 too? | 18:22 |
| corvus | i will manually edit bosconfig on 01 | 18:22 |
| corvus | ya | 18:22 |
| clarkb | ++ | 18:22 |
| corvus | oh and ord | 18:23 |
| *** raschid has joined #openstack-infra | 18:23 | |
| corvus | #status log restarted afs02.dfw.openstack.org which was unresponsive (suspected live migration issue); manually fixed syntax error in BosConfig files on all fileservers | 18:23 |
| openstackstatus | corvus: finished logging | 18:24 |
| *** factor has joined #openstack-infra | 18:24 | |
| *** markvoelker has quit IRC | 18:26 | |
| *** ralonsoh has quit IRC | 18:27 | |
| *** markvoelker has joined #openstack-infra | 18:28 | |
| paladox | fyi if you pull from gerrit.wikimedia.org, there is currently network issues affecting the data center. So you may have intermittent access. | 18:36 |
| clarkb | paladox: thank you for the heads up | 18:38 |
| *** markvoelker has quit IRC | 18:38 | |
| paladox | it's a confirmed DDOS attack too | 18:38 |
| clarkb | After lunch today i'm going to go and pick up some real life servers. Not the virtual kind (so I'll be afk) | 18:38 |
| clarkb | paladox: :/ | 18:38 |
| *** igordc has joined #openstack-infra | 18:53 | |
| paladox | yeh :( | 18:54 |
| *** roman_g has quit IRC | 18:59 | |
| clarkb | on the zuul queue side of things we are catching up | 19:00 |
| openstackgerrit | Clark Boylan proposed zuul/zuul master: Web: rely on new attributes when determining task failure https://review.opendev.org/680498 | 19:06 |
| openstackgerrit | Clark Boylan proposed zuul/zuul master: Record handler tasks in json job output https://review.opendev.org/680726 | 19:06 |
| *** gagehugo has quit IRC | 19:07 | |
| *** markvoelker has joined #openstack-infra | 19:11 | |
| *** markvoelker has quit IRC | 19:15 | |
| openstackgerrit | David Shrewsbury proposed zuul/nodepool master: Fix node failures when at volume quota https://review.opendev.org/671704 | 19:15 |
| *** guimaluf has joined #openstack-infra | 19:21 | |
| *** markvoelker has joined #openstack-infra | 19:41 | |
| *** markvoelker has quit IRC | 19:45 | |
| *** yamamoto has joined #openstack-infra | 19:53 | |
| *** diga has quit IRC | 19:57 | |
| *** yamamoto has quit IRC | 19:59 | |
| *** jcoufal has joined #openstack-infra | 20:01 | |
| *** diablo_rojo__ has joined #openstack-infra | 20:06 | |
| *** ricolin has joined #openstack-infra | 20:08 | |
| openstackgerrit | Merged opendev/system-config master: Remove newlines in afs server params variable https://review.opendev.org/680754 | 20:09 |
| *** prometheanfire has quit IRC | 20:10 | |
| *** prometheanfire has joined #openstack-infra | 20:11 | |
| *** jtomasek has quit IRC | 20:17 | |
| *** markvoelker has joined #openstack-infra | 20:19 | |
| openstackgerrit | Jeremy Stanley proposed zuul/zuul-website master: Update to page titles and Users https://review.opendev.org/680459 | 20:22 |
| *** markvoelker has quit IRC | 20:29 | |
| *** gagehugo has joined #openstack-infra | 20:43 | |
| *** xenos76 has quit IRC | 20:47 | |
| *** rfolco has quit IRC | 20:50 | |
| *** markvoelker has joined #openstack-infra | 20:50 | |
| *** markvoelker has quit IRC | 20:54 | |
| *** ociuhandu has joined #openstack-infra | 21:04 | |
| *** ricolin has quit IRC | 21:14 | |
| *** ociuhandu has quit IRC | 21:15 | |
| *** rcernin|brb has joined #openstack-infra | 21:19 | |
| *** rcernin|brb has quit IRC | 21:33 | |
| *** jcoufal has quit IRC | 21:38 | |
| *** KeithMnemonic1 has quit IRC | 21:40 | |
| *** mriedem has quit IRC | 21:41 | |
| *** EvilienM is now known as EmilienM | 21:47 | |
| *** yamamoto has joined #openstack-infra | 21:50 | |
| openstackgerrit | James E. Blair proposed zuul/zuul master: WIP: Add support for the Gerrit checks plugin https://review.opendev.org/680778 | 21:50 |
| *** iurygregory has quit IRC | 21:55 | |
| *** yamamoto has quit IRC | 21:55 | |
| *** e0ne has joined #openstack-infra | 22:02 | |
| *** jcoufal has joined #openstack-infra | 22:06 | |
| clarkb | ok I'm back from errands | 22:12 |
| *** diablo_rojo__ has quit IRC | 22:13 | |
| clarkb | fungi: or any other infra-root want to double check neutron's branch deletion request? | 22:14 |
| clarkb | I was having a hard time verifying that the feature/lbaasv2 branch merged into master | 22:15 |
| *** markvoelker has joined #openstack-infra | 22:15 | |
| clarkb | then I had to head out | 22:15 |
| clarkb | however other than the graphql branch the other branches are all ~4 years old | 22:16 |
| clarkb | so I guess I'm not too concerned aboutl osing history there | 22:16 |
| clarkb | but would be good if we can double check that | 22:16 |
| * clarkb learns him a git log | 22:16 | |
| clarkb | if run git log --graph I don't get a feature/lbaasv2 according to my pager search | 22:19 |
| fungi | lemme try | 22:19 |
| clarkb | pecan and qos show up | 22:20 |
| fungi | though i lost access to my workstation after power dropped to the house, so not sure how long it'll take me to clone neutron deep into appalachia | 22:20 |
| *** markvoelker has quit IRC | 22:20 | |
| corvus | #status log upgraded exim on lists.openstack.org | 22:23 |
| openstackstatus | corvus: finished logging | 22:23 |
| clarkb | and ya graphql never seems to have merged anything | 22:23 |
| corvus | clarkb, fungi: ^ i ran apt-get update/upgrade manually on lists.openstack.org and watched it upgrade exim, so we should be good there. if kata still has its own listserver you might want to check on it | 22:24 |
| clarkb | I believe it does | 22:24 |
| corvus | i think that's the only public listener we have? | 22:24 |
| clarkb | corvus: apt-get update && apt-get install exim4 ? | 22:24 |
| corvus | clarkb: i'd just go with upgrade | 22:24 |
| corvus | update and upgrade | 22:25 |
| clarkb | k | 22:25 |
| corvus | wasn't there something with lpmqtt? | 22:25 |
| corvus | do we have an smtp server listening for that? | 22:25 |
| clarkb | corvus: I thought it used imap to some otherwise hosted email but ya it gets launchpad events via email somehow | 22:25 |
| clarkb | corvus: the firehose server iirc | 22:26 |
| fungi | yeah, i think launchpad delivers to it via smtp | 22:26 |
| corvus | does anything use that? | 22:26 |
| fungi | and then a local process checks a local imap server running on it and converts the messages into mqtt | 22:26 |
| fungi | i think it may only listen on the submission socket | 22:27 |
| clarkb | lists.katacontainers.io should be done now. watched it update exim4 | 22:27 |
| clarkb | corvus: the mqtt bus which probably notno | 22:27 |
| fungi | hrm, or my parents' isp is blocking outbound smtp | 22:27 |
| corvus | exim and mosquitto have been upgraded on firehose | 22:28 |
| fungi | thanks, i was seconds away from doing that | 22:29 |
| clarkb | 4.86.2-2ubuntu2.5 is the version we want right? | 22:29 |
| corvus | hope the mosquitto upgrade doesn't break anything | 22:29 |
| clarkb | (just double checking what was installed on lists.katacontainers | 22:29 |
| corvus | clarkb: https://usn.ubuntu.com/4124-1/ | 22:29 |
| fungi | and yeah, exim and iptables have smtp open on firehose, so i guess at&t dsl here blocks outbound smtp | 22:30 |
| corvus | my grep of system-config for the string "- 25" shows only mailman and firehose | 22:32 |
| corvus | and "- 465" only on mailman, and no instances of "- 587" | 22:33 |
| *** e0ne has quit IRC | 22:34 | |
| *** rfolco has joined #openstack-infra | 22:38 | |
| *** michael-beaver has quit IRC | 22:38 | |
| fungi | yay, finally finished cloning neutron and performing a remote update in it. now i just need to remember what i was going to do with it. oh! right, branch comparisons ;) | 22:38 |
| fungi | clarkb: git merge-base origin/feature/lbaasv2 origin/master | 22:49 |
| fungi | using `git log --oneline --no-merges ...` between that commit and the branch tip turns up 6 commits most of which are basically noise | 22:50 |
| fungi | the only substantive commits seem to be: | 22:51 |
| fungi | 4ed88624b1 New extension for version 2 of LBaaS API | 22:51 |
| clarkb | ah ok so it diverged and was updated | 22:51 |
| fungi | 02c01a3378 Plugin/DB additions for version 2 of LBaaS API | 22:51 |
| fungi | cfa4a86ebe Tests for extension, db and plugin for LBaaS V2 | 22:51 |
| clarkb | but substantive changes were in master? | 22:51 |
| fungi | also maybe this one: | 22:51 |
| fungi | 7147389bad Implement Jinja templates for haproxy config | 22:51 |
| fungi | i'll check that next | 22:51 |
| fungi | i don't find any of those 4 commits in origin/master history, no | 22:53 |
| fungi | so may be worth confirming with mlavalle that they're irrelevant | 22:53 |
| clarkb | k did you want to respond to the email thread on -discuss? I'm going to do similar checking for pecan and qos | 22:54 |
| fungi | will do | 22:54 |
| clarkb | qos and pecan only have the opendev migration changes in them | 22:55 |
| clarkb | after the merge base | 22:55 |
| clarkb | similar with graphql nothing substantice so ya I think just lbaasv2 to worry about | 22:56 |
| fungi | replired | 22:58 |
| fungi | er, replied too | 22:58 |
| clarkb | thanks! | 22:58 |
| clarkb | cmurphy: you might want to bump the timeout on keystone's unittests looks like they may timeout occasionally | 22:59 |
| clarkb | (current gate reset train seems to have happened due to that) | 22:59 |
| cmurphy | clarkb: yeah we've been struggling with that | 22:59 |
| cmurphy | clarkb: what's the best way to do that - we just use the train tox project template | 22:59 |
| cmurphy | would we need to override all the jobs in that template to set our own timeout? | 23:00 |
| clarkb | cmurphy: either that or bump the timeout in the template or the jobs themselves | 23:01 |
| *** yamamoto has joined #openstack-infra | 23:02 | |
| cmurphy | clarkb: this is shared by every openstack project, i don't think i'd want to bump the timeout for everyone? | 23:03 |
| openstackgerrit | Michael Johnson proposed openstack/diskimage-builder master: Add output for mis-configured element scripts https://review.opendev.org/680785 | 23:03 |
| cmurphy | i think a plan was hatched while i was gone to split the tests so they wouldn't reach the timeout but not sure where that ended up | 23:04 |
| clarkb | cmurphy: the timeout is mostly a sanity check that jobs don't run overly long if they end up going sideways | 23:05 |
| clarkb | its probably ok to incrase the timeout a reaosnable amount globally but ya if it needs a big bump might be better to constrain that to keystone | 23:05 |
| *** mattw4 has quit IRC | 23:05 | |
| mlavalle | fungi: thanks for the thoroughness. Please look at this message: http://lists.openstack.org/pipermail/openstack-discuss/2019-May/006142.html | 23:06 |
| mlavalle | lbaas doesn't exist anymore | 23:06 |
| fungi | okay, so those commits aren't valuable? | 23:06 |
| openstackgerrit | Michael Johnson proposed openstack/diskimage-builder master: Add output for mis-configured element scripts https://review.opendev.org/680785 | 23:06 |
| cmurphy | clarkb: okay i'll figure out if we can work around it quickly and if not propose a bump | 23:07 |
| *** yamamoto has quit IRC | 23:07 | |
| mlavalle | fungi: nope | 23:07 |
| fungi | mlavalle: and lbaasv2 is not octavia, right? | 23:07 |
| fungi | or octavia was an lbaasv2 backend? | 23:08 |
| johnsom | lbaasv2 is an API spec | 23:08 |
| mlavalle | fungi: octavia replaces lbass and is a superset | 23:08 |
| johnsom | neutron-lbaas and octavia both implement that spec | 23:08 |
| johnsom | octavia is a superset | 23:08 |
| mlavalle | yeap | 23:08 |
| fungi | thanks. it's been so long i've lost the brain cells which once held that information | 23:08 |
| johnsom | I have a handy FAQ: https://wiki.openstack.org/wiki/Neutron/LBaaS/Deprecation | 23:08 |
| clarkb | right so the question is whether we need to preserve those commits otherwise they will get garbage collected and be deleted | 23:09 |
| clarkb | historically we've avoided deleting commits and used tags to ensure that history remains | 23:09 |
| clarkb | whether or not it is currently relevant | 23:09 |
| johnsom | What commits? I'm coming to the party late | 23:10 |
| clarkb | johnsom: http://lists.openstack.org/pipermail/openstack-discuss/2019-September/009197.html those commits | 23:10 |
| johnsom | Oh, ok. Yeah, I don't think there is any value in saving the feature/lbaasv2 branch. If there was it would have been rolled into the neutron-lbaas repo long ago. | 23:12 |
| fungi | clarkb: i do wonder though, since there are named refs for those (thank you gerrit) they wouldn't be garbage-collected anyway, right? | 23:13 |
| clarkb | fungi: oh maybe | 23:13 |
| clarkb | sounds like we should proceed with deleting those brances as the other 3 are fine and no one is concerned about the lbaasv2 one | 23:14 |
| fungi | that seems to be the case, yes | 23:14 |
| mlavalle | Thanks! | 23:15 |
| *** ociuhandu has joined #openstack-infra | 23:15 | |
| clarkb | mlavalle: done, wantot double check it? | 23:17 |
| *** ociuhandu has quit IRC | 23:20 | |
| *** signed8b_ has joined #openstack-infra | 23:21 | |
| mlavalle | clarkb, fungi: yes, they are gone. Thanks! | 23:22 |
| *** signed8bit has quit IRC | 23:23 | |
| *** jcoufal has quit IRC | 23:28 | |
| openstackgerrit | Michael Johnson proposed openstack/diskimage-builder master: Add output for mis-configured element scripts https://review.opendev.org/680785 | 23:29 |
| *** Goneri has quit IRC | 23:31 | |
| corvus | cmurphy: to override the timeout you can still use the template and just also list the job individually with the timeout value | 23:35 |
| clarkb | TIL | 23:36 |
| cmurphy | corvus: oh that is handy | 23:36 |
| cmurphy | corvus: is that documented somewhere? i was looking for it a couple of weeks ago | 23:36 |
| corvus | good question... i'm not sure that we even state that explicitly; it's sort of a construct that you can derive from the combination of https://zuul-ci.org/docs/zuul/user/config.html#attr-project.%3Cpipeline%3E.jobs and paragraph 5 of https://zuul-ci.org/docs/zuul/user/config.html#job | 23:40 |
| corvus | which sounds like a not entirely convenient way of having that documented :) | 23:40 |
| *** yamamoto has joined #openstack-infra | 23:40 | |
| cmurphy | :) | 23:40 |
| corvus | yeah, i'm thinking we could use a dedicated section on "these are practical ways you can inherit/override/etc jobs" | 23:42 |
| *** yamamoto has quit IRC | 23:45 | |
| *** gyee has quit IRC | 23:45 | |
| *** mtreinish has quit IRC | 23:49 | |
| *** sthussey has quit IRC | 23:49 | |
| openstackgerrit | Michael Johnson proposed openstack/diskimage-builder master: Add output for mis-configured element scripts https://review.opendev.org/680785 | 23:54 |
| *** gyee has joined #openstack-infra | 23:55 | |
| *** gyee has quit IRC | 23:56 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!