| *** mattw4 has quit IRC | 00:00 | |
| *** goldyfruit___ has joined #openstack-infra | 00:01 | |
| *** igordc has quit IRC | 00:10 | |
| *** jamesmcarthur has joined #openstack-infra | 00:21 | |
| *** mriedem_away has quit IRC | 00:22 | |
| *** mrda has quit IRC | 00:28 | |
| *** gyee has quit IRC | 00:28 | |
| *** elod has quit IRC | 00:29 | |
| *** jamesmcarthur has quit IRC | 00:30 | |
| *** jcoufal has joined #openstack-infra | 00:31 | |
| TheJulia | Out of curiosity, are there any known issues for mirror.regionone.limestone.openstack.org ? | 00:37 |
|---|---|---|
| openstackgerrit | Kerby proposed zuul/nodepool master: AWS driver: add ability to determine AMI id using filters https://review.opendev.org/683205 | 00:37 |
| TheJulia | or have there been? | 00:37 |
| *** rosmaita has quit IRC | 00:37 | |
| *** markvoelker has joined #openstack-infra | 00:46 | |
| *** elod has joined #openstack-infra | 00:49 | |
| *** markvoelker has quit IRC | 00:50 | |
| *** rosmaita has joined #openstack-infra | 00:51 | |
| openstackgerrit | Kerby proposed zuul/nodepool master: AWS driver: add ability to determine AMI id using filters https://review.opendev.org/683205 | 00:52 |
| clarkb | TheJulia: yes we pulled it out of service due to routing issues | 01:01 |
| fungi | i wonder how we could do a better job of reflecting stuff like that at https://wiki.openstack.org/wiki/Infrastructure_Status | 01:13 |
| *** larainema has quit IRC | 01:15 | |
| *** whoami-rajat has joined #openstack-infra | 01:17 | |
| *** yamamoto has quit IRC | 01:24 | |
| *** yamamoto has joined #openstack-infra | 01:30 | |
| *** rkukura has quit IRC | 01:50 | |
| *** jamesmcarthur has joined #openstack-infra | 01:54 | |
| *** ricolin has joined #openstack-infra | 01:55 | |
| ianw | remember the days when "shutdown -h now" meant, shutdown, now ... not shutdown unless systemd thinks there's something i shouldn't shutdown for but don't worry there's a separate systemd command to override that now | 01:55 |
| *** jcoufal has quit IRC | 01:56 | |
| *** yamamoto has quit IRC | 02:07 | |
| *** jamesmcarthur has quit IRC | 02:15 | |
| rm_work | hey, uhh... is https://github.com/openstack/ospurge not actually configured for gerrit? did it move somewhere? not seeing it in gerrit | 02:15 |
| rm_work | also: anyone happen to know if it's active / who owns it? | 02:16 |
| rm_work | ahhh nm i found it, it got moved under x/ospurge | 02:16 |
| clarkb | rm_work: one trick is to go to opendev.org/openstack/ospurge and see where it redirects you | 02:18 |
| rm_work | ah cool | 02:19 |
| rm_work | gonna try to track down the team | 02:21 |
| rm_work | there's a high probability our org would like to start contributing to it significantly | 02:22 |
| *** prometheanfire has quit IRC | 02:23 | |
| *** jamesmcarthur has joined #openstack-infra | 02:26 | |
| *** roman_g has quit IRC | 02:34 | |
| *** rlandy|bbl is now known as rlandy | 02:37 | |
| *** tinwood has quit IRC | 02:50 | |
| *** tinwood has joined #openstack-infra | 02:52 | |
| *** jamesmcarthur has quit IRC | 02:52 | |
| *** jamesmcarthur has joined #openstack-infra | 02:54 | |
| *** prometheanfire has joined #openstack-infra | 02:57 | |
| *** ricolin_ has joined #openstack-infra | 03:04 | |
| *** rfolco has quit IRC | 03:05 | |
| *** ricolin has quit IRC | 03:06 | |
| openstackgerrit | Logan V proposed openstack/diskimage-builder master: Ensure machine-id is not included in images https://review.opendev.org/683272 | 03:15 |
| openstackgerrit | Logan V proposed openstack/diskimage-builder master: Ensure machine-id is not included in images https://review.opendev.org/683272 | 03:17 |
| openstackgerrit | Logan V proposed openstack/diskimage-builder master: Ensure machine-id is not included in images https://review.opendev.org/683272 | 03:17 |
| *** jamesmcarthur has quit IRC | 03:18 | |
| *** jamesmcarthur has joined #openstack-infra | 03:18 | |
| *** jamesmcarthur has quit IRC | 03:22 | |
| *** jamesmcarthur has joined #openstack-infra | 03:22 | |
| *** jamesmcarthur has quit IRC | 03:26 | |
| *** psachin has joined #openstack-infra | 03:35 | |
| *** rlandy has quit IRC | 03:38 | |
| *** ykarel|away has joined #openstack-infra | 03:39 | |
| *** ramishra has joined #openstack-infra | 03:47 | |
| AJaeger | config-core, could I get another +2a on https://review.opendev.org/682972 , please? | 04:00 |
| *** jamesmcarthur has joined #openstack-infra | 04:10 | |
| *** ykarel|away is now known as ykarel | 04:11 | |
| *** dave-mccowan has quit IRC | 04:19 | |
| *** rh-jelabarre has quit IRC | 04:32 | |
| *** jamesmcarthur has quit IRC | 04:41 | |
| *** weshay has quit IRC | 04:52 | |
| *** yamamoto has joined #openstack-infra | 04:58 | |
| *** yamamoto has quit IRC | 05:01 | |
| *** kjackal has joined #openstack-infra | 05:08 | |
| *** armax has joined #openstack-infra | 05:14 | |
| *** armax has quit IRC | 05:18 | |
| *** armax_ has joined #openstack-infra | 05:18 | |
| *** pcaruana has joined #openstack-infra | 05:28 | |
| *** yamamoto has joined #openstack-infra | 05:36 | |
| *** yamamoto has quit IRC | 05:37 | |
| *** yamamoto has joined #openstack-infra | 05:37 | |
| openstackgerrit | Merged openstack/project-config master: Remove obsolete docs publish jobs https://review.opendev.org/682972 | 05:39 |
| *** ykarel has quit IRC | 05:43 | |
| *** armax_ has quit IRC | 05:50 | |
| *** armax has joined #openstack-infra | 05:51 | |
| *** pcaruana has quit IRC | 05:53 | |
| *** ricolin_ is now known as ricolin | 05:54 | |
| *** ykarel has joined #openstack-infra | 06:03 | |
| *** gfidente has joined #openstack-infra | 06:06 | |
| *** apetrich has quit IRC | 06:08 | |
| *** yamamoto has quit IRC | 06:15 | |
| *** Tengu has quit IRC | 06:16 | |
| *** Tengu has joined #openstack-infra | 06:17 | |
| *** slaweq has joined #openstack-infra | 06:19 | |
| *** Tengu has quit IRC | 06:20 | |
| *** Tengu has joined #openstack-infra | 06:21 | |
| *** psachin has quit IRC | 06:23 | |
| *** kjackal has quit IRC | 06:25 | |
| *** Tengu has quit IRC | 06:28 | |
| *** kopecmartin|off is now known as kopecmartin | 06:36 | |
| *** xek has joined #openstack-infra | 06:36 | |
| *** psachin has joined #openstack-infra | 06:38 | |
| *** pgaxatte has joined #openstack-infra | 06:40 | |
| *** dciabrin has joined #openstack-infra | 06:40 | |
| *** Tengu has joined #openstack-infra | 06:42 | |
| *** Tengu has quit IRC | 06:44 | |
| openstackgerrit | Ian Wienand proposed openstack/diskimage-builder master: Install Python 3 libselinux packages for Fedora https://review.opendev.org/683295 | 06:45 |
| *** zbr is now known as zbr|ruck | 06:48 | |
| *** trident has quit IRC | 06:49 | |
| *** Tengu has joined #openstack-infra | 06:50 | |
| *** yamamoto has joined #openstack-infra | 06:55 | |
| *** markvoelker has joined #openstack-infra | 06:57 | |
| *** trident has joined #openstack-infra | 07:01 | |
| *** markvoelker has quit IRC | 07:01 | |
| *** gfidente has quit IRC | 07:04 | |
| *** trident has quit IRC | 07:07 | |
| *** ccamacho has joined #openstack-infra | 07:13 | |
| *** xenos76 has joined #openstack-infra | 07:14 | |
| *** trident has joined #openstack-infra | 07:17 | |
| *** tosky has joined #openstack-infra | 07:18 | |
| *** udesale has joined #openstack-infra | 07:20 | |
| *** yamamoto has quit IRC | 07:20 | |
| *** yolanda has quit IRC | 07:21 | |
| *** yolanda has joined #openstack-infra | 07:23 | |
| *** rcernin has quit IRC | 07:29 | |
| *** rpittau|afk is now known as rpittau | 07:31 | |
| *** yolanda has quit IRC | 07:32 | |
| openstackgerrit | Ian Wienand proposed openstack/diskimage-builder master: Install Python 3 libselinux packages for Fedora https://review.opendev.org/683295 | 07:33 |
| *** apetrich has joined #openstack-infra | 07:41 | |
| *** jpena|off is now known as jpena | 07:42 | |
| *** jamesmcarthur has joined #openstack-infra | 07:43 | |
| *** ralonsoh has joined #openstack-infra | 07:44 | |
| *** e0ne has joined #openstack-infra | 07:46 | |
| *** gfidente has joined #openstack-infra | 07:47 | |
| *** jamesmcarthur has quit IRC | 07:47 | |
| *** yolanda has joined #openstack-infra | 07:47 | |
| *** iurygregory has joined #openstack-infra | 07:51 | |
| *** ykarel is now known as ykarel|lunch | 07:52 | |
| *** e0ne has quit IRC | 07:59 | |
| *** apetrich has quit IRC | 08:07 | |
| *** tkajinam has quit IRC | 08:09 | |
| *** dchen has quit IRC | 08:17 | |
| *** e0ne has joined #openstack-infra | 08:24 | |
| *** e0ne has quit IRC | 08:42 | |
| *** roman_g has joined #openstack-infra | 08:44 | |
| *** exsdev0 has joined #openstack-infra | 08:44 | |
| *** xenos76 has quit IRC | 08:45 | |
| *** exsdev has quit IRC | 08:45 | |
| *** exsdev0 is now known as exsdev | 08:45 | |
| *** xenos76 has joined #openstack-infra | 08:47 | |
| *** ykarel|lunch is now known as ykarel | 08:50 | |
| *** ociuhandu has joined #openstack-infra | 08:52 | |
| *** rcernin has joined #openstack-infra | 08:55 | |
| openstackgerrit | Merged opendev/irc-meetings master: Reduce frequency of Sahara team meeting https://review.opendev.org/683171 | 09:01 |
| *** pcaruana has joined #openstack-infra | 09:12 | |
| *** igordc has joined #openstack-infra | 09:15 | |
| *** yamamoto has joined #openstack-infra | 09:19 | |
| *** igordc has quit IRC | 09:20 | |
| *** Tuor has joined #openstack-infra | 09:22 | |
| *** yamamoto has quit IRC | 09:26 | |
| *** yamamoto has joined #openstack-infra | 09:26 | |
| *** derekh has joined #openstack-infra | 09:27 | |
| *** dtantsur|afk is now known as dtantsur | 09:27 | |
| *** ociuhandu has quit IRC | 09:30 | |
| *** ociuhandu_ has joined #openstack-infra | 09:30 | |
| *** igordc has joined #openstack-infra | 09:39 | |
| *** apetrich has joined #openstack-infra | 09:41 | |
| *** apetrich has quit IRC | 09:45 | |
| *** rcernin has quit IRC | 09:49 | |
| lxkong | hi guys, can we customize the timer for periodic jobs? | 09:50 |
| lxkong | here https://github.com/openstack/project-config/blob/a26c532cd5bab556dbde3d15ee3e538a229c2c13/zuul.d/pipelines.yaml#L191 | 09:50 |
| lxkong | the default is every 6 hours, can i change that to per day? | 09:51 |
| *** e0ne has joined #openstack-infra | 09:54 | |
| *** sean-k-mooney has quit IRC | 09:57 | |
| *** kjackal has joined #openstack-infra | 09:58 | |
| *** apetrich has joined #openstack-infra | 09:59 | |
| AJaeger | lxkong: that says : Run at 6:00 UTC every day. | 10:00 |
| AJaeger | lxkong: so, it does what you want - why do you think it runs every 6 hours? What do you want to do? | 10:00 |
| lxkong | AJaeger: yeah, that's exactly what i want ,hah | 10:01 |
| lxkong | i thougt it's */6 | 10:01 |
| *** e0ne has quit IRC | 10:02 | |
| lxkong | AJaeger: just out of curiosity, can that be overwritten? | 10:02 |
| *** jaosorior has quit IRC | 10:03 | |
| *** gfidente has quit IRC | 10:05 | |
| AJaeger | lxkong: no, it cannot | 10:06 |
| AJaeger | still, what do you want to do? | 10:06 |
| lxkong | AJaeger: i want to manually trigger a job that could build an image (trove) | 10:06 |
| *** ociuhandu_ has quit IRC | 10:07 | |
| lxkong | at the moment, afaik, i can only define the job either in promote or periodic pipeline | 10:07 |
| *** ociuhandu has joined #openstack-infra | 10:07 | |
| lxkong | because i need to upload the image to openstack tarball website | 10:07 |
| lxkong | i am using publish-openstack-artifacts | 10:08 |
| *** yamamoto has quit IRC | 10:11 | |
| AJaeger | why manualy trigger? | 10:13 |
| *** ociuhandu has quit IRC | 10:14 | |
| lxkong | because i only want to create a new image after there are some 'real' patches | 10:14 |
| AJaeger | lxkong: and why is post/promote not working for that? | 10:15 |
| lxkong | sometime the image build may fail, i'm not sure in the post/promote job, if i can re-trigger the job when job fails | 10:16 |
| AJaeger | lxkong: I'm curious on what problem you are trying to solve - what is not working today and what to do you want to see instead? | 10:16 |
| lxkong | AJaeger: that's fine, i'm happy to let people know my pain :-) | 10:16 |
| AJaeger | lxkong: do the following: Build the image in the gate queue and use a promote job to only push the image from the gate job to the site | 10:16 |
| AJaeger | that way if the build fails, you get informed (zuul gives a -1) and no rebuild is needed in post | 10:17 |
| lxkong | AJaeger: is there a example show me how i can refer to the image built in the gate pipeline in a promote job? | 10:17 |
| *** ociuhandu has joined #openstack-infra | 10:18 | |
| AJaeger | check the promote jobs run today, for example zuul-promote-image | 10:20 |
| AJaeger | http://zuul.opendev.org/t/zuul/builds?pipeline=promote or http://zuul.opendev.org/t/openstack/builds?pipeline=promote | 10:20 |
| lxkong | AJaeger: thanks a lot! | 10:20 |
| AJaeger | lxkong: if you do it with docker images, best talk to corvus later today. | 10:21 |
| lxkong | AJaeger: it's not docker :-) | 10:21 |
| AJaeger | ok | 10:21 |
| *** ociuhandu has quit IRC | 10:22 | |
| AJaeger | lxkong: important is to use a zuul_return to tell Zuul about an artifact like in https://opendev.org/zuul/zuul-jobs/src/branch/master/roles/fetch-sphinx-tarball/tasks/pdf.yaml#L46 - then you can retrieve it from post job. | 10:23 |
| AJaeger | promote I mean | 10:23 |
| lxkong | AJaeger: sure, i will have a try | 10:24 |
| *** kjackal has quit IRC | 10:24 | |
| *** ociuhandu has joined #openstack-infra | 10:32 | |
| *** ociuhandu has quit IRC | 10:33 | |
| *** ociuhandu has joined #openstack-infra | 10:34 | |
| *** ociuhandu has quit IRC | 10:39 | |
| *** ociuhandu has joined #openstack-infra | 10:40 | |
| *** yamamoto has joined #openstack-infra | 10:48 | |
| *** pcaruana has quit IRC | 10:56 | |
| *** yamamoto has quit IRC | 10:59 | |
| *** yamamoto has joined #openstack-infra | 10:59 | |
| *** jpena is now known as jpena|lunch | 11:01 | |
| *** e0ne has joined #openstack-infra | 11:03 | |
| *** panda is now known as panda|lunch | 11:03 | |
| *** Tuor has left #openstack-infra | 11:06 | |
| *** apetrich has quit IRC | 11:07 | |
| *** lucasagomes has joined #openstack-infra | 11:19 | |
| *** ccamacho has quit IRC | 11:21 | |
| openstackgerrit | Jan Kubovy proposed zuul/zuul master: Github connection timeout configuration https://review.opendev.org/683366 | 11:24 |
| *** e0ne has quit IRC | 11:24 | |
| *** igordc has quit IRC | 11:26 | |
| *** ociuhandu has quit IRC | 11:27 | |
| *** jaosorior has joined #openstack-infra | 11:27 | |
| *** pcaruana has joined #openstack-infra | 11:29 | |
| *** rh-jelabarre has joined #openstack-infra | 11:32 | |
| *** chandankumar is now known as raukadah | 11:33 | |
| openstackgerrit | Jean-Philippe Evrard proposed openstack/project-config master: Add tooling to update python jobs on branch creation https://review.opendev.org/673019 | 11:38 |
| *** yamamoto has quit IRC | 11:39 | |
| *** pcaruana has quit IRC | 11:42 | |
| *** pcaruana has joined #openstack-infra | 11:42 | |
| *** dmellado has quit IRC | 11:45 | |
| *** jaosorior has quit IRC | 11:47 | |
| *** dmellado has joined #openstack-infra | 11:48 | |
| *** apetrich has joined #openstack-infra | 11:51 | |
| *** udesale has quit IRC | 11:52 | |
| *** jpena|lunch is now known as jpena | 11:53 | |
| *** udesale has joined #openstack-infra | 11:54 | |
| *** xenos76 has quit IRC | 11:59 | |
| *** xeno_os76_xyz has joined #openstack-infra | 11:59 | |
| *** weshay has joined #openstack-infra | 12:00 | |
| *** ociuhandu has joined #openstack-infra | 12:06 | |
| *** ociuhandu has quit IRC | 12:06 | |
| *** markvoelker has joined #openstack-infra | 12:07 | |
| *** ociuhandu has joined #openstack-infra | 12:07 | |
| *** yamamoto has joined #openstack-infra | 12:12 | |
| *** goldyfruit___ has quit IRC | 12:12 | |
| *** jamesmcarthur has joined #openstack-infra | 12:13 | |
| *** panda|lunch is now known as panda | 12:16 | |
| *** psachin has quit IRC | 12:28 | |
| *** jamesmcarthur has quit IRC | 12:29 | |
| *** jamesmcarthur has joined #openstack-infra | 12:30 | |
| *** jamesmcarthur has quit IRC | 12:35 | |
| *** jaosorior has joined #openstack-infra | 12:38 | |
| *** rlandy has joined #openstack-infra | 12:38 | |
| *** ociuhandu has quit IRC | 12:40 | |
| *** apetrich has quit IRC | 12:42 | |
| *** ociuhandu has joined #openstack-infra | 12:45 | |
| *** ociuhandu has quit IRC | 12:45 | |
| *** ociuhandu has joined #openstack-infra | 12:46 | |
| *** rfolco has joined #openstack-infra | 12:46 | |
| *** fdegir has quit IRC | 12:47 | |
| *** georgk has quit IRC | 12:47 | |
| *** georgk has joined #openstack-infra | 12:47 | |
| *** fdegir has joined #openstack-infra | 12:47 | |
| *** mriedem has joined #openstack-infra | 12:52 | |
| *** dave-mccowan has joined #openstack-infra | 12:54 | |
| *** jamesmcarthur has joined #openstack-infra | 12:55 | |
| *** rcernin has joined #openstack-infra | 12:56 | |
| *** eharney has joined #openstack-infra | 12:56 | |
| *** ociuhandu has quit IRC | 12:57 | |
| *** ociuhandu has joined #openstack-infra | 12:57 | |
| *** jamesmcarthur has quit IRC | 13:02 | |
| *** jamesmcarthur has joined #openstack-infra | 13:03 | |
| *** jamesmcarthur has quit IRC | 13:08 | |
| *** ociuhandu has quit IRC | 13:09 | |
| *** ociuhandu has joined #openstack-infra | 13:13 | |
| *** ociuhandu has quit IRC | 13:16 | |
| *** ociuhandu has joined #openstack-infra | 13:16 | |
| *** jamesmcarthur has joined #openstack-infra | 13:24 | |
| *** goldyfruit___ has joined #openstack-infra | 13:28 | |
| *** gfidente has joined #openstack-infra | 13:31 | |
| *** gfidente has quit IRC | 13:33 | |
| openstackgerrit | David Shrewsbury proposed zuul/zuul master: Don't use urllib3 1.25.4 https://review.opendev.org/683250 | 13:35 |
| *** goldyfruit_ has joined #openstack-infra | 13:35 | |
| *** goldyfruit___ has quit IRC | 13:35 | |
| *** aaronsheffield has joined #openstack-infra | 13:36 | |
| *** goldyfruit___ has joined #openstack-infra | 13:36 | |
| *** apetrich has joined #openstack-infra | 13:39 | |
| *** goldyfruit_ has quit IRC | 13:39 | |
| mriedem | clarkb: do you know if we're still using http://status.openstack.org/elastic-recheck/#1808010 to track failures like this? https://zuul.opendev.org/t/openstack/build/e3f767dbd32247ffa5aa7daa79e0e5af/log/job-output.txt#32748 | 13:40 |
| *** weshay is now known as weshay|ruck | 13:47 | |
| *** sthussey has joined #openstack-infra | 13:49 | |
| *** apetrich has quit IRC | 13:51 | |
| AJaeger | config-core, please review https://review.opendev.org/682544 to finish retiring os-acc | 13:52 |
| AJaeger | thanks, mnaser | 13:53 |
| mnaser | AJaeger: if you have any quick ones i can look htrough feel freee to ping em throughout the day | 13:53 |
| mriedem | clarkb: maybe not b/c the logstash query for that bug is looking for "failed to get http://169.254.169.254/2009-04-04/meta-data" but that doesn't show up in this failure | 13:54 |
| AJaeger | mnaser: https://review.opendev.org/682954 is one | 13:54 |
| mriedem | clarkb: looks like we don't even get to cloud-init trying to hit the metadata api because we fail settin gup the root disk | 13:54 |
| mriedem | in this test, we create a server, attach a volume, stop the server, start the server, and then try to read disks off the guest and that's what blows up | 13:58 |
| mriedem | so maybe "EXT4-fs (vda1): group descriptors corrupted!" is the thing to focus on | 13:59 |
| *** gfidente has joined #openstack-infra | 14:01 | |
| clarkb | ya that sounds like a dofferent issue. In the metadata caseit was falling back from cofnig drive to metadata then failing to talk to the api server aiui | 14:01 |
| mriedem | this is also showing up in logstash on this one change, | 14:03 |
| mriedem | so maybe it's just a super rare failure | 14:03 |
| openstackgerrit | Merged openstack/project-config master: Retiring project os-acc. https://review.opendev.org/682544 | 14:05 |
| openstackgerrit | Merged openstack/project-config master: Set requireContributorAgreement false for windmill roles https://review.opendev.org/682954 | 14:05 |
| *** openstackgerrit has quit IRC | 14:06 | |
| *** rascasoft has quit IRC | 14:10 | |
| *** whoami-rajat has quit IRC | 14:17 | |
| *** iurygregory_ has joined #openstack-infra | 14:21 | |
| *** xeno_os76_xyz has quit IRC | 14:21 | |
| *** iurygregory has quit IRC | 14:21 | |
| *** rcernin has quit IRC | 14:23 | |
| *** openstackgerrit has joined #openstack-infra | 14:26 | |
| openstackgerrit | James E. Blair proposed zuul/zuul master: Fix gerrit errors from production https://review.opendev.org/683006 | 14:26 |
| *** jaosorior has quit IRC | 14:30 | |
| *** dtantsur is now known as dtantsur|afk | 14:33 | |
| mwhahaha | is https://releases.openstack.org/constraints/upper/stable/stein no longer valid? it's currently returnning a 404 for me | 14:34 |
| *** liuyulong has joined #openstack-infra | 14:34 | |
| redrobot | Ohai Infra friends! | 14:34 |
| redrobot | I finally got the F29 gate to work on Barbican master (minus an upstream bug that we're waitin on a fix for) | 14:34 |
| redrobot | I was wondering if I need to backport the fix to stable/XXXXX ? That is to say, are older branches also going to be running on F29 ? | 14:35 |
| *** pgaxatte has quit IRC | 14:36 | |
| *** rcernin has joined #openstack-infra | 14:38 | |
| *** ociuhandu has quit IRC | 14:39 | |
| *** xenos76 has joined #openstack-infra | 14:39 | |
| AJaeger | mwhahaha: remove "upper" | 14:44 |
| mwhahaha | this is a change? | 14:44 |
| mwhahaha | cause this is a stable branch | 14:44 |
| * mwhahaha doesn't want to have to fix all the tox.ini's | 14:44 | |
| AJaeger | wait,remove "stable" -> this works https://releases.openstack.org/constraints/upper/stein | 14:44 |
| *** mattw4 has joined #openstack-infra | 14:45 | |
| AJaeger | and that's what should be in thre | 14:45 |
| AJaeger | there was no change for this | 14:46 |
| *** mattw4 has quit IRC | 14:46 | |
| mwhahaha | hrm | 14:46 |
| *** mattw4 has joined #openstack-infra | 14:46 | |
| AJaeger | mwhahaha: note that in the gate we pass in the file via environment variable, so you might have added something that was ignored in Zuul but only run in local tests | 14:46 |
| mwhahaha | must have been it | 14:46 |
| * AJaeger just check nova's stein branch, it has:-c{env:UPPER_CONSTRAINTS_FILE:https://releases.openstack.org/constraints/upper/stein} | 14:47 | |
| mwhahaha | yea | 14:48 |
| mwhahaha | it's the fact it's not tested in CI | 14:48 |
| *** mattw4 has quit IRC | 14:48 | |
| *** mattw4 has joined #openstack-infra | 14:48 | |
| mwhahaha | thanks | 14:49 |
| *** xenos76 has quit IRC | 14:49 | |
| *** ociuhandu has joined #openstack-infra | 14:49 | |
| *** ociuhandu has quit IRC | 14:49 | |
| *** ociuhandu has joined #openstack-infra | 14:50 | |
| *** udesale has quit IRC | 14:50 | |
| *** udesale has joined #openstack-infra | 14:51 | |
| fungi | right, because in ci we want to be able to substitute gerrit changes which alter that file to support depends-on and proper dependent pipeline gating. but client-side that would be impossibly complicated so the url there is a local-dev-only fallback | 14:51 |
| openstackgerrit | Merged zuul/zuul master: Don't use urllib3 1.25.4 https://review.opendev.org/683250 | 14:52 |
| *** iurygregory_ is now known as iurygregory | 14:53 | |
| mwhahaha | makes sense | 14:53 |
| *** xenos76 has joined #openstack-infra | 15:03 | |
| *** jamesmcarthur has quit IRC | 15:05 | |
| *** bnemec is now known as beekneemech | 15:06 | |
| *** jamesmcarthur has joined #openstack-infra | 15:07 | |
| *** gyee has joined #openstack-infra | 15:07 | |
| *** prometheanfire has quit IRC | 15:11 | |
| *** prometheanfire has joined #openstack-infra | 15:11 | |
| *** trident has quit IRC | 15:14 | |
| *** goldyfruit___ has quit IRC | 15:21 | |
| *** xeno_os76_xyz has joined #openstack-infra | 15:23 | |
| *** mattw4 has quit IRC | 15:24 | |
| *** trident has joined #openstack-infra | 15:24 | |
| *** xenos76 has quit IRC | 15:25 | |
| *** goldyfruit has joined #openstack-infra | 15:26 | |
| *** xeno_os76_xyz has quit IRC | 15:29 | |
| *** trident has quit IRC | 15:29 | |
| *** xeno_os76_xyz has joined #openstack-infra | 15:29 | |
| * mordred is stepping out for a few - will be back in a bit | 15:30 | |
| *** kopecmartin is now known as kopecmartin|off | 15:35 | |
| *** apetrich has joined #openstack-infra | 15:37 | |
| openstackgerrit | James E. Blair proposed zuul/zuul-registry master: Add README and license https://review.opendev.org/683426 | 15:38 |
| *** jamesmcarthur has quit IRC | 15:38 | |
| *** trident has joined #openstack-infra | 15:40 | |
| openstackgerrit | David Shrewsbury proposed zuul/nodepool master: Do not overwrite image upload ZK data on delete https://review.opendev.org/681857 | 15:41 |
| openstackgerrit | David Shrewsbury proposed zuul/nodepool master: Do not overwrite image upload ZK data on delete https://review.opendev.org/681857 | 15:43 |
| *** jamesmcarthur has joined #openstack-infra | 15:43 | |
| *** rcernin has quit IRC | 15:44 | |
| *** jamesmcarthur has quit IRC | 15:45 | |
| *** sgw has quit IRC | 15:49 | |
| openstackgerrit | sean mooney proposed openstack/project-config master: add common nested-virt lables https://review.opendev.org/683431 | 15:49 |
| *** iurygregory has quit IRC | 15:52 | |
| *** jamesmcarthur has joined #openstack-infra | 15:54 | |
| *** rpittau is now known as rpittau|afk | 15:57 | |
| *** elod has quit IRC | 15:58 | |
| openstackgerrit | Merged zuul/zuul master: Store autohold requests in zookeeper https://review.opendev.org/661114 | 15:58 |
| openstackgerrit | Merged zuul/zuul master: Add caching of autohold requests https://review.opendev.org/663412 | 15:58 |
| *** ykarel is now known as ykarel|away | 16:00 | |
| *** sean-k-mooney has joined #openstack-infra | 16:01 | |
| *** cmurphy is now known as cmorpheus | 16:01 | |
| sean-k-mooney | donnyd: mnaser logan- if you have not seen it i just pushed the node pool patch for the nested-virt lables. https://review.opendev.org/#/c/683431/ this is seperate form the multi-numa flavor we talkabout before but is intend to allow testing of cpu pinning/dpdk/hugepages. | 16:03 |
| sean-k-mooney | donnyd: mnaser logan- can ye confirm that it is correct and ack or nack it | 16:03 |
| *** jamesmcarthur_ has joined #openstack-infra | 16:04 | |
| *** yamamoto has quit IRC | 16:05 | |
| donnyd | The label looks to be using the standard flavor so if that is how you want to use it, makes no difference to me. For FN I would have put it in the custom bucket so your jobs could be run in the seperated pool | 16:06 |
| donnyd | But if that's how you want to run it, I am just happy FN can be of use | 16:07 |
| *** jamesmcarthur has quit IRC | 16:07 | |
| sean-k-mooney | donnyd: i want the lable to be the same as the standard one | 16:08 |
| sean-k-mooney | but just select clouds that have nested virt | 16:08 |
| donnyd | Yea that works for me | 16:09 |
| donnyd | :) | 16:09 |
| donnyd | Just happy to help | 16:09 |
| sean-k-mooney | the other numa lable we have then can be in the seperate pool | 16:09 |
| *** signed8bit has quit IRC | 16:09 | |
| sean-k-mooney | but this one does not need to be special | 16:09 |
| *** elod has joined #openstack-infra | 16:13 | |
| *** promethe- has joined #openstack-infra | 16:14 | |
| *** prometheanfire has quit IRC | 16:14 | |
| *** promethe- is now known as prometheanfire | 16:14 | |
| *** yamamoto has joined #openstack-infra | 16:14 | |
| *** yamamoto has quit IRC | 16:14 | |
| *** signed8bit has joined #openstack-infra | 16:15 | |
| *** yamamoto has joined #openstack-infra | 16:15 | |
| *** pcaruana has quit IRC | 16:19 | |
| *** yamamoto has quit IRC | 16:23 | |
| *** signed8b_ has joined #openstack-infra | 16:23 | |
| *** signed8bit has quit IRC | 16:25 | |
| *** jpena is now known as jpena|off | 16:31 | |
| *** sgw has joined #openstack-infra | 16:37 | |
| *** mattw4 has joined #openstack-infra | 16:38 | |
| *** yamamoto has joined #openstack-infra | 16:38 | |
| *** ykarel|away has quit IRC | 16:41 | |
| *** sean-k-mooney has quit IRC | 16:42 | |
| *** derekh has quit IRC | 16:42 | |
| *** ociuhandu has quit IRC | 16:43 | |
| openstackgerrit | Mark Meyer proposed zuul/zuul master: Change reporter pr_version handling https://review.opendev.org/683588 | 16:50 |
| openstackgerrit | James E. Blair proposed zuul/zuul-jobs master: Retry pushes to dockerhub https://review.opendev.org/683589 | 16:51 |
| *** signed8b_ has quit IRC | 16:51 | |
| *** pcrews has quit IRC | 16:52 | |
| openstackgerrit | James E. Blair proposed zuul/zuul-registry master: Add README, license, and Zuul config https://review.opendev.org/683426 | 16:53 |
| *** igordc has joined #openstack-infra | 16:55 | |
| *** ramishra has quit IRC | 16:55 | |
| *** udesale has quit IRC | 16:57 | |
| *** ykarel|away has joined #openstack-infra | 16:58 | |
| openstackgerrit | Paul Belanger proposed zuul/zuul-jobs master: Allow upload-logs to toggle compression of console logs https://review.opendev.org/567696 | 17:00 |
| *** ociuhandu has joined #openstack-infra | 17:00 | |
| *** goldyfruit has quit IRC | 17:01 | |
| *** goldyfruit has joined #openstack-infra | 17:01 | |
| openstackgerrit | Andreas Jaeger proposed zuul/zuul-jobs master: Allow upload-logs to toggle compression of console logs https://review.opendev.org/567696 | 17:04 |
| *** ociuhandu has quit IRC | 17:04 | |
| openstackgerrit | Merged zuul/nodepool master: Reduce upload threads in tests from 4 to 1 https://review.opendev.org/682977 | 17:09 |
| openstackgerrit | Paul Belanger proposed zuul/zuul master: Remove support for ansible 2.5 https://review.opendev.org/650431 | 17:18 |
| openstackgerrit | Paul Belanger proposed zuul/zuul master: Switch ansible_default to 2.8 https://review.opendev.org/676695 | 17:18 |
| openstackgerrit | Paul Belanger proposed zuul/zuul master: WIP: Support Ansible 2.9 https://review.opendev.org/674854 | 17:18 |
| openstackgerrit | Paul Belanger proposed zuul/zuul master: Switch ansible_default to 2.8 https://review.opendev.org/676695 | 17:22 |
| openstackgerrit | Paul Belanger proposed zuul/zuul master: WIP: Support Ansible 2.9 https://review.opendev.org/674854 | 17:22 |
| *** e0ne has joined #openstack-infra | 17:22 | |
| openstackgerrit | Mark Meyer proposed zuul/zuul master: Change reporter pr_version handling https://review.opendev.org/683588 | 17:26 |
| *** goldyfruit has quit IRC | 17:31 | |
| *** roman_g has quit IRC | 17:31 | |
| *** sgw has quit IRC | 17:33 | |
| *** rkukura has joined #openstack-infra | 17:37 | |
| *** sgw has joined #openstack-infra | 17:47 | |
| *** tosky_ has joined #openstack-infra | 17:48 | |
| *** markvoelker has quit IRC | 17:49 | |
| *** pcrews has joined #openstack-infra | 17:49 | |
| *** whoami-rajat has joined #openstack-infra | 17:50 | |
| *** tosky is now known as Guest87259 | 17:51 | |
| *** tosky_ is now known as tosky | 17:51 | |
| *** Guest87259 has quit IRC | 17:52 | |
| *** markvoelker has joined #openstack-infra | 17:52 | |
| openstackgerrit | Jeremy Stanley proposed opendev/system-config master: Clean up logging for abandoned IRC channels https://review.opendev.org/682500 | 17:52 |
| openstackgerrit | Jeremy Stanley proposed opendev/system-config master: Log and report infra status in #openstack-election https://review.opendev.org/682442 | 17:52 |
| AJaeger | fungi: do you include openstack-fwaas, openstack-vpnaas and networking-sfc ? | 17:53 |
| AJaeger | fungi, amotoki proposed to remove those and we merged https://review.opendev.org/#/c/682771/ already | 17:54 |
| AJaeger | fungi: Ah, your commit message is so long that I need to scroll down to read about those three ;) All good | 17:56 |
| fungi | sorry about that! sometimes my commit messages wax poetic | 17:56 |
| AJaeger | I rather have poetic ones than oneliners ;) | 17:57 |
| tosky | it's definitely better when a commit message is detailed | 18:05 |
| *** eharney has quit IRC | 18:05 | |
| *** goldyfruit has joined #openstack-infra | 18:07 | |
| *** Tengu has quit IRC | 18:07 | |
| *** Tengu has joined #openstack-infra | 18:07 | |
| *** goldyfruit_ has joined #openstack-infra | 18:14 | |
| *** goldyfruit has quit IRC | 18:17 | |
| *** igordc has quit IRC | 18:20 | |
| openstackgerrit | Merged zuul/zuul master: Pagure - handle Pull Request tags (labels) metadata https://review.opendev.org/681050 | 18:22 |
| openstackgerrit | Paul Belanger proposed zuul/zuul master: WIP: Support Ansible 2.9 https://review.opendev.org/674854 | 18:25 |
| openstackgerrit | Albin Vass proposed ttygroup/gertty master: urllib 1.25.4 is broken with gerrit https://review.opendev.org/683689 | 18:26 |
| *** igordc has joined #openstack-infra | 18:26 | |
| *** yamamoto has quit IRC | 18:26 | |
| *** snapiri has quit IRC | 18:26 | |
| *** auristor has quit IRC | 18:27 | |
| *** ricolin has quit IRC | 18:27 | |
| *** yamamoto has joined #openstack-infra | 18:27 | |
| *** snapiri has joined #openstack-infra | 18:27 | |
| *** ricolin has joined #openstack-infra | 18:27 | |
| *** adriant has quit IRC | 18:28 | |
| *** adriant has joined #openstack-infra | 18:28 | |
| *** jistr has quit IRC | 18:28 | |
| *** redrobot has quit IRC | 18:28 | |
| *** jistr has joined #openstack-infra | 18:28 | |
| *** auristor has joined #openstack-infra | 18:28 | |
| openstackgerrit | Merged zuul/zuul master: Pagure - reference pipelines add open: True requirement https://review.opendev.org/681252 | 18:29 |
| *** dansmith has quit IRC | 18:29 | |
| *** dansmith has joined #openstack-infra | 18:29 | |
| openstackgerrit | Paul Belanger proposed zuul/zuul master: Remove support for ansible 2.5 https://review.opendev.org/650431 | 18:32 |
| openstackgerrit | Paul Belanger proposed zuul/zuul master: Switch ansible_default to 2.8 https://review.opendev.org/676695 | 18:32 |
| openstackgerrit | Paul Belanger proposed zuul/zuul master: WIP: Support Ansible 2.9 https://review.opendev.org/674854 | 18:32 |
| *** e0ne has quit IRC | 18:34 | |
| *** e0ne has joined #openstack-infra | 18:36 | |
| *** e0ne has quit IRC | 18:36 | |
| *** e0ne has joined #openstack-infra | 18:37 | |
| *** mriedem has quit IRC | 18:40 | |
| *** mriedem has joined #openstack-infra | 18:40 | |
| *** e0ne has quit IRC | 18:41 | |
| weshay|ruck | pabelanger, can you take a peek at https://review.opendev.org/#/c/683240/ when you have a sec please ;) | 18:44 |
| weshay|ruck | Add proxy for quay | 18:44 |
| pabelanger | weshay|ruck: will have to be next week | 18:45 |
| weshay|ruck | k | 18:45 |
| *** eharney has joined #openstack-infra | 18:52 | |
| *** armax has joined #openstack-infra | 18:56 | |
| openstackgerrit | Merged zuul/zuul master: Pagure - handles pull-request.closed event https://review.opendev.org/681279 | 19:00 |
| *** rkukura has quit IRC | 19:01 | |
| openstackgerrit | Merged zuul/zuul-jobs master: Retry pushes to dockerhub https://review.opendev.org/683589 | 19:02 |
| openstackgerrit | Colleen Murphy proposed openstack/project-config master: Use iptables for openSUSE https://review.opendev.org/683236 | 19:06 |
| *** jamesmcarthur_ has quit IRC | 19:10 | |
| *** igordc has quit IRC | 19:13 | |
| gouthamr | TheJulia: fungi was helping clarify the lost logs/builds question on #zuul | 19:16 |
| *** ricolin has quit IRC | 19:16 | |
| gouthamr | TheJulia: apparently, builds get reported only if the buildsets complete, and not if they get dropped | 19:17 |
| *** ykarel|away has quit IRC | 19:17 | |
| fungi | yeah, but the logs are still stored, just without any easy way to find them short of hunting down the log base url in the zuul service logs, which i can do here in a moment given sufficient details on the change, like its url | 19:18 |
| openstackgerrit | Merged zuul/zuul master: Add autohold-info CLI command https://review.opendev.org/662487 | 19:19 |
| *** igordc has joined #openstack-infra | 19:19 | |
| gouthamr | ack ty fungi - i don't have a concrete case at the moment, TheJulia might - i just tried a throwaway change on manila to see if i can see the builds or the logs myself | 19:19 |
| fungi | ahh, okay | 19:19 |
| gouthamr | i know who to ask in case i do though :) | 19:19 |
| *** jamesmcarthur has joined #openstack-infra | 19:20 | |
| fungi | well if there's a particular one you need looked up i should be able to dig it up. we keep roughly a month of logging from the zuul services, it'll just take some time to narrow down | 19:21 |
| *** jamesmcarthur has quit IRC | 19:23 | |
| *** lucasagomes has quit IRC | 19:25 | |
| *** lewo has quit IRC | 19:26 | |
| *** slaweq has quit IRC | 19:32 | |
| openstackgerrit | Merged zuul/zuul master: Record held node IDs with autohold request https://review.opendev.org/662498 | 19:48 |
| *** ociuhandu has joined #openstack-infra | 19:49 | |
| *** whoami-rajat has quit IRC | 20:00 | |
| *** ociuhandu has quit IRC | 20:03 | |
| *** markvoelker has quit IRC | 20:05 | |
| *** ralonsoh has quit IRC | 20:05 | |
| openstackgerrit | Merged zuul/zuul master: Add scheduler config options for hold expiration https://review.opendev.org/682675 | 20:17 |
| *** kjackal has joined #openstack-infra | 20:17 | |
| *** KeithMnemonic1 has quit IRC | 20:23 | |
| pabelanger | fungi: gouthamr: you should be able to find the logs via logstash too. | 20:25 |
| *** jamesmcarthur has joined #openstack-infra | 20:27 | |
| *** jamesmcarthur has quit IRC | 20:27 | |
| fungi | pabelanger: ooh, that's a great point | 20:28 |
| fungi | i didn't even think about that option | 20:28 |
| *** jtomasek has quit IRC | 20:32 | |
| *** redrobot has joined #openstack-infra | 20:34 | |
| openstackgerrit | Merged zuul/zuul master: Mark nodes as USED when deleting autohold https://review.opendev.org/664060 | 20:37 |
| openstackgerrit | Merged zuul/zuul master: Auto-delete expired autohold requests https://review.opendev.org/663762 | 20:59 |
| *** jamesmcarthur has joined #openstack-infra | 21:00 | |
| *** jamesmcarthur has quit IRC | 21:03 | |
| *** rfolco has quit IRC | 21:08 | |
| openstackgerrit | Scott Hussey proposed openstack/project-config master: Add airshipui project https://review.opendev.org/679806 | 21:12 |
| *** gfidente has quit IRC | 21:16 | |
| *** apetrich has quit IRC | 21:19 | |
| *** liuyulong has quit IRC | 21:20 | |
| *** rkukura has joined #openstack-infra | 21:20 | |
| openstackgerrit | Merged zuul/zuul master: Add autohold delete/info commands to web API https://review.opendev.org/679057 | 21:20 |
| *** mriedem is now known as mriedem_away | 21:21 | |
| *** rlandy has quit IRC | 21:25 | |
| *** markvoelker has joined #openstack-infra | 21:32 | |
| *** eharney has quit IRC | 21:39 | |
| openstackgerrit | Paul Belanger proposed zuul/zuul master: WIP: Support Ansible 2.9 https://review.opendev.org/674854 | 21:39 |
| *** markvoelker has quit IRC | 21:42 | |
| *** rh-jelabarre has quit IRC | 21:43 | |
| openstackgerrit | Merged zuul/zuul master: Remove outdated TODO https://review.opendev.org/682421 | 21:43 |
| *** rh-jelabarre has joined #openstack-infra | 21:44 | |
| *** dave-mccowan has quit IRC | 21:55 | |
| *** rkukura has quit IRC | 22:04 | |
| *** kjackal has quit IRC | 22:05 | |
| *** xek has quit IRC | 22:12 | |
| *** rh-jelabarre has quit IRC | 22:22 | |
| openstackgerrit | Merged zuul/zuul master: Fix gerrit errors from production https://review.opendev.org/683006 | 22:24 |
| openstackgerrit | Colleen Murphy proposed openstack/project-config master: Update Grafana dashboard for openSUSE https://review.opendev.org/683738 | 22:28 |
| TheJulia | pabelanger: but people find logstash hard and just posting the links back on a directly aborted run on the change for the already posted logs into swift would be awesome such that if there is something we see in a queue that we know is horribly broken we can kill the job and make it easy to find logs as opposed to somebody just coming along and rechecking the job. | 22:31 |
| TheJulia | fungi: ^^^ | 22:31 |
| TheJulia | the alternative, is to sit and wait for the job to completely fail and timeout in order to preserve the links being posted back | 22:32 |
| fungi | TheJulia: there's been discussion about ways to record the build results as each build completes, but right now the database writes are implemented as a zuul "reporter" so tied to when a buildset is completed | 22:35 |
| fungi | also there are a lot of ways builds get aborted which you wouldn't want reported, for example gate resets, automatic retries due to network connectivity issues, shifts in change dependencies, and so on | 22:36 |
| TheJulia | So maybe a button that can be congizent of gerrit group memberships to imply cores can kill jobs? pretty please with sugar on top? | 22:37 |
| TheJulia | Truthfully, if we could post results back in that case, I would have clicked it at least three times today. | 22:39 |
| TheJulia | But I'm also trying to get broken gate fixed | 22:39 |
| fungi | yeah, something like that may be increasingly possible now that zuul is growing more of a writeable rest api | 22:40 |
| *** goldyfruit_ has quit IRC | 22:42 | |
| fungi | if you're trying to iterate rapidly on a single job, make a dnm change and comment out all the other jobs so only that one runs | 22:42 |
| fungi | if it's the only job, it'll report immediately as it represents the entire buildset | 22:43 |
| fungi | report immediately on failure that is | 22:43 |
| TheJulia | fungi: no, just get stuff out of the queue while preserving the failed job logs for others | 22:44 |
| TheJulia | and by stuff, I mean failed stuff that is pointless to try and continue to try and test | 22:45 |
| TheJulia | if your looking at the openstack tenant console right now, a perfect example would be 680585,1 | 22:47 |
| fungi | clicking on the failed job names right now takes you to their logs | 23:01 |
| fungi | you don't get the fancy html-ized rendering, but the raw logs are all there still | 23:02 |
| * TheJulia sighs | 23:02 | |
| TheJulia | we're on two different pages | 23:02 |
| fungi | likely | 23:03 |
| fungi | sounded like you were trying to investigate job failures, but i probably misunderstood | 23:03 |
| TheJulia | I'm just wanting easy access for other people to be able to investigate their changes's failures should we have a way to abort the running job | 23:06 |
| TheJulia | And it would be really handy to be able to kill running jobs when there are broken gate situations because job a should work, it may have failed, job b could be known broken and have failed and as such there is no real reason to let that to continue to run except maybe to ensure logs are easily accessible | 23:09 |
| fungi | yep, i get that the current procrastinated feature freeze rush coupled with countless bugs throughout openstack are leading to a lot of frustration on the parts of many folks involved. if it helps, panicked developers trying to explain why the ci system isn't meeting their needs in such pathological scenarios gets really demotivating for the people trying to keep that system running | 23:12 |
| *** notmyname has quit IRC | 23:12 | |
| fungi | we've tried to design to maximize throughput for projects whose jobs pass fairly consistently, with the expectation that failures are an exceptional condition | 23:13 |
| *** rledisez has quit IRC | 23:15 | |
| fungi | zuul wants to complete all builds and report them together, to improve efficiency and not waste all running builds when one hits a failure condition | 23:15 |
| fungi | as each build has an opportunity to expose different failure conditions in a change | 23:16 |
| fungi | we're trying to assemble an faq document about such topics here https://docs.openstack.org/infra/manual/testing.html#why-are-jobs-for-changes-queued-for-a-long-time | 23:17 |
| TheJulia | I get that, I really do which is partially why I'm trying to be really mellow and told people to stop rechecking on ironic. I also recognize it is not the maintainers fault that for some reason, the physnet mtu getting assigned on neutron is now fifty bytes smaller on fortnebula which basically tanked any jobs running on it in virtualized networking. | 23:17 |
| *** mgoddard has quit IRC | 23:17 | |
| fungi | yeah, i believe that's because the ipv6 connection there (which is the only globally routable addressing those nodes have) is provided over gre to a tunnel broker | 23:19 |
| clarkb | note we specifically warn for that here https://docs.openstack.org/infra/manual/testing.html#known-differences-to-watch-out-for | 23:19 |
| TheJulia | That being said, the effort to report together ignores the human state knowledge that can exist, and your right on the previous point, its not possible now but maybe later to have a capability to feedback that human state knowledge to help save resources and time from being wasted | 23:19 |
| *** mgoddard has joined #openstack-infra | 23:19 | |
| clarkb | this isnt tye first cloud to need to do that | 23:19 |
| TheJulia | fungi: that would explain it.... oh wow | 23:19 |
| * TheJulia checks the other window to see if happiness has resumed | 23:20 | |
| TheJulia | clarkb: I was thinking the same thing when I was looking at it late last night | 23:20 |
| TheJulia | fungi: is the gre connection brought up as part of the job somepalce? | 23:21 |
| TheJulia | someplace | 23:21 |
| fungi | fortnebula could probably get by with a 1500-byte mtu on the nodes and then rely on pmtud or, worst case, fragmentation and reassembly | 23:21 |
| fungi | TheJulia: no, it's on the other side of the router | 23:21 |
| fungi | which is why theoretically pmtud or fragmentation would be viable | 23:22 |
| clarkb | https://opendev.org/zuul/zuul-jobs/src/branch/master/roles/multi-node-bridge/tasks/common.yaml#L78 is how we lookup the mtu in jobs using the standard multinode bridge role | 23:22 |
| TheJulia | okay, weird... pmtu likely would have kept things working. Ended up being the kernel was dropping packets on the floor because neutron was picking up the 1350 byte physnet mtu and running with it | 23:22 |
| fungi | but regardless, we do warn that the details of the local devices on the test nodes can vary, and jobs should try to test with abstracted devices if they require such low-level interactions | 23:23 |
| TheJulia | clarkb: nice! | 23:23 |
| fungi | i wonder if devstack is hard-coding an mtu somewhere | 23:24 |
| TheJulia | fungi: yeah, except neutron picking up the base physnet mtu is where things went sideways | 23:24 |
| TheJulia | it is not, we've long had a hardcoded override mtu and even calculation code, just neither were low enough for the physical broadcast/interface MTU | 23:25 |
| fungi | i thought we had worked out a solution to make the neutron network mtus dynamically sized based on what the node interfaces have, but that may have been in devstack-gate so i have no idea if it's still the case now | 23:25 |
| * TheJulia is going to rip the override out likely tomorrow, but set it just to get CI un-bricked | 23:25 | |
| TheJulia | (this will include... hopefully... fixing ironic's mtu detection code to handle or at least exist somewhat happily in fortnebula | 23:26 |
| TheJulia | ) | 23:26 |
| TheJulia | fungi: I didn't see it when I was looking nor in the logs. I looked at the base neutron code and it was dynamically setting it based on the public physical network | 23:26 |
| fungi | ahh, got it, so neutron is dynamically sizing its mtus based on the node's interface mtus, but then ironic is making flat assumptions about what those will be? | 23:27 |
| *** tosky has quit IRC | 23:28 | |
| TheJulia | well, neutron was also creating all ports with that dynamic mtu, and we've long had a static number to drop that mtu down to handle cases along these lines, just even if we didn't our autodetection code for handling it when we're building the simulated networking would have never spotted it nor is it even aware of the networking neutron is using since they are build/used independently of each other until at | 23:29 |
| TheJulia | some point when they are connected for a flat network or networking-generic-switch does it as part of a port binding action | 23:29 |
| TheJulia | At which point, linux is evil and unlike a switch which by spec for ethernet is supposed to still take in a 1500 byte packet | 23:30 |
| TheJulia | linux drops it on the floor and *kaboom* | 23:30 |
| *** xeno_os76_xyz has quit IRC | 23:33 | |
| TheJulia | I guess I really should have just complained about the issues we were encountering.... | 23:36 |
| * TheJulia shrugs | 23:36 | |
| fungi | yeah, mtu is the maximum *transmission* unit, if linux also treats it as a maximum *reception* unit then that's questionable behavior | 23:36 |
| TheJulia | sadly it does :( | 23:38 |
| TheJulia | at least on tap interfaces, but taps and vnets have slightly different operating rules | 23:39 |
| fungi | anyway, to summarize, there are discussions ongoing about moving build result recording to happen at build completion instead of buildset completion, no real design work has been done on it that i know of, but there are other benefits we'd get from doing that so i have a feeling it will eventually happen. the writeable rest api for zuul is also coming along, very recently authentication got added | 23:40 |
| fungi | which was the biggest prerequisite, and that may eventually give us more flexibility for things like dequeuing and rechecking | 23:40 |
| TheJulia | fungi: that is awesome news! thanks! | 23:43 |
| fungi | also "fail-fast" mode comes up fairly regularly. people seem to want it when things are going sideways, but then prefer not to have it when everything's roses, so it's one of those discussions we get to have over and over | 23:43 |
| *** markvoelker has joined #openstack-infra | 23:43 | |
| TheJulia | yeah.... I've been on the fence about fail-fast before personally. There are good times for it, and bad times for it. :( | 23:43 |
| TheJulia | Anyway, fix merged finally, world is a happier place. I'm going to go find some dinner! Have an awesome evening fungi! | 23:44 |
| fungi | enjoy your evening | 23:44 |
| fungi | thanks! | 23:44 |
| *** sthussey has quit IRC | 23:47 | |
| *** markvoelker has quit IRC | 23:47 | |
| *** mattw4 has quit IRC | 23:48 | |
| *** rfolco has joined #openstack-infra | 23:57 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!