| *** signed8b_ has joined #openstack-infra | 00:23 | |
| *** signed8bit has quit IRC | 00:26 | |
| *** jamesmcarthur has joined #openstack-infra | 01:11 | |
| *** jamesmcarthur has quit IRC | 01:14 | |
| *** jamesmcarthur has joined #openstack-infra | 01:21 | |
| *** hongbin has joined #openstack-infra | 01:39 | |
| *** diablo_rojo has quit IRC | 01:43 | |
| *** hongbin_ has joined #openstack-infra | 01:44 | |
| *** hongbin has quit IRC | 01:46 | |
| ianw | rm_work: you may need devstack_services for octavia? but it looks in the ballpark to me | 02:11 |
|---|---|---|
| *** yamamoto has joined #openstack-infra | 02:16 | |
| openstackgerrit | Ian Wienand proposed zuul/zuul-jobs master: Remove invalid argument from format_exc() calls https://review.opendev.org/684167 | 02:17 |
| *** weshay|rover has quit IRC | 02:20 | |
| *** yamamoto has quit IRC | 02:21 | |
| *** jamesmcarthur has quit IRC | 02:22 | |
| *** jamesmcarthur has joined #openstack-infra | 02:23 | |
| *** jamesmcarthur_ has joined #openstack-infra | 02:27 | |
| *** jamesmcarthur_ has quit IRC | 02:27 | |
| *** rfolco has quit IRC | 02:27 | |
| *** jamesmcarthur has quit IRC | 02:28 | |
| *** larainema has joined #openstack-infra | 02:28 | |
| *** rlandy has quit IRC | 02:33 | |
| *** roman_g has quit IRC | 02:34 | |
| rm_work | ianw: hmm k, but then probably also for designate, right? I guess i was hoping since the plugin does all its own enable_services stuff that it'd be fine | 02:38 |
| rm_work | though right now the weirdest bit is that the config directory isn't getting created and it's exploding on the custom config bit | 02:39 |
| rm_work | which works fine when i do my own devstack | 02:39 |
| ianw | oh, yeah, if the plugin is doing it i think you're right | 02:39 |
| rm_work | but here it says there's no /etc/octavia/octavia.conf when it tries to do post-config | 02:39 |
| rm_work | haven't had that issue before | 02:40 |
| ianw | hrm, permissions maybe? | 02:40 |
| rm_work | hmm could be i guess | 02:40 |
| rm_work | gonna try my own devstack again and see if something changed | 02:41 |
| *** markvoelker has joined #openstack-infra | 02:41 | |
| openstackgerrit | Merged zuul/zuul-jobs master: Add more languages for build-releasenotes https://review.opendev.org/683923 | 02:43 |
| *** ricolin has joined #openstack-infra | 02:48 | |
| *** yamamoto has joined #openstack-infra | 03:27 | |
| *** hongbin has joined #openstack-infra | 03:28 | |
| *** hongbin_ has quit IRC | 03:30 | |
| *** ykarel has joined #openstack-infra | 03:32 | |
| *** hongbin has quit IRC | 03:38 | |
| *** hongbin has joined #openstack-infra | 03:39 | |
| *** hongbin has quit IRC | 03:48 | |
| *** hongbin_ has joined #openstack-infra | 03:49 | |
| *** hongbin_ has quit IRC | 03:49 | |
| *** jamesdenton has quit IRC | 04:07 | |
| *** eharney has joined #openstack-infra | 04:21 | |
| *** ykarel has quit IRC | 04:24 | |
| *** whoami-rajat has joined #openstack-infra | 04:33 | |
| *** eharney has quit IRC | 04:33 | |
| *** jamesden_ has joined #openstack-infra | 04:42 | |
| *** ykarel has joined #openstack-infra | 04:55 | |
| *** udesale has joined #openstack-infra | 05:00 | |
| *** bnemec has quit IRC | 05:09 | |
| *** igordc has quit IRC | 05:12 | |
| *** markvoelker has quit IRC | 05:14 | |
| *** bnemec has joined #openstack-infra | 05:15 | |
| *** hwoarang has quit IRC | 05:29 | |
| *** jtomasek has joined #openstack-infra | 05:30 | |
| *** hwoarang has joined #openstack-infra | 05:32 | |
| *** exsdev0 has joined #openstack-infra | 05:41 | |
| *** exsdev has quit IRC | 05:41 | |
| *** exsdev0 is now known as exsdev | 05:41 | |
| *** dpawlik has joined #openstack-infra | 05:44 | |
| *** exsdev0 has joined #openstack-infra | 05:45 | |
| *** exsdev has quit IRC | 05:46 | |
| *** exsdev0 is now known as exsdev | 05:46 | |
| *** yamamoto has quit IRC | 05:48 | |
| AJaeger | ianw: could you look at https://review.opendev.org/683236 and https://review.opendev.org/683738 , please? A dib change for openSUSE and one for grafana | 05:56 |
| openstackgerrit | Jan Kubovy proposed zuul/zuul master: Report retried builds in a build set via mqtt. https://review.opendev.org/632727 | 05:59 |
| openstackgerrit | Jan Kubovy proposed zuul/zuul master: Report retried builds via sql reporter. https://review.opendev.org/633501 | 05:59 |
| *** kopecmartin|off is now known as kopecmartin | 06:05 | |
| *** slaweq has joined #openstack-infra | 06:09 | |
| openstackgerrit | Merged openstack/project-config master: Update Grafana dashboard for openSUSE https://review.opendev.org/683738 | 06:18 |
| openstackgerrit | Ian Wienand proposed opendev/infra-specs master: Spec to retire static.openstack.org https://review.opendev.org/683852 | 06:21 |
| *** kjackal has joined #openstack-infra | 06:21 | |
| *** pgaxatte has joined #openstack-infra | 06:31 | |
| *** yolanda has quit IRC | 06:34 | |
| *** rpittau|afk is now known as rpittau | 06:43 | |
| *** rosmaita has joined #openstack-infra | 06:46 | |
| *** lpetrut has joined #openstack-infra | 06:47 | |
| *** eharney has joined #openstack-infra | 06:52 | |
| *** tosky has joined #openstack-infra | 06:53 | |
| *** pcaruana has joined #openstack-infra | 06:56 | |
| *** gfidente has joined #openstack-infra | 06:59 | |
| *** e0ne has joined #openstack-infra | 07:04 | |
| *** donnyd has quit IRC | 07:07 | |
| *** tesseract has joined #openstack-infra | 07:07 | |
| *** xenos76 has joined #openstack-infra | 07:08 | |
| *** donnyd has joined #openstack-infra | 07:08 | |
| *** e0ne has quit IRC | 07:09 | |
| *** soniya29 has joined #openstack-infra | 07:12 | |
| *** markvoelker has joined #openstack-infra | 07:15 | |
| *** xek has joined #openstack-infra | 07:16 | |
| rm_work | Yeah weird, my devstack (with what I assume would be the exact same config) works fine... Very odd. Not sure how that directory isn't getting created. | 07:17 |
| *** markvoelker has quit IRC | 07:19 | |
| rm_work | I should be able to just use https://zuul.opendev.org/t/openstack/build/88b40f4a0b6c432fa8c944a77bfb7b2c/log/controller/logs/_.localrc_auto.txt.gz and it should exactly mirror what zuul does? | 07:20 |
| rm_work | Though designate shows up in https://zuul.opendev.org/t/openstack/build/88b40f4a0b6c432fa8c944a77bfb7b2c/log/controller/logs/__stackenv.txt.gz while Octavia does not... | 07:22 |
| rm_work | Err sorry rather, I'd stack with this to replicate: https://zuul.opendev.org/t/openstack/build/88b40f4a0b6c432fa8c944a77bfb7b2c/log/controller/logs/local_conf.txt.gz | 07:24 |
| *** tobiash has quit IRC | 07:27 | |
| rm_work | Yeah this looks fine, can't imagine why the Octavia config directory doesn't get created in time for the config merge | 07:29 |
| rm_work | The rest of them do... | 07:29 |
| rm_work | Even designate config files show up, and that's also added via this same method :/ | 07:30 |
| *** tobiash has joined #openstack-infra | 07:31 | |
| *** ralonsoh has joined #openstack-infra | 07:34 | |
| *** auristor has quit IRC | 07:38 | |
| *** gfidente has quit IRC | 07:46 | |
| *** iurygregory has joined #openstack-infra | 07:47 | |
| openstackgerrit | pengyuesheng proposed openstack/diskimage-builder master: Update the constraints url https://review.opendev.org/684216 | 07:53 |
| *** roman_g has joined #openstack-infra | 07:56 | |
| *** priteau has joined #openstack-infra | 07:58 | |
| *** psachin has joined #openstack-infra | 07:59 | |
| openstackgerrit | Tobias Henkel proposed zuul/zuul master: Evaluate CODEOWNERS settings during canMerge check https://review.opendev.org/644557 | 08:01 |
| frickler | rm_work: commented on the review, the octavia devstack plugin seems to be ... special | 08:03 |
| *** dchen has quit IRC | 08:03 | |
| *** ykarel is now known as ykarel|lunch | 08:08 | |
| *** apetrich has quit IRC | 08:12 | |
| *** tkajinam has quit IRC | 08:14 | |
| *** apetrich has joined #openstack-infra | 08:22 | |
| yoctozepto | hey infra, gerrit links in http://zuul.openstack.org/status are broken (point to internal paths) | 08:27 |
| *** derekh has joined #openstack-infra | 08:28 | |
| openstackgerrit | Merged zuul/zuul-jobs master: Remove invalid argument from format_exc() calls https://review.opendev.org/684167 | 08:29 |
| *** jpena|off is now known as jpena | 08:30 | |
| *** jbadiapa has joined #openstack-infra | 08:35 | |
| jrosser | is there any known reason for random looking POST_FAILURE atm? I have a few turning up | 08:36 |
| jrosser | like here https://zuul.opendev.org/t/openstack/build/f9801b0d8bbf43c6ab4c35526ecd655b | 08:37 |
| *** jbadiapa has quit IRC | 08:37 | |
| *** jbadiapa has joined #openstack-infra | 08:38 | |
| *** kjackal_v2 has joined #openstack-infra | 08:41 | |
| *** kjackal has quit IRC | 08:41 | |
| *** jbadiapa has quit IRC | 08:42 | |
| *** freerunner has quit IRC | 08:43 | |
| *** amorin has quit IRC | 08:44 | |
| *** dirk has quit IRC | 08:44 | |
| *** stephenfin has quit IRC | 08:44 | |
| *** dosaboy has quit IRC | 08:45 | |
| *** frickler has quit IRC | 08:45 | |
| *** olaph has quit IRC | 08:45 | |
| *** freerunner has joined #openstack-infra | 08:45 | |
| *** stephenfin has joined #openstack-infra | 08:45 | |
| *** dirk has joined #openstack-infra | 08:45 | |
| *** dosaboy has joined #openstack-infra | 08:45 | |
| *** amorin has joined #openstack-infra | 08:46 | |
| *** frickler has joined #openstack-infra | 08:46 | |
| *** ykarel|lunch is now known as ykarel | 08:46 | |
| *** prometheanfire has quit IRC | 08:50 | |
| *** prometheanfire has joined #openstack-infra | 08:50 | |
| *** ccamacho has joined #openstack-infra | 09:01 | |
| *** rcernin has quit IRC | 09:04 | |
| *** ricolin_ has joined #openstack-infra | 09:05 | |
| *** xenos76 has quit IRC | 09:08 | |
| *** ricolin has quit IRC | 09:08 | |
| openstackgerrit | Merged opendev/system-config master: Revert to rit.edu mirror for Suse https://review.opendev.org/683978 | 09:10 |
| openstackgerrit | Merged opendev/system-config master: Clean up logging for abandoned IRC channels https://review.opendev.org/682500 | 09:10 |
| openstackgerrit | Merged opendev/system-config master: Log and report infra status in #openstack-election https://review.opendev.org/682442 | 09:10 |
| *** kjackal_v2 has quit IRC | 09:11 | |
| *** kjackal has joined #openstack-infra | 09:18 | |
| *** ricolin_ is now known as ricolin | 09:23 | |
| *** ociuhandu has joined #openstack-infra | 09:30 | |
| *** e0ne has joined #openstack-infra | 09:54 | |
| *** e0ne has quit IRC | 09:55 | |
| *** e0ne has joined #openstack-infra | 09:55 | |
| *** openstackstatus has quit IRC | 10:12 | |
| *** openstack has joined #openstack-infra | 10:14 | |
| *** ChanServ sets mode: +o openstack | 10:14 | |
| *** dtantsur|afk is now known as dtantsur | 10:17 | |
| *** pcaruana has quit IRC | 10:24 | |
| *** itxaka has joined #openstack-infra | 10:26 | |
| *** markvoelker has joined #openstack-infra | 10:40 | |
| *** pcaruana has joined #openstack-infra | 10:48 | |
| *** weshay has joined #openstack-infra | 10:50 | |
| *** ociuhandu has quit IRC | 11:00 | |
| *** ociuhandu has joined #openstack-infra | 11:01 | |
| *** iurygregory_ has joined #openstack-infra | 11:03 | |
| *** iurygregory has quit IRC | 11:05 | |
| *** xenos76 has joined #openstack-infra | 11:07 | |
| *** udesale has quit IRC | 11:08 | |
| *** udesale has joined #openstack-infra | 11:09 | |
| *** ociuhandu has quit IRC | 11:09 | |
| *** ociuhandu has joined #openstack-infra | 11:09 | |
| donnyd | jrosser: those are usually from failures to upload the logs to swift | 11:14 |
| donnyd | But could also be something else | 11:14 |
| jrosser | indeed - just trying to keep the jobs passing :) | 11:14 |
| *** lucasagomes has joined #openstack-infra | 11:21 | |
| *** jpena is now known as jpena|lunch | 11:34 | |
| *** jamesmcarthur has joined #openstack-infra | 11:39 | |
| *** noama has joined #openstack-infra | 11:45 | |
| *** zhurong has joined #openstack-infra | 11:48 | |
| *** jamesmcarthur has quit IRC | 11:49 | |
| *** rh-jelabarre has joined #openstack-infra | 11:54 | |
| *** fdegir has quit IRC | 11:57 | |
| *** fdegir has joined #openstack-infra | 11:58 | |
| *** roman_g has quit IRC | 11:59 | |
| *** roman_g has joined #openstack-infra | 11:59 | |
| *** derekh has quit IRC | 12:00 | |
| *** jamesmcarthur has joined #openstack-infra | 12:02 | |
| *** signed8b_ has quit IRC | 12:02 | |
| *** jamesmcarthur_ has joined #openstack-infra | 12:03 | |
| *** larainema has quit IRC | 12:03 | |
| *** signed8bit has joined #openstack-infra | 12:03 | |
| *** jamesmcarthur has quit IRC | 12:07 | |
| *** rfolco has joined #openstack-infra | 12:10 | |
| openstackgerrit | Merged zuul/zuul master: Unify Github/Github Enterpise read timeout https://review.opendev.org/683366 | 12:13 |
| *** goldyfruit has quit IRC | 12:13 | |
| clarkb | yoctozepto: that is an issue related to updating zuul to support the gerrit checks api. corvus waslooking at it | 12:15 |
| clarkb | jrosser: donnyd ya no logs means swift either failed or couldnt run typically I'll check executor logs after breakfast | 12:16 |
| clarkb | keep in mind if the hostcrashes or breaks its own networking that can happen (not sure what that change is updating) | 12:17 |
| *** jamesmcarthur_ has quit IRC | 12:20 | |
| *** aaronsheffield has joined #openstack-infra | 12:23 | |
| *** hashar has joined #openstack-infra | 12:31 | |
| clarkb | jrosser: http://paste.openstack.org/show/779168/ it looks like either swift uploads were taking a significant amount of time (> half an hour maybe due to large file count or large files?) or the host's networking stopped working around the time it tried to upload logs. Considering that the cleanup phase failed to ssh into the host I think the host likely went away at around the time of uploads | 12:33 |
| clarkb | that job ran on ze10 for any other infra-roots wanting to look at logs | 12:33 |
| *** jamesden_ has quit IRC | 12:34 | |
| *** jamesden_ has joined #openstack-infra | 12:34 | |
| *** jpena|lunch is now known as jpena | 12:35 | |
| *** rlandy has joined #openstack-infra | 12:39 | |
| clarkb | that job ran on fortnebula | 12:41 |
| *** jamesden_ is now known as jamesdenton | 12:44 | |
| jrosser | clarkb: reason i asked is that failed job was a depends-on for this one https://review.opendev.org/#/c/681003/ | 12:45 |
| jrosser | and that also failed with a similar POST_FAILURE but for a different test | 12:46 |
| *** gfidente has joined #openstack-infra | 12:47 | |
| jrosser | clarkb: actually for any of these jobs would you expect the log output to just get truncated midway through the log upload? https://zuul.opendev.org/t/openstack/build/bd8c8e7721df433ea3a13f752d385523/log/job-output.txt#15381 | 12:48 |
| *** psachin has quit IRC | 12:49 | |
| clarkb | jrosser: we cant actually log the upload to swift becausethat happens after uploading to swift | 12:57 |
| *** jaosorior has quit IRC | 12:59 | |
| openstackgerrit | Carlos Goncalves proposed openstack/diskimage-builder master: WIP: CentOS 8 support https://review.opendev.org/684308 | 13:02 |
| *** ociuhandu has quit IRC | 13:03 | |
| donnyd | clarkb: do we know which swift was in the rotation? | 13:06 |
| *** jcoufal has joined #openstack-infra | 13:07 | |
| donnyd | clarkb: maybe I should lower to the load FN, I don't recall as many issues when the load was a little lower | 13:07 |
| clarkb | donnyd: I dont know. I can look abit later to see if welogged that | 13:07 |
| donnyd | I know we were having swift issues with OVH, but in the last week or so I had a large uptick in FN related issues | 13:07 |
| donnyd | so maybe I will turn it back down to 60 or so nodes | 13:08 |
| openstackgerrit | Donny Davis proposed openstack/project-config master: decrease FN test nodes to 65 https://review.opendev.org/684312 | 13:10 |
| *** iurygregory_ is now known as iurygregory | 13:11 | |
| donnyd | at 65 test nodes the jobs will not be overcommitted at all and they can all run at 100% without being a noisy neighbor | 13:14 |
| *** liuyulong has joined #openstack-infra | 13:17 | |
| *** psachin has joined #openstack-infra | 13:19 | |
| *** sthussey has joined #openstack-infra | 13:22 | |
| *** mriedem has joined #openstack-infra | 13:24 | |
| *** hashar has quit IRC | 13:33 | |
| *** dpawlik has quit IRC | 13:34 | |
| *** ociuhandu has joined #openstack-infra | 13:36 | |
| *** goldyfruit has joined #openstack-infra | 13:36 | |
| *** ociuhandu has quit IRC | 13:36 | |
| *** ociuhandu has joined #openstack-infra | 13:37 | |
| *** soniya29 has quit IRC | 13:38 | |
| openstackgerrit | Tobias Henkel proposed zuul/zuul master: Support emitting warnings via zuul_return https://review.opendev.org/651526 | 13:38 |
| openstackgerrit | Tobias Henkel proposed zuul/zuul master: Support emitting warnings via zuul_return https://review.opendev.org/651526 | 13:40 |
| *** ociuhandu has quit IRC | 13:40 | |
| *** ociuhandu has joined #openstack-infra | 13:41 | |
| *** lpetrut has quit IRC | 13:41 | |
| openstackgerrit | Paul Belanger proposed zuul/zuul-jobs master: DNM - ansiblefest 2019 demo! https://review.opendev.org/684322 | 13:49 |
| roman_g | mordred: listening to the recording of a Python podcast with you. Enjoying it a lot. Thank you. | 13:51 |
| roman_g | mordred: podcast about Zuul | 13:51 |
| *** hashar has joined #openstack-infra | 13:51 | |
| fungi | jrosser: clarkb: donnyd: that build ran from ze10... looks like the executor spontaneously lost contact with that remote node? http://paste.openstack.org/show/779181/ | 13:52 |
| clarkb | fungi yup and earlier in the job the post run playbook timed out | 13:53 |
| *** ociuhandu has quit IRC | 13:53 | |
| openstackgerrit | Carlos Goncalves proposed openstack/diskimage-builder master: WIP: CentOS 8 support https://review.opendev.org/684308 | 13:55 |
| fungi | ahh, yep, http://paste.openstack.org/show/779182/ | 14:00 |
| *** ociuhandu has joined #openstack-infra | 14:05 | |
| *** whoami-rajat has quit IRC | 14:06 | |
| fungi | do we not actually log the swift api endpoint which gets picked? | 14:07 |
| fungi | or am i just overlooking it? | 14:07 |
| clarkb | fungi: we do when it breaks but a timeout isnt a break in the upload script | 14:07 |
| fungi | i guess we're letting ansible do the random choice there still, right? so not something the executor code can just echo | 14:08 |
| clarkb | ya | 14:08 |
| *** bdodd has joined #openstack-infra | 14:09 | |
| *** lennyb has quit IRC | 14:11 | |
| jrosser | these are jobs with embedded ARA so there are many many files | 14:11 |
| dmsimard | :( | 14:14 |
| fungi | i'm wondering if the timeout is just too short for jobs which upload a massive number of files | 14:16 |
| *** whoami-rajat has joined #openstack-infra | 14:17 | |
| *** liuyulong has quit IRC | 14:22 | |
| *** JorgeFranco has joined #openstack-infra | 14:22 | |
| clarkb | that doesnt explain the ssh issue though | 14:22 |
| *** lennyb has joined #openstack-infra | 14:23 | |
| *** hashar has quit IRC | 14:31 | |
| donnyd | well if the job tinkers with the network we had that issue frickler was fixing with ipv6 and FN https://review.opendev.org/#/c/683166/ | 14:32 |
| donnyd | we were seeing the same thing where it would drop ssh inbound because when it creates the public ipv6 side of the network it drops the def route. In other clouds that doesn't really matter.. .but in FN it does | 14:33 |
| *** petevg has joined #openstack-infra | 14:36 | |
| *** pgaxatte has quit IRC | 14:38 | |
| openstackgerrit | David Shrewsbury proposed zuul/nodepool master: Do not overwrite image upload ZK data on delete https://review.opendev.org/681857 | 14:39 |
| *** kopecmartin is now known as kopecmartin|off | 14:40 | |
| openstackgerrit | David Shrewsbury proposed zuul/nodepool master: Do not overwrite image upload ZK data on delete https://review.opendev.org/681857 | 14:43 |
| openstackgerrit | Merged openstack/project-config master: decrease FN test nodes to 65 https://review.opendev.org/684312 | 14:43 |
| openstackgerrit | Carlos Goncalves proposed openstack/diskimage-builder master: WIP: CentOS 8 support https://review.opendev.org/684308 | 14:58 |
| *** eharney has quit IRC | 14:58 | |
| *** tosky has quit IRC | 15:00 | |
| *** rosmaita has quit IRC | 15:01 | |
| *** pcaruana has quit IRC | 15:01 | |
| *** ccamacho has quit IRC | 15:05 | |
| *** gfidente has quit IRC | 15:05 | |
| openstackgerrit | Tobias Henkel proposed zuul/zuul master: Annotate cancelJobInQueue with build id https://review.opendev.org/684373 | 15:09 |
| openstackgerrit | Kostyantyn Kalynovskyi proposed openstack/project-config master: Add airship/apis project https://review.opendev.org/684374 | 15:09 |
| *** auristor has joined #openstack-infra | 15:10 | |
| *** hashar has joined #openstack-infra | 15:10 | |
| *** xenos76 has quit IRC | 15:19 | |
| donnyd | I'm thinking if the reduction doesn't help, then something else is wrong jrosser | 15:23 |
| donnyd | clarkb: ^^^ | 15:24 |
| jrosser | donnyd: are those all failing on FN? | 15:25 |
| *** lennyb has quit IRC | 15:25 | |
| donnyd | no | 15:26 |
| clarkb | fungi and I are going to test a thing | 15:27 |
| donnyd | The failures on FN seem to load based as they aren't always consistent. There are a couple of failure that are directly related... like when the actual def route for v6 is purged | 15:27 |
| donnyd | jrosser: if you look at the load here https://grafana.fortnebula.com/d/9MMqh8HWk/openstack-utilization?orgId=2&refresh=30s&from=now-24h&to=now | 15:29 |
| donnyd | There are some coincidences that the load of FN sometimes correlates to failing jobs | 15:30 |
| donnyd | so the reduction should help with that | 15:30 |
| *** lennyb has joined #openstack-infra | 15:30 | |
| *** iurygregory has quit IRC | 15:32 | |
| donnyd | For the most the load is well under the capacity FN can handle.. but the procs are older so I think maybe keeping it to around 1:1 ratios ensures the jobs are more reliable | 15:33 |
| *** diablo_rojo has joined #openstack-infra | 15:33 | |
| donnyd | I would think that reliability in job success is more important that # of jobs per hour | 15:33 |
| donnyd | jrosser: does that job tinker with the network? IE does it create a ipv6 public side network? | 15:35 |
| jrosser | OSA hasnt done anything specific at all for ipv6 yet | 15:35 |
| jrosser | it does fiddle with the network at the start quite a lot, but that doesnt seem troublesome | 15:35 |
| donnyd | well troublesome on other non ipv6 providers.. or not troublesome | 15:36 |
| donnyd | and has this job passed on FN before or does it fail every time? | 15:36 |
| jrosser | donnyd: well i had two very similar things https://review.opendev.org/#/c/684125/ and https://review.opendev.org/#/c/681003/ | 15:38 |
| jrosser | each of which runs ~10 jobs which in themselves are similar | 15:39 |
| jrosser | and two failed, seemingly randomly during log upload | 15:39 |
| donnyd | Looks to me like its about a 50/50 shot | 15:40 |
| donnyd | node_provider:"fortnebula-regionone" AND filename:"job-output.txt" AND build_name:"openstack-ansible-deploy-aio_lxc-ubuntu-bionic" AND message:"Upload logs to swift" | 15:40 |
| donnyd | if you look in logstash I see about a half pass rate | 15:40 |
| donnyd | https://openstack.fortnebula.com:13808/v1/AUTH_e8fd161dc34c421a979a9e6421f823e9/zuul_opendev_logs_b98/periodic/opendev.org/openstack/openstack-ansible/stable/stein/openstack-ansible-deploy-aio_lxc-ubuntu-bionic/b98e926/job-output.txt | 15:41 |
| donnyd | That one passes | 15:41 |
| donnyd | passed | 15:41 |
| donnyd | but my query could be off | 15:41 |
| *** rosmaita has joined #openstack-infra | 15:43 | |
| donnyd | The POST_FAILURE could be for a couple reasons... but its unlikely to be related specifically to FN | 15:43 |
| *** gyee has joined #openstack-infra | 15:44 | |
| *** goldyfruit_ has joined #openstack-infra | 15:47 | |
| donnyd | is there a good way to see what provider https://zuul.opendev.org/t/openstack/builds?uuid=f7687852eb6345dc850b2d4c1e82402e and https://zuul.opendev.org/t/openstack/builds?uuid=f9801b0d8bbf43c6ab4c35526ecd655b ran on ? | 15:47 |
| *** goldyfruit has quit IRC | 15:50 | |
| openstackgerrit | Kostyantyn Kalynovskyi proposed openstack/project-config master: Add airship/apis project https://review.opendev.org/684374 | 15:51 |
| *** hashar has quit IRC | 15:52 | |
| *** ykarel is now known as ykarel|afk | 15:54 | |
| dtantsur | folks, is there a way to figure out why some jobs are not running? | 15:54 |
| dtantsur | what I see is that some jobs that run on master do not run on stable/train | 15:55 |
| AJaeger | dtantsur: example? | 15:56 |
| AJaeger | dtantsur: you can always use debug, let me find a link... | 15:56 |
| AJaeger | https://zuul-ci.org/docs/zuul/user/config.html#attr-project.%3Cpipeline%3E.debug | 15:56 |
| dtantsur | AJaeger: compare https://review.opendev.org/#/c/684363/ with https://review.opendev.org/#/c/682666/ | 15:56 |
| AJaeger | check: debug: true will report at the *end* for each job why it was run/not run | 15:57 |
| AJaeger | dtantsur: you wonder about ironicclient-functional ? | 15:58 |
| dtantsur | AJaeger: both functional and tempest | 15:58 |
| AJaeger | it searches the job on stable/train in other repos and cannot find it... | 15:58 |
| dtantsur | shouldn't it fall back to master? | 15:58 |
| AJaeger | so, if that job is defined in another repo that has stable branches but no stable/train branch yet | 15:58 |
| AJaeger | dtantsur: only if the repo is branchless AFAIK - but please double check the docs | 15:59 |
| dtantsur | OMG | 15:59 |
| dtantsur | that's a huge issue | 15:59 |
| AJaeger | dtantsur: https://zuul-ci.org/docs/zuul/user/config.html#attr-job.branches | 16:00 |
| AJaeger | dtantsur: so, where is ironicclient-functional defined? Can't you branch that? | 16:01 |
| dtantsur | AJaeger: ironicclient-functional is defined in the same repo. ironicclient-tempest uses ironic-base, and ironic gets branched 2-3 weeks after ironicclient (because of the openstack schedule) | 16:02 |
| dtantsur | this seems a big problem to me | 16:02 |
| *** panda is now known as panda|bbl | 16:02 | |
| dtantsur | and I still don't quite get why the functional test doesn't run | 16:02 |
| * dtantsur uses debug | 16:02 | |
| AJaeger | and functional has devstack-minimal as parent which comes from devstack repo. | 16:03 |
| AJaeger | and devstack is not branched... | 16:03 |
| AJaeger | smcginnis, ttx, fungi, clarkb, how did we handle this in previous releases? ^ I think we just waited for all repos to branch, correct? | 16:04 |
| dtantsur | it means that all stable branches do not have integration tests until devstack is branched. which is.. bad | 16:04 |
| dtantsur | especially in the case like ours when we need to make an urgent release | 16:04 |
| fungi | it means that all stable branches use master integration tests until devstack is branched (or should) | 16:05 |
| dtantsur | fungi: what I'm seeing is that some jobs simply do not run | 16:05 |
| fungi | and sorry, the zuul booth at ansiblefest is mobbed by attendees so i'll have to catch up on scrollback later | 16:05 |
| dtantsur | I would expect that they use master as you say | 16:06 |
| fungi | dtantsur: that sounds like you have explicit branch conditions for them maybe? | 16:06 |
| dtantsur | fungi: not that I'm aware of | 16:06 |
| fungi | not in a place to be able to investigate, sorry | 16:06 |
| dtantsur | no worries.. I'm using debug now. | 16:07 |
| dtantsur | fungi: make'em all use zuul for greater good! | 16:07 |
| AJaeger | dtantsur: which change? | 16:07 |
| dtantsur | AJaeger: https://review.opendev.org/#/c/684394/ | 16:07 |
| *** rpittau is now known as rpittau|afk | 16:08 | |
| AJaeger | dtantsur: we might have had this discussion last release as well, want to check eavesdrop archives? Or releaseteam has better memory than old me ... | 16:08 |
| dtantsur | I don't think this happened in the past. but maybe we just did not check. | 16:09 |
| * AJaeger waits for 684394 to *finish* and will be back later... | 16:09 | |
| AJaeger | dtantsur: I doubt you are the first to run into this | 16:09 |
| *** hashar has joined #openstack-infra | 16:09 | |
| dtantsur | with my luck I may well be the first :D | 16:10 |
| *** udesale has quit IRC | 16:10 | |
| openstackgerrit | Kostyantyn Kalynovskyi proposed openstack/project-config master: Add airship/apis project https://review.opendev.org/684374 | 16:10 |
| dtantsur | hmm, looking at the history, it may have happened in the past as well | 16:11 |
| ttx | dtantsur: process calls for devstack to be branched once all projects using devstack have been branched. Which is end of this week | 16:11 |
| dtantsur | ttx: it ends in a chicken-and-egg problem. we need to release ironicclient to unbreak ironic-inspector. we need to release ironic-inspector for devstack to get branched. we need devstack to get branched to test ironicclient and release it. | 16:12 |
| dtantsur | we will probably just trust the master CI and assume that nothing has changed in train (which is true) | 16:13 |
| ttx | yeah, that is how we traditionally broke that loop | 16:13 |
| dtantsur | k | 16:14 |
| *** dtantsur is now known as dtantsur|afk | 16:15 | |
| dtantsur|afk | I need to go, will catch up with the scrollback tomorrow | 16:15 |
| *** priteau has quit IRC | 16:19 | |
| *** xenos76 has joined #openstack-infra | 16:26 | |
| openstackgerrit | Kostyantyn Kalynovskyi proposed openstack/project-config master: Add airship/apis project https://review.opendev.org/684374 | 16:28 |
| *** jaosorior has joined #openstack-infra | 16:29 | |
| *** yoctozepto has quit IRC | 16:35 | |
| *** yoctozepto has joined #openstack-infra | 16:40 | |
| *** markvoelker has quit IRC | 16:40 | |
| *** eernst has joined #openstack-infra | 16:44 | |
| *** tesseract has quit IRC | 16:46 | |
| rm_work | frickler: so could we just move the octavia config step to the last part of the `install`? | 16:50 |
| rm_work | seems like that's probably what others do? | 16:50 |
| rm_work | but I don't know why the post-config runs fine in MY devstack using the same mechanism, and not in that one | 16:50 |
| rm_work | why would the order be different | 16:51 |
| *** jaosorior has quit IRC | 16:51 | |
| rm_work | I think historically we had to do ours late because we needed to wait for neutron-lbaas to be configured first and had no way to guarantee that otherwise | 16:51 |
| rm_work | but now that we got rid of neutron-lbaas, we could probably move the config stage back | 16:51 |
| rm_work | hmm maybe... it also needs to guarantee running after neutron I think | 16:54 |
| *** panda|bbl is now known as panda | 16:56 | |
| *** ramishra has quit IRC | 16:57 | |
| *** e0ne has quit IRC | 16:57 | |
| *** markvoelker has joined #openstack-infra | 16:59 | |
| rm_work | looking at designate tho, it also does its configure stage during post-config -- but that works fine!? how does the ordering work, that octavia is getting pushed back to after the config-merge step? >_> | 17:00 |
| *** KeithMnemonic has quit IRC | 17:00 | |
| *** KeithMnemonic has joined #openstack-infra | 17:00 | |
| *** ricolin has quit IRC | 17:03 | |
| AJaeger | rm_work: that'S probably a discussion for #openstack-qa | 17:15 |
| rm_work | hmm, are they the owners of devstack? | 17:15 |
| *** ociuhandu has quit IRC | 17:15 | |
| AJaeger | rm_work: yes! | 17:18 |
| AJaeger | rm_work: some of them are here as well like frickler ;) | 17:18 |
| rm_work | ah, i guess i just assumed it was ya'll since folks here normally know every single nook and cranny of that code :D | 17:19 |
| AJaeger | some do ;) But you won't get every experts help since those are listening more to the qa channel :) | 17:19 |
| *** jamesmcarthur has joined #openstack-infra | 17:25 | |
| *** armax has quit IRC | 17:26 | |
| *** jamesmcarthur has quit IRC | 17:31 | |
| *** jpena is now known as jpena|off | 17:32 | |
| *** sthussey has quit IRC | 17:39 | |
| *** ralonsoh has quit IRC | 17:42 | |
| *** e0ne has joined #openstack-infra | 17:47 | |
| *** igordc has joined #openstack-infra | 17:48 | |
| *** jamesmcarthur has joined #openstack-infra | 17:54 | |
| rm_work | frickler: ah yeah i figured out a different way to do it, without post-config | 17:56 |
| rm_work | forgot we added vars for all that stuff that our plugin reads during its own config step | 17:56 |
| *** psachin has quit IRC | 17:59 | |
| *** ykarel|afk is now known as ykarel|away | 18:00 | |
| *** e0ne has quit IRC | 18:09 | |
| *** Goneri has joined #openstack-infra | 18:10 | |
| openstackgerrit | Kostyantyn Kalynovskyi proposed openstack/project-config master: Add airship/apis project https://review.opendev.org/684374 | 18:13 |
| *** ykarel|away has quit IRC | 18:15 | |
| *** ociuhandu has joined #openstack-infra | 18:18 | |
| *** jtomasek has quit IRC | 18:21 | |
| *** bdodd has quit IRC | 18:22 | |
| *** ociuhandu has quit IRC | 18:23 | |
| *** bdodd has joined #openstack-infra | 18:24 | |
| *** bdodd has quit IRC | 18:24 | |
| *** e0ne has joined #openstack-infra | 18:32 | |
| *** e0ne has quit IRC | 18:33 | |
| *** adam_g has joined #openstack-infra | 18:35 | |
| *** hashar has quit IRC | 18:35 | |
| *** ykarel|away has joined #openstack-infra | 18:35 | |
| *** Goneri has quit IRC | 18:37 | |
| openstackgerrit | Tobias Henkel proposed zuul/zuul master: DNM: Try to reproduce cancel failure https://review.opendev.org/684414 | 18:38 |
| mriedem | clarkb: https://review.opendev.org/#/c/684042/ fixes mysql log collection in devstack-gate which grenade jobs still use and is needed to help debug http://status.openstack.org/elastic-recheck/#1844929 | 18:39 |
| *** ykarel has joined #openstack-infra | 18:42 | |
| *** jamesmcarthur has quit IRC | 18:42 | |
| clarkb | mriedem: trying to look via conference wifi | 18:42 |
| *** ykarel|away has quit IRC | 18:43 | |
| *** jamesmcarthur has joined #openstack-infra | 18:44 | |
| *** jamesmcarthur has quit IRC | 18:47 | |
| *** mriedem has quit IRC | 18:48 | |
| openstackgerrit | Kostyantyn Kalynovskyi proposed openstack/project-config master: Add airship/apis project https://review.opendev.org/684374 | 18:49 |
| *** mriedem has joined #openstack-infra | 18:49 | |
| *** hashar has joined #openstack-infra | 18:53 | |
| *** e0ne has joined #openstack-infra | 18:54 | |
| *** lucasagomes has quit IRC | 19:00 | |
| *** armax has joined #openstack-infra | 19:02 | |
| diablo_rojo | Is the infra meeting cancelled this week cause clarkb and fungi are at Ansiblefest? | 19:04 |
| clarkb | and monty and jim so yes | 19:05 |
| diablo_rojo | Got it. | 19:05 |
| clarkb | I sent email yesterday about it | 19:05 |
| diablo_rojo | Totally missed that. Sorry! | 19:05 |
| diablo_rojo | Thanks for the info. | 19:05 |
| clarkb | no worries | 19:06 |
| rm_work | ah that explains why it's so quiet :D | 19:11 |
| diablo_rojo | rm_work, yeah, all the loud people are at a conference ;) | 19:11 |
| rm_work | well, i'm sitting here being loud all by myself | 19:11 |
| diablo_rojo | rm_work, I can be loud too if you want company | 19:13 |
| rm_work | \o/ | 19:13 |
| diablo_rojo | I could just spam the channel with pictures of my new puppy. | 19:14 |
| jroll | um, yes please | 19:15 |
| jroll | why would that even be a question? :P | 19:16 |
| diablo_rojo | jroll, it wont let me send to the channel, but I can link you to her instagram lol | 19:19 |
| jroll | ++ | 19:19 |
| diablo_rojo | jroll, https://www.instagram.com/goose_the_stberdoodle/?hl=en | 19:20 |
| *** sthussey has joined #openstack-infra | 19:21 | |
| jroll | diablo_rojo: adorable :D | 19:23 |
| diablo_rojo | jroll, yeah she is :) | 19:23 |
| jroll | I am attempting to return the favor, but internet is lacking | 19:23 |
| diablo_rojo | jroll, I will wait for as long as it takes lol | 19:24 |
| jroll | diablo_rojo: not quite new, but here we go https://i.imgur.com/CJrDqpD.gifv | 19:25 |
| diablo_rojo | jroll, oh man, such a majestic pupper :) | 19:26 |
| diablo_rojo | cmurphy, you probably want in on this ^^ | 19:26 |
| jroll | :) | 19:26 |
| AJaeger | what a different kind of noise - I imagine all the barfing :) | 19:26 |
| AJaeger | diablo_rojo: nice images and narrative | 19:27 |
| diablo_rojo | AJaeger, thanks lol :) Trying to channel her voice and lack of opposable thumbs. | 19:27 |
| AJaeger | ;) | 19:28 |
| cmurphy | diablo_rojo: omg | 19:31 |
| rm_work | `tox_envlist: functional` should make zuul run `tox -e functional', right? | 19:33 |
| diablo_rojo | cmurphy, come to Seattle and you can play with her :) | 19:33 |
| rm_work | also... gotta love animals having their own insta :D | 19:33 |
| openstackgerrit | Tobias Henkel proposed zuul/zuul master: DNM: Try to reproduce cancel failure https://review.opendev.org/684414 | 19:33 |
| rm_work | oh i keep forgetting you're in seattle | 19:33 |
| rm_work | since I'm in Bellingham but make it down every so often, I should try to make it to some events or something | 19:34 |
| diablo_rojo | rm_work, we could do our own little meetup thing and get portland folks to come up too | 19:38 |
| *** xenos76 has quit IRC | 19:39 | |
| *** jamesmcarthur has joined #openstack-infra | 19:39 | |
| rm_work | i'm usually down for meetups | 19:39 |
| rm_work | or, am i up for meetdowns? | 19:39 |
| rm_work | one of those | 19:40 |
| openstackgerrit | Tobias Henkel proposed zuul/zuul master: DNM: Try to reproduce cancel failure https://review.opendev.org/684414 | 19:44 |
| rm_work | I have one majestic piggie (if you are of the persuasion that believes piggies can be majestic): https://i.imgur.com/Qs0ITOl.png | 19:46 |
| *** ykarel has quit IRC | 19:47 | |
| *** prometheanfire has quit IRC | 19:47 | |
| *** prometheanfire has joined #openstack-infra | 19:48 | |
| rm_work | and one not so majestic piggie... https://i.imgur.com/uZ5uudH.png | 19:48 |
| *** hwoarang has quit IRC | 19:50 | |
| *** rlandy has quit IRC | 19:50 | |
| jroll | lol awesome | 19:51 |
| *** hwoarang has joined #openstack-infra | 19:52 | |
| *** jamesmcarthur has quit IRC | 19:54 | |
| *** rlandy has joined #openstack-infra | 19:54 | |
| *** jamesmcarthur has joined #openstack-infra | 19:55 | |
| *** whoami-rajat has quit IRC | 19:56 | |
| *** e0ne has quit IRC | 19:59 | |
| *** hwoarang has quit IRC | 20:00 | |
| rm_work | diablo_rojo: how old is that pupper? | 20:01 |
| rm_work | oh wait that was jroll's\ | 20:02 |
| rm_work | i didn't even click through to the instagram yet! omg | 20:02 |
| rm_work | the derp-face is my favorite | 20:02 |
| *** hwoarang has joined #openstack-infra | 20:04 | |
| *** noama has quit IRC | 20:07 | |
| *** Lucas_Gray has joined #openstack-infra | 20:11 | |
| *** Lucas_Gray has quit IRC | 20:27 | |
| *** Lucas_Gray has joined #openstack-infra | 20:29 | |
| *** jcoufal has quit IRC | 20:34 | |
| *** e0ne has joined #openstack-infra | 20:38 | |
| rm_work | hmm, i thought `tox_envlist` would tell zuul to run the tox environment you list? but it doesn't seem to do anything. am I missing something else? | 20:51 |
| rm_work | https://review.opendev.org/#/c/683285/8/.zuul.yaml@37 | 20:52 |
| rm_work | tox never runs | 20:52 |
| *** xenos76 has joined #openstack-infra | 20:59 | |
| *** jamesmcarthur has quit IRC | 21:06 | |
| *** rfolco has quit IRC | 21:11 | |
| *** rfolco has joined #openstack-infra | 21:11 | |
| *** xek has quit IRC | 21:11 | |
| clarkb | I dont think it actually takes a list | 21:16 |
| clarkb | oh yours isnt a list so should be fine | 21:16 |
| openstackgerrit | Gary W. Smith proposed openstack/project-config master: Add airshipui project https://review.opendev.org/679806 | 21:31 |
| *** hashar has quit IRC | 21:33 | |
| rm_work | clarkb: well, it's not running tox at all :/ | 21:36 |
| rm_work | do i need to add more roles or something? | 21:36 |
| rm_work | the example just had tox_envlist set... | 21:36 |
| rm_work | looking at this job: https://review.opendev.org/#/c/500365/34/.zuul.yaml@40 | 21:37 |
| rm_work | hmm do I need to set the workdir? | 21:37 |
| rm_work | it's just weird because i can't even find any proof that it's *trying* | 21:40 |
| *** diablo_rojo_phon has joined #openstack-infra | 21:45 | |
| *** markvoelker has quit IRC | 21:52 | |
| *** xenos76 has quit IRC | 21:56 | |
| *** weshay has quit IRC | 21:58 | |
| *** slaweq has quit IRC | 22:00 | |
| *** e0ne has quit IRC | 22:04 | |
| *** jamesmcarthur has joined #openstack-infra | 22:06 | |
| *** rcernin has joined #openstack-infra | 22:07 | |
| *** aaronsheffield has quit IRC | 22:10 | |
| *** Lucas_Gray has quit IRC | 22:12 | |
| *** jamesmcarthur has quit IRC | 22:13 | |
| *** weshay has joined #openstack-infra | 22:19 | |
| *** rfolco has quit IRC | 22:29 | |
| *** rfolco has joined #openstack-infra | 22:29 | |
| openstackgerrit | Merged openstack/devstack-gate master: Fix MySQL log collection https://review.opendev.org/684042 | 22:41 |
| *** dtantsur|afk has quit IRC | 22:51 | |
| *** tkajinam has joined #openstack-infra | 22:51 | |
| *** eernst has quit IRC | 23:08 | |
| *** eernst_ has joined #openstack-infra | 23:12 | |
| *** rfolco has quit IRC | 23:13 | |
| *** rfolco has joined #openstack-infra | 23:14 | |
| *** goldyfruit_ has quit IRC | 23:14 | |
| *** eernst_ has quit IRC | 23:16 | |
| *** dchen has joined #openstack-infra | 23:19 | |
| *** sthussey has quit IRC | 23:20 | |
| *** weshay has quit IRC | 23:31 | |
| *** JorgeFranco has quit IRC | 23:32 | |
| *** diablo_rojo has quit IRC | 23:47 | |
| *** jamesmcarthur has joined #openstack-infra | 23:50 | |
| *** markvoelker has joined #openstack-infra | 23:54 | |
| *** jamesmcarthur has quit IRC | 23:55 | |
| *** markvoelker has quit IRC | 23:58 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!
