| *** mattw4 has quit IRC | 00:09 | |
| *** jamesmcarthur has quit IRC | 00:09 | |
| *** slaweq has joined #openstack-infra | 00:11 | |
| *** ociuhandu has joined #openstack-infra | 00:12 | |
| *** slaweq has quit IRC | 00:16 | |
| openstackgerrit | Ghanshyam Mann proposed openstack/hacking master: Remove pypy jobs https://review.opendev.org/710349 | 00:17 |
|---|---|---|
| *** tosky has quit IRC | 00:18 | |
| clarkb | nl01 has updated nodepool installation. I'm going to restart it to pick up the fix | 00:19 |
| clarkb | (then will proceed with the others once it looks happy) | 00:19 |
| clarkb | nodepool==3.11.1.dev35 # git sha aba9b4e is what we will be running | 00:19 |
| clarkb | monitoring 0014873052 as a building node | 00:22 |
| *** igordc has quit IRC | 00:23 | |
| clarkb | that node built successfully. Restarting nl02-04 now | 00:25 |
| clarkb | #status log Restarted nodepool launcers on nodepool==3.11.1.dev35 # git sha aba9b4e to pick up proper fix for missing node issue that was previously worked around with a restart. | 00:36 |
| openstackstatus | clarkb: finished logging | 00:36 |
| *** rlandy has quit IRC | 00:49 | |
| *** mpjetta has joined #openstack-infra | 00:50 | |
| *** jcapitao_off has quit IRC | 00:50 | |
| *** ociuhandu has quit IRC | 00:51 | |
| *** tetsuro has quit IRC | 00:57 | |
| *** tetsuro has joined #openstack-infra | 00:57 | |
| *** lbragstad has quit IRC | 01:06 | |
| *** slaweq has joined #openstack-infra | 01:11 | |
| *** slaweq has quit IRC | 01:16 | |
| openstackgerrit | Ian Wienand proposed opendev/system-config master: letsencrypt: force renewal on certificate change https://review.opendev.org/710360 | 01:17 |
| ianw | clarkb / mordred: ^ i think that will solve this longer term | 01:18 |
| ianw | for now, i'll move the config files to generate the new certs on static.opendev.org | 01:18 |
| *** mpjetta has quit IRC | 01:21 | |
| *** mpjetta has joined #openstack-infra | 01:24 | |
| *** stevebaker has joined #openstack-infra | 01:27 | |
| ianw | ok, that got it curl --resolve files.openstack.org:443:23.253.245.150 https://files.openstack.org | 01:31 |
| *** lbragstad has joined #openstack-infra | 01:33 | |
| clarkb | ianw fix change lgtm | 01:40 |
| clarkb | thank you for the descriptive commit message | 01:41 |
| *** apevec has quit IRC | 01:50 | |
| *** gyee has quit IRC | 01:52 | |
| ianw | #status log switched files.openstack.org CNAME to static.opendev.org | 02:00 |
| openstackstatus | ianw: finished logging | 02:00 |
| *** slaweq has joined #openstack-infra | 02:11 | |
| *** smarcet has joined #openstack-infra | 02:12 | |
| *** slaweq has quit IRC | 02:16 | |
| *** yamamoto has joined #openstack-infra | 02:22 | |
| *** sreejithp has joined #openstack-infra | 02:26 | |
| *** sreejithp_ has joined #openstack-infra | 02:29 | |
| *** sreejithp has quit IRC | 02:31 | |
| *** sreejithp_ has quit IRC | 02:41 | |
| *** yamamoto has quit IRC | 02:45 | |
| *** yamamoto has joined #openstack-infra | 02:47 | |
| *** jamesmcarthur has joined #openstack-infra | 02:53 | |
| *** yamamoto has quit IRC | 03:04 | |
| *** slaweq has joined #openstack-infra | 03:11 | |
| *** apetrich has quit IRC | 03:11 | |
| *** apetrich has joined #openstack-infra | 03:15 | |
| *** slaweq has quit IRC | 03:16 | |
| *** jamesmcarthur has quit IRC | 03:22 | |
| * prometheanfire thinks 710309 just timed out :( | 03:24 | |
| prometheanfire | stupid tempest | 03:24 |
| *** diablo_rojo has quit IRC | 03:25 | |
| openstackgerrit | Ghanshyam Mann proposed openstack/hacking master: Remove pypy jobs https://review.opendev.org/710349 | 03:27 |
| openstackgerrit | Merged opendev/system-config master: static: implement legacy redirect sites https://review.opendev.org/710160 | 03:30 |
| openstackgerrit | Merged opendev/system-config master: Update redirects for legacy sides https://review.opendev.org/710195 | 03:30 |
| *** psachin has joined #openstack-infra | 03:36 | |
| *** ykarel|away is now known as ykarel | 03:57 | |
| *** yamamoto has joined #openstack-infra | 03:58 | |
| *** stevebaker has quit IRC | 04:03 | |
| *** yamamoto has quit IRC | 04:07 | |
| *** yamamoto has joined #openstack-infra | 04:08 | |
| *** slaweq has joined #openstack-infra | 04:11 | |
| *** slaweq has quit IRC | 04:16 | |
| * prometheanfire wonders if it's just slower now | 04:25 | |
| *** smarcet has quit IRC | 04:32 | |
| *** imacdonn has quit IRC | 04:48 | |
| *** imacdonn has joined #openstack-infra | 04:48 | |
| *** yamamoto has quit IRC | 04:55 | |
| *** yamamoto has joined #openstack-infra | 04:57 | |
| *** rh-jelabarre has quit IRC | 05:08 | |
| *** slaweq has joined #openstack-infra | 05:11 | |
| *** udesale has joined #openstack-infra | 05:13 | |
| *** slaweq has quit IRC | 05:15 | |
| *** roman_g has quit IRC | 05:16 | |
| *** dave-mccowan has quit IRC | 05:19 | |
| ianw | #status log domains from https://review.opendev.org/#/c/710160/ switched to CNAMEs for static.opendev.org | 05:20 |
| openstackstatus | ianw: finished logging | 05:20 |
| openstackgerrit | Ian Wienand proposed opendev/system-config master: ssl cert check: add recent domains https://review.opendev.org/710386 | 05:21 |
| *** evrardjp has quit IRC | 05:35 | |
| *** evrardjp has joined #openstack-infra | 05:35 | |
| *** jamesdenton has quit IRC | 05:38 | |
| *** yamamoto has quit IRC | 05:39 | |
| *** jamesdenton has joined #openstack-infra | 05:39 | |
| *** yamamoto has joined #openstack-infra | 05:39 | |
| ianw | static.openstack.org is also now a cname for static.opendev.org. the old server is at static-old.openstack.org until we want to remove it | 05:41 |
| *** ijw has quit IRC | 05:46 | |
| openstackgerrit | Ian Wienand proposed opendev/system-config master: [wip] remove static site puppet https://review.opendev.org/710388 | 05:49 |
| *** yamamoto has quit IRC | 05:59 | |
| *** yamamoto has joined #openstack-infra | 06:00 | |
| *** slaweq has joined #openstack-infra | 06:11 | |
| openstackgerrit | Andreas Jaeger proposed opendev/system-config master: Kill qa.o.o https://review.opendev.org/710390 | 06:11 |
| AJaeger | ianw, gmann, this removes qa.o.o ^ | 06:12 |
| *** kozhukalov has joined #openstack-infra | 06:15 | |
| *** ijw has joined #openstack-infra | 06:15 | |
| *** slaweq has quit IRC | 06:16 | |
| *** ijw has quit IRC | 06:20 | |
| *** matt_kosut has joined #openstack-infra | 06:26 | |
| *** lmiccini has joined #openstack-infra | 06:47 | |
| *** ijw has joined #openstack-infra | 06:47 | |
| *** ijw has quit IRC | 06:52 | |
| *** matt_kosut has quit IRC | 06:54 | |
| *** matt_kosut has joined #openstack-infra | 06:55 | |
| *** ykarel is now known as ykarel|afk | 06:55 | |
| *** matt_kosut has quit IRC | 07:00 | |
| *** dpawlik has joined #openstack-infra | 07:09 | |
| *** slaweq has joined #openstack-infra | 07:11 | |
| *** slaweq has quit IRC | 07:16 | |
| *** ijw has joined #openstack-infra | 07:19 | |
| *** ralonsoh has joined #openstack-infra | 07:20 | |
| *** ijw has quit IRC | 07:23 | |
| *** ykarel|afk is now known as ykarel | 07:30 | |
| *** slaweq has joined #openstack-infra | 07:30 | |
| *** pgaxatte has joined #openstack-infra | 07:40 | |
| *** ijw has joined #openstack-infra | 07:51 | |
| *** ijw has quit IRC | 07:55 | |
| *** psachin has quit IRC | 08:00 | |
| *** tesseract has joined #openstack-infra | 08:05 | |
| *** ociuhandu has joined #openstack-infra | 08:07 | |
| *** jcapitao_off has joined #openstack-infra | 08:11 | |
| *** pkopec has joined #openstack-infra | 08:13 | |
| *** jcapitao_off is now known as jcapitao | 08:19 | |
| *** tosky has joined #openstack-infra | 08:21 | |
| *** ijw has joined #openstack-infra | 08:22 | |
| *** rpittau|afk is now known as rpittau | 08:22 | |
| *** ijw has quit IRC | 08:26 | |
| openstackgerrit | Felix Edel proposed zuul/zuul master: WIP Visualize retry builds as expandable rows on buildset page https://review.opendev.org/710408 | 08:26 |
| openstackgerrit | Felix Edel proposed zuul/zuul master: WIP Visualize retry builds as expandable rows on buildset page https://review.opendev.org/710408 | 08:28 |
| *** ociuhandu has quit IRC | 08:29 | |
| *** ociuhandu has joined #openstack-infra | 08:29 | |
| openstackgerrit | Andreas Jaeger proposed opendev/system-config master: Kill qa.o.o https://review.opendev.org/710390 | 08:31 |
| *** ykarel is now known as ykarel|lunch | 08:37 | |
| *** jpena|off is now known as jpena | 08:38 | |
| *** ociuhandu has quit IRC | 08:38 | |
| *** ociuhandu has joined #openstack-infra | 08:42 | |
| *** ociuhandu has quit IRC | 08:42 | |
| openstackgerrit | Tobias Henkel proposed zuul/zuul master: Optimize canMerge using graphql https://review.opendev.org/709836 | 08:44 |
| *** amoralej|off is now known as amoralej | 08:54 | |
| *** tetsuro has quit IRC | 08:55 | |
| *** rcernin has quit IRC | 08:56 | |
| *** ykarel|lunch is now known as ykarel | 09:08 | |
| *** lucasagomes has joined #openstack-infra | 09:10 | |
| *** ijw has joined #openstack-infra | 09:24 | |
| *** ijw has quit IRC | 09:28 | |
| *** ramishra has quit IRC | 09:30 | |
| *** ramishra has joined #openstack-infra | 09:31 | |
| *** derekh has joined #openstack-infra | 09:40 | |
| *** xek__ has joined #openstack-infra | 09:44 | |
| openstackgerrit | Jan Kubovy proposed zuul/zuul master: Scheduler test app manager https://review.opendev.org/708812 | 09:45 |
| *** yamamoto has quit IRC | 09:48 | |
| *** yamamoto has joined #openstack-infra | 09:52 | |
| *** yamamoto has quit IRC | 10:01 | |
| *** tesseract has quit IRC | 10:11 | |
| *** tesseract has joined #openstack-infra | 10:12 | |
| *** tesseract has quit IRC | 10:16 | |
| *** tesseract has joined #openstack-infra | 10:16 | |
| *** shivank01 has joined #openstack-infra | 10:19 | |
| shivank01 | Hey, I am setting up openstack for my organisation. The work is mostly of training Deep Learning models. I just want to know which GPU should I install in my server which is not so costly? | 10:19 |
| *** jcapitao has quit IRC | 10:19 | |
| *** jcapitao has joined #openstack-infra | 10:19 | |
| *** jcapitao has quit IRC | 10:20 | |
| *** jcapitao has joined #openstack-infra | 10:20 | |
| *** ykarel is now known as ykarel|afk | 10:25 | |
| *** ijw has joined #openstack-infra | 10:27 | |
| *** trident has quit IRC | 10:31 | |
| *** ijw has quit IRC | 10:32 | |
| *** trident has joined #openstack-infra | 10:34 | |
| mnaser | shivank01: this is a good question to post on our mailing lists | 10:48 |
| *** ijw has joined #openstack-infra | 10:58 | |
| *** stephenfin has quit IRC | 10:59 | |
| *** stephenfin has joined #openstack-infra | 11:00 | |
| *** ijw has quit IRC | 11:03 | |
| *** ykarel|afk is now known as ykarel | 11:11 | |
| *** stephenfin has quit IRC | 11:15 | |
| *** stephenfin has joined #openstack-infra | 11:16 | |
| openstackgerrit | Felix Edel proposed zuul/zuul master: Dequeue changes via github checks API https://review.opendev.org/709135 | 11:17 |
| *** ijw has joined #openstack-infra | 11:30 | |
| *** ijw has quit IRC | 11:34 | |
| openstackgerrit | Jan Kubovy proposed zuul/zuul master: Make test setup_config more pure https://review.opendev.org/709084 | 11:37 |
| openstackgerrit | Jan Kubovy proposed zuul/zuul master: Scheduler test app manager https://review.opendev.org/708812 | 11:37 |
| *** stephenfin has quit IRC | 11:52 | |
| *** stephenfin has joined #openstack-infra | 11:52 | |
| openstackgerrit | Felix Edel proposed zuul/zuul master: WIP Dequeue changes via github checks API https://review.opendev.org/709135 | 11:54 |
| openstackgerrit | Jan Kubovy proposed zuul/zuul master: Use scheduler manager consistently in tests https://review.opendev.org/709542 | 11:59 |
| *** ijw has joined #openstack-infra | 12:01 | |
| *** jcapitao is now known as jcapitao_lunch | 12:03 | |
| *** ijw has quit IRC | 12:06 | |
| *** yamamoto has joined #openstack-infra | 12:09 | |
| openstackgerrit | Jan Kubovy proposed zuul/zuul master: Scheduler test app manager https://review.opendev.org/708812 | 12:16 |
| *** yamamoto has quit IRC | 12:17 | |
| amoralej | would it be possible to get the job which is running a specific nodepool node by its ip ? | 12:18 |
| fungi | i'm not sure what that question means | 12:19 |
| fungi | you mean you know an ip address of a nodepool node and want to find out what job is running on it? | 12:20 |
| amoralej | fungi, i'm checking rdo logs and i see ip 23.253.216.84 fetching some packages | 12:20 |
| amoralej | i'd like to know what job is taht | 12:20 |
| amoralej | that | 12:20 |
| amoralej | fungi, related to http://lists.openstack.org/pipermail/openstack-discuss/2020-February/012907.html | 12:21 |
| fungi | with access to the nodepool launcher logs and zuul scheduler logs you can match an ip address in a specific timeframe to a node request and then find out what build was associated with that request | 12:21 |
| fungi | amoralej: what timeframe? | 12:25 |
| fungi | are you suspecting that's a nodepool node in the opendev ci system? | 12:25 |
| amoralej | fungi, last log is 10 minutes ago | 12:26 |
| amoralej | yes | 12:26 |
| amoralej | it's rackspace ip | 12:26 |
| amoralej | i suspect from some opendev ci, but not sure, of course | 12:26 |
| fungi | i don't see it winding up in our nodepool over the past few days | 12:27 |
| amoralej | fungi, ok, then it must be something else | 12:27 |
| amoralej | thx | 12:27 |
| *** cgoncalves has quit IRC | 12:29 | |
| fungi | amoralej: looks like it's hosting a mirror, seems to be mirror01.ord.rax.opendev.org but for some reason there's no reverse dns set up for it | 12:30 |
| fungi | i'll add that to my list of things to fix this morning | 12:30 |
| amoralej | fungi, yeah i tried reverse dns | 12:30 |
| amoralej | no problem | 12:30 |
| amoralej | it makes sense | 12:30 |
| amoralej | fungi++ thanks a bunch | 12:30 |
| fungi | yw | 12:30 |
| *** ijw has joined #openstack-infra | 12:32 | |
| *** jpena is now known as jpena|lunch | 12:33 | |
| *** tkajinam has quit IRC | 12:34 | |
| *** nicolasbock has joined #openstack-infra | 12:34 | |
| *** gfidente|afk is now known as gfidente | 12:36 | |
| *** ijw has quit IRC | 12:37 | |
| frickler | amoralej: will your changes just be moving things inside trunk.rdoproject.org or will they involve some other host? in the latter case we might want to update our mirror config https://opendev.org/opendev/system-config/src/branch/master/modules/openstack_project/templates/mirror.vhost.erb#L190-L193 | 12:37 |
| *** matt_kosut has joined #openstack-infra | 12:37 | |
| amoralej | frickler, everything is inside trunk.rdoproject.org | 12:38 |
| frickler | amoralej: also, if it would be really needed, we might be able to identify the node running the job from proxy logs on the mirror | 12:38 |
| frickler | amoralej: o.k., so that should be transparent to our setup then, good | 12:38 |
| amoralej | frickler, no, i don't need it | 12:38 |
| amoralej | frickler, the trikiest part is that i may replace same URLs with different content | 12:39 |
| amoralej | mirroring check update date or something? | 12:39 |
| amoralej | to refresh content? | 12:39 |
| frickler | amoralej: it should, iiuc the "ttl=120" in our config will imply that we will check only every 2 minutes (I'm hoping it's not 120 minutes) | 12:40 |
| amoralej | frickler, ok, i think should be fine | 12:40 |
| amoralej | frickler, do you have a list of mirrors that i can manually check? | 12:41 |
| frickler | amoralej: not sure all of those are active https://opendev.org/opendev/system-config/src/branch/master/inventory/openstack.yaml#L450-L585 | 12:43 |
| amoralej | wow 137 mirrors | 12:44 |
| openstackgerrit | Jan Kubovy proposed zuul/zuul master: Scheduler test app manager https://review.opendev.org/708812 | 12:46 |
| frickler | I'm counting 19, there are 7 or more lines of inventory per mirror | 12:48 |
| *** cgoncalves has joined #openstack-infra | 12:59 | |
| *** rlandy has joined #openstack-infra | 12:59 | |
| *** auristor has quit IRC | 13:00 | |
| *** rh-jelabarre has joined #openstack-infra | 13:00 | |
| *** ijw has joined #openstack-infra | 13:04 | |
| *** auristor has joined #openstack-infra | 13:08 | |
| *** ijw has quit IRC | 13:09 | |
| *** jamesmcarthur has joined #openstack-infra | 13:09 | |
| *** nicolasbock has quit IRC | 13:11 | |
| *** udesale_ has joined #openstack-infra | 13:13 | |
| *** jcapitao_lunch is now known as jcapitao | 13:14 | |
| *** nicolasbock has joined #openstack-infra | 13:14 | |
| *** udesale has quit IRC | 13:15 | |
| *** jamesmcarthur has quit IRC | 13:18 | |
| *** TViernion has quit IRC | 13:21 | |
| *** jamesmcarthur has joined #openstack-infra | 13:21 | |
| *** Lucas_Gray has joined #openstack-infra | 13:26 | |
| *** piotrowskim has quit IRC | 13:31 | |
| *** Lucas_Gray has quit IRC | 13:32 | |
| *** udesale_ has quit IRC | 13:33 | |
| *** ijw has joined #openstack-infra | 13:35 | |
| *** jamesmcarthur has quit IRC | 13:36 | |
| *** jamesmcarthur has joined #openstack-infra | 13:37 | |
| *** jpena|lunch is now known as jpena | 13:38 | |
| *** toabctl has quit IRC | 13:38 | |
| *** ijw has quit IRC | 13:39 | |
| *** tesseract has quit IRC | 13:40 | |
| openstackgerrit | Merged opendev/system-config master: letsencrypt: force renewal on certificate change https://review.opendev.org/710360 | 13:41 |
| openstackgerrit | Merged zuul/zuul master: Support file comments via Github checks API https://review.opendev.org/708611 | 13:41 |
| *** tesseract has joined #openstack-infra | 13:42 | |
| *** jamesmcarthur has quit IRC | 13:53 | |
| *** jamesmcarthur has joined #openstack-infra | 13:55 | |
| openstackgerrit | Jan Kubovy proposed zuul/zuul master: Scheduler test app manager https://review.opendev.org/708812 | 13:55 |
| *** stephenfin has quit IRC | 13:56 | |
| *** stephenfin has joined #openstack-infra | 13:57 | |
| *** Goneri has joined #openstack-infra | 14:00 | |
| *** matt_kosut has quit IRC | 14:03 | |
| *** ijw has joined #openstack-infra | 14:07 | |
| *** stephenfin has quit IRC | 14:07 | |
| *** stephenfin has joined #openstack-infra | 14:08 | |
| *** ijw has quit IRC | 14:11 | |
| *** ociuhandu has joined #openstack-infra | 14:19 | |
| *** ociuhandu has quit IRC | 14:20 | |
| *** ociuhandu has joined #openstack-infra | 14:21 | |
| *** matt_kosut has joined #openstack-infra | 14:24 | |
| *** ociuhandu has quit IRC | 14:24 | |
| *** yamamoto has joined #openstack-infra | 14:24 | |
| *** ociuhandu has joined #openstack-infra | 14:24 | |
| *** yamamoto has quit IRC | 14:25 | |
| *** stephenfin has quit IRC | 14:25 | |
| *** stephenfin has joined #openstack-infra | 14:25 | |
| *** gagehugo has quit IRC | 14:28 | |
| *** matt_kosut has quit IRC | 14:28 | |
| *** gagehugo has joined #openstack-infra | 14:29 | |
| *** amoralej is now known as amoralej|lunch | 14:30 | |
| *** yamamoto has joined #openstack-infra | 14:30 | |
| *** dave-mccowan has joined #openstack-infra | 14:32 | |
| *** yamamoto has quit IRC | 14:33 | |
| *** yamamoto has joined #openstack-infra | 14:34 | |
| *** ijw has joined #openstack-infra | 14:38 | |
| *** jamesmcarthur has quit IRC | 14:41 | |
| *** ijw has quit IRC | 14:42 | |
| *** yamamoto has quit IRC | 14:44 | |
| *** jamesmcarthur has joined #openstack-infra | 14:47 | |
| *** ociuhandu has quit IRC | 14:54 | |
| *** nicolasbock has quit IRC | 14:58 | |
| *** nicolasbock has joined #openstack-infra | 14:58 | |
| *** ykarel is now known as ykarel|away | 14:59 | |
| *** yamamoto has joined #openstack-infra | 15:01 | |
| *** ociuhandu has joined #openstack-infra | 15:03 | |
| openstackgerrit | Jan Kubovy proposed zuul/zuul master: Use scheduler manager consistently in tests https://review.opendev.org/709542 | 15:04 |
| openstackgerrit | Jan Kubovy proposed zuul/zuul master: Refactor executor_client in tests https://review.opendev.org/709672 | 15:04 |
| *** tesseract has quit IRC | 15:05 | |
| *** tesseract has joined #openstack-infra | 15:07 | |
| *** ijw has joined #openstack-infra | 15:09 | |
| corvus | infra-root, config-core: fyi, the gerrit project has reversed their deprecation of numeric change ids; they will continue to be supported: https://gerrit-review.googlesource.com/c/gerrit/+/256792 | 15:11 |
| fungi | awesome, much nicer | 15:12 |
| *** njohnston has quit IRC | 15:12 | |
| *** ijw has quit IRC | 15:13 | |
| *** amoralej|lunch is now known as amoralej | 15:13 | |
| *** bnemec is now known as beekneemech | 15:16 | |
| *** njohnston_ has joined #openstack-infra | 15:16 | |
| *** jackedin has joined #openstack-infra | 15:21 | |
| *** ramishra has quit IRC | 15:28 | |
| *** yamamoto has quit IRC | 15:30 | |
| *** yamamoto has joined #openstack-infra | 15:31 | |
| *** ociuhandu has quit IRC | 15:31 | |
| *** TViernion has joined #openstack-infra | 15:36 | |
| *** ociuhandu has joined #openstack-infra | 15:38 | |
| *** ijw has joined #openstack-infra | 15:40 | |
| *** jamesmcarthur has quit IRC | 15:41 | |
| *** jcoufal has joined #openstack-infra | 15:42 | |
| *** ijw has quit IRC | 15:44 | |
| *** jcapitao is now known as jcapitao_afk | 15:47 | |
| openstackgerrit | wes hayutin proposed openstack/openstack-zuul-jobs master: add debug for swapfile mount status https://review.opendev.org/710470 | 15:47 |
| *** jamesmcarthur has joined #openstack-infra | 15:52 | |
| clarkb | amoralej: fungi fwiw that info should be indexed in elasticserach so is publicly available to everyone | 15:56 |
| clarkb | oh if it is a mirror then that won't get indexed :) | 15:56 |
| *** ykarel|away is now known as ykarel | 15:56 | |
| amoralej | i looked in logstash and didn't found it, but yeah, it was no nodepool so maybe the info is there | 15:56 |
| *** jamesmcarthur has quit IRC | 15:57 | |
| * fungi goes to find lunch, should be back soon | 15:57 | |
| clarkb | frickler: amoralej we should respect the ttls listed by the servers as well as having a 24 hour cap on caching | 15:57 |
| clarkb | that means if the server's ttl is longer than 24 hours we'll still refresh at the 24 hour mark | 15:58 |
| amoralej | clarkb, ok, good to know | 15:58 |
| openstackgerrit | Jan Kubovy proposed zuul/zuul master: Refactor merge_client in tests https://review.opendev.org/709676 | 16:00 |
| *** yamamoto has quit IRC | 16:01 | |
| clarkb | mordred: I've taken a quick look at the puppetry needed to use the new review.o.o cert and I think its actually pretty straight forward (we support simply setting a file path in the puppet and not managing contents) | 16:02 |
| clarkb | mordred: I'll get a patch up for that as it should be easy to write and review and gives us breathing room if necessary | 16:02 |
| *** jcapitao_afk is now known as jcapitao | 16:05 | |
| *** ramishra has joined #openstack-infra | 16:10 | |
| openstackgerrit | Jan Kubovy proposed zuul/zuul master: Refactor nodepool in tests https://review.opendev.org/709703 | 16:10 |
| openstackgerrit | Clark Boylan proposed opendev/system-config master: Use LE cert on review.open*.org https://review.opendev.org/710478 | 16:11 |
| clarkb | infra-root ^ I think that will switch us over to the LE cert well ahead of old cert expiring | 16:11 |
| *** ijw has joined #openstack-infra | 16:11 | |
| *** ijw has quit IRC | 16:16 | |
| *** gyee has joined #openstack-infra | 16:16 | |
| *** tesseract has quit IRC | 16:17 | |
| *** mattw4 has joined #openstack-infra | 16:20 | |
| *** jackedin has quit IRC | 16:21 | |
| *** slaweq has quit IRC | 16:23 | |
| *** slaweq has joined #openstack-infra | 16:25 | |
| *** jamesmcarthur has joined #openstack-infra | 16:26 | |
| *** lmiccini has quit IRC | 16:26 | |
| clarkb | roman_g isn't here but following up on the node failure questions from yesterday. It appears that their jobs are failing properly more often than not after we settled the zk situation. However there is at least one NODE_FAILURE due to 2020-02-28 07:20:37,378 ERROR nodepool.NodeLauncher: [e: c2984a28bac9404e8a3eeb788573cddc] [node_request: 300-0007677230] [node: 0014880727] Detailed node error: No | 16:27 |
| clarkb | valid host was found. There are not enough hosts available. | 16:27 |
| clarkb | that is the "cloud can't schedule your request due to lack of hypervisors" error | 16:27 |
| *** cgoncalves has quit IRC | 16:27 | |
| mordred | clarkb: cool. I also looked in - and we're missing manage-projects stuff in the ansible - so I'll be working on that today | 16:27 |
| *** cgoncalves has joined #openstack-infra | 16:28 | |
| *** pgaxatte has quit IRC | 16:28 | |
| AJaeger | frickler, fungi, yes, https://review.opendev.org/#/c/710390 (system-config change to remove qa.o.o) is not ready, if anybody could help to remove the file, it would be great | 16:35 |
| clarkb | AJaeger: looking | 16:36 |
| AJaeger | thanks | 16:37 |
| corvus | infra-root: i'm going to restart the zuul executors | 16:37 |
| corvus | i think we can leave the scheduler up and running | 16:37 |
| clarkb | corvus: ++ | 16:37 |
| clarkb | AJaeger: re your W-1 comment you mean it has to be removed on the server correct? | 16:37 |
| clarkb | AJaeger: I can do that if you want to remove the W-1 | 16:38 |
| *** roman_g has joined #openstack-infra | 16:38 | |
| openstackgerrit | Jan Kubovy proposed zuul/zuul master: Refactor zookeeper in tests https://review.opendev.org/709709 | 16:39 |
| AJaeger | clarkb: that is one way of doing it ;) | 16:41 |
| AJaeger | clarkb: removed the WIP. | 16:41 |
| AJaeger | fungi: would that work for you as well on 710390 ? | 16:42 |
| *** ijw has joined #openstack-infra | 16:43 | |
| *** ijw has quit IRC | 16:47 | |
| *** ykarel is now known as ykarel|afk | 16:48 | |
| *** ociuhandu has quit IRC | 16:49 | |
| openstackgerrit | Andreas Jaeger proposed openstack/project-config master: Rework stackviz elements publishing https://review.opendev.org/710488 | 16:50 |
| AJaeger | config-core, the stackviz publishing failed again, I had to rework, could you put this on your review queue, please? ^ | 16:50 |
| openstackgerrit | Jan Kubovy proposed zuul/zuul master: Consolidate scheduler pause/exit as hibernation https://review.opendev.org/709723 | 16:51 |
| clarkb | AJaeger: why is https://review.opendev.org/#/c/710488/1/playbooks/javascript/post-stackviz.yaml using pre_tasks? | 16:51 |
| clarkb | AJaeger: I think regular tasks is fine since we don't run any roles in that same play? | 16:52 |
| *** lucasagomes has quit IRC | 16:53 | |
| *** ociuhandu has joined #openstack-infra | 16:54 | |
| AJaeger | clarkb, I first want to run the task, then the fetch-java role | 16:54 |
| clarkb | AJaeger: yes I think that will happen even wtih tasks because they are in separate - hosts: play blocks | 16:55 |
| clarkb | if you put them in the same - hosts: all block then pre_tasks would be the way to do that | 16:55 |
| AJaeger | I guess I thought I removed the second "hosts: all" ;) | 16:55 |
| AJaeger | let me remove the line | 16:55 |
| clarkb | ok | 16:55 |
| AJaeger | thanks, clarkb | 16:56 |
| openstackgerrit | Andreas Jaeger proposed openstack/project-config master: Rework stackviz elements publishing https://review.opendev.org/710488 | 16:56 |
| openstackgerrit | Monty Taylor proposed openstack/project-config master: Add gerrit's zuul/jobs repo https://review.opendev.org/710489 | 16:56 |
| mordred | corvus: ^ | 16:56 |
| corvus | mordred: ++ | 16:59 |
| corvus | waiting on 2 executors to finish stopping | 16:59 |
| corvus | down to 1 | 17:07 |
| corvus | and i'm not sure what it's doing | 17:07 |
| *** ijw has joined #openstack-infra | 17:07 | |
| corvus | ok, looks like really slow unlinks | 17:08 |
| *** rpittau is now known as rpittau|afk | 17:09 | |
| *** tosky has quit IRC | 17:09 | |
| fungi | i/o contention from noisy neighbors (who might just also be our nodepool nodes)? | 17:10 |
| *** michael-beaver has joined #openstack-infra | 17:11 | |
| corvus | seems plausible [probable?] | 17:11 |
| corvus | it is making progress | 17:11 |
| fungi | clouds | 17:11 |
| openstackgerrit | Jan Kubovy proposed zuul/zuul master: Refactor `self.event_queues` in tests https://review.opendev.org/709990 | 17:12 |
| *** ociuhandu has quit IRC | 17:13 | |
| *** jcapitao has quit IRC | 17:14 | |
| *** gagehugo has quit IRC | 17:21 | |
| *** ociuhandu has joined #openstack-infra | 17:32 | |
| openstackgerrit | Jan Kubovy proposed zuul/zuul master: Scheduler's pause/resume functionality https://review.opendev.org/709735 | 17:34 |
| *** evrardjp has quit IRC | 17:35 | |
| *** evrardjp has joined #openstack-infra | 17:35 | |
| *** ociuhandu has quit IRC | 17:35 | |
| *** ociuhandu has joined #openstack-infra | 17:36 | |
| *** ociuhandu has quit IRC | 17:38 | |
| *** ociuhandu has joined #openstack-infra | 17:39 | |
| *** ykarel|afk is now known as ykarel | 17:39 | |
| *** ociuhandu has quit IRC | 17:44 | |
| corvus | #status log restarted zuul executors on commit a25bab2f3c3b190fce5b872790dd60f955e5d29c | 17:45 |
| openstackstatus | corvus: finished logging | 17:45 |
| *** roman_g has quit IRC | 17:48 | |
| *** jpena is now known as jpena|off | 17:50 | |
| openstackgerrit | Jan Kubovy proposed zuul/zuul master: WIP: Store unparsed branch config in Zookeeper https://review.opendev.org/705716 | 17:50 |
| AJaeger | clarkb: fungi gave +2 on 710390 (thanks, fungi) - if you want to merge and delete the file afterwards... | 18:02 |
| *** derekh has quit IRC | 18:02 | |
| clarkb | AJaeger: I don't see the +2 | 18:03 |
| clarkb | is my browser caching poorly? | 18:03 |
| AJaeger | clarkb: neither do I - I misread my email ;( | 18:03 |
| AJaeger | clarkb: sorry, complete different change | 18:03 |
| * AJaeger needs a weekend ;( | 18:03 | |
| clarkb | no worries, and you should enjoy your weekend :) | 18:03 |
| AJaeger | thanks | 18:04 |
| AJaeger | config-core, I would appreciate a second review on https://review.opendev.org/710488 to hopefully finish this, please. | 18:04 |
| fungi | i'm happy to give a +2 on it if that's addressed, sorry | 18:05 |
| fungi | spread very thin today | 18:05 |
| *** jamesmcarthur has quit IRC | 18:06 | |
| AJaeger | fungi: clarkb will remove the file manually... | 18:09 |
| *** kozhukalov has quit IRC | 18:10 | |
| clarkb | ya I'll remove it then run apache reload | 18:10 |
| clarkb | (or systemctl reload apache2 | 18:11 |
| fungi | approved it | 18:11 |
| *** imacdonn has quit IRC | 18:13 | |
| AJaeger | thanks | 18:16 |
| openstackgerrit | Merged openstack/project-config master: Add gerrit's zuul/jobs repo https://review.opendev.org/710489 | 18:16 |
| *** kozhukalov has joined #openstack-infra | 18:20 | |
| *** amoralej is now known as amoralej|off | 18:22 | |
| *** ccamacho has quit IRC | 18:24 | |
| *** gfidente has quit IRC | 18:25 | |
| openstackgerrit | Zen Kurosaky proposed zuul/zuul master: This patch extends zuul deployment from scratch documentation. it adds guide on MySQL setup it extends zuul-from-scratch setup by adding mysql connection to zuul.conf file it extends gerrit setup by adding Verified label to gerrit it exten https://review.opendev.org/710504 | 18:31 |
| *** ociuhandu has joined #openstack-infra | 18:34 | |
| *** diablo_rojo has joined #openstack-infra | 18:34 | |
| *** ociuhandu has quit IRC | 18:39 | |
| *** ralonsoh has quit IRC | 18:44 | |
| *** ykarel is now known as ykarel|away | 18:48 | |
| clarkb | infra-root https://review.opendev.org/#/c/710478/ passed check testing and will update review.o.o to use the new LE cert. I'm happy to hit +A when I'm ready to babysit if yall review it and are happy with it | 18:48 |
| * fungi takes a break to look at that | 18:49 | |
| *** jamesmcarthur has joined #openstack-infra | 18:49 | |
| clarkb | I expect I'll approve it (assuming it looks good) after lunch and a bike ride | 18:49 |
| *** jamesmcarthur_ has joined #openstack-infra | 18:50 | |
| fungi | clarkb: approved it now | 18:50 |
| fungi | i can keep an eye on it | 18:50 |
| fungi | go enjoy your lunch/outside | 18:50 |
| clarkb | that also works, thanks | 18:51 |
| clarkb | fwiw I'll be around for the next little bit but was unsure of how long it would take to land | 18:51 |
| fungi | i need an excuse to not be staring at ancient vulnerability reports for a bit | 18:51 |
| *** jamesmcarthur has quit IRC | 18:54 | |
| *** jamesmcarthur_ has quit IRC | 18:54 | |
| *** mattw4 has quit IRC | 18:54 | |
| clarkb | we've had spring early here but the rain is supposed to return early afternoon. need to get outside before then :) | 18:54 |
| fungi | go for it | 18:55 |
| clarkb | ya current plan is to find food and sun after cleaning up static once that qa vhost change lands | 18:55 |
| prometheanfire | I don't think it's a thing, but is there an email header that gerrit sends out that shows the owner of the review (I'd like to sort my review into a folder separate than the ones I'm reviewing) | 18:55 |
| *** jamesmcarthur has joined #openstack-infra | 18:56 | |
| clarkb | prometheanfire: lack of header info was one of my concerns with zbr's proposal that we remove data from the emails themselves | 18:56 |
| clarkb | prometheanfire: I think you have to filter on the subject and content | 18:56 |
| prometheanfire | it looks like 'my reviews' don't contain the reply-to header | 18:56 |
| *** jamesmcarthur_ has joined #openstack-infra | 18:56 | |
| clarkb | you get change id and project and branch in the header iirc | 18:56 |
| fungi | prometheanfire: i don't recall. have you looked at the headers on messages you've received? | 18:56 |
| clarkb | and thats it | 18:56 |
| prometheanfire | fungi: ya, I'm looking at them now | 18:57 |
| fungi | i tend to delete them after looking, so don't have any examples handy | 18:57 |
| zbr | afaik, gerrit provides a good set of msg headers to allow filtering | 18:57 |
| fungi | i do some fairly specific filtering and discarding on gerrit e-mails myself | 18:58 |
| * fungi checks his .forward filters | 18:58 | |
| zbr | but in the end we need to experiment a little bit various subject formats till we get it right | 18:58 |
| openstackgerrit | Monty Taylor proposed opendev/system-config master: Use prepare-gerrit-repo role for gerrit submodules https://review.opendev.org/709602 | 18:58 |
| zbr | i am willing to provide feedback on that. | 18:58 |
| zbr | in fact is not only about subject but also about the body | 18:59 |
| zbr | we need to priorietize important information on the first part of them. | 18:59 |
| prometheanfire | I'll try to see if I can have a sieve filter for a header or missing header | 18:59 |
| zbr | like the first ~100chars of each. | 19:00 |
| fungi | yeah, actually it looks like most of my exim .forward filters are against patterns for $message_body | 19:00 |
| *** jamesmcarthur has quit IRC | 19:00 | |
| clarkb | zbr: it only sets a small number of them. I noted them on your email thread when I responded | 19:01 |
| fungi | i apparantly played a bit of whack-a-mole with patterns in gerrit messages i didn't want to see, and then sent anything else where $h_list-id: contains "review.opendev.org" into my reviews inbox | 19:01 |
| clarkb | "I don't think Gerrit's email templating system allows us to edit headers. Looking at an email from Gerrit today the only headers it seems to set are X-Gerrit-Change-Id, X-Gerrit-ChangeURL, and X-Gerrit-Commit." | 19:01 |
| fungi | so my experience suggests you need a mail filtering solution which will let you match on patterns in the message body | 19:02 |
| clarkb | fungi: yup | 19:02 |
| fungi | because there's just not rich enough metadata in the headers | 19:02 |
| fungi | but i agree, that's some great feedback/feature request for gerrit upstream | 19:03 |
| prometheanfire | I'm gonna try with the missing header for that | 19:03 |
| *** imacdonn has joined #openstack-infra | 19:04 | |
| openstackgerrit | Clark Boylan proposed opendev/system-config master: Increase timeout for run static job https://review.opendev.org/710507 | 19:07 |
| openstackgerrit | James E. Blair proposed zuul/zuul master: web ui: fix buildset display when no builds https://review.opendev.org/710508 | 19:07 |
| clarkb | infra-root ^ fyi the qa vhost cleanup failed in the gate due to a timeout on that job. I've increased it there | 19:07 |
| clarkb | but also I've rechecked as check worked fine. I Think we are close to the edge but not well over it | 19:08 |
| AJaeger | mordred: could I trouble you with reviewing https://review.opendev.org/710488 , please? I'd like to get it tested on the next periodic run... | 19:11 |
| *** gagehugo has joined #openstack-infra | 19:11 | |
| openstackgerrit | Zen Kurosaky proposed zuul/zuul master: This patch extends zuul deployment from scratch documentation. https://review.opendev.org/710504 | 19:17 |
| *** jamesmcarthur_ has quit IRC | 19:19 | |
| *** ijw has quit IRC | 19:19 | |
| openstackgerrit | Merged opendev/system-config master: Use LE cert on review.open*.org https://review.opendev.org/710478 | 19:19 |
| *** mattw4 has joined #openstack-infra | 19:22 | |
| openstackgerrit | Zen Kurosaky proposed zuul/zuul master: This patch extends zuul deployment from scratch documentation. https://review.opendev.org/710504 | 19:22 |
| prometheanfire | someone mind commenting on https://review.opendev.org/710510 | 19:22 |
| *** jamesmcarthur has joined #openstack-infra | 19:26 | |
| *** jamesmcarthur_ has joined #openstack-infra | 19:26 | |
| AJaeger | prometheanfire: done | 19:27 |
| prometheanfire | oh, when I get it I should hopefully have it marked as important | 19:27 |
| prometheanfire | doesn't look like that did it | 19:28 |
| prometheanfire | Guess I parse the body for Gerrit-Owner | 19:29 |
| *** jamesmcarthur has quit IRC | 19:30 | |
| *** EmilienM is now known as EvilienM | 19:30 | |
| *** roman_g has joined #openstack-infra | 19:30 | |
| openstackgerrit | Monty Taylor proposed opendev/system-config master: Use prepare-gerrit-repo role for gerrit submodules https://review.opendev.org/709602 | 19:31 |
| prometheanfire | AJaeger: mind one more time? | 19:32 |
| *** jamesmcarthur_ has quit IRC | 19:33 | |
| roman_g | Hello team. Has Zuul recovered from problems? I'm still seeing NODE_FAILURE messages quite often. However no more RETRY_LIMIT messages. | 19:36 |
| roman_g | NODE_FAILURE should be provider end problems. How do I get full logs? | 19:36 |
| clarkb | roman_g: I looked it up earlier today and confirmed its the same error I sent email about a few weeks back. Nova reports no valid host is found | 19:38 |
| roman_g | clarkb: Good. Thank you. | 19:39 |
| clarkb | roman_g: one thing we can check with robert and jan-erik is if a different region would work better | 19:39 |
| fungi | to clarify, we've been reulctant to expose our nodepool launcher logs directly to the public, over concerns that openstacksdk might expose credential details, though i think we've all come around to deciding that's not a risk | 19:40 |
| *** lyarwood has quit IRC | 19:40 | |
| fungi | so maybe it's time to revisit that option | 19:40 |
| clarkb | fungi: also zuul should expose these errors too | 19:42 |
| *** dpawlik has quit IRC | 19:43 | |
| clarkb | roman_g: but basically I understand this to mean that cloud region doesnot have capacity for the requested nodes | 19:43 |
| fungi | oh, right, we talked about ways to plumb exception details back through from nodepool errors through zuul to reporting | 19:43 |
| *** dpawlik has joined #openstack-infra | 19:43 | |
| roman_g | clarkb, yes, that's right. Quota != reservation. | 19:43 |
| roman_g | I'm fine with getting logs exposed on e.g. http://logstash.openstack.org, not necessarily in Zuul. | 19:44 |
| prometheanfire | AJaeger: one last time? ordering issue | 19:44 |
| *** mattw4 has quit IRC | 19:44 | |
| prometheanfire | ah, you did | 19:44 |
| AJaeger | prometheanfire: I can do one more ;) | 19:45 |
| *** mattw4 has joined #openstack-infra | 19:46 | |
| clarkb | roman_g: ya its just something that is planned for exposing on the zuul side as it will help other zuul installs too | 19:47 |
| prometheanfire | for some reason this isn't marking, AJaeger maybe now? | 19:49 |
| *** jamesmcarthur has joined #openstack-infra | 19:58 | |
| *** jamesmcarthur_ has joined #openstack-infra | 19:59 | |
| prometheanfire | doesn't seem to flag :( | 20:00 |
| *** jamesmcarthur has quit IRC | 20:03 | |
| prometheanfire | so, the filter matches but it doesn't flag, I guess that's good enough for my use case | 20:03 |
| prometheanfire | https://gist.github.com/prometheanfire/4e200db2de609b4922afa37bdcef6f7c | 20:03 |
| prometheanfire | hmm, no, it'd still hit both rules without the stop | 20:05 |
| prometheanfire | AJaeger: ok, lets try without the catchall :D | 20:06 |
| prometheanfire | AJaeger: does the body check look right to you? | 20:06 |
| openstackgerrit | Monty Taylor proposed opendev/system-config master: Use prepare-gerrit-repo role for gerrit submodules https://review.opendev.org/709602 | 20:10 |
| AJaeger | prometheanfire: I commented again, not sure what you want to achieve to comment on your gist | 20:11 |
| prometheanfire | that's the filter | 20:11 |
| mordred | fungi: yah - I'm definitely confident at this point that we will not leak creds if we expose those | 20:11 |
| prometheanfire | AJaeger: anyway, it looks like the filter works, but the flagging does not | 20:12 |
| *** jamesmcarthur_ has quit IRC | 20:18 | |
| *** jamesmcarthur has joined #openstack-infra | 20:19 | |
| *** jamesmcarthur_ has joined #openstack-infra | 20:20 | |
| *** jamesmcarthur has quit IRC | 20:24 | |
| fungi | clarkb: ianw: https://review.opendev.org/ "Verified by: Let's Encrypt" | 20:36 |
| fungi | lgtm | 20:36 |
| corvus | mordred: why is gerrit_additional_plugins required? with the new role, shouldn't anything added to required-projects work? | 20:36 |
| fungi | #status log added missing reverse dns entries for mirror01.ord.rax.opendev.org | 20:40 |
| openstackstatus | fungi: finished logging | 20:40 |
| mordred | corvus: yup! I came around to figuring that out | 20:41 |
| mordred | corvus: there is an edge case where we'll break - which is if a plugin doesn't have a matching branch and is also not a submodule | 20:41 |
| mordred | but we don't have that case | 20:41 |
| mordred | in this case, the issue I was having was lack of an updated mapping dict containing system-config | 20:41 |
| mordred | corvus: I spoke too soon: error: pathspec 'plugins/its-storyboard' did not match any file(s) known to git. | 20:43 |
| mordred | corvus: I'll post a link to more info in a bit when it's finished and published | 20:44 |
| *** ociuhandu has joined #openstack-infra | 20:48 | |
| AJaeger | config-core, I would appreciate a second review on https://review.opendev.org/710488 to hopefully finish this, please. | 20:49 |
| *** ociuhandu has quit IRC | 20:52 | |
| AJaeger | thanks, fungi | 20:53 |
| *** slaweq has quit IRC | 20:53 | |
| *** jamesmcarthur_ has quit IRC | 21:03 | |
| mordred | corvus: so that theoretical issue I mentioned is not theoretical | 21:05 |
| mordred | corvus: we use plugins that have neither submodules nor matching branches | 21:06 |
| corvus | mordred: so what's it hitting that's a problem? | 21:06 |
| mordred | corvus: https://zuul.opendev.org/t/openstack/build/f6ecc430aba54e6b853a9c27ed4094ac/log/job-output.txt#6137-6146 | 21:06 |
| mordred | corvus: the fallback case for non-matching branch is to do a submodule update | 21:06 |
| openstackgerrit | Merged openstack/project-config master: Rework stackviz elements publishing https://review.opendev.org/710488 | 21:07 |
| mordred | but if it's not a submodule, then we shoudl just copy it into the repo | 21:07 |
| mordred | (into the plugins dir that is) | 21:07 |
| *** nicolasbock has quit IRC | 21:07 | |
| corvus | mordred: okay, so we need a test to see if the project is in gitmodules, it sounds like? | 21:11 |
| mordred | yeah | 21:11 |
| mordred | corvus: oh! neat - new failure in a different way (although this one is fine) | 21:11 |
| *** sreejithp has joined #openstack-infra | 21:12 | |
| corvus | and we'll do the submodule update: when: ["not project_branch_exists", "project_in_gitmodules"] and the move otherwise | 21:12 |
| mordred | on the branches where the gitmodules thing wasnt' an issue (2.15 and master fwiw) - we failed because the bazelisk-build role from gerrit overrode the bazelisk-build role from system-confifg | 21:12 |
| mordred | corvus: yah | 21:12 |
| corvus | mordred: are they different? | 21:13 |
| mordred | corvus: yeah - the one in system-config moves the release.war somewhere | 21:13 |
| mordred | I can obviously move that difference | 21:13 |
| corvus | mordred: should upstream do that? | 21:14 |
| corvus | oh, it moves it to an opendev-container specific place | 21:14 |
| corvus | nm | 21:14 |
| corvus | mordred: i haven't paged this in -- but should we go ahead and have that role move it to zuul-output/artifacts and then have the opendev thing move it from there? | 21:15 |
| mordred | corvus: that's a good idae | 21:15 |
| mordred | corvus: I can push up a patch with the move to zuul-output/artifacts | 21:17 |
| mordred | corvus: one other thing - don't know if we care -but bazelisk-build in gerrit runs tests - which we don't do in opendev | 21:18 |
| mordred | should we add a flag for skipping tests? or just go ahead and run them anyway | 21:18 |
| mordred | (in our case it's going to make us run the whole gerrit suite I think) | 21:18 |
| openstackgerrit | Monty Taylor proposed opendev/system-config master: Use prepare-gerrit-repo role for gerrit submodules https://review.opendev.org/709602 | 21:27 |
| mordred | corvus: that should do the trick on the cp - and pushed https://gerrit-review.googlesource.com/c/zuul/jobs/+/257252 upstream | 21:27 |
| clarkb | fungi: yay | 21:31 |
| corvus | mordred: -1 on the gerrit change -- bu regarding tests -- does it? i thought that only happens if the tests argument is != '' ? | 21:31 |
| clarkb | infra-root https://review.opendev.org/#/c/710507/ is a quick review to address a timeout issue on the newly expanded system-config-run-static job | 21:31 |
| clarkb | if we get that in we can land the qa vhost removal more reliably | 21:32 |
| corvus | clarkb: ouch | 21:32 |
| clarkb | I think it is the afs build that hurts us there | 21:33 |
| clarkb | because it does the dkms thing and builds against the current kernel | 21:33 |
| corvus | yeah. i guess it's for the best. | 21:35 |
| clarkb | mordred: I feel like http://lists.openstack.org/pipermail/openstack-discuss/2020-February/012914.html is something you probably know the answer to | 21:35 |
| clarkb | mordred knower of all esoteric cloud behaviors | 21:35 |
| *** dciabrin has quit IRC | 21:39 | |
| *** ijw has joined #openstack-infra | 21:44 | |
| *** mattw4 has quit IRC | 21:46 | |
| * paladox voted in the labour leadership contest today | 21:47 | |
| *** dwalt has joined #openstack-infra | 21:51 | |
| *** imacdonn has quit IRC | 21:54 | |
| *** imacdonn has joined #openstack-infra | 21:54 | |
| mordred | clarkb: I actually don't - but I can say gtema has a patch up to make OSC use SDK for image stuff | 21:56 |
| mordred | clarkb: https://review.opendev.org/#/c/699416/ | 21:56 |
| mordred | clarkb: so - I have no idea what OSC is doing now | 21:57 |
| openstackgerrit | James E. Blair proposed opendev/system-config master: DNM: Test: Use explicit provides/requires for container jobs https://review.opendev.org/710532 | 21:59 |
| openstackgerrit | Merged opendev/system-config master: Kill qa.o.o https://review.opendev.org/710390 | 21:59 |
| *** matt_kosut has joined #openstack-infra | 22:01 | |
| clarkb | infra-root I'm going to take the host vars/hiera lock now and cleanup the gerrit ssl cert data | 22:02 |
| *** imacdonn has quit IRC | 22:02 | |
| *** imacdonn has joined #openstack-infra | 22:03 | |
| *** xek_ has joined #openstack-infra | 22:03 | |
| *** xek__ has quit IRC | 22:05 | |
| *** matt_kosut has quit IRC | 22:05 | |
| clarkb | and done | 22:06 |
| dwalt | hey all! It seems as though our post jobs are not running over on airship/airshipctl. Our config hasn't changed in a few days. Are there any known issues? These are the jobs that typically run https://opendev.org/airship/airshipctl/src/branch/master/zuul.d/projects.yaml | 22:14 |
| clarkb | dwalt: http://zuul.openstack.org/build/5f89939b27704c2c8499d2cfc55da26b seems to be the most recent run from the 25th and it failed | 22:16 |
| clarkb | dwalt: do you expect more recent runs? | 22:16 |
| clarkb | looks like there have been more recent merges so yes probably | 22:16 |
| dwalt | clarkb: indeed. Especially the git mirroring job | 22:17 |
| dwalt | that was successful in its last run | 22:17 |
| *** sreejithp has quit IRC | 22:20 | |
| *** rlandy has quit IRC | 22:21 | |
| *** matt_kosut has joined #openstack-infra | 22:21 | |
| clarkb | (I'm looking in logs so may be a few minutes) | 22:23 |
| clarkb | 2020-02-28 22:11:23,315 DEBUG zuul.Pipeline.openstack.post: [e: b2a507bbc6194a78ace197d0ed0e3aa3] Event <GerritTriggerEvent change-merged opendev.org/airship/airshipctl master 710454,5> for change <Change 0x7fd2f11da940 airship/airshipctl 710454,5> does not match <GerritEventFilter types: ref-updated refs: ^refs/heads/.*$ ignore_deletes: True> in pipeline <SupercedentPipelineManager post> because | 22:23 |
| clarkb | False | 22:23 |
| clarkb | I think that is zuul telling us why it didn't run any post jobs. But I'm not sure I understand the underlying why yet | 22:23 |
| clarkb | oh right bceause post is for ref updated not change-merged events | 22:24 |
| * clarkb looks for more events | 22:24 | |
| *** tosky has joined #openstack-infra | 22:24 | |
| *** matt_kosut has quit IRC | 22:25 | |
| clarkb | if I do something like `grep 710454 /var/log/zuul/debug.log | grep Event | grep -v change-merged | grep -v comment-added` I see no events around 22:11UTC which is when we should get the ref updated event | 22:26 |
| clarkb | corvus: ^ did we maybe start filtering those out? | 22:26 |
| clarkb | though we get the ref updated when the patchset is created | 22:26 |
| clarkb | as a sanity check we've run post jobs on other repos 7 minutes ago and other recent jobs | 22:27 |
| clarkb | both gerrit and gitea know about that commit | 22:28 |
| clarkb | https://review.opendev.org/#/c/710454/ ^ is "that commit" | 22:28 |
| *** kozhukalov has quit IRC | 22:28 | |
| clarkb | oh right I can't filter on change number to see those | 22:30 |
| clarkb | dwalt: 2020-02-28 22:11:23,792 DEBUG zuul.Pipeline.openstack.post: [e: 2bb67d3ad8a34ae2ad07a982e6a2f815] <QueueItem 0x7fd222b90320 for <Branch 0x7fd2f30fff60 airship/airshipctl refs/heads/master updated 84e677344d51035eb2ea9d848eeb0cc31aade1e1..f66f94dc80c4b54a0d0b34c3a33193e4e91eb99b> in post> is a failing item because ['it has an invalid configuration'] | 22:32 |
| clarkb | dwalt: I don't know yet what hte invalid configuration is but that is a good breadcrumb | 22:32 |
| dwalt | clarkb: thanks for tracking this down :) | 22:32 |
| dwalt | the docs and image jobs may need work | 22:32 |
| dwalt | would they impact the success of the git mirroring job? | 22:33 |
| clarkb | dwalt: Exception: Unable to modify final job <Job trigger-readthedocs-webhook branches: None source: opendev/base-jobs/zuul.d/jobs.yaml@master#25> attribute roles=(<ZuulRole opendev.org/airship/airshipctl airshipctl>,) with variant <Job airship-airshipctl-publish-docs branches: None source: airship/airshipctl/zuul.d/jobs.yaml@master#79' | 22:33 |
| clarkb | dwalt: in this case it impacts that job beacuse it can't create the post set of jobs | 22:33 |
| clarkb | the issue is you've modified a job marked final | 22:33 |
| clarkb | I'm looking for an example of how you should use that docs job | 22:34 |
| clarkb | dwalt: https://opendev.org/openstack/project-config/src/branch/master/zuul.d/projects.yaml#L72-L78 basically you use the job as is then set a project level vars value which will be passed into the job without modifying it | 22:35 |
| dwalt | clarkb: thanks so much | 22:36 |
| clarkb | dwalt: to start if you drop that job entirely (in the pipeline config and its definition) that should get the other jobs going | 22:36 |
| clarkb | then you can add it back in using the project level var (or just fix it in a go) | 22:36 |
| clarkb | *in one go | 22:36 |
| dwalt | I will do that. We don't have a tox file in that project (it's written in go), so that's why the config is weird. But I'll just go ahead and add one | 22:37 |
| clarkb | dwalt: you don't need a tox file | 22:37 |
| clarkb | this is all in zuul config | 22:37 |
| clarkb | the zuul config is composed in layers. You have the top level project which defines the pipelines that are used and what jobs to execute in those pipelines. What we want to do is set a var at the project level which the job set in the pipeline can consume | 22:38 |
| clarkb | (if you'd like I can push up the change that I think will work) | 22:38 |
| dwalt | Oh, I thought that it brought in the openstack-tox-docs. My misunderstanding | 22:42 |
| clarkb | dwalt: something like remote: https://review.opendev.org/710535 Fix read the docs post job | 22:42 |
| clarkb | (whether or not rtd can build your docs without some python infrastructure I don't know but that would be up to your project config there and their apis | 22:43 |
| clarkb | I expect that change will make zuul happy though | 22:43 |
| dwalt | clarkb: That's awesome. Thanks for the help and change | 22:44 |
| dwalt | much appreciated | 22:44 |
| clarkb | you're welcome | 22:45 |
| clarkb | dwalt: I also expect that merging that change will trigger all the post jobs and get things synced up again | 22:46 |
| *** xek__ has joined #openstack-infra | 22:46 | |
| *** xek_ has quit IRC | 22:49 | |
| *** stevebaker has joined #openstack-infra | 22:50 | |
| clarkb | I'm going to single core approve https://review.opendev.org/#/c/710507/ since that should make system-config jobs more stable and its late on friday and everyone must be weekending :) | 22:54 |
| *** mattw4 has joined #openstack-infra | 22:54 | |
| *** EvilienM is now known as EmilienM | 22:56 | |
| openstackgerrit | Merged zuul/zuul master: Stream output from kubectl pods https://review.opendev.org/709261 | 23:07 |
| *** dpawlik has quit IRC | 23:13 | |
| clarkb | qa.openstack.org has been removed from apache's config on static.opendev.org. If you hit that url you end up at the root of static (default vhost) | 23:18 |
| *** kozhukalov has joined #openstack-infra | 23:18 | |
| clarkb | I think we should delete dns for this site too, but maybe we double check on that? | 23:18 |
| clarkb | ianw: gmann AJaeger ^ I think that is a question for yall, no rush | 23:18 |
| clarkb | #status log Removed qa.openstack.org apache config from static.opendev.org. DNS still needs cleanup if we are comfortable with that. | 23:21 |
| openstackstatus | clarkb: finished logging | 23:21 |
| clarkb | thank you helpful bot | 23:21 |
| openstackgerrit | Merged opendev/system-config master: Increase timeout for run static job https://review.opendev.org/710507 | 23:28 |
| *** kozhukalov has quit IRC | 23:28 | |
| *** mattw4 has quit IRC | 23:29 | |
| *** kozhukalov has joined #openstack-infra | 23:34 | |
| *** stevebaker has quit IRC | 23:40 | |
| *** jcoufal has quit IRC | 23:42 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!