| *** ryohayakawa has joined #openstack-infra | 00:02 | |
| *** tetsuro has joined #openstack-infra | 00:02 | |
| *** markvoelker has joined #openstack-infra | 00:08 | |
| *** markvoelker has quit IRC | 00:12 | |
| openstackgerrit | Andrii Ostapenko proposed zuul/zuul-jobs master: Add ability to use (upload|promote)-docker-image roles in periodic jobs https://review.opendev.org/740560 | 00:13 |
|---|---|---|
| openstackgerrit | Andrii Ostapenko proposed zuul/zuul-jobs master: Add ability to use (upload|promote)-docker-image roles in periodic jobs https://review.opendev.org/740560 | 00:14 |
| *** markvoelker has joined #openstack-infra | 00:28 | |
| *** rfolco has quit IRC | 00:32 | |
| *** markvoelker has quit IRC | 00:33 | |
| *** Lucas_Gray has joined #openstack-infra | 00:33 | |
| *** jamesmcarthur has joined #openstack-infra | 00:50 | |
| openstackgerrit | Merged openstack/project-config master: Update ceph grafana for nova https://review.opendev.org/742516 | 00:59 |
| *** Goneri has quit IRC | 01:49 | |
| *** Goneri has joined #openstack-infra | 01:50 | |
| *** Goneri has quit IRC | 01:50 | |
| *** jamesmcarthur has quit IRC | 01:51 | |
| *** jamesmcarthur has joined #openstack-infra | 01:55 | |
| *** Goneri has joined #openstack-infra | 01:59 | |
| *** Goneri has quit IRC | 01:59 | |
| *** Goneri has joined #openstack-infra | 02:02 | |
| *** Goneri has quit IRC | 02:02 | |
| *** Goneri has joined #openstack-infra | 02:04 | |
| *** jamesmcarthur has quit IRC | 02:15 | |
| *** Goneri has quit IRC | 02:20 | |
| *** Goneri has joined #openstack-infra | 02:23 | |
| openstackgerrit | Ian Wienand proposed openstack/project-config master: Add opendev-3pci/project-config https://review.opendev.org/743143 | 02:25 |
| openstackgerrit | Ian Wienand proposed openstack/project-config master: Add OpenDev 3rd-party CI tenant https://review.opendev.org/743144 | 02:25 |
| *** Lucas_Gray has quit IRC | 02:25 | |
| *** jamesmcarthur has joined #openstack-infra | 02:27 | |
| *** Goneri has quit IRC | 02:31 | |
| *** rcernin has quit IRC | 02:31 | |
| *** jamesmcarthur has quit IRC | 02:33 | |
| *** armax has joined #openstack-infra | 02:35 | |
| *** jamesmcarthur has joined #openstack-infra | 02:36 | |
| *** armax has quit IRC | 02:37 | |
| *** jamesmcarthur has quit IRC | 02:37 | |
| *** ramishra has joined #openstack-infra | 02:39 | |
| *** jamesmcarthur has joined #openstack-infra | 02:39 | |
| *** rcernin has joined #openstack-infra | 02:40 | |
| *** rcernin has quit IRC | 02:40 | |
| *** rcernin has joined #openstack-infra | 02:41 | |
| *** jamesmcarthur has quit IRC | 02:53 | |
| *** dchen is now known as dchen|away | 02:57 | |
| *** jamesmcarthur has joined #openstack-infra | 03:35 | |
| *** psachin has joined #openstack-infra | 03:39 | |
| *** ricolin has joined #openstack-infra | 03:41 | |
| *** chandankumar has joined #openstack-infra | 03:44 | |
| *** jamesmcarthur has quit IRC | 03:45 | |
| *** jamesmcarthur has joined #openstack-infra | 03:45 | |
| *** fdegir2 has joined #openstack-infra | 03:48 | |
| *** fdegir has quit IRC | 03:49 | |
| *** jamesmcarthur has quit IRC | 03:52 | |
| *** udesale has joined #openstack-infra | 04:07 | |
| *** ykarel has joined #openstack-infra | 04:26 | |
| *** dchen|away is now known as dchen | 04:27 | |
| *** dave-mccowan has quit IRC | 04:37 | |
| *** markvoelker has joined #openstack-infra | 04:38 | |
| *** markvoelker has quit IRC | 04:42 | |
| *** dchen is now known as dchen|away | 04:44 | |
| *** Goneri has joined #openstack-infra | 04:52 | |
| *** Goneri has quit IRC | 05:03 | |
| *** Goneri has joined #openstack-infra | 05:16 | |
| *** ysandeep|away is now known as ysandeep | 05:19 | |
| *** Goneri has quit IRC | 05:22 | |
| *** Goneri has joined #openstack-infra | 05:23 | |
| *** dchen|away is now known as dchen | 05:25 | |
| *** Goneri has quit IRC | 05:30 | |
| *** Goneri has joined #openstack-infra | 05:32 | |
| *** lmiccini has joined #openstack-infra | 05:32 | |
| *** Goneri has quit IRC | 05:34 | |
| *** Goneri has joined #openstack-infra | 05:37 | |
| *** Goneri has quit IRC | 05:43 | |
| *** Goneri has joined #openstack-infra | 05:46 | |
| *** rcernin has quit IRC | 05:50 | |
| *** rcernin has joined #openstack-infra | 05:51 | |
| *** markvoelker has joined #openstack-infra | 05:58 | |
| *** markvoelker has quit IRC | 06:03 | |
| *** marios has joined #openstack-infra | 06:03 | |
| *** udesale has quit IRC | 06:08 | |
| *** udesale has joined #openstack-infra | 06:10 | |
| *** rcernin has quit IRC | 06:21 | |
| *** rcernin has joined #openstack-infra | 06:22 | |
| *** Goneri has quit IRC | 06:26 | |
| *** xek has joined #openstack-infra | 06:38 | |
| *** tosky has joined #openstack-infra | 06:42 | |
| *** eolivare has joined #openstack-infra | 06:43 | |
| *** ysandeep is now known as ysandeep|rover | 06:46 | |
| *** Goneri has joined #openstack-infra | 06:59 | |
| *** jcapitao has joined #openstack-infra | 07:10 | |
| *** lpetrut has joined #openstack-infra | 07:10 | |
| *** andrewbonney has joined #openstack-infra | 07:11 | |
| *** jpena|off is now known as jpena | 07:21 | |
| *** dpawlik2 has joined #openstack-infra | 07:23 | |
| *** hashar has joined #openstack-infra | 07:24 | |
| *** rcernin has quit IRC | 07:28 | |
| *** Goneri has quit IRC | 07:35 | |
| *** ricolin has quit IRC | 07:43 | |
| *** ociuhandu has joined #openstack-infra | 07:49 | |
| *** Tengu has quit IRC | 07:51 | |
| *** nightmare_unreal has joined #openstack-infra | 07:52 | |
| *** ralonsoh has joined #openstack-infra | 07:53 | |
| openstackgerrit | Ian Wienand proposed openstack/project-config master: Add opendev-3pci/project-config https://review.opendev.org/743143 | 07:56 |
| openstackgerrit | Ian Wienand proposed openstack/project-config master: Add OpenDev 3rd-party CI tenant https://review.opendev.org/743144 | 07:56 |
| *** lucasagomes has joined #openstack-infra | 07:58 | |
| *** Tengu has joined #openstack-infra | 07:58 | |
| *** ramishra has quit IRC | 07:59 | |
| *** ramishra has joined #openstack-infra | 08:01 | |
| *** tosky has quit IRC | 08:03 | |
| *** Tengu has quit IRC | 08:04 | |
| *** dtantsur|afk is now known as dtantsur | 08:05 | |
| *** Tengu has joined #openstack-infra | 08:06 | |
| *** Tengu has quit IRC | 08:11 | |
| *** dchen is now known as dchen|away | 08:14 | |
| *** ysandeep|rover is now known as ysandeep|lunch | 08:15 | |
| *** vishalmanchanda has joined #openstack-infra | 08:29 | |
| *** rcernin has joined #openstack-infra | 08:30 | |
| *** rpittau has joined #openstack-infra | 08:34 | |
| *** rcernin has quit IRC | 08:35 | |
| *** fdegir2 is now known as fdegir | 08:40 | |
| *** psachin has quit IRC | 08:42 | |
| *** ysandeep|lunch is now known as ysandeep|rover | 08:42 | |
| *** pkopec has joined #openstack-infra | 08:43 | |
| *** derekh has joined #openstack-infra | 08:48 | |
| *** xek has quit IRC | 09:06 | |
| *** tetsuro has quit IRC | 09:08 | |
| *** tetsuro has joined #openstack-infra | 09:09 | |
| *** Lucas_Gray has joined #openstack-infra | 09:10 | |
| *** Tengu has joined #openstack-infra | 09:12 | |
| *** Lucas_Gray has quit IRC | 09:16 | |
| *** tetsuro has quit IRC | 09:16 | |
| *** rcernin has joined #openstack-infra | 09:25 | |
| *** rcernin has quit IRC | 09:30 | |
| *** tosky has joined #openstack-infra | 09:33 | |
| *** Lucas_Gray has joined #openstack-infra | 09:52 | |
| *** tkajinam has quit IRC | 10:10 | |
| *** rcernin has joined #openstack-infra | 10:11 | |
| *** rcernin has quit IRC | 10:15 | |
| *** lpetrut has quit IRC | 10:17 | |
| *** psachin has joined #openstack-infra | 10:22 | |
| *** artom has joined #openstack-infra | 10:35 | |
| *** markvoelker has joined #openstack-infra | 10:49 | |
| *** dchen|away is now known as dchen | 10:50 | |
| *** markvoelker has quit IRC | 10:54 | |
| *** iurygregory has quit IRC | 10:59 | |
| *** dchen is now known as dchen|away | 11:00 | |
| *** iurygregory has joined #openstack-infra | 11:01 | |
| *** jcapitao is now known as jcapitao_lunch | 11:02 | |
| *** dchen|away is now known as dchen | 11:03 | |
| *** ryohayakawa has quit IRC | 11:03 | |
| *** ysandeep|rover is now known as ysandeep|afk | 11:04 | |
| *** xek has joined #openstack-infra | 11:10 | |
| *** xek has quit IRC | 11:21 | |
| *** rcernin has joined #openstack-infra | 11:23 | |
| *** ysandeep|afk is now known as ysandeep|rover | 11:29 | |
| *** psachin has quit IRC | 11:34 | |
| *** pkopec has quit IRC | 11:36 | |
| *** jpena is now known as jpena|lunch | 11:38 | |
| *** pkopec has joined #openstack-infra | 11:48 | |
| *** xek has joined #openstack-infra | 11:51 | |
| *** udesale_ has joined #openstack-infra | 12:00 | |
| *** rfolco has joined #openstack-infra | 12:00 | |
| *** udesale has quit IRC | 12:03 | |
| *** rlandy has joined #openstack-infra | 12:04 | |
| *** psachin has joined #openstack-infra | 12:07 | |
| *** dchen is now known as dchen|away | 12:08 | |
| *** mordred has joined #openstack-infra | 12:21 | |
| *** markvoelker has joined #openstack-infra | 12:23 | |
| *** derekh has quit IRC | 12:25 | |
| *** dave-mccowan has joined #openstack-infra | 12:30 | |
| *** evrardjp has quit IRC | 12:32 | |
| *** evrardjp has joined #openstack-infra | 12:32 | |
| *** evrardjp has quit IRC | 12:32 | |
| *** evrardjp has joined #openstack-infra | 12:33 | |
| *** jcapitao_lunch is now known as jcapitao | 12:33 | |
| *** jpena|lunch is now known as jpena | 12:34 | |
| *** dave-mccowan has quit IRC | 12:37 | |
| *** KeithMnemonic has joined #openstack-infra | 12:46 | |
| *** lbragstad has joined #openstack-infra | 12:53 | |
| openstackgerrit | Artom Lifshitz proposed openstack/project-config master: Add nested-virt-ubuntu-focal label https://review.opendev.org/743217 | 12:55 |
| *** eharney has joined #openstack-infra | 12:57 | |
| openstackgerrit | Artom Lifshitz proposed openstack/project-config master: Add nested-virt-ubuntu-focal label https://review.opendev.org/743217 | 13:02 |
| *** Lucas_Gray has quit IRC | 13:10 | |
| *** Lucas_Gray has joined #openstack-infra | 13:21 | |
| *** derekh has joined #openstack-infra | 13:22 | |
| *** dtantsur is now known as dtantsur|brb | 13:22 | |
| *** rcernin has quit IRC | 13:29 | |
| *** ramishra has quit IRC | 13:46 | |
| *** ramishra has joined #openstack-infra | 13:48 | |
| *** mnasiadka has joined #openstack-infra | 13:49 | |
| *** d34dh0r53 has joined #openstack-infra | 13:52 | |
| *** ramishra has quit IRC | 13:58 | |
| *** ramishra has joined #openstack-infra | 14:00 | |
| *** zxiiro has joined #openstack-infra | 14:05 | |
| *** __ministry1 has joined #openstack-infra | 14:11 | |
| *** lpetrut has joined #openstack-infra | 14:14 | |
| *** hashar has quit IRC | 14:16 | |
| *** ysandeep|rover is now known as ysandeep|away | 14:17 | |
| *** rcernin has joined #openstack-infra | 14:22 | |
| *** bhagyashris is now known as bhagyashris|away | 14:24 | |
| *** __ministry1 has quit IRC | 14:25 | |
| *** psachin has quit IRC | 14:31 | |
| *** ricolin has joined #openstack-infra | 14:31 | |
| *** lpetrut has quit IRC | 14:49 | |
| *** rcernin has quit IRC | 14:56 | |
| *** Lucas_Gray has quit IRC | 14:59 | |
| *** beekneemech is now known as bnemec | 15:00 | |
| *** dtantsur|brb is now known as dtantsur | 15:06 | |
| *** Lucas_Gray has joined #openstack-infra | 15:14 | |
| *** dklyle has joined #openstack-infra | 15:18 | |
| *** __ministry1 has joined #openstack-infra | 15:22 | |
| *** __ministry1 has quit IRC | 15:22 | |
| *** armax has joined #openstack-infra | 15:26 | |
| melwitt | we merged an update for the ceph grafana to point at the "nova-ceph-multistore" job since we tweaked the job and updated the name https://review.opendev.org/742516 but the dashboard is still showing the old job name http://grafana.openstack.org/d/-iKINcImz/ceph-failure-rate?orgId=1 | 15:29 |
| melwitt | anyone have any ideas why it's not reflecting the change? | 15:29 |
| clarkb | melwitt: my first hunch was it merged after the zuul update andbefore we fixed our application jobs but that merged after the fix | 15:30 |
| clarkb | perhaps a bug with the containerized grafana | 15:31 |
| *** ykarel is now known as ykarel|away | 15:33 | |
| melwitt | hm ok | 15:39 |
| melwitt | I found I have a copy-paste error in the stable/ussuri stuff that I forgot to replace train -> ussuri but I also notice the stable/ussuri panel isn't on grafana.o.o either. so something not picking up the new thing | 15:40 |
| clarkb | I'm finishing breakfast then can look at the server | 15:41 |
| *** jamesmcarthur has joined #openstack-infra | 15:42 | |
| *** lpetrut has joined #openstack-infra | 15:42 | |
| melwitt | thanks | 15:44 |
| *** gyee has joined #openstack-infra | 15:47 | |
| clarkb | melwitt: https://zuul.opendev.org/t/openstack/builds?job_name=infra-prod-service-grafana that shows the job I would expect to update things hasn't run recently | 15:50 |
| clarkb | I think the issue is that we update the configs in openstack/project-config but update the server via job in opendev/system-config and have to tie the two together | 15:50 |
| clarkb | previously this was all driven by puppet and would have worked, so I do suspect the containerization just missed this | 15:51 |
| melwitt | oh hm | 15:51 |
| clarkb | I'm going to see what we do for other project-config bits that related to container things and see if I can fix it | 15:51 |
| openstackgerrit | Clark Boylan proposed openstack/project-config master: Run the grafana ansnible when grafana dashboard change https://review.opendev.org/743270 | 15:56 |
| clarkb | melwitt: I think ^ fixes that disconnect, then we can land something trivial and see that it works | 15:56 |
| melwitt | ok great. I have something trivial I need to run bc I made the copy-paste error (forgot to replace train with ussuri for the ussuri panel) | 15:57 |
| melwitt | thank you! | 15:57 |
| clarkb | cool we can use that fix to test it then | 15:57 |
| clarkb | maybe push it with a depends on to https://review.opendev.org/743270 ? | 15:57 |
| clarkb | or restack it I guess since same repo | 15:58 |
| melwitt | yeah, was about to ask if I should stack it on top | 15:58 |
| *** lucasagomes has quit IRC | 16:00 | |
| *** marios has quit IRC | 16:02 | |
| openstackgerrit | melanie witt proposed openstack/project-config master: Fix copy-paste error for the ceph grafana nova ussuri panel https://review.opendev.org/743273 | 16:03 |
| *** markvoelker has quit IRC | 16:03 | |
| *** pkopec has quit IRC | 16:08 | |
| *** ramishra has quit IRC | 16:09 | |
| *** jackedin has joined #openstack-infra | 16:14 | |
| *** ykarel|away has quit IRC | 16:15 | |
| *** jcapitao has quit IRC | 16:15 | |
| *** ricolin has quit IRC | 16:16 | |
| *** lmiccini has quit IRC | 16:16 | |
| *** udesale_ has quit IRC | 16:21 | |
| melwitt | clarkb: woot http://grafana.openstack.org/d/-iKINcImz/ceph-failure-rate?orgId=1 | 16:27 |
| clarkb | hrm the fixes didn't merge, but maybe the job ran periodically or something? | 16:28 |
| melwitt | oh heh, I wasn't sure if the proposal itself could kick something off? | 16:28 |
| *** hashar has joined #openstack-infra | 16:29 | |
| *** Goneri has joined #openstack-infra | 16:38 | |
| *** jpena is now known as jpena|off | 16:41 | |
| *** xek has quit IRC | 16:46 | |
| *** rcernin has joined #openstack-infra | 16:54 | |
| *** hamalq has joined #openstack-infra | 16:55 | |
| *** rcernin has quit IRC | 16:58 | |
| *** ociuhandu has quit IRC | 16:59 | |
| *** dtantsur is now known as dtantsur|afk | 16:59 | |
| *** nicolasbock has joined #openstack-infra | 16:59 | |
| *** Lucas_Gray has quit IRC | 16:59 | |
| *** derekh has quit IRC | 17:00 | |
| *** andrewbonney has quit IRC | 17:09 | |
| *** ociuhandu has joined #openstack-infra | 17:15 | |
| *** ociuhandu has quit IRC | 17:20 | |
| *** nightmare_unreal has quit IRC | 17:20 | |
| *** ralonsoh has quit IRC | 17:23 | |
| *** ralonsoh has joined #openstack-infra | 17:23 | |
| *** jamesmcarthur has quit IRC | 17:26 | |
| *** jamesmcarthur has joined #openstack-infra | 17:27 | |
| *** eolivare has quit IRC | 17:45 | |
| *** ociuhandu has joined #openstack-infra | 17:50 | |
| *** ykarel has joined #openstack-infra | 17:53 | |
| *** __ministry1 has joined #openstack-infra | 17:54 | |
| *** ociuhandu has quit IRC | 17:55 | |
| *** ykarel has quit IRC | 18:01 | |
| *** ralonsoh has quit IRC | 18:08 | |
| *** __ministry1 has quit IRC | 18:10 | |
| *** ociuhandu has joined #openstack-infra | 18:14 | |
| *** vishalmanchanda has quit IRC | 18:18 | |
| *** ociuhandu has quit IRC | 18:19 | |
| *** evrardjp has quit IRC | 18:21 | |
| *** evrardjp has joined #openstack-infra | 18:24 | |
| *** lpetrut has quit IRC | 18:33 | |
| *** tosky has quit IRC | 18:33 | |
| openstackgerrit | Ghanshyam Mann proposed openstack/openstack-zuul-jobs master: Migrate unit tests jobs to run on Ubuntu Focal https://review.opendev.org/738323 | 18:40 |
| openstackgerrit | Ghanshyam Mann proposed openstack/openstack-zuul-jobs master: Migrate unit tests jobs to run on Ubuntu Focal https://review.opendev.org/738323 | 18:40 |
| openstackgerrit | Ghanshyam Mann proposed openstack/openstack-zuul-jobs master: Migrate openstack-tox-cover jobs on Ubuntu Focal https://review.opendev.org/738324 | 18:40 |
| openstackgerrit | Ghanshyam Mann proposed openstack/openstack-zuul-jobs master: Migrate functional tests jobs to Ubuntu Focal https://review.opendev.org/738325 | 18:41 |
| openstackgerrit | Ghanshyam Mann proposed openstack/openstack-zuul-jobs master: Migrate openstack-tox-docs jobs to Ubuntu Focal https://review.opendev.org/738326 | 18:41 |
| openstackgerrit | Ghanshyam Mann proposed openstack/openstack-zuul-jobs master: Pin ubuntu bionic for tox-py27 and tox-py36 in no-constraints template https://review.opendev.org/738327 | 18:41 |
| openstackgerrit | Ghanshyam Mann proposed openstack/openstack-zuul-jobs master: Pin nodejs6 & nodejs8 jobs on ubuntu-bionic https://review.opendev.org/738328 | 18:41 |
| clarkb | gmann: what we have done before is kept the old python testing on the old platform | 18:41 |
| clarkb | gmann: then switch newer python testing to the new platform. Eventually if/when you stop caring about the old python you can drop those jobs | 18:42 |
| gmann | clarkb: yeah I update that way now. running python3.8 jobs on focal and older on odler distro (bionic or xenial) | 18:43 |
| *** gouthamr has left #openstack-infra | 19:10 | |
| *** gouthamr has joined #openstack-infra | 19:11 | |
| gouthamr | hello o/ i've a job that didn't fire, and was looking to understand why: https://review.opendev.org/742987 introduces a new check pipeline job called "manila-tempest-dsvm-lvm-em-only" on an older stable branch, but it didn't show up | 19:12 |
| gouthamr | it was backported from stable/rocky, where it worked fine: https://review.opendev.org/742724/ | 19:13 |
| clarkb | gouthamr: if the parent jobs are undefined on the current brnach it will be skipped. I expect that is what is happening here | 19:14 |
| clarkb | however its parented to manila-tempest-base which seems like the sort of thing that would be present | 19:14 |
| gouthamr | clarkb: oh, you're right!~ | 19:14 |
| clarkb | though it isn't in the change itself so maybe not | 19:15 |
| gouthamr | clarkb: that base job was called "legacy-manila-tempest-dsvm-base" in that branch | 19:15 |
| gouthamr | clarkb: ty, was a head-scratcher - i'll be happy to get rid of these legacy jobs soon | 19:15 |
| clarkb | no worries | 19:16 |
| *** jackedin has quit IRC | 19:32 | |
| bnemec | Any idea what happened with the promote job in https://review.opendev.org/#/c/741029/ ? | 19:39 |
| bnemec | The error is: "There was an issue creating /afs/.openstack.org as requested: [Errno 13] Permission denied: b'/afs/.openstack.org'" | 19:40 |
| clarkb | bnemec: yes, that was a service start ordering issue on the zuul executor. We needed openafs to be ready before starting docker containers for zuul but that wasn't happening. We've since fixed that by restarting the containers and have a change in flight to ensure the order on boot going forward | 19:40 |
| clarkb | if you land a new docs change it should update properly going forward. If its important and a new docs change isn't appropriate we can probably reenqueue that job | 19:41 |
| clarkb | fungi: ^ fyi | 19:41 |
| bnemec | Should be fine. It doesn't look like there was any actual docs change in that patch anyway. | 19:42 |
| bnemec | Just wanted to make sure it wasn't a recurring problem. | 19:42 |
| *** jamesmcarthur has quit IRC | 19:43 | |
| fungi | it was definitely a recurring problem | 19:44 |
| fungi | we've restarted the containers for the affected executors for now, and https://review.opendev.org/743316 should prevent it from recurring (once it merges) | 19:44 |
| fungi | if anyone finds builds which need their refs reenqueued let us know. i think we're finally in the clear with all the release job cleanup | 19:45 |
| *** SpamapS has quit IRC | 19:46 | |
| *** xek has joined #openstack-infra | 19:48 | |
| *** SpamapS has joined #openstack-infra | 19:50 | |
| *** jamesmcarthur has joined #openstack-infra | 19:51 | |
| *** jamesmcarthur has quit IRC | 19:55 | |
| *** jamesmcarthur has joined #openstack-infra | 19:56 | |
| *** hashar has quit IRC | 20:02 | |
| *** ociuhandu has joined #openstack-infra | 20:10 | |
| *** jamesmcarthur has quit IRC | 20:14 | |
| *** samueldmq has joined #openstack-infra | 20:17 | |
| *** jamesmcarthur has joined #openstack-infra | 20:17 | |
| weshay|ruck | anything going on w/ upstream mirrors? TripleO is seeing a spike in failures trying to pull containers http://logstash.openstack.org/#/dashboard/file/logstash.json?query=message:%5C%22_attach_volume%5C%22%20AND%20module:%5C%22oslo_messaging.rpc.server%5C%22%20AND%20tags:%5C%22screen-c-bak.txt%5C%22%20AND%20voting:1&from=864000s | 20:26 |
| weshay|ruck | bah.. bad link | 20:26 |
| weshay|ruck | better link http://logstash.openstack.org/#dashboard/file/logstash.json?query=message%3A%5C%22429%20Client%20Error%3A%20Too%20Many%20Requests%20for%20url%3A%5C%22%20AND%20voting%3A1 | 20:27 |
| *** Lucas_Gray has joined #openstack-infra | 20:31 | |
| *** sean-k-mooney has joined #openstack-infra | 20:31 | |
| sean-k-mooney | o/ | 20:31 |
| sean-k-mooney | i just noticed once of the nova docs promote jobs failed with an afs issue | 20:32 |
| sean-k-mooney | https://zuul.opendev.org/t/openstack/build/52ac4a6fbc7643989e3828a5be25c7b2/log/job-output.txt#129-133 | 20:32 |
| sean-k-mooney | msg": "There was an issue creating /afs/.openstack.org as requested: [Errno 13] Permission denied: b'/afs/.openstack.org'", | 20:32 |
| sean-k-mooney | just wanted to check if ye were aware of that | 20:33 |
| sean-k-mooney | incase its affecting others too | 20:33 |
| *** ociuhandu has quit IRC | 20:33 | |
| weshay|ruck | sean-k-mooney, sounds like that may impact mirrors too? | 20:33 |
| sean-k-mooney | the next time a nova patch merges teh promote jobs will run so its not really an issue | 20:33 |
| clarkb | sean-k-mooney: weshay|ruck I think they are separate issues | 20:34 |
| weshay|ruck | k | 20:34 |
| clarkb | sean-k-mooney: your issue was related to afs mounts on the zuul executors not being done before containers started. We've fixed that by restarting the containers and added systemd boot ordering ot have docker start after afs | 20:34 |
| sean-k-mooney | clarkb: cool | 20:34 |
| clarkb | weshay|ruck: your issue is going through the mirrors which run no containers so shouldn't have the same issue | 20:35 |
| sean-k-mooney | i assumed it was a trasiant issue i just got a zull email and noticed it failed. | 20:35 |
| clarkb | weshay|ruck: your issue is dockerhub rate limiting us | 20:35 |
| weshay|ruck | OH NO | 20:35 |
| clarkb | weshay|ruck: last week I think that came up because rdo was doing a million releases or something | 20:36 |
| weshay|ruck | the day has come?? | 20:36 |
| *** slaweq has quit IRC | 20:36 | |
| weshay|ruck | crap | 20:36 |
| clarkb | and ya'll were going to address that by not trying to consume every single one of them or something | 20:36 |
| clarkb | but the too many requests is coming from docker hub | 20:36 |
| weshay|ruck | ok.. well tbh I'm suprised it took so long | 20:36 |
| weshay|ruck | k | 20:36 |
| clarkb | and generally we don't see it if we're able to use cached layers because we rarely hit docker hub | 20:36 |
| clarkb | but if a ton of new layers are all published and we need to pull all of them ... | 20:37 |
| sean-k-mooney | clarkb: does infra have a registry? it might make sense for rdo to push to an infra registry and then to an hourly or dayly sync to docker hub | 20:37 |
| clarkb | sean-k-mooney: no at least not for that purpose | 20:37 |
| clarkb | we run a small registry just for speculate states | 20:37 |
| clarkb | we don't intend to actually host production images | 20:37 |
| fungi | the dockerhub rate limiting has been cropping up regularly for at least a couple of months | 20:38 |
| sean-k-mooney | oh im not talking about production images | 20:38 |
| sean-k-mooney | i ment for them to pull for use in ci | 20:38 |
| clarkb | sean-k-mooney: thats what the mirror is | 20:38 |
| clarkb | sean-k-mooney: we cache the image layers from docker hub | 20:38 |
| fungi | likely we just don't have enough room to cache all those images and keep rewarming | 20:38 |
| clarkb | fungi: well in this case it was specifically related to like many many many new layers | 20:38 |
| clarkb | fungi: which ya makes it hard to cache but also likely to hit rate limits anyway | 20:38 |
| sean-k-mooney | clarkb: i think triplo are currenly trying to remove there use of kolla and instead intoduce there own image building system | 20:39 |
| sean-k-mooney | maybe its related to that | 20:39 |
| clarkb | there are a number of issues related to running a proper registry though, major one is pruning. Since we only keep speculative states in the registry we do run we can simply rm those that are older than $time and be fine | 20:41 |
| clarkb | granted that also required zuul writing its own registry because the existing ones don't do that | 20:42 |
| sean-k-mooney | i think the quay registry at least support rm | 20:42 |
| sean-k-mooney | i always found it odd that the docker one does not | 20:42 |
| clarkb | quay isn't open source ? | 20:43 |
| sean-k-mooney | i think after redhat aquired coreos it became opensources | 20:43 |
| sean-k-mooney | https://github.com/quay/quay | 20:43 |
| clarkb | https://www.openshift.com/products/quay | 20:44 |
| clarkb | huh weird that they don't even talk about the open source version there | 20:44 |
| sean-k-mooney | its now apache2 | 20:44 |
| sean-k-mooney | its a redhat product page | 20:44 |
| sean-k-mooney | they tend not to tell you where to get the free opensouce versions | 20:45 |
| clarkb | anyway I have very little interest in running another registry that is intended to replace docker hub | 20:45 |
| clarkb | for basiclly the same reasons we don't mirror pypi anymore | 20:45 |
| clarkb | ENODISK | 20:45 |
| sean-k-mooney | well not really a replacment i was thinking of a buffer | 20:45 |
| clarkb | yes we hvae that already is my point | 20:45 |
| clarkb | we cache layers with apache | 20:45 |
| sean-k-mooney | you have a read cache | 20:45 |
| sean-k-mooney | not a write cache | 20:45 |
| clarkb | yes, we don't need a write cache | 20:45 |
| sean-k-mooney | is the failure on downloading the images | 20:46 |
| clarkb | yes | 20:46 |
| sean-k-mooney | i tought it was uploading, oh ok dont mind me then :) | 20:46 |
| clarkb | aiui its all upstream of tripleo the layers are being made | 20:46 |
| clarkb | then tripleo trying to pull and consume them | 20:46 |
| sean-k-mooney | well that is changing | 20:46 |
| sean-k-mooney | tripleo used to use kolla images | 20:46 |
| sean-k-mooney | but they are looking into replacing that | 20:47 |
| clarkb | and writes wouldn't be a huge concern because tehy would be direct | 20:47 |
| sean-k-mooney | with there own images | 20:47 |
| clarkb | and so you'd get automagic IP randomization | 20:47 |
| clarkb | I wonder if we could give mirrors a second IP then default route balance out them to double our requestsl imits | 20:54 |
| clarkb | I imagine that afs udp may get sad about that | 20:54 |
| clarkb | thtas probably overthinking it, the short term fix is likely to stop using the mirror while there is thrashing | 20:54 |
| weshay|ruck | master and ussuri are no longer using kolla images.. centos8 train is being worked now.. the images are smaller now.. but still a lot there to xfer | 20:57 |
| *** mnencia has quit IRC | 20:59 | |
| *** mnencia_ has joined #openstack-infra | 20:59 | |
| *** mnencia_ is now known as mnencia | 20:59 | |
| *** jamesmcarthur has quit IRC | 21:07 | |
| *** jamesmcarthur has joined #openstack-infra | 21:08 | |
| *** jamesmcarthur has quit IRC | 21:22 | |
| *** tosky has joined #openstack-infra | 21:25 | |
| *** jamesmcarthur has joined #openstack-infra | 21:27 | |
| *** jamesmcarthur has quit IRC | 21:32 | |
| *** jamesmcarthur has joined #openstack-infra | 21:34 | |
| *** zxiiro has quit IRC | 21:45 | |
| *** SpamapS has quit IRC | 21:52 | |
| fungi | yeah, it's unfortunate that us putting a caching proxy in front of hundreds of machines (and thousands of ip addresses) causes dockerhub to throttle access when *not* having that caching proxy would put *far* more load on their service | 21:52 |
| fungi | they're basically discouraging us from relieving some of that load | 21:53 |
| *** rcernin has joined #openstack-infra | 21:53 | |
| clarkb | oh weshay|ruck's email mentions using quay as an option and that would likely also help | 21:54 |
| clarkb | we should be proxying quay already iirc | 21:54 |
| fungi | yeah, pretty sure we have a port for it | 21:55 |
| weshay|ruck | aye.. I think Emilien added that a while ago.. :) | 21:55 |
| weshay|ruck | quay also rate limits | 21:55 |
| weshay|ruck | going to keep discussing it while I put up some patches to mitigate it as much as possible | 21:56 |
| *** SpamapS has joined #openstack-infra | 21:57 | |
| *** rcernin has quit IRC | 21:58 | |
| *** rcernin has joined #openstack-infra | 21:59 | |
| *** d34dh0r53 has quit IRC | 22:11 | |
| openstackgerrit | Ian Wienand proposed openstack/project-config master: Add opendev-3pci/project-config https://review.opendev.org/743143 | 22:30 |
| openstackgerrit | Ian Wienand proposed openstack/project-config master: Add OpenDev 3rd-party CI tenant https://review.opendev.org/743144 | 22:30 |
| *** jamesmcarthur has quit IRC | 22:31 | |
| *** jamesmcarthur has joined #openstack-infra | 22:31 | |
| *** tkajinam has joined #openstack-infra | 22:52 | |
| *** xek has quit IRC | 22:59 | |
| *** rlandy is now known as rlandy|bbl | 23:03 | |
| *** dtroyer has quit IRC | 23:03 | |
| openstackgerrit | Merged openstack/project-config master: Run the grafana ansnible when grafana dashboard change https://review.opendev.org/743270 | 23:07 |
| openstackgerrit | Merged openstack/project-config master: Fix copy-paste error for the ceph grafana nova ussuri panel https://review.opendev.org/743273 | 23:07 |
| *** tosky has quit IRC | 23:25 | |
| *** jamesmcarthur has quit IRC | 23:28 | |
| *** jamesmcarthur has joined #openstack-infra | 23:29 | |
| *** jamesmcarthur has quit IRC | 23:33 | |
| *** dchen|away is now known as dchen | 23:36 | |
Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!