| *** mattw4 has quit IRC | 00:00 | |
| *** weifan has joined #openstack-infra | 00:01 | |
| *** weifan has quit IRC | 00:01 | |
| *** weifan has joined #openstack-infra | 00:01 | |
| *** gyee has quit IRC | 00:05 | |
| *** weifan has quit IRC | 00:06 | |
| clarkb | mordred: we do get things like ./api/compute/GET/e8fd161dc34c421a979a9e6421f823e9/25d702bc-6c34-4fcd-8296-eda3001afd85/sum_squares_90.wsp | 00:07 |
|---|---|---|
| clarkb | is that GET/tenant/compute-uuid/ ? | 00:08 |
| clarkb | if so then compute may also suffer this | 00:08 |
| calebb | clarkb: i think im not far from being able to test the mirror thing on a vm | 00:08 |
| clarkb | calebb: ok like tomorrow? | 00:09 |
| calebb | if you wanna wait until im super ready (if it's annoying for you to have that quota taken up or whatever) i can wait more, but otherwise i might be able to test things over the weekend | 00:09 |
| calebb | yeah, i think so | 00:09 |
| clarkb | can probably spin one up tomorrow | 00:09 |
| calebb | i basically have a few small things to clean up | 00:09 |
| calebb | ok, sounds good | 00:10 |
| mordred | clarkb: weird. ok. I mean - that is not a very useful thing to track | 00:14 |
| mordred | clarkb: I'm going to mull ways to make that better overall | 00:14 |
| *** jamesmcarthur has joined #openstack-infra | 00:14 | |
| clarkb | mordred: reskimming that function I think https://opendev.org/openstack/openstacksdk/src/branch/master/openstack/proxy.py#L133-L136 may be broken. Maybe there is a type difference? | 00:14 |
| mordred | clarkb, corvus: I ran the stack ending in https://review.opendev.org/690766 from bridge against review-dev and all seems happy | 00:15 |
| clarkb | but also we don't actually exclude the id of the resource I don't think | 00:15 |
| clarkb | whihc is why we end up with the id in the comput epaths | 00:15 |
| clarkb | oh wait the consume function is what should pull out the uuid of the actual resource so that must be buggy too | 00:15 |
| *** rfolco|rover has joined #openstack-infra | 00:16 | |
| clarkb | https://opendev.org/openstack/openstacksdk/src/branch/master/openstack/proxy.py#L176 we only look at every other entry? | 00:16 |
| clarkb | that seems like a bug | 00:17 |
| *** slaweq has joined #openstack-infra | 00:17 | |
| mordred | clarkb, corvus: so I think we can land that stack | 00:17 |
| mordred | well - it was an early attempt to exclude the ids from the urls - since what you want are the signatures of teh general calls, not the specific calls to objects | 00:18 |
| *** goldyfruit has quit IRC | 00:18 | |
| clarkb | sure but why would it be every other entry? | 00:18 |
| clarkb | that doesn't seem like a reliable metric | 00:18 |
| *** andreaf has joined #openstack-infra | 00:18 | |
| clarkb | I think that is one bug | 00:18 |
| mordred | because the pattern is usually /server/{id}/action or /image/{id} or whatnot | 00:18 |
| mordred | you can't just exclude ids based on them being a uuid because there's non-uuid ids | 00:19 |
| clarkb | mordred: then you need to flip the condition then | 00:19 |
| clarkb | oh wait its adding them to a list ok | 00:19 |
| mordred | I think what we need to do, now that we have per-server proxy subclasses, is not try to do things like "skip every other url segment" but instead have the compute proxy have a method that says "this is how to report my urls" | 00:20 |
| clarkb | oh wait no that is the bug | 00:20 |
| clarkb | I think because we aren't removingthe project/tenant id properly we get shifted by one | 00:20 |
| mordred | yeah. I agree with that | 00:20 |
| clarkb | and then we evaluate the uuid instead of the action in that function | 00:20 |
| mordred | yup | 00:20 |
| clarkb | being specific in the subclasses makes sense to me | 00:21 |
| mordred | way more scalable - because there's likely to be a bazillion edge cases | 00:21 |
| *** slaweq has quit IRC | 00:22 | |
| mordred | oh - you know what - I bet self.session.get_project_id() in the call to _extract_names is wrong and it should be self.get_project_id() | 00:22 |
| * mordred goes to check | 00:23 | |
| clarkb | you think it is returning none or something like that? | 00:23 |
| clarkb | none is the default value to the method too | 00:23 |
| mordred | nope. nevermind -they both have it | 00:24 |
| mordred | yeah- my thought was that we were not stripping the project_id because we weren't passing it in | 00:24 |
| mordred | but self.session.get_project_id() totally has the project id - so that shouldn't be it | 00:24 |
| mordred | (like, I think we should rework this - but if we can fix the lack of stripping the project id properly, that should buy us a little more space to work in) | 00:25 |
| clarkb | ya | 00:26 |
| clarkb | I'm fairly certain it is the list comprehension but the logic there apperas corret to me | 00:26 |
| clarkb | which is why I think it must be a typing issue? | 00:26 |
| mordred | yeah - like unicode vs bytes or something? | 00:27 |
| *** markvoelker has joined #openstack-infra | 00:27 | |
| *** jamesmcarthur has quit IRC | 00:27 | |
| *** jamesmcarthur has joined #openstack-infra | 00:28 | |
| clarkb | or uuid is an int or has the -'s stripped out when we get project id but in the url its got the -'s | 00:28 |
| mordred | project_id should always be a string - it comes from a field of a json dict | 00:29 |
| mordred | but - worth looking at | 00:29 |
| *** markvoelker has quit IRC | 00:30 | |
| *** markvoelker has joined #openstack-infra | 00:30 | |
| mordred | clarkb: do we have any things like ./api/compute/GET/e8fd161dc34c421a979a9e6421f823e9/25d702bc-6c34-4fcd-8296-eda3001afd85/sum_squares_90.wsp that are attached to a specific cloud? | 00:31 |
| clarkb | mordred: no it does http method then uuids or servers or os-volumes-boot etc | 00:32 |
| clarkb | I believe the uuids are project/tenant uuids | 00:32 |
| clarkb | I need to figure out dinner | 00:35 |
| mordred | dinner is important | 00:35 |
| clarkb | the whisper resize is running in a root screen | 00:35 |
| clarkb | I expect it will be a while | 00:35 |
| *** weifan has joined #openstack-infra | 00:45 | |
| *** benj_ has quit IRC | 00:50 | |
| *** weifan has quit IRC | 00:50 | |
| *** ricolin has joined #openstack-infra | 00:50 | |
| *** jamesmcarthur has quit IRC | 00:53 | |
| *** markvoelker has quit IRC | 00:54 | |
| *** ricolin has quit IRC | 00:55 | |
| mordred | clarkb: I can't reproduce this locally - making the various calls both with and without additional logging and print statements - the things are doing what they should (this is both on clouds with project_id in the url and without) | 00:56 |
| *** markvoelker has joined #openstack-infra | 00:57 | |
| clarkb | mordred: maybe we can figure out which cloud theseproject ids come from and work backward from there? | 00:58 |
| mordred | yeah - I tried with one of those clouds | 00:58 |
| mordred | (rax is one of the ones with projectid/uuid whisper files) | 00:59 |
| *** diablo_rojo has quit IRC | 01:01 | |
| *** benj_ has joined #openstack-infra | 01:05 | |
| *** jamesmcarthur has joined #openstack-infra | 01:07 | |
| *** markvoelker has quit IRC | 01:08 | |
| *** Goneri has joined #openstack-infra | 01:10 | |
| *** jamesmcarthur has quit IRC | 01:19 | |
| *** Goneri has quit IRC | 01:20 | |
| *** weifan has joined #openstack-infra | 01:21 | |
| *** weifan has quit IRC | 01:26 | |
| *** jamesmcarthur has joined #openstack-infra | 01:43 | |
| *** jamesmcarthur has quit IRC | 01:46 | |
| openstackgerrit | Sean McGinnis proposed openstack/project-config master: Increase tag-releases timeout https://review.opendev.org/691139 | 01:50 |
| *** rfolco|rover has quit IRC | 01:51 | |
| openstackgerrit | zhurong proposed openstack/project-config master: Grant permissions for murano-dashboard to horizon-core https://review.opendev.org/691140 | 01:51 |
| *** rosmaita has quit IRC | 01:53 | |
| openstackgerrit | zhurong proposed openstack/project-config master: Grant permissions for solum-dashboard to horizon-core https://review.opendev.org/691142 | 01:54 |
| *** weifan has joined #openstack-infra | 02:01 | |
| openstackgerrit | zhurong proposed openstack/project-config master: Grant permissions for murano-dashboard to horizon-core https://review.opendev.org/691140 | 02:04 |
| *** rosmaita has joined #openstack-infra | 02:05 | |
| *** yamamoto has joined #openstack-infra | 02:06 | |
| *** weifan has quit IRC | 02:07 | |
| *** yamamoto has quit IRC | 02:11 | |
| *** ramishra has joined #openstack-infra | 02:28 | |
| openstackgerrit | zhurong proposed openstack/project-config master: Grant permissions for solum-dashboard to horizon-core https://review.opendev.org/691142 | 02:30 |
| *** yamamoto has joined #openstack-infra | 02:32 | |
| *** roman_g has quit IRC | 02:34 | |
| *** jerryz has quit IRC | 02:47 | |
| *** markvoelker has joined #openstack-infra | 03:09 | |
| *** benj_ has quit IRC | 03:10 | |
| *** yamamoto has quit IRC | 03:11 | |
| openstackgerrit | Monty Taylor proposed opendev/system-config master: Add service playbook and test run for prod gerrit https://review.opendev.org/691171 | 03:12 |
| openstackgerrit | Monty Taylor proposed opendev/system-config master: Add launchpadlib credentials to gerrit ansible https://review.opendev.org/691172 | 03:12 |
| openstackgerrit | Monty Taylor proposed opendev/system-config master: Add replication config to gerrit ansible https://review.opendev.org/691173 | 03:12 |
| *** markvoelker has quit IRC | 03:13 | |
| *** yamamoto has joined #openstack-infra | 03:13 | |
| openstackgerrit | zhangboye proposed openstack/os-testr master: Switch to Ussuri jobs https://review.opendev.org/691176 | 03:29 |
| *** yamamoto has quit IRC | 03:33 | |
| *** psachin has joined #openstack-infra | 03:34 | |
| *** yamamoto has joined #openstack-infra | 03:34 | |
| *** yamamoto has quit IRC | 03:35 | |
| *** benj_ has joined #openstack-infra | 03:40 | |
| *** benj_ has quit IRC | 03:53 | |
| *** janki has joined #openstack-infra | 03:56 | |
| *** ykarel|away has joined #openstack-infra | 03:57 | |
| *** yamamoto has joined #openstack-infra | 04:01 | |
| *** igordc has quit IRC | 04:01 | |
| *** benj_ has joined #openstack-infra | 04:05 | |
| *** ykarel|away is now known as ykarel | 04:06 | |
| *** yamamoto has quit IRC | 04:08 | |
| *** benj_ has quit IRC | 04:10 | |
| *** benj_ has joined #openstack-infra | 04:14 | |
| *** janki has quit IRC | 04:19 | |
| *** janki has joined #openstack-infra | 04:22 | |
| *** ricolin has joined #openstack-infra | 04:24 | |
| openstackgerrit | Merged openstack/diskimage-builder master: Ensure machine-id is not included in images https://review.opendev.org/683272 | 04:28 |
| *** kjackal has joined #openstack-infra | 04:31 | |
| *** ykarel is now known as ykarel|afk | 04:37 | |
| *** udesale has joined #openstack-infra | 04:44 | |
| openstackgerrit | Ian Wienand proposed openstack/diskimage-builder master: pip-and-virtualenv: include python3-venv for Debuntu https://review.opendev.org/691182 | 04:46 |
| *** ianychoi has quit IRC | 04:59 | |
| *** ykarel|afk is now known as ykarel | 05:00 | |
| *** ianychoi has joined #openstack-infra | 05:01 | |
| *** tkajinam has quit IRC | 05:01 | |
| openstackgerrit | Merged openstack/diskimage-builder master: Add security suite name override in debian-minimal https://review.opendev.org/688816 | 05:16 |
| *** sgw has joined #openstack-infra | 05:18 | |
| *** kjackal has quit IRC | 05:18 | |
| *** hamzy has quit IRC | 05:19 | |
| *** hamzy has joined #openstack-infra | 05:21 | |
| *** surpatil has joined #openstack-infra | 05:25 | |
| openstackgerrit | Ian Wienand proposed openstack/diskimage-builder master: pip-and-virtualenv: include python3-venv for Debuntu https://review.opendev.org/691182 | 05:27 |
| openstackgerrit | Ian Wienand proposed opendev/system-config master: [wip] test arm64 mirror https://review.opendev.org/690798 | 05:28 |
| openstackgerrit | Ian Wienand proposed opendev/system-config master: install-ansible: Pre-install build deps for aarch64 https://review.opendev.org/691187 | 05:28 |
| *** janki has quit IRC | 05:30 | |
| *** janki has joined #openstack-infra | 05:33 | |
| *** janki has quit IRC | 05:33 | |
| *** janki has joined #openstack-infra | 05:33 | |
| *** yamamoto has joined #openstack-infra | 05:34 | |
| *** dchen has quit IRC | 05:38 | |
| *** janki has quit IRC | 05:39 | |
| *** slaweq has joined #openstack-infra | 05:39 | |
| *** dchen has joined #openstack-infra | 05:40 | |
| *** dchen has quit IRC | 05:44 | |
| *** tkajinam has joined #openstack-infra | 05:44 | |
| *** slaweq has quit IRC | 05:44 | |
| *** slaweq has joined #openstack-infra | 05:45 | |
| *** dchen has joined #openstack-infra | 05:45 | |
| *** yamamoto has quit IRC | 05:51 | |
| *** yamamoto has joined #openstack-infra | 05:53 | |
| *** lmiccini has joined #openstack-infra | 05:57 | |
| *** yamamoto has quit IRC | 06:13 | |
| *** dpawlik has joined #openstack-infra | 06:17 | |
| *** jerryz has joined #openstack-infra | 06:18 | |
| *** tesseract has joined #openstack-infra | 06:19 | |
| *** jerryz has quit IRC | 06:24 | |
| *** udesale has quit IRC | 06:24 | |
| *** udesale has joined #openstack-infra | 06:25 | |
| *** rcernin has quit IRC | 06:26 | |
| *** udesale has quit IRC | 06:27 | |
| *** jtomasek has joined #openstack-infra | 06:36 | |
| *** kjackal has joined #openstack-infra | 06:42 | |
| *** apetrich has joined #openstack-infra | 06:50 | |
| openstackgerrit | Merged openstack/diskimage-builder master: Drop vhdutil dependency, use qemu-img https://review.opendev.org/688849 | 06:52 |
| *** udesale has joined #openstack-infra | 06:52 | |
| *** apetrich has quit IRC | 06:53 | |
| *** apetrich has joined #openstack-infra | 06:54 | |
| *** lmiccini has quit IRC | 06:57 | |
| *** sshnaidm is now known as sshnaidm|off | 06:58 | |
| *** lmiccini has joined #openstack-infra | 06:59 | |
| AJaeger | config-core, could you review https://review.opendev.org/#/c/688997/ to allow moving forward with python2 drop for Ussuri, please? | 07:00 |
| *** yamamoto has joined #openstack-infra | 07:00 | |
| *** ccamacho has joined #openstack-infra | 07:01 | |
| *** trident has quit IRC | 07:02 | |
| *** trident has joined #openstack-infra | 07:06 | |
| *** otsukahy has joined #openstack-infra | 07:07 | |
| AJaeger | thanks, mordred | 07:09 |
| *** markvoelker has joined #openstack-infra | 07:11 | |
| *** slaweq has quit IRC | 07:12 | |
| *** yamamoto has quit IRC | 07:16 | |
| *** markvoelker has quit IRC | 07:16 | |
| *** tosky has joined #openstack-infra | 07:18 | |
| *** gfidente has joined #openstack-infra | 07:19 | |
| openstackgerrit | Merged openstack/openstack-zuul-jobs master: Update openstack-python3-ussuri-jobs-* templates to test without Python2 https://review.opendev.org/688997 | 07:20 |
| openstackgerrit | zhurong proposed openstack/project-config master: Grant permissions for murano-dashboard to horizon-core https://review.opendev.org/691140 | 07:26 |
| *** yamamoto has joined #openstack-infra | 07:39 | |
| *** kopecmartin|off is now known as kopecmartin | 07:45 | |
| *** udesale has quit IRC | 07:49 | |
| *** udesale has joined #openstack-infra | 07:49 | |
| *** rpittau|afk is now known as rpittau | 07:51 | |
| openstackgerrit | Fabien Boucher proposed zuul/zuul master: Keep tenant_last_reconfigured float precision (time.sleep removal) https://review.opendev.org/689394 | 07:51 |
| openstackgerrit | Fabien Boucher proposed zuul/zuul master: Pagure - Manage project connector refresh when EINVALIDTOK https://review.opendev.org/687259 | 07:52 |
| *** zbr has joined #openstack-infra | 07:53 | |
| *** ralonsoh has joined #openstack-infra | 07:55 | |
| *** otsukahy has quit IRC | 07:55 | |
| *** yamamoto has quit IRC | 07:59 | |
| *** amoralej|off is now known as amoralej | 08:01 | |
| *** efried has quit IRC | 08:02 | |
| *** efried has joined #openstack-infra | 08:02 | |
| openstackgerrit | Felix Schmidt proposed zuul/zuul master: Make reporting asynchronous https://review.opendev.org/691253 | 08:03 |
| *** yamamoto has joined #openstack-infra | 08:04 | |
| *** rcernin has joined #openstack-infra | 08:09 | |
| *** lucasagomes has joined #openstack-infra | 08:09 | |
| *** tkajinam has quit IRC | 08:10 | |
| openstackgerrit | Fabien Boucher proposed zuul/zuul master: Pagure: Fix handling of threshold_reached: None https://review.opendev.org/691254 | 08:11 |
| *** whoami-rajat has joined #openstack-infra | 08:14 | |
| *** yolanda has quit IRC | 08:14 | |
| *** yolanda has joined #openstack-infra | 08:14 | |
| openstackgerrit | Felix Schmidt proposed zuul/zuul master: WIP: Make reporting asynchronous https://review.opendev.org/691253 | 08:16 |
| *** rcernin has quit IRC | 08:28 | |
| *** trident has quit IRC | 08:29 | |
| *** dpawlik has quit IRC | 08:34 | |
| *** trident has joined #openstack-infra | 08:35 | |
| *** derekh has joined #openstack-infra | 08:43 | |
| openstackgerrit | Luka Peschke proposed openstack/project-config master: Grant core permissions to horizon-core on cloudkitty-dashboard https://review.opendev.org/691263 | 08:46 |
| *** yamamoto has quit IRC | 08:59 | |
| *** dpawlik has joined #openstack-infra | 09:02 | |
| *** ykarel is now known as ykarel|lunch | 09:03 | |
| *** yamamoto has joined #openstack-infra | 09:04 | |
| *** markvoelker has joined #openstack-infra | 09:12 | |
| *** markvoelker has quit IRC | 09:17 | |
| *** xek_ has joined #openstack-infra | 09:19 | |
| *** prometheanfire has quit IRC | 09:20 | |
| *** prometheanfire has joined #openstack-infra | 09:21 | |
| *** otsukahy has joined #openstack-infra | 09:23 | |
| *** pkopec has joined #openstack-infra | 09:26 | |
| *** FlorianFa has joined #openstack-infra | 09:28 | |
| *** ykarel|lunch is now known as ykarel | 09:30 | |
| *** ykarel is now known as ykarel|meeting | 09:36 | |
| *** lpetrut has joined #openstack-infra | 09:42 | |
| *** otsukahy has quit IRC | 09:42 | |
| *** e0ne has joined #openstack-infra | 09:43 | |
| zbr | i observed that openstack-tox-linters is often starting slowly, in many cases after other heavy-duty jobs, can we update the min-ready for it? | 09:43 |
| zbr | the issue is that I tried to identify which nodeset it uses but got lost into the inheritance and never managed to find where it comes from | 09:44 |
| AJaeger | zbr: min-ready is per node type - not per job | 09:44 |
| zbr | yeha, i know. but i was not able to find the nodeset used by it by looking at all parents. | 09:45 |
| zbr | i went up to unittests, no nodeset | 09:45 |
| AJaeger | zbr: just look at the log of one run | 09:45 |
| zbr | ubuntu-bionic | 09:47 |
| AJaeger | and xenial for older stable branches, see https://opendev.org/openstack/openstack-zuul-jobs/src/branch/master/zuul.d/jobs.yaml#L79 | 09:47 |
| *** dpawlik has quit IRC | 09:48 | |
| *** rcernin has joined #openstack-infra | 09:49 | |
| *** otsukahy has joined #openstack-infra | 09:49 | |
| zbr | AJaeger: i think i found where to bump it, but i still not sure where is defined, probably i missed it / not important. | 09:50 |
| *** otsukahy has quit IRC | 09:50 | |
| *** otsukahy has joined #openstack-infra | 09:51 | |
| *** yamamoto has quit IRC | 09:51 | |
| AJaeger | nodepool/nl01.openstack.org.yaml in openstack/project-config - and having there 1 and 10 for xenial: Yes, bumping up a bit is fine IMHO | 09:51 |
| zbr | i am proposing 1-3, do you want me to lower the 10? | 09:55 |
| *** lpetrut has quit IRC | 09:55 | |
| openstackgerrit | Sorin Sbarnea proposed openstack/project-config master: Bump min-ready for ubuntu-bionic https://review.opendev.org/691275 | 09:56 |
| *** otsukahy has quit IRC | 09:56 | |
| *** rpittau is now known as rpittau|bbl | 10:08 | |
| *** otsukahy has joined #openstack-infra | 10:12 | |
| *** whoami-rajat has quit IRC | 10:14 | |
| *** dtantsur|afk has quit IRC | 10:16 | |
| *** dtantsur has joined #openstack-infra | 10:16 | |
| *** dpawlik has joined #openstack-infra | 10:19 | |
| *** dpawlik has quit IRC | 10:24 | |
| *** otsukahy has quit IRC | 10:31 | |
| *** otsukahy has joined #openstack-infra | 10:37 | |
| *** otsukahy has quit IRC | 10:50 | |
| *** ykarel|meeting is now known as ykarel | 10:54 | |
| *** ccamacho has quit IRC | 10:55 | |
| *** lpetrut has joined #openstack-infra | 11:02 | |
| *** lpetrut has quit IRC | 11:05 | |
| *** dpawlik has joined #openstack-infra | 11:10 | |
| *** whoami-rajat_ has joined #openstack-infra | 11:17 | |
| donnyd | looks like something may be broken in grafana http://grafana.openstack.org/d/3Bwpi5SZk/nodepool-fortnebula?orgId=1&from=now-24h&to=now | 11:18 |
| *** rcernin has quit IRC | 11:18 | |
| donnyd | only shows data till 22:30 | 11:18 |
| *** otsukahy has joined #openstack-infra | 11:21 | |
| *** whoami-rajat_ is now known as whoami-rajat | 11:22 | |
| *** ccamacho has joined #openstack-infra | 11:27 | |
| *** otsukahy has quit IRC | 11:32 | |
| *** ykarel is now known as ykarel|afk | 11:35 | |
| *** iurygregory has joined #openstack-infra | 11:36 | |
| *** iurygregory has quit IRC | 11:49 | |
| *** otsukahy has joined #openstack-infra | 11:50 | |
| *** ramishra has quit IRC | 11:52 | |
| *** otsukahy has quit IRC | 11:54 | |
| *** rh-jelabarre has joined #openstack-infra | 11:57 | |
| *** otsukahy has joined #openstack-infra | 12:00 | |
| *** rpittau|bbl is now known as rpittau | 12:04 | |
| *** jamesmcarthur has joined #openstack-infra | 12:10 | |
| *** ramishra has joined #openstack-infra | 12:11 | |
| zbr | just made same same discovery! see https://zuul.opendev.org/t/openstack/builds?job_name=tripleo-ci-centos-7-scenario003-standalone --- last execution is 5-6 hours ago! which is not true. | 12:14 |
| *** rosmaita has left #openstack-infra | 12:14 | |
| zbr | apparently zuul history is no longer updated | 12:15 |
| *** aedc has quit IRC | 12:15 | |
| *** slaweq has joined #openstack-infra | 12:15 | |
| *** otsukahy has quit IRC | 12:15 | |
| *** dave-mccowan has joined #openstack-infra | 12:16 | |
| *** ykarel|afk is now known as ykarel | 12:17 | |
| *** otsukahy has joined #openstack-infra | 12:17 | |
| *** rlandy has joined #openstack-infra | 12:20 | |
| *** rosmaita has joined #openstack-infra | 12:20 | |
| *** jamesmcarthur has quit IRC | 12:20 | |
| openstackgerrit | Sagi Shnaidman proposed zuul/zuul-jobs master: DNM: test openvswitch on centos8 https://review.opendev.org/691294 | 12:22 |
| *** otsukahy has quit IRC | 12:23 | |
| *** aedc has joined #openstack-infra | 12:25 | |
| *** otsukahy has joined #openstack-infra | 12:26 | |
| openstackgerrit | Felix Schmidt proposed zuul/zuul master: WIP: Make reporting asynchronous https://review.opendev.org/691253 | 12:27 |
| *** surpatil has quit IRC | 12:35 | |
| *** otsukahy has quit IRC | 12:37 | |
| *** rfolco|rover has joined #openstack-infra | 12:43 | |
| *** jamesmcarthur has joined #openstack-infra | 12:45 | |
| *** FlorianFa has quit IRC | 12:48 | |
| openstackgerrit | David Shrewsbury proposed zuul/zuul master: Handle upgrade of autohold held nodes https://review.opendev.org/691297 | 12:51 |
| *** aedc has quit IRC | 12:53 | |
| *** dpawlik has quit IRC | 12:57 | |
| *** aedc has joined #openstack-infra | 13:00 | |
| *** yamamoto has joined #openstack-infra | 13:01 | |
| *** jamesmcarthur has quit IRC | 13:05 | |
| *** yamamoto has quit IRC | 13:07 | |
| *** ykarel is now known as ykarel|afk | 13:10 | |
| *** yamamoto has joined #openstack-infra | 13:10 | |
| *** markvoelker has joined #openstack-infra | 13:15 | |
| *** mriedem has joined #openstack-infra | 13:17 | |
| AJaeger | zbr, btw. the bump of min-ready will only help on an idle system. AFAIK If we are at the limit, then min-ready is ignored and nodes are booted according to demand | 13:17 |
| *** goldyfruit has joined #openstack-infra | 13:17 | |
| AJaeger | zbr: do you have a link to a later run of that job? | 13:18 |
| zbr | AJaeger: https://zuul.opendev.org/t/openstack/build/e6e6847012474ebe83380a42177f1389 -- i observed this one being triggered later than others. | 13:19 |
| *** markvoelker has quit IRC | 13:19 | |
| *** amoralej is now known as amoralej|lunch | 13:19 | |
| zbr | it was not a huge delay but enough to observe it. i seen this happening quite often and the min-ready=1 explains why I seen it. | 13:20 |
| zbr | in fact is very easy to reproduce, just do a recheck on it and you will see how most of the other jobs are started right away but this one starts much later. | 13:22 |
| *** yamamoto has quit IRC | 13:22 | |
| *** yamamoto has joined #openstack-infra | 13:26 | |
| *** eharney has quit IRC | 13:30 | |
| *** tesseract has quit IRC | 13:36 | |
| *** yamamoto has quit IRC | 13:37 | |
| *** dpawlik has joined #openstack-infra | 13:37 | |
| *** Goneri has joined #openstack-infra | 13:47 | |
| *** otsukahy has joined #openstack-infra | 13:57 | |
| *** ykarel|afk is now known as ykarel | 13:57 | |
| *** amoralej|lunch is now known as amoralej | 13:57 | |
| *** rascasoft has quit IRC | 13:57 | |
| *** priteau has joined #openstack-infra | 13:59 | |
| *** benj_ has quit IRC | 14:08 | |
| *** pkopec has quit IRC | 14:10 | |
| *** benj has joined #openstack-infra | 14:11 | |
| *** benj is now known as Guest14611 | 14:12 | |
| clarkb | donnyd:we ran out of disk on the server so I stopped the services and am reducing whisper file sizes and retention | 14:13 |
| *** yamamoto has joined #openstack-infra | 14:13 | |
| clarkb | zbr node assignment is not a strict fifo and is effectively random | 14:13 |
| clarkb | nodepools assigns a launcher thread to each zuul request in order but the launchers may not produce nodes in order | 14:14 |
| *** markvoelker has joined #openstack-infra | 14:16 | |
| *** goldyfruit_ has joined #openstack-infra | 14:17 | |
| *** dpawlik has quit IRC | 14:17 | |
| *** goldyfruit has quit IRC | 14:19 | |
| *** yamamoto has quit IRC | 14:19 | |
| *** piotrowskim has quit IRC | 14:19 | |
| *** eharney has joined #openstack-infra | 14:23 | |
| donnyd | clarkb: thanks for the update | 14:25 |
| TheJulia | fungi: diablo_rojo_phon: or anyone else with back-end storyboard access. I'm trying to submit a new story to storyboard and I either get a "sql transation is being rolled back" or "unable to convert input to story" | 14:29 |
| fungi | TheJulia: let me guess, trying to add more than one initial task when creating the story? | 14:30 |
| TheJulia | yes | 14:30 |
| TheJulia | two tasks to be precise | 14:30 |
| fungi | workaround for now is to create the story with one task and then add additional tasks once it's created | 14:30 |
| fungi | there is a write lock contention with task additions happening in one transaction, trying to work out a solution | 14:30 |
| TheJulia | k | 14:30 |
| *** mattw4 has joined #openstack-infra | 14:31 | |
| TheJulia | fungi: deleting the 2nd task and it still went boom, starting over completely | 14:34 |
| fungi | yes, i think the webclient gets into an inconsistent state following database write errors, the error handling there needs improvement as well (though ideally that wouldn't be hit if the insert/update queries didn't step on one another) | 14:35 |
| *** jaosorior has joined #openstack-infra | 14:38 | |
| *** ccamacho has quit IRC | 14:39 | |
| TheJulia | \o/ it created a duplicate. I marked it as invalid | 14:40 |
| openstackgerrit | James E. Blair proposed zuul/zuul-jobs master: Test buildset registry with k8s and docker https://review.opendev.org/689280 | 14:40 |
| fungi | TheJulia: ahh, yeah, that's probably why it went splodey when you retried to save it after removing the second task, it didn't realize it had successfully created the story | 14:43 |
| *** jangutter has quit IRC | 14:43 | |
| fungi | hopefully we get this fixed soon. we had a prospective intern write up a patch, but that turned out to not be a great solution and we had to walk it back | 14:44 |
| *** goldyfruit___ has joined #openstack-infra | 14:45 | |
| *** eernst_ has joined #openstack-infra | 14:47 | |
| *** goldyfruit_ has quit IRC | 14:48 | |
| *** eernst_ has quit IRC | 14:49 | |
| *** eernst_ has joined #openstack-infra | 14:49 | |
| clarkb | my whisper resize of just the poenstack timings is still running | 14:51 |
| clarkb | 51GB free now | 14:51 |
| clarkb | I think the stats per resource are really hurting us there | 14:52 |
| *** mattw4 has quit IRC | 14:52 | |
| clarkb | I was hoping I'd wake and be able to run the resize over the zuul files | 14:53 |
| *** yamamoto has joined #openstack-infra | 14:58 | |
| corvus | clarkb: did you get to the bottom of that? | 14:59 |
| corvus | when i checked out, it seemed like everyone was surprised that was happening... | 14:59 |
| clarkb | corvus: I think we understand the failure mode in sdk. But we don't know why it happens. mordred was trying to reprpduce it yesterday when I ran off to dinner | 15:00 |
| clarkb | corvus: basically the sdk code has code to remove the project id from the url, then from that it has a check every other url path entry. Problem is it isn't removing the project id then its every other check is offset by one which means it looks at the resource id instead of the actions | 15:01 |
| openstackgerrit | Matt Riedemann proposed opendev/elastic-recheck master: Add query for ssh fail bug 1849857 https://review.opendev.org/691374 | 15:01 |
| openstack | bug 1849857 in OpenStack-Gate "ssh fails to guest in tempest run, "Starting dropbear sshd: WARN: generating key of type ecdsa failed!" in guest console log" [Undecided,New] https://launchpad.net/bugs/1849857 | 15:01 |
| clarkb | What we weren't understanding is why that remove project id code isn't working | 15:01 |
| clarkb | calebb: I have an instance booted in ovh bhs1 (chose that region because it has a 200GB disk flavor we use for the mirror nodes and they don't charge customers for network io so figure doing a bunch of network io there is ok) | 15:02 |
| clarkb | corvus: need ssh keys whenever you are ready | 15:03 |
| *** yamamoto has quit IRC | 15:03 | |
| clarkb | er calebb ^ | 15:03 |
| *** xek__ has joined #openstack-infra | 15:03 | |
| clarkb | corvus: mordred the other thing I've realized sleeping on it is that without a cloud/project prefix that data is going to be not very useful to us anyway | 15:04 |
| *** tosky_ has joined #openstack-infra | 15:04 | |
| clarkb | because it will be 8 different regions all mixed together | 15:04 |
| *** Benibr has joined #openstack-infra | 15:04 | |
| *** yamamoto has joined #openstack-infra | 15:04 | |
| *** xek_ has quit IRC | 15:06 | |
| *** tosky has quit IRC | 15:06 | |
| *** jamesmcarthur has joined #openstack-infra | 15:08 | |
| clarkb | basically there are two bugs here. The software isn't doing what is intended but the intended behavior isn't useful to users. Both things will need to be addressed. Fixing the "not doing what is intended" bug should help with the disk usage | 15:09 |
| clarkb | corvus: https://opendev.org/openstack/openstacksdk/src/branch/master/openstack/proxy.py#L130-L138 is the code that attempts to remove the project id from the url parts. Maybe the bug stands out to you? Best I can figure is the logic is correct so we may have a type error | 15:12 |
| *** roman_g has joined #openstack-infra | 15:13 | |
| *** gyee has joined #openstack-infra | 15:14 | |
| *** e0ne has quit IRC | 15:15 | |
| corvus | clarkb: what are the paths that are ending up on disk? (example?) | 15:17 |
| clarkb | stats/times/openstack/api/compute/DELETE/dcaab5e32b234d56b626f72581e3644c/fe945625-f7ca-4dcc-99c3-0bca1fa7ab96/sum.wsp | 15:18 |
| clarkb | corvus: another idea I had is that first id lacks -'s | 15:18 |
| clarkb | and its possible that the sdk internal representation of that string includes them | 15:18 |
| *** kjackal has quit IRC | 15:19 | |
| *** kopecmartin is now known as kopecmartin|off | 15:19 | |
| calebb | clarkb: my github keys should be good | 15:20 |
| calebb | https://github.com/squidboylan.keys | 15:20 |
| clarkb | calebb: ssh ubuntu@158.69.66.151 | 15:23 |
| openstackgerrit | Merged opendev/elastic-recheck master: Add query for ssh fail bug 1849857 https://review.opendev.org/691374 | 15:24 |
| openstackgerrit | Merged opendev/elastic-recheck master: Hard-code os-brick into TestQueries.openstack_projects https://review.opendev.org/689039 | 15:24 |
| openstack | bug 1849857 in OpenStack-Gate "ssh fails to guest in tempest run, "Starting dropbear sshd: WARN: generating key of type ecdsa failed!" in guest console log" [Undecided,Confirmed] https://launchpad.net/bugs/1849857 | 15:24 |
| calebb | clarkb: ty | 15:24 |
| corvus | clarkb: or that's being called without a project_id argument | 15:25 |
| corvus | clarkb: do you know what those 2 ids in the example path are? | 15:26 |
| corvus | dca... and fe9... | 15:26 |
| openstackgerrit | Merged zuul/zuul-website master: Update case studies on users page https://review.opendev.org/691130 | 15:27 |
| clarkb | corvus: I believe dca is a rackspace project id (based on conversation with mordred yesterday) and fe9 is a nova instance id. Confirming the first should be straightforward. Confirming the second is not so easy | 15:27 |
| corvus | i thought rax projects were integers? | 15:27 |
| clarkb | hrm ya they are. maybe another cloud then (mordred mentioned rax yesterday and tested against them because the project id matched but maybe he meant for a different path). Let me see if I can track down what dcaa is | 15:29 |
| corvus | it doesn't show up in all-clouds.yaml | 15:29 |
| openstackgerrit | Merged opendev/elastic-recheck master: Remove H233 from flake8 ignore list https://review.opendev.org/689041 | 15:29 |
| openstackgerrit | Merged opendev/elastic-recheck master: Fix link formatting in CONTRIBUTING.rst https://review.opendev.org/689043 | 15:29 |
| clarkb | corvus: dcaa is the ovh openstack-jenkins project | 15:30 |
| clarkb | *project id | 15:30 |
| corvus | why can i not find that in all-clouds? | 15:30 |
| clarkb | I'm guessing because we use the name in all clouds /me looks | 15:30 |
| clarkb | yes we use a project name there not id | 15:31 |
| *** lmiccini has quit IRC | 15:31 | |
| clarkb | (I did openstack project list against clouds to look for the id) | 15:31 |
| corvus | i wonder if sdk does the same.... | 15:31 |
| *** otsukahy has quit IRC | 15:32 | |
| *** tosky_ is now known as tosky | 15:36 | |
| corvus | what's the env var to tell sdk where clouds.yaml is? | 15:39 |
| clarkb | corvus: OS_CLIENT_CONFIG_FILE=/etc/openstack/all-clouds.yaml | 15:40 |
| corvus | openstack.connect(cloud='openstackjenkins-ovh').session.get_project_id() returns 'dcaab5e32b234d56b626f72581e3644c' so i think that's working as expected. | 15:40 |
| clarkb | fwiw conn.session.get_project_id() gives me 'dcaab5e32b234d56b626f72581e3644c' | 15:41 |
| clarkb | ya | 15:41 |
| *** mattw4 has joined #openstack-infra | 15:46 | |
| *** diablo_rojo has joined #openstack-infra | 15:46 | |
| *** mattw4 has quit IRC | 15:50 | |
| *** mattw4 has joined #openstack-infra | 15:51 | |
| *** gfidente has quit IRC | 15:51 | |
| *** jaosorior has quit IRC | 15:51 | |
| *** mattw4 has quit IRC | 15:54 | |
| *** xek_ has joined #openstack-infra | 15:56 | |
| fungi | gonna take a break from banging my head against openid/cors interactions in storyboard-webclient drafts and take a long lunch to run some errands | 15:57 |
| clarkb | hrm so I'm trying to debug it with opreations against server resources and I used osc to create an instance for that | 15:57 |
| clarkb | problem is sdk can't get that server instance | 15:57 |
| *** mattw4 has joined #openstack-infra | 15:57 | |
| *** STEWIE925 has joined #openstack-infra | 15:58 | |
| STEWIE925 | requesting review and approval for https://review.opendev.org/#/c/691038/ - thank you so much! | 15:58 |
| clarkb | ah that is because it is only looking at gra1 and not bhs1 even though I toldit a region at connect time. I'm betting I didn't use thecorrect parameter for that and it doens't care what you pass in | 15:59 |
| *** xek__ has quit IRC | 15:59 | |
| clarkb | and from that I'm not able to reproduce having the project id in the url parts list that ends up being part of the statsd stuff | 16:01 |
| clarkb | corvus: perhaps this is an older bug not present in current sdk? and we've just go old stale data as a result? | 16:02 |
| corvus | clarkb: oh, are these not current timestamps? | 16:02 |
| *** lucasagomes has quit IRC | 16:02 | |
| clarkb | I'm not sure how to check properly (via a garphite query?) | 16:03 |
| corvus | clarkb: file creation time? | 16:04 |
| clarkb | I bet that the whisper file size reduction is mangling that | 16:04 |
| clarkb | but could check | 16:04 |
| corvus | it's unfortunate we did that. i bet the original retentions were fine | 16:05 |
| *** rpittau is now known as rpittau|afk | 16:05 | |
| clarkb | I've found one with timestamps from august 12 | 16:06 |
| clarkb | october 2 | 16:06 |
| clarkb | (we wouldn't have noticed hte problem without the resizing so chicken and egg I guess) | 16:07 |
| *** derekh has quit IRC | 16:07 | |
| corvus | i think i repro'd; patch incoming | 16:07 |
| *** otsukahy has joined #openstack-infra | 16:08 | |
| corvus | clarkb: remote: https://review.opendev.org/691386 WIP: test extract_name | 16:08 |
| corvus | clarkb: that produces: b'testtools.matchers._impl.MismatchError: !=:' | 16:08 |
| corvus | b"reference = ['servers']" | 16:08 |
| corvus | b"actual = ['dcaab5e32b234d56b626f72581e3644c', 'a82af292-bb2e-44a4-bf31-fb8145b66e6b']" | 16:08 |
| corvus | which, based on my read of the calling site, would produce a path just like what we're seeing | 16:09 |
| mriedem | fyi docs jobs might be failing globally https://zuul.opendev.org/t/openstack/build/10344924bca44849b06b8fae29b5494f/log/job-output.txt#652 | 16:10 |
| clarkb | yup that is what I would expect given our filesystem entries | 16:10 |
| clarkb | STEWIE925: can you respond to fungi's question? | 16:10 |
| clarkb | STEWIE925: just to make sure we are setting the correct acls | 16:10 |
| STEWIE925 | thank you clarkb - let me look | 16:10 |
| clarkb | corvus: in my testing against ovh what I've been getting after the first pass of parts is ['v2.1', 'servers', '8a84deaa-8b94-4f3d-9584-1f71a9938605'] | 16:11 |
| corvus | clarkb: i'm going to mention this in openstack-sdks | 16:11 |
| clarkb | corvus: the project id is getting removed but the rest of the url is still there as expected. It is possible the bug is not in the removal of project id and is further down | 16:12 |
| clarkb | corvus: ++ fwiw I think this affects all the reporting systems too and it doesn't produce useful reporting :( | 16:12 |
| *** otsukahy has quit IRC | 16:12 | |
| clarkb | your reproduction is an easier way to test so I'm switching to that locally now instead of talking to ovh | 16:15 |
| corvus | clarkb: i've got it down to self._extract_name_consume_url_parts(url_parts) | 16:16 |
| corvus | oh wait | 16:17 |
| clarkb | corvus: ya the modulus there breaks if project id isn't removed properly | 16:17 |
| corvus | the initial project id filtering is failing too | 16:17 |
| corvus | right that | 16:17 |
| clarkb | that seems to happen because project_id is None in your example | 16:18 |
| clarkb | which gives support to the idea that maybe we aren't passing in project_id always | 16:18 |
| corvus | oh, hrm. project_id wasn't supposed to be none | 16:18 |
| corvus | oh i forgot to pass in the arg | 16:18 |
| corvus | all right -- so if i fix the test, it does pass | 16:19 |
| clarkb | that does make me wonder though if the production calls aren't always passing in project_id | 16:20 |
| corvus | right. i wonder how that could happen? | 16:20 |
| clarkb | maybe get_project_id() fails? | 16:20 |
| clarkb | figuring out what the actual definition of these methods is always fun | 16:21 |
| clarkb | is session a keystoneauth1 session? | 16:22 |
| corvus | yes | 16:22 |
| corvus | it's ultimately calling get_session_id on keystoneauth1.identity.generic.password.Password | 16:23 |
| corvus | which inherits that method | 16:23 |
| clarkb | and reading the source there it seems that the project id comes out of the token | 16:24 |
| clarkb | it would be really weird for us to not have a project id in that case | 16:24 |
| openstackgerrit | Merged zuul/zuul master: Annotate further log messages in pipeline processing https://review.opendev.org/684759 | 16:24 |
| corvus | clarkb: did you figure out how to call _extract_name in a repl? | 16:28 |
| corvus | cloud.compute._extract_name is what i would expect | 16:29 |
| clarkb | corvus: I did the nasty thing of removing the if statsd_client conditional then commented out the bits that send statsd at the end | 16:29 |
| clarkb | corvus: then started making real quests to ovh :/ | 16:29 |
| *** ykarel is now known as ykarel|away | 16:30 | |
| openstackgerrit | Merged zuul/zuul master: Handle upgrade of autohold held nodes https://review.opendev.org/691297 | 16:33 |
| corvus | oh i think my sdk version may be old | 16:35 |
| corvus | launchers seem to run 0.36 | 16:37 |
| clarkb | corvus: da45a449dc047e25a295069991b42f1e431e0f3c updates stats reporting post 0.36.0 release | 16:40 |
| clarkb | includes "fix metric names in weird cases" I wish that was its own commit but reading it now | 16:40 |
| clarkb | I'm going to try rebasing your test on 0.36.0 | 16:41 |
| corvus | it works in my repl: cloud.compute._extract_name('https://compute.gra1.cloud.ovh.net/v2.1/dcaab5e32b234d56b626f72581e3644c/servers/a82af292-bb2e-44a4-bf31-fb8145b66e6b', cloud.compute.service_type, cloud.compute.session.get_project_id()) -> ['server'] | 16:42 |
| corvus | that's with openstacksdk 0.36.0 and keystoneauth1 3.18.0 | 16:42 |
| clarkb | ya it was working for me too talking to ovh directly | 16:43 |
| corvus | clarkb: 0.36.0 was sept 18 | 16:43 |
| corvus | but we may have run an older version for a while | 16:44 |
| clarkb | I think its unreleased code that may fix it though | 16:44 |
| clarkb | based on the "fix metric names in weird cases" commit message item | 16:44 |
| clarkb | your test still works when rebased on 0.36.0 though | 16:45 |
| corvus | and my repl test works on .36 | 16:45 |
| corvus | so it must have mostly worked before? | 16:45 |
| clarkb | ya it is possible these are very corner casey | 16:45 |
| clarkb | and the specific request details matter a lot | 16:45 |
| corvus | it looks like we restarted launches on 0.35.0 on sept 9, and then next in october (which should have picked up 0.36) | 16:47 |
| clarkb | what date in october? I was able to find cases of the bug on graphite.o.o from october 2 | 16:47 |
| *** udesale has quit IRC | 16:48 | |
| corvus | no earlier than oct 15 | 16:48 |
| corvus | (01 and 04 on oct 15, all launchers on oct 17) | 16:48 |
| corvus | so if it was fixed in 0.36, we've only been running that for ~ a week | 16:48 |
| clarkb | maybe the easiest thing to do here is rm all the resource specific graphite whisper files and dirs then chcek it again in a week or so (course that then runs into summit travel) | 16:49 |
| *** roman_g has quit IRC | 16:49 | |
| clarkb | but something like that to be able to check easily if it is still a problem | 16:49 |
| *** yamamoto has quit IRC | 16:49 | |
| corvus | clarkb: da45a449dc047e25a295069991b42f1e431e0f3c is in 0.36 | 16:49 |
| corvus | so if we assume that fixes it ("fix metric name build when url contains project id") then i think everything lines up | 16:50 |
| clarkb | oh so it is. I see what happened I tried to check it out but that failed becuase I had local edits | 16:50 |
| clarkb | and so when I did git log I had tip of master which is after 0.36.0 | 16:51 |
| openstackgerrit | Merged zuul/zuul master: Keep tenant_last_reconfigured float precision (time.sleep removal) https://review.opendev.org/689394 | 16:51 |
| corvus | clarkb: you want to rm all the things? | 16:55 |
| clarkb | corvus: ya if people think that is a useful next step I can rm all the timers/openstack/VERB/project_id dirs | 16:55 |
| corvus | ++ | 16:55 |
| clarkb | I'll stop my whisper resize first, rm the dirs then rerun the whisper resize? It should run much more quickly | 16:56 |
| clarkb | ok resize stopped. working on rms now | 16:56 |
| *** jamesmcarthur has quit IRC | 16:57 | |
| *** otsukahy has joined #openstack-infra | 17:04 | |
| clarkb | ok compute and object store are the only ones that work this way | 17:08 |
| clarkb | I expect that the object store one will still be an issue but its far less data overall | 17:10 |
| clarkb | I've cleared them out and am restarting the resize. It should go quick now | 17:10 |
| clarkb | yup thats done now | 17:11 |
| clarkb | corvus: ok, do you think it is worth running the resize on the zuul timers? | 17:12 |
| *** priteau has quit IRC | 17:12 | |
| clarkb | I expect that will be a sloewr one based on total file size consumed by zuul timers. But we have 273GB free now so could start the service up again and probably be ok | 17:12 |
| clarkb | then watch for openstack timer weirdness | 17:12 |
| corvus | i'm not sure it's safe to resize with carbon running | 17:13 |
| clarkb | ya carbon is off now which is why I'm wondering if we should attempt the resize while we've got it off | 17:13 |
| clarkb | or maybe just not bother resizing zuul for now since we've freed a bunch of disk | 17:13 |
| corvus | oh i see what you're asking | 17:13 |
| *** goldyfruit_ has joined #openstack-infra | 17:13 | |
| *** igordc has joined #openstack-infra | 17:14 | |
| corvus | i think it'd be good for the db to be coherent with the config | 17:14 |
| clarkb | ok why don't I start it in my screen then and if it is still running on say monday we can reconsider? | 17:15 |
| clarkb | worst case we stop it | 17:15 |
| *** goldyfruit___ has quit IRC | 17:15 | |
| corvus | k | 17:16 |
| clarkb | that is running now against stats/timers/zuul. stats/timers/openstack is done | 17:16 |
| clarkb | there are a couple other timers as well but they are comparatively tiny so can be done quickly when zuul is done | 17:17 |
| corvus | maybe there's enough io bandwidth to do them in parallel? | 17:17 |
| corvus | (looking at top, i'd say we have lots of headroom) | 17:18 |
| *** jerryz has joined #openstack-infra | 17:18 | |
| *** otsukahy has quit IRC | 17:19 | |
| *** yamamoto has joined #openstack-infra | 17:20 | |
| clarkb | k I'll start the others now then | 17:20 |
| clarkb | they are all running now | 17:24 |
| clarkb | nodepool and zuul are the only ones that haven'tfinished | 17:24 |
| *** jerryz has quit IRC | 17:24 | |
| *** yamamoto has quit IRC | 17:26 | |
| *** weifan has joined #openstack-infra | 17:30 | |
| openstackgerrit | James E. Blair proposed zuul/zuul-jobs master: Test buildset registry with k8s and docker https://review.opendev.org/689280 | 17:32 |
| *** xek_ has quit IRC | 17:35 | |
| *** xek_ has joined #openstack-infra | 17:35 | |
| clarkb | now nodepool is done. Zuul is the only timers subdir still getting resized | 17:38 |
| *** ralonsoh has quit IRC | 17:39 | |
| *** zbr has quit IRC | 17:45 | |
| openstackgerrit | Merged zuul/zuul master: Pagure - Manage project connector refresh when EINVALIDTOK https://review.opendev.org/687259 | 17:48 |
| *** amoralej is now known as amoralej|off | 18:09 | |
| *** psachin has quit IRC | 18:15 | |
| *** aedc has quit IRC | 18:15 | |
| *** bnemec is now known as beekneemech | 18:16 | |
| *** takamatsu has quit IRC | 18:24 | |
| openstackgerrit | James E. Blair proposed zuul/zuul-jobs master: Test buildset registry with k8s and docker https://review.opendev.org/689280 | 18:27 |
| *** ricolin has quit IRC | 18:31 | |
| *** markvoelker has quit IRC | 18:36 | |
| *** markvoelker has joined #openstack-infra | 18:38 | |
| *** ykarel|away has quit IRC | 18:38 | |
| *** efried has quit IRC | 18:44 | |
| *** weifan has quit IRC | 18:52 | |
| *** weifan has joined #openstack-infra | 18:52 | |
| *** weifan has quit IRC | 18:52 | |
| *** weifan has joined #openstack-infra | 18:53 | |
| *** jerryz has joined #openstack-infra | 18:53 | |
| *** weifan has quit IRC | 18:53 | |
| *** weifan has joined #openstack-infra | 18:53 | |
| *** weifan has quit IRC | 18:54 | |
| *** weifan has joined #openstack-infra | 18:54 | |
| *** weifan has quit IRC | 18:55 | |
| *** weifan has joined #openstack-infra | 18:55 | |
| *** weifan has quit IRC | 18:56 | |
| *** weifan has joined #openstack-infra | 18:56 | |
| *** weifan has quit IRC | 18:56 | |
| *** weifan has joined #openstack-infra | 18:57 | |
| *** gmann is now known as gmann_afk | 18:57 | |
| *** weifan has quit IRC | 18:57 | |
| *** weifan has joined #openstack-infra | 18:57 | |
| *** weifan has quit IRC | 18:58 | |
| *** weifan has joined #openstack-infra | 18:58 | |
| *** weifan has quit IRC | 18:59 | |
| *** weifan has joined #openstack-infra | 18:59 | |
| *** weifan has quit IRC | 19:00 | |
| *** weifan has joined #openstack-infra | 19:00 | |
| *** weifan has quit IRC | 19:00 | |
| *** weifan has joined #openstack-infra | 19:01 | |
| *** zbr has joined #openstack-infra | 19:01 | |
| *** weifan has quit IRC | 19:01 | |
| *** weifan has joined #openstack-infra | 19:01 | |
| *** weifan has quit IRC | 19:02 | |
| *** weifan has joined #openstack-infra | 19:02 | |
| *** weifan has quit IRC | 19:03 | |
| *** weifan has joined #openstack-infra | 19:03 | |
| *** weifan has quit IRC | 19:04 | |
| *** weifan has joined #openstack-infra | 19:04 | |
| *** weifan has quit IRC | 19:04 | |
| *** weifan has joined #openstack-infra | 19:05 | |
| *** eharney has quit IRC | 19:05 | |
| *** xek_ has quit IRC | 19:05 | |
| *** weifan has quit IRC | 19:05 | |
| *** xek_ has joined #openstack-infra | 19:05 | |
| *** weifan has joined #openstack-infra | 19:05 | |
| *** weifan has quit IRC | 19:06 | |
| *** weifan has joined #openstack-infra | 19:06 | |
| *** weifan has quit IRC | 19:07 | |
| *** weifan has joined #openstack-infra | 19:07 | |
| *** weifan has quit IRC | 19:08 | |
| *** weifan has joined #openstack-infra | 19:08 | |
| *** weifan has quit IRC | 19:08 | |
| *** weifan has joined #openstack-infra | 19:09 | |
| *** yamamoto has joined #openstack-infra | 19:09 | |
| *** weifan has quit IRC | 19:09 | |
| *** weifan has joined #openstack-infra | 19:09 | |
| *** KeithMnemonic has quit IRC | 19:10 | |
| *** KeithMnemonic has joined #openstack-infra | 19:11 | |
| *** yamamoto has quit IRC | 19:13 | |
| *** KeithMnemonic has quit IRC | 19:14 | |
| *** KeithMnemonic has joined #openstack-infra | 19:15 | |
| *** beekneemech has quit IRC | 19:16 | |
| *** bnemec has joined #openstack-infra | 19:18 | |
| *** lifeless has quit IRC | 19:29 | |
| sshnaidm|off | please take a look at os_image fix in ansible: https://github.com/ansible/ansible/pull/63959 | 19:35 |
| *** jamesmcarthur has joined #openstack-infra | 19:51 | |
| *** goldyfruit_ has quit IRC | 20:01 | |
| *** e0ne has joined #openstack-infra | 20:02 | |
| *** goldyfruit has joined #openstack-infra | 20:02 | |
| *** eharney has joined #openstack-infra | 20:08 | |
| *** gmann_afk is now known as gmann | 20:11 | |
| openstackgerrit | James E. Blair proposed zuul/zuul-jobs master: Test buildset registry with k8s and docker https://review.opendev.org/689280 | 20:16 |
| *** prometheanfire has quit IRC | 20:19 | |
| *** prometheanfire has joined #openstack-infra | 20:20 | |
| *** e0ne has quit IRC | 20:22 | |
| *** jamesmcarthur has quit IRC | 20:24 | |
| *** jamesmcarthur has joined #openstack-infra | 20:24 | |
| *** jamesmcarthur has quit IRC | 20:24 | |
| *** EmilienM has quit IRC | 20:32 | |
| *** goldyfruit has quit IRC | 20:38 | |
| *** jamesmcarthur has joined #openstack-infra | 20:40 | |
| openstackgerrit | James E. Blair proposed zuul/zuul-jobs master: Test buildset registry with k8s and docker https://review.opendev.org/689280 | 20:42 |
| openstackgerrit | Adam Coldrick proposed opendev/storyboard master: Add support for POST requests to /v1/openid/authorize_return https://review.opendev.org/691428 | 20:49 |
| *** markvoelker has quit IRC | 20:50 | |
| clarkb | dtroyer: fyi remote: https://review.opendev.org/691430 Switch to the opensuse-15 nodeset | 20:53 |
| *** xarses has quit IRC | 20:54 | |
| *** xarses has joined #openstack-infra | 20:54 | |
| dtroyer | clarkb: thanks for the heads-up, I'll make sure Saul (sgw) sees it | 20:55 |
| clarkb | dtroyer: I wouldn't call it a rush at this point, but want to call it out as I push them up so people are aware of the planned change | 20:56 |
| clarkb | eventually we'll likely merge changes if we hvae to but better for you all to transition pain free ahead of time :) | 20:56 |
| dtroyer | I don't think it will be a problem for them, they are still working on packaging and builds there | 20:56 |
| *** jamesmcarthur has quit IRC | 20:58 | |
| *** goldyfruit_ has joined #openstack-infra | 21:00 | |
| clarkb | mriedem: related to ^ https://review.opendev.org/#/c/690437/ is a straightforward stable change against keystone | 21:00 |
| *** yamamoto has joined #openstack-infra | 21:01 | |
| STEWIE925 | hello Infra team - thanks for the review - would like to request workflow for https://review.opendev.org/#/c/691038/ | 21:01 |
| clarkb | https://review.opendev.org/#/c/690440/ and https://review.opendev.org/#/c/690439/ for freezer stable branches | 21:01 |
| clarkb | STEWIE925: done | 21:02 |
| *** KeithMnemonic has quit IRC | 21:02 | |
| mriedem | clarkb: that's not a cherry-pick? | 21:02 |
| STEWIE925 | clarkb: thank you! | 21:02 |
| clarkb | mriedem: no because it doesn't apply to master in the keystone case and otherwise its branch specific config | 21:03 |
| mriedem | does it apply to train? | 21:03 |
| clarkb | nope | 21:03 |
| mriedem | alright then | 21:04 |
| *** yamamoto has quit IRC | 21:05 | |
| clarkb | each branch's zuul config is independent of the others | 21:06 |
| clarkb | in general we shouldn't be afraid of modifying older branches to ccomdate changes (eol of platform as in this case or maybe job simply stops working etc) | 21:06 |
| openstackgerrit | Jeremy Stanley proposed opendev/storyboard master: Regular expression support for CORS and OAuth ACLs https://review.opendev.org/691034 | 21:06 |
| mordred | clarkb: oh good, it's fixed in newer sdk - that explains why I couldn't reproduce it yesterday | 21:07 |
| clarkb | mordred: well maybe | 21:07 |
| clarkb | we haven't turned on statsd yet to confirm | 21:07 |
| clarkb | still reducing zuul timers retention | 21:07 |
| *** mriedem has quit IRC | 21:08 | |
| mordred | nod | 21:14 |
| openstackgerrit | Merged openstack/project-config master: Update Project Configs for x/ranger https://review.opendev.org/691038 | 21:16 |
| *** xarses has quit IRC | 21:17 | |
| *** xarses has joined #openstack-infra | 21:17 | |
| openstackgerrit | James E. Blair proposed zuul/zuul-jobs master: Test buildset registry with k8s and docker https://review.opendev.org/689280 | 21:20 |
| openstackgerrit | Monty Taylor proposed opendev/system-config master: Add service playbook and test run for prod gerrit https://review.opendev.org/691171 | 21:22 |
| openstackgerrit | Monty Taylor proposed opendev/system-config master: Add launchpadlib credentials to gerrit ansible https://review.opendev.org/691172 | 21:22 |
| openstackgerrit | Monty Taylor proposed opendev/system-config master: Add replication config to gerrit ansible https://review.opendev.org/691173 | 21:22 |
| *** markvoelker has joined #openstack-infra | 21:23 | |
| openstackgerrit | Jeremy Stanley proposed opendev/system-config master: Use a wildcard regex for storyboard-dev OAuth/CORS https://review.opendev.org/691440 | 21:24 |
| fungi | SotK: diablo_rojo: ^ that's the config change we'll need corresponding to the regex implementation | 21:24 |
| *** diablo_rojo has quit IRC | 21:25 | |
| SotK | nice, thanks | 21:25 |
| *** rlandy has quit IRC | 21:25 | |
| mordred | fungi: lgtm - I didn't +A in case you want to land in a coordinated fashion | 21:27 |
| fungi | mordred: thanks! care to review https://review.opendev.org/691428 too? that is also needed because our object storage urls are so long that launchpad switches from get to post when connecting back to the api | 21:28 |
| mordred | oh neat | 21:29 |
| fungi | i have that set of three changes (the regex implementation, the post addition, and the config change) hand-applied on storyboard-dev and working | 21:29 |
| mordred | all lgtm | 21:30 |
| SotK | thanks for reviewing | 21:30 |
| fungi | much obliged! i'll approve the lot of them now and keep an eye on everything | 21:30 |
| *** markvoelker has quit IRC | 21:33 | |
| * SotK reviews the config change | 21:36 | |
| fungi | yeah, i'm not approving that one until the other two merge | 21:37 |
| *** goldyfruit_ has quit IRC | 21:37 | |
| fungi | since i don't think they share a change queue in zuul anyway | 21:37 |
| *** yamamoto has joined #openstack-infra | 21:38 | |
| *** xek_ has quit IRC | 21:41 | |
| *** xek has joined #openstack-infra | 21:42 | |
| *** yamamoto has quit IRC | 21:43 | |
| *** xarses has quit IRC | 22:05 | |
| *** xarses has joined #openstack-infra | 22:05 | |
| openstackgerrit | James E. Blair proposed zuul/zuul-jobs master: Test buildset registry with k8s and docker https://review.opendev.org/689280 | 22:06 |
| openstackgerrit | Monty Taylor proposed opendev/system-config master: Add service playbook and test run for prod gerrit https://review.opendev.org/691171 | 22:08 |
| *** EmilienM has joined #openstack-infra | 22:10 | |
| clarkb | infra-root care to look at https://review.opendev.org/#/c/691275/1 for updating min-ready values? If there is consensus we should make my suggeted change I'll probably go ahead and push that up and we can get that in | 22:12 |
| mordred | clarkb: I think you are correct | 22:14 |
| *** xarses has quit IRC | 22:15 | |
| clarkb | fungi: on https://review.opendev.org/#/c/690757/2/nodepool/nodepool.yaml do we need apt-https or whateve rthe pacakge is called? | 22:15 |
| clarkb | or does it come by default on all the distros now? | 22:16 |
| *** xarses has joined #openstack-infra | 22:17 | |
| *** xarses has quit IRC | 22:17 | |
| *** xarses has joined #openstack-infra | 22:17 | |
| clarkb | looks like it is there by default in buster but not stretch | 22:18 |
| * clarkb tries to figure it out for ubuntu | 22:18 | |
| corvus | clarkb: ++minready | 22:18 |
| clarkb | k I'll get a new ps up shortly | 22:18 |
| openstackgerrit | Clark Boylan proposed openstack/project-config master: Bump min-ready for ubuntu-bionic https://review.opendev.org/691275 | 22:19 |
| clarkb | er I hsould update the commit message too one moment | 22:20 |
| openstackgerrit | Merged openstack/project-config master: Grant core permissions to horizon-core on cloudkitty-dashboard https://review.opendev.org/691263 | 22:20 |
| openstackgerrit | Merged openstack/project-config master: Increase tag-releases timeout https://review.opendev.org/691139 | 22:21 |
| openstackgerrit | Merged openstack/project-config master: Grant permissions for solum-dashboard to horizon-core https://review.opendev.org/691142 | 22:21 |
| openstackgerrit | Merged openstack/project-config master: Update ceph grafana dashboard for cinder master branch https://review.opendev.org/690980 | 22:21 |
| openstackgerrit | Clark Boylan proposed openstack/project-config master: Bump min-ready for ubuntu-bionic https://review.opendev.org/691275 | 22:21 |
| clarkb | there | 22:21 |
| openstackgerrit | Merged openstack/project-config master: Grant permissions for Monasca UI to horizon-core https://review.opendev.org/690918 | 22:22 |
| openstackgerrit | Jeremy Stanley proposed opendev/storyboard master: Regular expression support for CORS and OAuth ACLs https://review.opendev.org/691034 | 22:23 |
| openstackgerrit | Jeremy Stanley proposed opendev/system-config master: Use a wildcard regex for storyboard-dev OAuth/CORS https://review.opendev.org/691440 | 22:23 |
| openstackgerrit | Merged opendev/storyboard master: Add support for POST requests to /v1/openid/authorize_return https://review.opendev.org/691428 | 22:24 |
| *** rosmaita has left #openstack-infra | 22:34 | |
| openstackgerrit | Merged openstack/project-config master: Override security suite for Debian image builds https://review.opendev.org/688817 | 22:36 |
| openstackgerrit | Merged openstack/project-config master: Bump min-ready for ubuntu-bionic https://review.opendev.org/691275 | 22:38 |
| openstackgerrit | James E. Blair proposed zuul/zuul-jobs master: Test buildset registry with k8s and docker https://review.opendev.org/689280 | 22:41 |
| *** ociuhandu has joined #openstack-infra | 22:43 | |
| *** yamamoto has joined #openstack-infra | 22:53 | |
| *** gyee has quit IRC | 22:53 | |
| openstackgerrit | James E. Blair proposed zuul/zuul-jobs master: Test buildset registry with k8s and docker https://review.opendev.org/689280 | 22:56 |
| *** yamamoto has quit IRC | 22:57 | |
| *** jtomasek has quit IRC | 22:58 | |
| *** jtomasek has joined #openstack-infra | 22:59 | |
| *** jtomasek has quit IRC | 22:59 | |
| *** ociuhandu has quit IRC | 23:04 | |
| *** rfolco|rover has quit IRC | 23:04 | |
| *** mattw4 has quit IRC | 23:06 | |
| openstackgerrit | Merged opendev/storyboard master: Regular expression support for CORS and OAuth ACLs https://review.opendev.org/691034 | 23:14 |
| *** markvoelker has joined #openstack-infra | 23:16 | |
| openstackgerrit | Monty Taylor proposed opendev/system-config master: Run a gerrit container (test only) https://review.opendev.org/630406 | 23:28 |
| openstackgerrit | Monty Taylor proposed opendev/system-config master: Run gerrit with podman instead https://review.opendev.org/689890 | 23:28 |
| openstackgerrit | Monty Taylor proposed opendev/system-config master: Apply review container to review-dev https://review.opendev.org/690483 | 23:28 |
| openstackgerrit | Monty Taylor proposed opendev/system-config master: Use explicit image paths https://review.opendev.org/690511 | 23:28 |
| openstackgerrit | Monty Taylor proposed opendev/system-config master: Update pip3 role to install from get-pip.py https://review.opendev.org/690766 | 23:28 |
| openstackgerrit | Monty Taylor proposed opendev/system-config master: Add service playbook and test run for prod gerrit https://review.opendev.org/691171 | 23:28 |
| openstackgerrit | Monty Taylor proposed opendev/system-config master: Add launchpadlib credentials to gerrit ansible https://review.opendev.org/691172 | 23:28 |
| openstackgerrit | Monty Taylor proposed opendev/system-config master: Add replication config to gerrit ansible https://review.opendev.org/691173 | 23:28 |
| openstackgerrit | Monty Taylor proposed opendev/system-config master: Fix gerrit 2.13 execution path https://review.opendev.org/691451 | 23:28 |
| *** markvoelker has quit IRC | 23:28 | |
| *** yamamoto has joined #openstack-infra | 23:30 | |
| *** tosky has quit IRC | 23:42 | |
| *** igordc has quit IRC | 23:42 | |
| *** rh-jelabarre has quit IRC | 23:47 | |
| *** eernst_ has quit IRC | 23:56 | |
| *** weifan has quit IRC | 23:57 | |
| *** jamesmcarthur has joined #openstack-infra | 23:59 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!