| * clarkb looks at grafana docs to see if they differ | 00:00 | |
| ianw | i think in this case it may be "poke at grafana source" | 00:01 |
|---|---|---|
| clarkb | ianw: https://grafana.com/docs/grafana/latest/variables/templates-and-variables/ doesn't show a definition (but does query). I think you may be right about rtfsing | 00:01 |
| openstackgerrit | Mohammed Naser proposed openstack/project-config master: gerrit: change retired.config acls https://review.opendev.org/737649 | 00:09 |
| ianw | i feel like this was the last thing that touched it -> https://github.com/grafana/grafana/commit/cc813d71c6c0fa92f438be4c5bff63109475f282 | 00:13 |
| ianw | Showing with 8,298 additions and 232 deletions. | 00:14 |
| ianw | i.e. good luck | 00:14 |
| ianw | https://github.com/grafana/grafana/blob/master/public/app/features/variables/query/reducer.ts#L31 | 00:15 |
| clarkb | ianw: wow is that not a merge commit? | 00:17 |
| clarkb | ianw: https://github.com/grafana/grafana/commit/cc813d71c6c0fa92f438be4c5bff63109475f282#diff-f97144bba3190d0259b13e0e6f5c430cR72 | 00:18 |
| ianw | it definitely feels like definition is the thing here ... why i'm not sure | 00:19 |
| clarkb | and no documentation update | 00:21 |
| clarkb | ianw: https://github.com/grafana/grafana/commit/cc813d71c6c0fa92f438be4c5bff63109475f282#diff-2da0f7d413588f993c6a59c2d4c69770R107 that is interesting because it seems to say I prefer definition but I fall back to query | 00:24 |
| clarkb | ianw: maybe its a half baked transition of terms that ended up breaking the old thing | 00:25 |
| ianw | it seems we should write both ... i can maybe test this | 00:27 |
| *** ryohayakawa has joined #opendev | 00:35 | |
| ianw | clarkb: i think it's the "refresh" actually | 00:43 |
| ianw | yeah it is, it shouldn't be a bool; it doesn't update. it's still ... odd in the UI | 00:53 |
| ianw | (the variable update UI0 | 00:53 |
| openstackgerrit | Ian Wienand proposed openstack/project-config master: grafana: don't use bool for refresh https://review.opendev.org/737662 | 00:55 |
| clarkb | ianw: doesthat have more than two value options? | 00:58 |
| ianw | it's got 0=never 1=on load 2=on time range change | 00:58 |
| clarkb | huh | 00:58 |
| ianw | i mean, grafyaml kind of falls down here because htf is anyone supposed to know that | 00:59 |
| fungi | i take it the config specification for grafana is "whatever the webui produces" | 01:00 |
| ianw | fungi: well insert react/angular/typescript/blah layers somewhere in there and, yeah, i think so | 01:01 |
| openstackgerrit | Hirotaka Wakabayashi proposed openstack/diskimage-builder master: Fixes to use the latest centos-8 release image https://review.opendev.org/737663 | 01:01 |
| *** diablo_rojo has quit IRC | 01:01 | |
| ianw | i get the feeling they keep the json compatible so you can import old dashboards | 01:01 |
| *** xiaolin has joined #opendev | 01:05 | |
| openstackgerrit | Ian Wienand proposed opendev/grafyaml master: Remove query variable refresh deprecation https://review.opendev.org/737664 | 01:13 |
| ianw | clarkb / fungi : ^ that's my suggestion to avoid future confusion | 01:13 |
| ianw | system-config-run-grafanaSUCCESS in 16m 05s | 01:14 |
| ianw | so that's good ... just need to beat up documentation into shape and it will be gtg | 01:14 |
| *** mrunge_ has joined #opendev | 01:20 | |
| *** mrunge has quit IRC | 01:21 | |
| openstackgerrit | Merged openstack/project-config master: grafana: don't use bool for refresh https://review.opendev.org/737662 | 01:23 |
| fungi | ianw: 737664 seems to have some job failures | 01:26 |
| fungi | i didn't dig into them, but likely bitrot | 01:26 |
| ianw | yeah, requirements stuff and pep failures ... i will yak shave :) | 01:28 |
| *** DSpider has quit IRC | 01:46 | |
| ianw | AJaeger / clarkb: I think I like option 2 in https://etherpad.opendev.org/p/-CBx0IaMT37oFBHdt8iV if i've understood. basically don't put pep8/publish jobs lumped in with tox jobs? | 01:48 |
| openstackgerrit | Ian Wienand proposed openstack/project-config master: grafyaml: drop python2 jobs https://review.opendev.org/737666 | 01:51 |
| openstackgerrit | Ian Wienand proposed opendev/grafyaml master: Drop Python 2 support https://review.opendev.org/737667 | 01:54 |
| *** shtepanie has quit IRC | 02:20 | |
| *** sgw1 has quit IRC | 02:50 | |
| *** sgw1 has joined #opendev | 03:06 | |
| *** sgw1 has quit IRC | 03:12 | |
| *** sgw1 has joined #opendev | 03:15 | |
| *** sgw1 has quit IRC | 03:58 | |
| *** sgw1 has joined #opendev | 04:00 | |
| *** xiaolin has quit IRC | 04:11 | |
| *** ysandeep|away is now known as ysandeep | 04:28 | |
| *** aannuusshhkkaa has quit IRC | 04:45 | |
| openstackgerrit | Ian Wienand proposed openstack/project-config master: wheel-cache: convert release to a loop https://review.opendev.org/737678 | 04:59 |
| ianw | AJaeger: ^ rather unsatisfying, but I can not replicate this | 05:00 |
| AJaeger | ianw: https://review.opendev.org/#/c/737525/ should have fixed it already | 05:19 |
| *** sshnaidm|afk is now known as sshnaidm|off | 05:34 | |
| ianw | AJaeger: arrrggghhhh!!!! i should have looked! | 05:54 |
| ianw | no wonder it looked right, it was :) | 05:55 |
| *** ysandeep is now known as ysandeep|afk | 06:30 | |
| AJaeger | :9 | 06:33 |
| AJaeger | :) | 06:33 |
| *** xiaolin has joined #opendev | 06:56 | |
| *** rpittau|afk is now known as rpittau | 06:57 | |
| openstackgerrit | Albin Vass proposed zuul/zuul-jobs master: Test multiarch release builds https://review.opendev.org/737315 | 07:17 |
| *** hashar has joined #opendev | 07:30 | |
| openstackgerrit | Albin Vass proposed zuul/zuul-jobs master: Test multiarch release builds https://review.opendev.org/737315 | 07:34 |
| *** ysandeep|afk is now known as ysandeep | 07:42 | |
| openstackgerrit | Albin Vass proposed zuul/zuul-jobs master: Test multiarch release builds https://review.opendev.org/737315 | 07:47 |
| *** priteau has joined #opendev | 07:47 | |
| *** tosky has joined #opendev | 07:49 | |
| *** bhagyashris is now known as bhagyashris|lunc | 07:57 | |
| *** moppy has quit IRC | 08:01 | |
| openstackgerrit | Vishal Manchanda proposed openstack/project-config master: Upadting horizon nodejs job name https://review.opendev.org/737457 | 08:03 |
| *** moppy has joined #opendev | 08:03 | |
| openstackgerrit | Vishal Manchanda proposed openstack/project-config master: Upadting horizon nodejs job name https://review.opendev.org/737457 | 08:12 |
| openstackgerrit | Albin Vass proposed zuul/zuul-jobs master: Test multiarch release builds https://review.opendev.org/737315 | 08:50 |
| openstackgerrit | Albin Vass proposed zuul/zuul-jobs master: Test multiarch release builds https://review.opendev.org/737315 | 09:04 |
| *** bhagyashris|lunc is now known as bhagyashris | 09:08 | |
| *** mrunge_ is now known as mrunge | 09:22 | |
| *** hashar has quit IRC | 09:48 | |
| *** tkajinam has quit IRC | 10:03 | |
| *** rpittau is now known as rpittau|bbl | 10:11 | |
| *** xiaolin has quit IRC | 10:12 | |
| *** DSpider has joined #opendev | 10:23 | |
| openstackgerrit | Jonathan Rosser proposed openstack/project-config master: Refresh openstack-ansible grafana dashboards https://review.opendev.org/737742 | 10:39 |
| *** hashar has joined #opendev | 10:41 | |
| openstackgerrit | Vishal Manchanda proposed openstack/project-config master: Upadting horizon nodejs job name https://review.opendev.org/737457 | 11:43 |
| openstackgerrit | Vishal Manchanda proposed openstack/project-config master: Upadting horizon nodejs job name https://review.opendev.org/737457 | 11:44 |
| *** ysandeep is now known as ysandeep|brb | 11:58 | |
| *** rpittau|bbl is now known as rpittau | 12:13 | |
| *** ysandeep|brb is now known as ysandeep | 12:14 | |
| openstackgerrit | Jonathan Rosser proposed openstack/project-config master: Refresh openstack-ansible grafana dashboards https://review.opendev.org/737742 | 12:29 |
| *** ryohayakawa has quit IRC | 12:29 | |
| *** rchurch has quit IRC | 12:34 | |
| *** rchurch has joined #opendev | 12:37 | |
| openstackgerrit | Albin Vass proposed zuul/zuul-jobs master: Test multiarch release builds https://review.opendev.org/737315 | 12:43 |
| openstackgerrit | Thierry Carrez proposed zuul/zuul-jobs master: upload-git-mirror: check after mirror operation https://review.opendev.org/737533 | 12:49 |
| openstackgerrit | Thierry Carrez proposed zuul/zuul-jobs master: upload-git-mirror: check after mirror operation https://review.opendev.org/737533 | 12:50 |
| openstackgerrit | Thierry Carrez proposed zuul/zuul-jobs master: upload-git-mirror: check after mirror operation https://review.opendev.org/737533 | 12:51 |
| fungi | infra-root: we may want to consider scaling down the number of builds we're accepting on executors, and maybe increase the number of executors. they're consistently hovering around 2gb swap in use | 13:11 |
| fungi | when under load | 13:11 |
| frickler | fungi: I'm assuming this is related to the failures mentioned over in #-infra? | 13:20 |
| fungi | frickler: i don't think it's related at all, just something i noticed while trying to debug them | 13:21 |
| frickler | fungi: ah o.k., things permanently swapped out wouldn't seem so bad, but it looks like there's some non-significant amount of swapping going on and cpu usage is permanently around 70% at least on ze01 where I just looked | 13:27 |
| frickler | so yes, planning for more executors seems reasonable | 13:28 |
| fungi | frickler: yeah, the "swapping activity" graphs indicate this isn't just unused pages which have been shuffled to swap | 13:28 |
| corvus | true, though it's not new behavior: http://cacti.openstack.org/cacti/graph.php?action=view&local_graph_id=64005&rra_id=all | 13:30 |
| fungi | right, i didn't mean to imply it was | 13:37 |
| fungi | in fact, first i thought maybe the switch to ansible 2.9 was the cause, but seems the behavior long predates that | 13:37 |
| corvus | yeah, i was wondering the same | 13:38 |
| frickler | infra-root: we seem to have almost 300 nodes in deleting for like 18 hours, maybe also ovh related? can someone dig deeper? | 13:43 |
| mordred | frickler: I will look right now | 13:43 |
| fungi | i can take a look in an hour or so | 13:43 |
| frickler | http://grafana.openstack.org/d/rZtIH5Imz/nodepool?orgId=1&from=now-24h&to=now | 13:43 |
| fungi | aha, thanks mordred! | 13:43 |
| fungi | i need to disappear for my grocery pickup appointment, but will be back as soon as i can | 13:44 |
| mordred | I betcha anything it is ovh related | 13:44 |
| mordred | yup. too old sdk on launchers | 13:47 |
| * mordred checking ot make sure latest image has latest code | 13:47 | |
| mordred | yes. they do | 13:48 |
| corvus | mordred: so we just need a launcher restart? | 13:48 |
| mordred | infra-root: I'm going to restart the launcher containers to pick up latest | 13:48 |
| mordred | yes | 13:48 |
| mordred | corvus: I guess I should really restart all of nodepool - builders will also have issues with old sdk | 13:51 |
| corvus | yeah | 13:54 |
| openstackgerrit | Aurelien Lourot proposed openstack/project-config master: Add Neutron Arista plugin charm to OpenStack charms https://review.opendev.org/737791 | 13:54 |
| mordred | #status log restarted nodepool to pick up latest openstacksdk | 13:54 |
| openstackstatus | mordred: finished logging | 13:54 |
| mordred | k. hopefully we should be able to delete nodes now and get quota back | 13:55 |
| openstackgerrit | Mohammed Naser proposed openstack/project-config master: gerrit: change retired.config acls https://review.opendev.org/737649 | 13:58 |
| frickler | grafana nodepool looks good, seems that had blocked like 30% of our capacity, so things might go smoother now | 14:03 |
| *** mlavalle has joined #opendev | 14:29 | |
| *** ysandeep is now known as ysandeep|afk | 14:29 | |
| *** hashar has quit IRC | 14:52 | |
| *** ysandeep|afk is now known as ysandeep | 14:56 | |
| corvus | frickler, mordred: \o/ | 15:06 |
| clarkb | see I really thought we had done that switch months ago | 15:08 |
| clarkb | did we not apply it in the correct places or were the updates incomplete? | 15:08 |
| *** ysandeep is now known as ysandeep|away | 15:09 | |
| clarkb | hrm double checking the clouds.yaml files we never set domain info there? Maybe we tested it and then didn't switch or this got lost in the puppet to ansible transition? | 15:10 |
| clarkb | mordred: frickler: do we need to update any of the templates in system-config/playbooks/templates/clouds/ ? | 15:14 |
| clarkb | fungi: not sure if you saw but https://review.opendev.org/#/c/729649/ to do the mailing list owner change for starlingx hit a merge conflict | 15:16 |
| mordred | clarkb: no, we should not need to | 15:18 |
| mordred | clarkb: the ovh profile in latest sdk sets default values for those | 15:19 |
| mordred | infra-root: I'm going to land the zuul-executor-to-docker change, ze* is in the emergency file already | 15:20 |
| mordred | once it lands I'll start rolling that out | 15:20 |
| clarkb | mordred: the afs on ze image change did land yesterday so there are no other concerns with that I know of | 15:21 |
| mordred | ++ | 15:24 |
| *** priteau has quit IRC | 15:27 | |
| openstackgerrit | Sean McGinnis proposed openstack/project-config master: Make tox global for update proposal jobs https://review.opendev.org/737836 | 15:36 |
| *** aannuusshhkkaa has joined #opendev | 15:58 | |
| *** shtepanie has joined #opendev | 15:59 | |
| openstackgerrit | Thierry Carrez proposed zuul/zuul-jobs master: upload-git-mirror: check after mirror operation https://review.opendev.org/737533 | 16:02 |
| *** gmann has joined #opendev | 16:04 | |
| openstackgerrit | Merged opendev/system-config master: Run zuul-executor using docker https://review.opendev.org/733967 | 16:10 |
| *** ysandeep|away is now known as ysandeep | 16:16 | |
| *** ysandeep is now known as ysandeep|away | 16:26 | |
| *** rpittau is now known as rpittau|afk | 16:41 | |
| openstackgerrit | Merged openstack/project-config master: Make tox global for update proposal jobs https://review.opendev.org/737836 | 16:49 |
| openstackgerrit | Merged openstack/project-config master: Retire dragonflow project https://review.opendev.org/737566 | 17:01 |
| openstackgerrit | Jeremy Stanley proposed opendev/system-config master: Revert "Reject messages to starlingx-discuss-owner" https://review.opendev.org/729649 | 17:16 |
| fungi | clarkb: yeah, so i'm thinking maybe jgit can't follow git mv | 17:17 |
| fungi | though locally a rebase was able to automatically shift the diff to the new file | 17:18 |
| fungi | yet one more place jgit is inferior (or maybe just more conservative) | 17:18 |
| fungi | here's some opendevification of jeepyb i pushed a while back, if anyone's interested: https://review.opendev.org/728479 | 17:23 |
| fungi | in particular it updates the url for our manual from the old openstack docs site to ours, reducing the openstackiness of new contributor welcome messages | 17:24 |
| fungi | and rewords the message to be generally more project-neutral | 17:25 |
| openstackgerrit | Merged openstack/project-config master: gerrit: change retired.config acls https://review.opendev.org/737649 | 17:25 |
| fungi | er, i guess i did the rewording in an earlier patch, before we moved the manual | 17:26 |
| mordred | AJaeger: do you know the source for https://www.openstack.org/appdev is? | 17:35 |
| mordred | oh - I guess that's an openstack.org one | 17:35 |
| fungi | mordred: seems to be https://github.com/OpenStackweb/openstack-org/blob/c41955c5a6b283276978b991a7e34cf84eed1127/themes/openstack/templates/Layout/AppDevHomePage.ss | 17:40 |
| mordred | fungi: thanks! I nudged diablo_rojo_phon about it and she said she's going to ping someone | 17:43 |
| fungi | cool | 17:44 |
| fungi | mordred: related to the question from fitzgerald1337 in #openstack i suppose | 17:45 |
| *** priteau has joined #opendev | 17:48 | |
| mordred | fungi: yeah | 17:50 |
| mordred | fungi: because a) the link should work and then b) it should not be a link to a guide about libcloud | 17:50 |
| mordred | although c) I really should get around to making an openstacksdk-based drier for libcloud | 17:50 |
| clarkb | I'm going to pop out for a bike ride nowish. back in a bit. | 17:57 |
| clarkb | Happy to help wtih executor restarts when I'm back but don't feel like you have to wait for me | 17:57 |
| openstackgerrit | Albin Vass proposed zuul/zuul-jobs master: Test multiarch release builds https://review.opendev.org/737315 | 17:57 |
| openstackgerrit | Merged opendev/system-config master: Revert "Reject messages to starlingx-discuss-owner" https://review.opendev.org/729649 | 18:07 |
| *** priteau has quit IRC | 18:16 | |
| *** diablo_rojo has joined #opendev | 18:24 | |
| openstackgerrit | Rafael Folco proposed openstack/diskimage-builder master: DNM: Debug py3 on dib 7 https://review.opendev.org/736421 | 18:26 |
| *** mlavalle has quit IRC | 19:26 | |
| *** mlavalle has joined #opendev | 19:31 | |
| clarkb | mordred: looks like buildx has distracted, are we going yo do a single executor first or all at once or? | 19:36 |
| openstackgerrit | Rafael Folco proposed openstack/diskimage-builder master: DNM: Debug py3 on dib 7 https://review.opendev.org/736421 | 19:36 |
| openstackgerrit | Albin Vass proposed zuul/zuul-jobs master: Test multiarch release builds https://review.opendev.org/737315 | 19:54 |
| openstackgerrit | Jeremy Stanley proposed opendev/infra-specs master: Central Authentication Service https://review.opendev.org/731838 | 20:03 |
| openstackgerrit | Albin Vass proposed zuul/zuul-jobs master: Test multiarch release builds https://review.opendev.org/737315 | 20:19 |
| *** priteau has joined #opendev | 20:28 | |
| openstackgerrit | Merged openstack/project-config master: grafyaml: drop python2 jobs https://review.opendev.org/737666 | 20:34 |
| mordred | clarkb: one at a time | 20:35 |
| mordred | clarkb: or, at least a single one first | 20:35 |
| mordred | clarkb: but I'm fighting a different fire currently | 20:35 |
| openstackgerrit | Clark Boylan proposed opendev/grafyaml master: Remove query variable refresh deprecation https://review.opendev.org/737664 | 20:36 |
| clarkb | mordred: ok let me know if I can help. I'm just poking at reviews currently | 20:36 |
| mordred | clarkb: kk. thanks! | 20:38 |
| *** priteau has quit IRC | 20:39 | |
| fungi | so... starting early utc on 2020-06-19 the infra-prod-manage-projects job began failing. because something doesn't seem to be maintaniing log retention for this correctly on bridge.o.o i don't know for sure that the latest errors are consistent with the earlier ones, but currently we're seeing org creation failing on all gitea servers with | 21:48 |
| fungi | "409 Client Error: Conflict for url: https://localhost:3000/api/v1/org/x/repos fatal: [gitea08.opendev.org]: FAILED! | 21:48 |
| clarkb | fungi: what is the problem with log retention? | 21:49 |
| clarkb | we should be doing a 30 day rotation | 21:50 |
| fungi | the earliest entry in /var/log/ansible/manage-projects.yaml.log has a timestamp of 2020-06-24T18:41:45Z | 21:50 |
| clarkb | ya we only have 3 files there for some reason. We should do a log file per job and then keep 30 days of those files | 21:51 |
| fungi | the only other files matching /var/log/ansible/manage-projects.yaml.log* are /var/log/ansible/manage-projects.yaml.log.2020-04-08T19:29:57Z and /var/log/ansible/manage-projects.yaml.log.2020-04-09T15:13:08Z | 21:51 |
| fungi | so we seem to have started rotating them in early april, but then switched to just overwriting the file i guess? | 21:51 |
| clarkb | fungi: we log that job to zuul apparnetly | 21:54 |
| clarkb | that is why the files on disk are stale | 21:55 |
| fungi | interesting, the task details in the zuul console beg to differ | 21:55 |
| clarkb | https://zuul.openstack.org/build/ef79b73c7ab143a5bf365b98afc72183/log/manage-projects.yaml.log for example | 21:55 |
| clarkb | the failures on the 19th were different | 21:55 |
| fungi | https://zuul.opendev.org/t/openstack/build/ef79b73c7ab143a5bf365b98afc72183/console#1/2/13/bridge.openstack.org | 21:56 |
| fungi | no stdout or stderr, but just "non-zero return code" | 21:56 |
| clarkb | fungi: ya the way it works is we run ansible and write it out to a log file on bridge. Then for some jobs we copy that log file to swift | 21:57 |
| clarkb | but only those jobs we've audited to be clean | 21:57 |
| clarkb | they are not echo'd into the console log because we don't run a zuul console logger | 21:57 |
| fungi | and the _raw_params say we ran ansible-playbook for manage-projects.yaml with redirection to /var/log/ansible/manage-projects.yaml.log | 21:57 |
| clarkb | anyway I think that means this could be a gitea upgrade problem | 21:57 |
| clarkb | fungi: yup thats the file we copy into swift | 21:57 |
| fungi | ohh, okay, so the file is removed after each build completes | 21:58 |
| clarkb | yes | 21:58 |
| fungi | https://zuul.opendev.org/t/openstack/build/ef79b73c7ab143a5bf365b98afc72183/log/manage-projects.yaml.log#4655 | 21:58 |
| fungi | so it failed against review-test? | 21:59 |
| clarkb | looks like it (we maybe don't want to run it there? I dunno | 21:59 |
| clarkb | but also the failure with gitea is newer which makes me think maybe it is gitea upgrade related. I'm trying to see if I can find a log where just gitea01 fails | 21:59 |
| clarkb | since we upgraded it first | 21:59 |
| clarkb | but I don't think we have such a thing | 22:00 |
| fungi | fatal: [review-test.opendev.org]: FAILED! => { "changed": false, "cmd": "docker pull docker.io/opendevorg/gerrit:2.13", "rc": 2 } | 22:00 |
| fungi | yeah, so the failures starting on the 19th may be a red herring | 22:01 |
| clarkb | fungi: there are two places we use that gitea api path the first is to list repos and the second is the create a repo | 22:05 |
| clarkb | I'm assuming that a listing isn't going to 409 but a create will | 22:05 |
| clarkb | which makes me believe that we are trying to create repos that already exist and gitea is sad | 22:05 |
| clarkb | and that first call feeds itno the second for determining if we need to create a project | 22:07 |
| clarkb | so now i guess we need to see what that first call produces | 22:08 |
| * clarkb tries to figure this out | 22:08 | |
| fungi | or are we trying to create orgs? | 22:09 |
| clarkb | I don't think we are trying to recreate orgs based on the path | 22:11 |
| clarkb | its a different path for that | 22:11 |
| clarkb | I've managed to do the json listing and it looks like what we expect | 22:13 |
| clarkb | oh except maybe its not a complete listing now? | 22:14 |
| fungi | so maybe repo creation has ceased to be idempotent in newer gitea | 22:14 |
| fungi | ahh, pagination? | 22:14 |
| clarkb | I only get 30 repos back from the listing | 22:15 |
| clarkb | I think we have more x/ repos than that | 22:15 |
| fungi | so perhaps they started paginating the results | 22:15 |
| fungi | we query for existing repos to decide what to create, get back only the first page, and then proceed to try to create repos which actually exist on later pages | 22:15 |
| clarkb | ya I think that is what is happening | 22:15 |
| clarkb | yup I see it in the haeders | 22:17 |
| clarkb | there is a Link header with a next page and a last page | 22:17 |
| clarkb | I wonder if we can request a page size | 22:17 |
| mnaser | clarkb: https://github.com/go-gitea/gitea/commit/5b902e2368fdb67eda4ba033f7167fadf72e4350 ? | 22:17 |
| fungi | worst case we loop until next page == current | 22:17 |
| fungi | or something along those lines | 22:18 |
| clarkb | max page size is 50 | 22:18 |
| clarkb | so I think we'll need to do loops | 22:18 |
| fungi | yep, loops it is | 22:18 |
| clarkb | and we can test this by running manage projects twice in gate | 22:18 |
| clarkb | and I've confirmed the limit works :) | 22:19 |
| clarkb | if you go above 50 you get back 50 results | 22:19 |
| fungi | rather than an error? | 22:20 |
| clarkb | correct | 22:20 |
| clarkb | also if you go above the valid page count you get an empty list | 22:20 |
| clarkb | I'm working on a patch now | 22:20 |
| clarkb | fungi: maybe you can work on a change that runs the playbook twice to test it and I'll rebase on that? | 22:21 |
| clarkb | it should fail on the chagne you push up and pass o nthe one I push up? | 22:21 |
| fungi | do we have a working test for it? | 22:21 |
| fungi | that playbook normally runs against the production servers | 22:22 |
| * fungi hunts | 22:22 | |
| clarkb | yes we test it pre merge against a fake gitea | 22:22 |
| clarkb | but we only do one pass not two so this problem sneaks by | 22:22 |
| fungi | ahh, okay, but i guess we don't see this because we run once, got it | 22:22 |
| fungi | so it would be a modification to the system-config-run-gitea job i guess, or maybe the playbooks/manage-projects.yaml playbook, to duplicate what we run there | 22:25 |
| clarkb | ya one of those I think the gitea job actually | 22:26 |
| fungi | does including a playbook twice cause it to get run twice? | 22:26 |
| clarkb | not sure :) | 22:27 |
| clarkb | fungi: yes ti should | 22:30 |
| clarkb | so I think that is what we want manage-projects listed twice in the gitea job | 22:30 |
| clarkb | its doing a proper ansible loop over those contents. No funny zuul stuff there | 22:31 |
| openstackgerrit | Jeremy Stanley proposed opendev/system-config master: DNM: Test manage-projects idempotency https://review.opendev.org/737881 | 22:31 |
| fungi | that ought to do it at least | 22:31 |
| mordred | __ | 22:32 |
| mordred | ++ | 22:32 |
| fungi | i couldn't find where we were including the playbook in a test-specific scenario | 22:33 |
| fungi | instead we just seem to add job nodes named like production and point our production playbooks at them | 22:33 |
| openstackgerrit | Clark Boylan proposed opendev/system-config master: Deal with gitea pagination of repo lists https://review.opendev.org/737882 | 22:34 |
| clarkb | fungi: https://opendev.org/opendev/system-config/src/branch/master/zuul.d/system-config-run.yaml#L502 but I think your version will work too | 22:34 |
| fungi | oh, right in the vars list | 22:35 |
| clarkb | we won't want to merge your version but if this shows things are working we can flip the change order and update the list at the link I gave to test this going forward | 22:36 |
| fungi | it wasn't clear to me that list wasn't deduplicated, but i can also try that if preferred | 22:36 |
| clarkb | fungi: I don't think it is beacuse its a normal ansible list that is iterated through | 22:36 |
| clarkb | fungi: only zuul level stuff gets modified | 22:36 |
| fungi | got it. i can give that a shot too, for an actually mergeable alternative, in case we decide we want it | 22:37 |
| clarkb | maybe we'll just wait for results as we need to rebase my fix off of your change before we can merge it anyway | 22:37 |
| clarkb | (since we don't want manage projects to run twice in prod) | 22:37 |
| openstackgerrit | Jeremy Stanley proposed opendev/system-config master: Make sure manage-projects is idempotent https://review.opendev.org/737883 | 22:39 |
| fungi | the alternative ^ | 22:39 |
| *** mlavalle has quit IRC | 22:39 | |
| fungi | we can at least see if either or both ways yield the same failure | 22:40 |
| clarkb | I wonder if anything else got paginated | 22:44 |
| clarkb | I guess wecan try ti check | 22:44 |
| clarkb | we also get orgs and org teams | 22:47 |
| clarkb | I'm not having luck getting orgs through my browser like I did my repos | 22:47 |
| clarkb | mordred: corvus ^ any idea why that acts differently? | 22:48 |
| clarkb | what I did was login to gitea01 through the web ui then my reuqests for the org repo listing works. but /api/v1/user/orgs is 401'd | 22:49 |
| clarkb | oh I can just check the swagger instead | 22:50 |
| mordred | clarkb: nope - no clue - but yes swagger | 22:51 |
| clarkb | also there is a 1.12.1 forwhen all this settles down | 22:51 |
| *** tkajinam has joined #opendev | 22:51 | |
| clarkb | yup the others appear to be paginated too | 22:53 |
| corvus | clarkb: sorry, i've been fightining a private fire -- how can i help? | 22:55 |
| mordred | corvus: fun with paginated queries! | 22:55 |
| corvus | 1st tldr question: is there corrupt data in production? | 22:56 |
| *** cloudnull has quit IRC | 22:56 | |
| corvus | or is the script just harmlessly failing | 22:56 |
| clarkb | it is harmlessly failing | 22:57 |
| clarkb | I don't think this is an emergency | 22:57 |
| clarkb | the inconvenience is we aren't running manage projects to update gerrit acls | 22:57 |
| clarkb | but the existing state is stable | 22:57 |
| corvus | cool. so we're just wanting to find the right api sequence to fix this. | 22:57 |
| clarkb | yup and I think I've got it for the most immediate issue and am trying to figure it out for the other place we do this but don't hit it because all the iterables have less than 30 items | 22:58 |
| clarkb | I'll do the fixes for the additional items in a followup so can more carefully check those once we figure out how to manually exercise the api | 22:58 |
| *** tosky has quit IRC | 23:03 | |
| corvus | i'm glad that's in python :) | 23:04 |
| corvus | clarkb: you based your change on fungi's dnm change | 23:05 |
| openstackgerrit | Clark Boylan proposed opendev/system-config master: Paginate all the gitea get requests https://review.opendev.org/737885 | 23:05 |
| fungi | i also have a second non-dnm change as an alternative which we could probably keep if we want to run m-p twice in the job | 23:05 |
| clarkb | corvus: yup we want to confirm that this fixes the problem | 23:05 |
| clarkb | corvus: the dnm change is running manage projects twice which is sort of how it looks like in production when we run manage project periodically | 23:06 |
| mordred | I kind of think we should merge fungi's dnm change... yeah - I thin running twice in the gate is a good idea | 23:06 |
| clarkb | mordred: ++ but we want to do it for the alternative not the one I'm based on right now | 23:06 |
| clarkb | since the one I'm based on right now would run manage projects twice in production too | 23:06 |
| corvus | why not base on the alternative? | 23:06 |
| fungi | non-dnm change hopefully? 737883 | 23:06 |
| clarkb | corvus: the alternative came later | 23:06 |
| corvus | ok. i understand the plan now. :) | 23:06 |
| clarkb | corvus: we can rebase once we get first pass of results | 23:06 |
| fungi | yeah, it was an afterthought because i wasn't sure those don't get deduped | 23:07 |
| clarkb | basically concurrent optimizations happening | 23:07 |
| corvus | (we're avoiding wasting the sunk testing cost with a rebase) | 23:07 |
| mordred | ++ | 23:07 |
| fungi | correctamundo | 23:07 |
| clarkb | https://review.opendev.org/737885 is a followon to do pagination of all the things I can find that do it according to gitea swagger. Not yet verified in production liek I did for the repos listing | 23:07 |
| clarkb | https://zuul.opendev.org/t/openstack/stream/88e6de69e6334be6b35b4f4c7132d91f?logfile=console.log should run manage-projects twice with my fix | 23:09 |
| clarkb | once that completes I'll rebase and incorporate any necessary fixes | 23:09 |
| clarkb | I'm going to step out for a few while we wait on that | 23:13 |
| clarkb | hrm fungi's run manage projects twice change passed | 23:30 |
| clarkb | oh I see why | 23:30 |
| clarkb | the bit that runs the gitea stuff happens at the beginning of the file so we aren't testing it properly there | 23:30 |
| clarkb | the other change should test it properly though | 23:30 |
| clarkb | I'll rebase now | 23:30 |
| openstackgerrit | Clark Boylan proposed opendev/system-config master: Deal with gitea pagination of repo lists https://review.opendev.org/737882 | 23:32 |
| openstackgerrit | Clark Boylan proposed opendev/system-config master: Paginate all the gitea get requests https://review.opendev.org/737885 | 23:32 |
| clarkb | I expect ^ will be more reliable testing | 23:32 |
| fungi | yeah, 737881 succeeded (unfortunately), while 737883 failed like we wanted | 23:46 |
| fungi | i'll abandon the former | 23:46 |
| clarkb | https://zuul.opendev.org/t/openstack/build/87a58974f36140afbe380569cad878f6/log/job-output.txt#18307 yup here is our reproduction | 23:46 |
| clarkb | and I've already rebased so we should be good to see if these follow up changes help | 23:46 |
| *** ryohayakawa has joined #opendev | 23:58 | |
| *** ryohayakawa has joined #opendev | 23:58 | |
Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!