| -@gerrit:opendev.org- Anil Shashikumar Belur proposed: [opendev/system-config] 999195: Convert iptables role to native nftables https://review.opendev.org/c/opendev/system-config/+/999195 | 01:55 | |
| -@gerrit:opendev.org- Anil Shashikumar Belur proposed: [opendev/system-config] 999195: Convert iptables role to native nftables https://review.opendev.org/c/opendev/system-config/+/999195 | 01:57 | |
| -@gerrit:opendev.org- Clark Boylan proposed: [opendev/grafyaml] 999030: Use datasource uids when updating datasources https://review.opendev.org/c/opendev/grafyaml/+/999030 | 02:44 | |
| @clarkb:matrix.org | Anil Belur: ok that latest patchset updates the existing test cases to be more specific about their assertions. I think that is worth doing now. But adding additional testing makes more sense as a followup and not sure I'll get to that immediately if you want to take a stab at it if I don't | 02:44 |
|---|---|---|
| @clarkb:matrix.org | Anil Belur: looking at the nftables change it looks like it is all or nothing. I think it would be good to consider approaches that allow us to more gradually convert things so that we can convert the less risky hosts before converting those with higher risk levels. One idea I had previously was to only opt in new servers. But another option may be to keep the old implementation and add the new implementation as a new role then convert service by service to the new role | 02:49 |
| @clarkb:matrix.org | something to think about, but I think there are some servers where it would be a scramble to address firewall issues | 02:49 |
| @clarkb:matrix.org | so planning ahead and making it a more gradual roll out is good if we can do so without too much trouble. I'm open to other ideas too those were just the two approaches that came to mind when I thought about this once upon a time | 02:50 |
| @abelur:matrix.org | Clark: thanks for addressing my review comments. yeah, a follow up change works for additional tests for delete() etc! | 02:52 |
| @abelur:matrix.org | Clark: I'd share the same concern and wanted to get some insights and how to go about it. I try to split that into smaller chunks to address the less risky hosts first! I'll only be able to get to it next week. | 02:54 |
| @clarkb:matrix.org | ack sounds good. Ya another third option may be to use a new group that we add hosts to. I think there are a few approaches that would work and we just need to sketch a likely one out and if it looks good stick with it and if not try another | 02:55 |
| @clarkb:matrix.org | I'm learning that the periodic daily jobs really interrupt everything else during this time of day :) | 03:20 |
| @mnasiadka:matrix.org | Clark, fungi : I think I'll go with v3-standard-4 which is 4 vCPU and 16GB RAM, if that's not enough after launching all node_exporters we can bump to 8 vCPU and 32GB RAM (in VexxHost's ca-ymq-1 region) | 04:57 |
| @clarkb:matrix.org | Ack | 06:01 |
| @mnasiadka:matrix.org | Ok, booting prometheus01.opendev.org in ca-ymq-1 | 09:23 |
| -@gerrit:opendev.org- Michal Nasiadka proposed: [opendev/zone-opendev.org] 999250: Add prometheus01 https://review.opendev.org/c/opendev/zone-opendev.org/+/999250 | 09:28 | |
| -@gerrit:opendev.org- Michal Nasiadka proposed: [opendev/system-config] 999251: Add prometheus01 https://review.opendev.org/c/opendev/system-config/+/999251 | 09:29 | |
| @fungicide:matrix.org | mnasiadka: 999250 needs the serial increased | 14:22 |
| @fungicide:matrix.org | otherwise lgtm | 14:26 |
| @fungicide:matrix.org | i'm going out to run some lunch errands, won't be long, probably back by 16:30 utc | 14:46 |
| -@gerrit:opendev.org- Jeremy Stanley https://matrix.to/#/@fungicide:matrix.org proposed on behalf of Michal Nasiadka: [opendev/zone-opendev.org] 999250: Add prometheus01 https://review.opendev.org/c/opendev/zone-opendev.org/+/999250 | 16:27 | |
| @fungicide:matrix.org | trivially fixed my comment and approved it | 16:28 |
| @fungicide:matrix.org | once that deploys and resolves i'll approve the inventory addition | 16:28 |
| -@gerrit:opendev.org- Zuul merged on behalf of Michal Nasiadka: [opendev/zone-opendev.org] 999250: Add prometheus01 https://review.opendev.org/c/opendev/zone-opendev.org/+/999250 | 16:36 | |
| @mnasiadka:matrix.org | fungi: thanks for updating the dns change, I got carried away on drinking beer in 36 degrees celcius… | 16:39 |
| @fungicide:matrix.org | that's nothing to apologize for! i'd be right there beside you if i could | 16:43 |
| -@gerrit:opendev.org- Zuul merged on behalf of Michal Nasiadka: [opendev/system-config] 999251: Add prometheus01 https://review.opendev.org/c/opendev/system-config/+/999251 | 17:13 | |
| @fungicide:matrix.org | infra-prod-letsencrypt failed in deploy for that ^ | 17:59 |
| @fungicide:matrix.org | `TASK [letsencrypt-create-certs : Run acme.sh driver for prometheus-opendev-org-main certificate issue] *** fatal: [prometheus01.opendev.org]: FAILED!` | 18:01 |
| @fungicide:matrix.org | `prometheus.opendev.org:Verify error:DNS problem: NXDOMAIN looking up TXT for _acme-challenge.prometheus.opendev.org - check that a DNS record exists for this domain` | 18:02 |
| @fungicide:matrix.org | i can confirm via `host -t txt _acme-challenge.prometheus.opendev.org` that `Host _acme-challenge.prometheus.opendev.org not found: 3(NXDOMAIN)` for me as well | 18:02 |
| @fungicide:matrix.org | my bad, i missed that the cname for it was missing a trailing `.` when i reviewed and approved the addition | 18:05 |
| @fungicide:matrix.org | i'll also add a missing cname for _acme-challenge.prometheus01 while i'm in there in case we want both in the cert | 18:05 |
| -@gerrit:opendev.org- Jeremy Stanley https://matrix.to/#/@fungicide:matrix.org proposed: [opendev/zone-opendev.org] 999325: Correct ACME CNAME for prometheus https://review.opendev.org/c/opendev/zone-opendev.org/+/999325 | 18:06 | |
| @fungicide:matrix.org | i'll self-approve that ^ once zuul gives a +1 in order to keep things moving quickly, then i'll reenqueue the failed deploy buildset once it's resolving | 18:07 |
| @fungicide:matrix.org | pip 26.2 came out today (not over a weekend for a change!) so be on the lookout for new regressions: https://pip.pypa.io/en/stable/news/#v26-2 | 18:14 |
| -@gerrit:opendev.org- Zuul merged on behalf of Jeremy Stanley https://matrix.to/#/@fungicide:matrix.org: [opendev/zone-opendev.org] 999325: Correct ACME CNAME for prometheus https://review.opendev.org/c/opendev/zone-opendev.org/+/999325 | 18:21 | |
| @fungicide:matrix.org | overriding `104.239.142.36 opendev.org` for the held gitea-lb03 node to test the anubis 1.26.0 change is working for me, i get the anubis challenge when visiting https://opendev.org/ and it reports it's running 1.26.0 instead of 1.25.0 briefly at the bottom while the hash is being calculated | 18:38 |
| @fungicide:matrix.org | okay, infra-prod-letsencrypt is running now in the reenqueued buildset | 18:45 |
| @fungicide:matrix.org | as long as it succeeds this time, i expect we can move forward with next steps for prometheus | 18:46 |
| @fungicide:matrix.org | which it just did, so the rest of the buildset should be smooth sailing (but also doesn't touch the new prometheus01 even if there is an unexpected problem) | 18:48 |
| @fungicide:matrix.org | oh right, infra-prod-service-grafana is still failing because we need to finish up merging Clark's grafyaml fix, i'll revisit that next | 18:50 |
| @fungicide:matrix.org | looks like it's going to merge after a recheck, but Clark: heads up that `test_is_datasource_true` seems to have some nondeterministic behaviors when testing for a consistent uid | 19:28 |
| @fungicide:matrix.org | not sure what happened there in https://zuul.opendev.org/t/openstack/build/136a858cc01744fca6ec46facbb61223 | 19:28 |
| @fungicide:matrix.org | it worked 3/4 times | 19:29 |
| -@gerrit:opendev.org- Zuul merged on behalf of Clark Boylan: [opendev/grafyaml] 999030: Use datasource uids when updating datasources https://review.opendev.org/c/opendev/grafyaml/+/999030 | 19:34 | |
| -@gerrit:opendev.org- Zuul merged on behalf of Clark Boylan: [opendev/system-config] 999031: Run grafyaml twice in testing https://review.opendev.org/c/opendev/system-config/+/999031 | 20:24 | |
| @fungicide:matrix.org | infra-prod-service-grafana isn't triggered in deploy by ^ | 20:25 |
| @fungicide:matrix.org | which i guess makes sense as it only alters the behavior of the system-config-run-grafana job | 20:28 |
| @fungicide:matrix.org | we'll either need to reenqueue a change like 999251 or wait, which given the lack of urgency is probably fine | 20:29 |
| @clarkb:matrix.org | > <@fungicide:matrix.org> looks like it's going to merge after a recheck, but Clark: heads up that `test_is_datasource_true` seems to have some nondeterministic behaviors when testing for a consistent uid | 21:44 |
| This is weird since its using a mocked out python request which should consistently return the same data | ||
| @fungicide:matrix.org | yeah, and yet somehow... one time it didn't do that | 21:51 |
| @clarkb:matrix.org | I wonder if requests mocker is not multi thread safe and we are leaking state across multiple test cases. And maybe we never noticed before because the assertions were so vague | 21:57 |
| @clarkb:matrix.org | If that is the issue serializing the test cases should make it stable | 21:57 |
| @fungicide:matrix.org | i assumed it was something to do with a lack of thread safety leaking state across tests | 21:58 |
| @fungicide:matrix.org | with some race condition involved | 21:58 |
| @clarkb:matrix.org | The total test runtime is under a second in parallel so should only be a few seconds serially. I'll push up a change to do that shortly | 22:00 |
| -@gerrit:opendev.org- Clark Boylan proposed: [opendev/grafyaml] 999344: Run unittests with concurrency 1 https://review.opendev.org/c/opendev/grafyaml/+/999344 | 22:27 | |
| @clarkb:matrix.org | ok serializing things makes the failure more consistent :) | 22:31 |
| @clarkb:matrix.org | the hash that we're getting back that is "wrong" is the sha256 sum for the url which is what we set the uid to when creating a datasource | 22:41 |
| @clarkb:matrix.org | this test case shouldn't be creating a datasource though. It must be creating it in a previous case then getting that data back ratherthan the directly mocked data (maybe because the creation process consumes the mock?) | 22:42 |
| @clarkb:matrix.org | I think using that hash value is actually correct because that is what the rest of grafyaml would be using, but it should work to use the specific value here and I don't understand why it doesn't yet | 22:44 |
| @clarkb:matrix.org | requests.session does not cache things which removes that theory | 22:46 |
| -@gerrit:opendev.org- Clark Boylan proposed: [opendev/grafyaml] 999344: Run unittests with concurrency 1 https://review.opendev.org/c/opendev/grafyaml/+/999344 | 22:50 | |
| -@gerrit:opendev.org- Clark Boylan proposed: [opendev/grafyaml] 999344: Run unittests with concurrency 1 https://review.opendev.org/c/opendev/grafyaml/+/999344 | 22:58 | |
| @clarkb:matrix.org | sorry I'm doing a bit of debug through zuul to avoid killing local battery life | 22:58 |
| @clarkb:matrix.org | ok isolating to that specific test case does make things work | 23:02 |
| @clarkb:matrix.org | so this is a test order side effect issue | 23:02 |
| -@gerrit:opendev.org- Clark Boylan proposed: [opendev/grafyaml] 999344: Run unittests with concurrency 1 https://review.opendev.org/c/opendev/grafyaml/+/999344 | 23:12 | |
| -@gerrit:opendev.org- Clark Boylan proposed: [opendev/grafyaml] 999344: Run unittests with concurrency 1 https://review.opendev.org/c/opendev/grafyaml/+/999344 | 23:23 | |
| @clarkb:matrix.org | if the create test runs before the is datasource true test then we fail. The create method sets the uid to a value based on a sha256 sum hash of the datasource url. This is the value we are getting back in the datasource true test | 23:26 |
| @clarkb:matrix.org | something in the create test is leaking across test boundaries that I'm not understanding yet | 23:26 |
| @clarkb:matrix.org | requests session shouldn't cache things. And we should have a new session per test case anyway. Similarly we decorate each test case with a new requests-mock mocker object | 23:26 |
| @clarkb:matrix.org | the datasource object doesn't seem to store data globally either | 23:27 |
| @clarkb:matrix.org | oh I think its the data we pass to the mock | 23:27 |
| @clarkb:matrix.org | the create method is modifying it in place rather than modifying a copy | 23:27 |
| -@gerrit:opendev.org- Clark Boylan proposed: [opendev/grafyaml] 999344: Run unittests with concurrency 1 https://review.opendev.org/c/opendev/grafyaml/+/999344 | 23:34 | |
| -@gerrit:opendev.org- Clark Boylan proposed: [opendev/grafyaml] 999344: Use copies of data when creating datasources https://review.opendev.org/c/opendev/grafyaml/+/999344 | 23:46 | |
| @clarkb:matrix.org | fungi: ^ ok I think that will fix the issue | 23:46 |
Generated by irclog2html.py 4.1.0 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!