| *** jparoly has quit IRC | 00:14 | |
| *** hemanth_n has joined #openstack-qa | 00:55 | |
| openstackgerrit | melanie witt proposed openstack/tempest master: WIP Add configuration for compute unified limits feature https://review.opendev.org/c/openstack/tempest/+/790186 | 01:14 |
|---|---|---|
| openstackgerrit | Brian Rosmaita proposed openstack/tempest master: Add tests for "new" Transfers API (mv 3.55) https://review.opendev.org/c/openstack/tempest/+/790201 | 02:49 |
| *** ysirndjuro has quit IRC | 03:14 | |
| *** rcernin has quit IRC | 03:32 | |
| *** rcernin has joined #openstack-qa | 03:42 | |
| *** rcernin has quit IRC | 03:42 | |
| *** rcernin has joined #openstack-qa | 03:42 | |
| *** gcheresh has joined #openstack-qa | 04:12 | |
| *** Luzi has joined #openstack-qa | 05:41 | |
| *** psahoo has joined #openstack-qa | 05:45 | |
| *** ralonsoh has joined #openstack-qa | 05:46 | |
| *** ysandeep|away is now known as ysandeep | 06:02 | |
| *** slaweq has joined #openstack-qa | 06:05 | |
| *** jhesketh has quit IRC | 06:28 | |
| *** vishalmanchanda has joined #openstack-qa | 06:34 | |
| *** rpittau|afk is now known as rpittau | 07:05 | |
| *** psahoo has quit IRC | 07:32 | |
| *** rcernin has quit IRC | 07:34 | |
| *** psahoo has joined #openstack-qa | 07:45 | |
| *** tosky has joined #openstack-qa | 07:48 | |
| *** jpena|off is now known as jpena | 07:51 | |
| *** dtantsur|afk is now known as dtantsur | 07:58 | |
| *** lucasagomes has joined #openstack-qa | 08:02 | |
| *** dansmith has quit IRC | 08:08 | |
| *** dansmith has joined #openstack-qa | 08:09 | |
| *** rcernin has joined #openstack-qa | 08:16 | |
| *** ysandeep is now known as ysandeep|lunch | 08:17 | |
| *** rcernin has quit IRC | 08:22 | |
| *** gfidente|afk is now known as gfidente | 08:27 | |
| *** rcernin has joined #openstack-qa | 08:32 | |
| *** rcernin has quit IRC | 08:37 | |
| *** jhesketh has joined #openstack-qa | 09:01 | |
| *** rcernin has joined #openstack-qa | 09:06 | |
| *** rcernin has quit IRC | 09:11 | |
| *** ysandeep|lunch is now known as ysandeep | 10:06 | |
| *** brinzhang_ has quit IRC | 10:11 | |
| openstackgerrit | Anand Bhat proposed openstack/devstack-plugin-ceph master: Changed minversion in tox to 3.18.0 https://review.opendev.org/c/openstack/devstack-plugin-ceph/+/790215 | 10:15 |
| *** Luzi has quit IRC | 10:41 | |
| *** ccamposr__ has quit IRC | 10:45 | |
| *** ccamposr__ has joined #openstack-qa | 10:46 | |
| *** Luzi has joined #openstack-qa | 10:56 | |
| *** psahoo has quit IRC | 11:07 | |
| *** psahoo has joined #openstack-qa | 11:19 | |
| *** rcernin has joined #openstack-qa | 11:29 | |
| *** jpena is now known as jpena|lunch | 11:30 | |
| *** rcernin has quit IRC | 11:34 | |
| *** rcernin has joined #openstack-qa | 11:50 | |
| *** rcernin has quit IRC | 11:55 | |
| *** hemanth_n has quit IRC | 12:05 | |
| *** ysandeep is now known as ysandeep|brb | 12:05 | |
| *** brinzhang has joined #openstack-qa | 12:14 | |
| *** psahoo has quit IRC | 12:18 | |
| *** jpena|lunch is now known as jpena | 12:27 | |
| *** eliadcohen has quit IRC | 12:28 | |
| *** ysandeep|brb is now known as ysandeep | 12:29 | |
| *** psahoo has joined #openstack-qa | 12:32 | |
| *** eliadcohen has joined #openstack-qa | 12:34 | |
| openstackgerrit | Brian Rosmaita proposed openstack/tempest master: Add tests for "new" Transfers API (mv 3.55) https://review.opendev.org/c/openstack/tempest/+/790201 | 13:08 |
| *** jparoly has joined #openstack-qa | 13:10 | |
| *** paras333 has joined #openstack-qa | 13:20 | |
| *** nweinber has joined #openstack-qa | 13:23 | |
| *** elod is now known as elod_afk | 13:23 | |
| *** whoami-rajat has joined #openstack-qa | 13:39 | |
| *** rh-jelabarre has joined #openstack-qa | 13:44 | |
| *** paras333 has quit IRC | 13:52 | |
| *** paras333 has joined #openstack-qa | 13:55 | |
| *** paras333 has quit IRC | 13:58 | |
| *** Luzi has quit IRC | 14:00 | |
| *** rcernin has joined #openstack-qa | 14:08 | |
| *** rcernin has quit IRC | 14:13 | |
| *** nweinber has quit IRC | 14:23 | |
| *** nweinber has joined #openstack-qa | 14:25 | |
| *** rcernin has joined #openstack-qa | 14:25 | |
| *** rpittau is now known as rpittau|afk | 14:29 | |
| *** rcernin has quit IRC | 14:30 | |
| *** rcernin has joined #openstack-qa | 14:41 | |
| *** rcernin has quit IRC | 14:46 | |
| *** gcheresh has quit IRC | 15:20 | |
| *** ysandeep is now known as ysandeep|away | 16:01 | |
| *** lucasagomes has quit IRC | 16:07 | |
| dansmith | can someone tell me how to get out requirements hell here? https://pastebin.com/raw/v0SwPPuU | 16:09 |
| dansmith | trying to add oslo.limit to glance's lower-constraints | 16:09 |
| dansmith | oslo.limit has only one release AFAIK, so I would think this would be simple | 16:10 |
| dansmith | gmann: ^ | 16:10 |
| gmann | dansmith: keystoneauth1 need to be bumped at same time https://github.com/openstack/oslo.limit/blob/1.1.0/requirements.txt#L4 | 16:11 |
| gmann | that is why maintaining l-c is complex sometime due to these indirect deps | 16:12 |
| dansmith | so I bump keystonauthe to >=3.9 in requirements and to ==3.9.0 in l-c? | 16:12 |
| gmann | yeah | 16:12 |
| dansmith | gmann: yes, I'm very confused why people want to do this :) | 16:12 |
| gmann | i thought you removed l-c from glance? | 16:12 |
| gmann | or it was just stable | 16:12 |
| dansmith | gmann: just stable, still arguing about master, unfortunately :/ | 16:13 |
| dansmith | cc abhishekk ^ | 16:13 |
| gmann | humm | 16:13 |
| gmann | abhishekk: dansmith I agree, we are maintaining these without any value | 16:13 |
| dansmith | apparently nova and cinder have also decided to keep it on master, but remove from stable when it breaks, creating inconsistent lineages :/ | 16:13 |
| gmann | also from master, when it break and no fix then propose to remvoe | 16:14 |
| gmann | oslo removed from master | 16:15 |
| dansmith | ah, that's good to know.. hard to understand why it makes sense on the projects if oslo dropped it | 16:16 |
| gmann | exactly | 16:16 |
| dansmith | gmann: well, I may have created some infinite loop trying to bump keystoneauth, | 16:20 |
| dansmith | pip is sitting at 100% for a long time trying to run tox lower-constraints :/ | 16:20 |
| *** jpena is now known as jpena|off | 16:20 | |
| dansmith | yeah, nothing moving in the logs, so I think it's stuck | 16:22 |
| clarkb | the new resolver can definitely be slow. Not sure if it also suffers from infinite looping | 16:23 |
| clarkb | constraints is what addresses much of that problem for us in CI, but if you are trying to generate a new constraints list then you don't have that beneift | 16:24 |
| dansmith | clarkb: slow like how long? ten minutes of 100% cpu on a fast machine? | 16:24 |
| dansmith | well, eight minutes | 16:25 |
| dansmith | certainly didn't take this long with just the oslo.limit bump, just after the keystone one | 16:25 |
| gmann | you might see more conflict there so that might be taking time | 16:26 |
| gmann | but should not stuck | 16:26 |
| dansmith | okay | 16:27 |
| clarkb | ya it can take many minutes aiui | 16:27 |
| dansmith | cripes | 16:28 |
| *** psahoo has quit IRC | 16:30 | |
| dansmith | I blew away the .tox and restarted it and it's still going | 16:35 |
| dansmith | 14 minutes now | 16:35 |
| clarkb | I want to say pip was soliciting setups that caused the resolver to have fits. Not sure if they still want those | 16:40 |
| clarkb | fungi: ^ do you know? | 16:40 |
| dansmith | getting ready to hit 19 minutes | 16:40 |
| dansmith | this is the change I'm making against glance: https://termbin.com/tql8 | 16:42 |
| *** rcernin has joined #openstack-qa | 16:42 | |
| dansmith | that's legit right? | 16:42 |
| clarkb | oh wait you are manually editing lower-constraints (I thought you were solving for them) | 16:43 |
| clarkb | in that case I do wonder if pip is unable to satisfy the constraint for some reason | 16:43 |
| dansmith | I thought I had to edit it? | 16:43 |
| dansmith | if I don't, the l-c job fails and tells me that I can't have oslo.limit in my requirements.txt file without it in l-c | 16:44 |
| clarkb | I think these struggles are why projects dropped l-c, but I still thought there aws some solver for it (I'm probably wrong on that) | 16:44 |
| dansmith | unfortunately, | 16:44 |
| dansmith | the projects have not dropped them | 16:44 |
| clarkb | the way upper constraints work is you update requirements.txt and then install everything and then you take the result and set that as upper constraints | 16:44 |
| fungi | yes, so if you have an incomplete constraints list, you can basically ask pip's dep solver to come up with impossible solutions | 16:45 |
| clarkb | I thought there was something similar for l-c but maybe not | 16:45 |
| dansmith | if I'm doing the wrong thing, I'm happy to be educated | 16:45 |
| fungi | i contend that using constraints lists for lower-bounds testing is untenable. so if you ask me, that's what's wrong | 16:46 |
| dansmith | I totally agree | 16:46 |
| dansmith | and I have been arguing to remove the testing from master, | 16:46 |
| dansmith | but I seem to be in the minority | 16:46 |
| fungi | i can get into gödel's arguments for why, but would rather not break out the set theory proofs | 16:47 |
| gmann | yeah, lot of hack and indirect dep bump needed | 16:47 |
| *** rcernin has quit IRC | 16:47 | |
| dansmith | so past 25 minutes.. I can stop this and call it dead right? | 16:47 |
| fungi | it's dead, jim | 16:47 |
| dansmith | I'm just going to rebase my patches on my remove-l-c patch again :) | 16:47 |
| gmann | dansmith: I will say, to bring it again in glance that this is broken and not worth to maintain it compare to its value addition | 16:48 |
| gmann | +1 | 16:48 |
| dansmith | gmann: yep, I'm going to point that review at this log :) | 16:48 |
| fungi | the problem would be solveable if someone wrote an inverse version selector for pip and used that to generate the lower bounds list for the entire transitive set, but even then it would require regular care and feeding, regenerating any time a lower bound on one dep was raised | 16:49 |
| fungi | and that still assumes your dependencies are also testing and managing their lower bounds similarly, which they're pretty well guaranteed not to be | 16:50 |
| dansmith | since all but one of the packagers don't care about this, I'm going to say it's not worth it :) | 16:52 |
| dansmith | but that has been said over and over already | 16:52 |
| fungi | and then there's the historical issue... you're talking about extracting transitive dependency information out of the most ancient direct dependencies, and then iterating that process... those old versions almost certainly wouldn't have received the same care for lower-bounds tracking even if the new versions were | 16:52 |
| dansmith | well, another problem I have, is that keeping this on master (which nobody packages) and then deleting it from stable branches when they start to fail, creates this non-linear history where packagers can't depend on it being there anyway | 16:52 |
| fungi | i think the idea for that compromise was not to delete the lower-constraints.txt file, but just to stop testing it once the stable branch is created | 16:53 |
| fungi | but that assumed that making it work on master was even possible | 16:53 |
| dansmith | ya'll feel free to pile on! https://review.opendev.org/c/openstack/glance/+/782768/3 | 16:55 |
| dansmith | fungi: yeah I think that's broken because we then advertise that the list works, but aren't testing it | 16:55 |
| dansmith | fungi: so far, people seem most in favor of deleting the file when we remove the testing, | 16:56 |
| dansmith | which we have already done in all the glance stable trees, fwiw | 16:56 |
| fungi | it's not like anything in git is ever truly deleted anyway. it can be checked out from the branch history or tag history | 16:56 |
| dansmith | yup | 16:57 |
| dansmith | getting it from the point in time at which it stopped working is probably more instructive than keeping it in tree alongside changes that may have broken it | 16:57 |
| *** dtantsur is now known as dtantsur|afk | 17:12 | |
| *** gfidente is now known as gfidente|afk | 17:16 | |
| *** vishalmanchanda has quit IRC | 17:23 | |
| *** ralonsoh has quit IRC | 17:34 | |
| *** brinzhang_ has joined #openstack-qa | 17:53 | |
| *** nweinber has quit IRC | 17:54 | |
| *** brinzhang has quit IRC | 17:56 | |
| *** dviroel has quit IRC | 18:31 | |
| *** csatari has quit IRC | 18:31 | |
| *** rpioso has quit IRC | 18:31 | |
| *** knikolla has quit IRC | 18:32 | |
| *** lxkong has quit IRC | 18:32 | |
| openstackgerrit | Anand Bhat proposed openstack/devstack master: Changed minversion in tox to 3.18.0 https://review.opendev.org/c/openstack/devstack/+/790292 | 18:34 |
| *** knikolla has joined #openstack-qa | 18:35 | |
| *** csatari has joined #openstack-qa | 18:35 | |
| *** dviroel has joined #openstack-qa | 18:35 | |
| *** lxkong has joined #openstack-qa | 18:35 | |
| *** rpioso has joined #openstack-qa | 18:35 | |
| *** vishalmanchanda has joined #openstack-qa | 18:38 | |
| *** rcernin has joined #openstack-qa | 18:43 | |
| *** rcernin has quit IRC | 18:48 | |
| yoctozepto | yeah, drop l-c everywhere | 18:57 |
| *** artom has quit IRC | 20:01 | |
| *** artom has joined #openstack-qa | 20:02 | |
| dansmith | gmann: I don't have time to look at this today, but I've been seeing this lately: https://zuul.opendev.org/t/openstack/build/eaa99aee1d504ff792bd52011f2248db/log/controller/logs/syslog.txt#5180 | 20:38 |
| dansmith | mysql getting OOM killed | 20:38 |
| clarkb | dstat crashed prior to that too with OverflowError: cannot convert float infinity to integer | 20:41 |
| clarkb | one thing we did (though not super recently at this point) was reduce the size of swap because ext4 on newer machines cannot be sparse allocated and used as swap anymore | 20:43 |
| clarkb | and writing out 8GB of zeros is not fast | 20:43 |
| clarkb | possible that makes OOMKiller trip more quickly than previously | 20:43 |
| dansmith | because we're using a swap file right? | 20:44 |
| dansmith | unless something has changed, swapping to a file increases vm pressur | 20:44 |
| clarkb | yes | 20:44 |
| dansmith | *pressure | 20:44 |
| clarkb | but a smaller file in the last few months to prior file sizes | 20:45 |
| clarkb | also I think devstack-gate may have set swappiness to try and counteract this a bit, not sure if the zuulv3 jobs do similar | 20:45 |
| dansmith | ack | 20:45 |
| *** gfidente|afk has quit IRC | 20:46 | |
| *** jparoly has quit IRC | 20:47 | |
| clarkb | the devstack jobs do set swappiness https://zuul.opendev.org/t/openstack/build/eaa99aee1d504ff792bd52011f2248db/console#2/0/22/controller | 20:47 |
| openstackgerrit | James Parker proposed openstack/whitebox-tempest-plugin master: Add vgpu resize tests https://review.opendev.org/c/openstack/whitebox-tempest-plugin/+/790304 | 20:51 |
| *** vishalmanchanda has quit IRC | 20:58 | |
| clarkb | One thing I notice looking at the OOMKiller output is that we seem to be running quite a few nova-conductors as well as swift services. I think devstack(-gate) tried to tune those down to a size that made sense for small dev deployments. Not sure if that is working properly (though also that doesn't seem to be a massive cost maybe a couple hundred MB?) | 20:58 |
| dansmith | [conductor] | 20:59 |
| dansmith | workers = 2 | 20:59 |
| dansmith | should only be two actual processes | 20:59 |
| dansmith | those might be tasks for threadpooling? | 20:59 |
| clarkb | oh ya could be | 20:59 |
| clarkb | looks like three groups of 2 pids | 21:00 |
| clarkb | and maybe ~9 qemu processes which also seems high since tempest should only be running on 4 threads? | 21:01 |
| clarkb | maybe those aren't cleaning up as quickly as necessary to keep memory down | 21:01 |
| clarkb | I suspect that may actually be the biggest controllable cost | 21:02 |
| dansmith | this is a ceph job, so there's a lot going on | 21:03 |
| dansmith | but I think I was seeing it last week not in ceph jobs, but not positive | 21:04 |
| *** rcernin has joined #openstack-qa | 21:37 | |
| *** paras333 has joined #openstack-qa | 21:42 | |
| *** rcernin has quit IRC | 21:42 | |
| *** paras333 has quit IRC | 21:47 | |
| *** rcernin has joined #openstack-qa | 22:05 | |
| *** rcernin has quit IRC | 22:26 | |
| *** tosky has quit IRC | 23:08 | |
| *** jparker has quit IRC | 23:27 | |
Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!