| *** ysandeep|away is now known as ysandeep | 06:32 | |
| *** jpena|off is now known as jpena | 07:32 | |
| whoami-rajat | hi #openstack-infra , I'm facing an issue with glance_store project, I've removed the lower-constraints job but my patch (proposed before the job removal) is still running it, if i propose a new DNM patch then it's not running there, i tried rebasing and again cherry picking but it doesn't work, patch: https://review.opendev.org/c/openstack/glance_store/+/786789 | 07:44 |
|---|---|---|
| *** rpittau|afk is now known as rpittau | 07:54 | |
| *** ykarel is now known as ykarel|lunch | 09:27 | |
| *** ykarel|lunch is now known as ykarel | 10:38 | |
| *** jpena is now known as jpena|lunch | 11:39 | |
| *** rlandy is now known as rlandy|rover | 11:49 | |
| *** jpena|lunch is now known as jpena | 12:42 | |
| fungi | whoami-rajat: i agree something strange is happening there. i can see change https://review.opendev.org/804606 merged yesterday to remove openstack-lower-constraints-jobs from the templates list, and your 786789 change was tested after that (even has the removal as its parent git commit, though that shouldn't be necessary). the build claims it's getting it from that template though: | 13:25 |
| fungi | https://zuul.opendev.org/t/openstack/build/acb88b19c55047858741a72a022e9b81/log/zuul-info/inventory.yaml#49 | 13:25 |
| fungi | source: openstack/openstack-zuul-jobs/zuul.d/project-templates.yaml@master#1664 | 13:26 |
| fungi | that's the line where the definition of the template starts | 13:26 |
| fungi | so all that's to say i'm as confused as you are at the moment | 13:27 |
| fungi | i'll ask for some debugging tips | 13:27 |
| whoami-rajat | fungi, ok, one more thing i noticed, i proposed a DNM in which if i do any change in .zuul file then lower-constraints doesn't run and if not then it runs | 13:28 |
| fungi | yeah, normally if you change the zuul configuration in your project zuul explicitly selects to run all impacted jobs just to make sure you haven't made a configuration change that leads one to fail... i wonder if that's sidestepping some cached value | 13:30 |
| whoami-rajat | got it | 13:31 |
| whoami-rajat | so any way we can debug to find the root cause? | 13:32 |
| fungi | i've linked this discussion in #zuul to get some additional ideas | 13:32 |
| fungi | since it seems like it could be a bug in configuration caching | 13:33 |
| whoami-rajat | ok, thanks for looking into it | 13:35 |
| *** thelounge555 is now known as thelounge55 | 13:35 | |
| dansmith | clarkb: can you help me remember what the trick is for debugging hung tests from the subunit stream? grabbing the stream and running -ls on it, I thought there was "started" and "finished" for each test, but I'm not seeing that | 14:32 |
| *** ykarel is now known as ykarel|away | 14:41 | |
| clarkb | dansmith: sorry was (still in) a meeting | 15:00 |
| dansmith | np | 15:00 |
| clarkb | dansmith: I find it helps to run the file through the subunit2to1 translator as subunut v1 is far more human parseable | 15:01 |
| clarkb | and ya a hung test will sometimes have a started but not a finished | 15:01 |
| clarkb | oh and just view the v1 converted file with your text editor | 15:02 |
| dansmith | okay the thing I've got seems to already be in v1 format | 15:02 |
| clarkb | ah we might record it in v1 for this reason then | 15:02 |
| dansmith | but I don't see *any* started..finished indications | 15:03 |
| dansmith | unless the tags: -worker1 and tags: worker1 mean something like that | 15:03 |
| dansmith | when I run some stats on it, I see nothing from one actually, but all the other workers have something | 15:03 |
| dansmith | which seems relevant, if I knew what list of tests that one was doing, but... | 15:03 |
| clarkb | ok ya looks like there isn't started and finished | 15:04 |
| clarkb | I see time: then tags: for the worker then test: test.name.path.etc then time: then failure: or success:: | 15:05 |
| dansmith | yeah | 15:05 |
| clarkb | I think you want to look for test: foo.bar and then if there is no success: or failure: that one got stuck | 15:05 |
| dansmith | okay, but it seems like one whole worker is missing: | 15:07 |
| dansmith | - Worker 0 (55 tests) => 0:01:26.408110 | 15:07 |
| dansmith | - WARNING: missing Worker 1! Race in testr accounting | 15:07 |
| dansmith | I see zero results for worker1 in the subunit | 15:07 |
| dansmith | which I guess pretty much means this isn't going to help me | 15:07 |
| clarkb | ya I suspect the issue happened before data was getting recordred in the stream | 15:08 |
| clarkb | that one may have crashes its stream somehow? | 15:08 |
| clarkb | dansmith: what you might try is list all the tests that ran in your subunit file then remove them from the total test list then just run those remaining tests one by one to see if you can crash it | 15:08 |
| dansmith | yeah I'm guessing it hung or crashed before it did anything | 15:09 |
| dansmith | not sure where to chase that though | 15:09 |
| clarkb | that might be a bit of annoying text processing work but if the other tests all ran happily we can assume the failure is likely in the other portion of the test list | 15:09 |
| dansmith | yeah, was just thinking that might be all I could discern from this | 15:09 |
| dansmith | i.e the list of tests that didn't get run | 15:10 |
| clarkb | yup | 15:10 |
| *** jpena is now known as jpena|off | 15:26 | |
| *** jpena|off is now known as jpena | 15:27 | |
| *** jpena is now known as jpena|off | 15:48 | |
| *** ysandeep is now known as ysandeep|dinner | 16:02 | |
| opendevreview | Sorin Sbârnea proposed openstack/openstack-zuul-jobs master: tox: help py36 jobs use utf8 encoding https://review.opendev.org/c/openstack/openstack-zuul-jobs/+/804890 | 16:06 |
| opendevreview | Sorin Sbârnea proposed openstack/openstack-zuul-jobs master: tox: help py36 jobs use utf8 encoding https://review.opendev.org/c/openstack/openstack-zuul-jobs/+/804890 | 16:10 |
| clarkb | zbr: is the issue that ubuntu 18.04 and centos 7 don't set LANG/LOCALE/LC to a utf8 value by default and then python assumes ascii? Or is it that they do set LANG/LOCALE/LC to utf8 values and python doesn't properly handle them? | 16:19 |
| clarkb | I wonder if it would be worth considering updating the platform locale settings if it is the former | 16:19 |
| zbr | clarkb: they don't set it and this means that py36 (only) will use ascii as that is what it used until py37. | 16:20 |
| zbr | breaking pip is one issue, but anything python related is affected by this. | 16:20 |
| clarkb | ya so it might be more correct to fix this with a locale update, but then we aren't testing what people in the wild will run into | 16:20 |
| clarkb | if we are worried about what people in the wild will run into then you should set that value in the tox.ini files and not the zuul job config | 16:21 |
| *** ysandeep|dinner is now known as ysandeep|out | 16:21 | |
| zbr | not sure what you think about this but I would dare recommend even ensuring that all node images we have with py36 also have the environment tunned to help py36 find a more inclusive locale. | 16:21 |
| clarkb | I've +2'd it but I suspect the two best otpions are actually either to update the platform and just accept we aren't going to test exactly what centos7 and bionic look like in the wild. Or set this value in tox.ini instead | 16:21 |
| zbr | while i am not sure yet, i suspect tunning only tox.ini may not work, as pip is used before the env is even created (hint tox self-bootstrapping for example). | 16:22 |
| clarkb | the problem with setting it in the job like that is we aren't really testing what people get in the wild and we aren't providing them any clue for why it works in ci but not locally | 16:22 |
| clarkb | zbr: I'm not sure I understand tox self bootstrapping. I've always had to preinstall tox myself | 16:22 |
| zbr | self-bootstrapping is not always happening, is determined by what you have already installed and if tox.ini requires extra plugins. | 16:23 |
| zbr | basically is happening only when needed | 16:23 |
| clarkb | and setting tox_environment only applies after tox is installed | 16:23 |
| clarkb | zbr: ok, but the problem with this is if anyone runs tox locally on centos 7 or bionic it will fail if that env var is needed and they won't understand why it works in CI | 16:24 |
| zbr | indeed :D | 16:24 |
| zbr | but the funny bit is that pip fails due to bad encoding only with some packages, not all. | 16:24 |
| clarkb | it is better imo to force projects that need this to set it in their own tox.ini | 16:24 |
| clarkb | then it works locally and if it fails it fails for everyone and it is debuggable | 16:24 |
| zbr | imaging someone adds some "smart" things to their package description, and when tries to install it, it chokes not being able to print those chars. | 16:25 |
| clarkb | like I said I +2'd it. I don't feel that strongly about this. But imo it is better to not solve it this way | 16:25 |
| clarkb | yes it depends on whether or not the package metadata is pure ascii | 16:25 |
| zbr | i am close to leave for the day but I am glad you are aware of the issue. not sure which patch we should take. | 16:25 |
| clarkb | also this can fail if someone uses utf16 (though that is very unlikely) | 16:25 |
| zbr | afaik, nobody enforced package metadata to be ascii. | 16:25 |
| clarkb | I'm not sure if any of the specs specify an encoding. And if they do that may not be enforced | 16:26 |
| fungi | tangentially, https://github.com/pypa/pip/issues/10219 is the first time i've seen anyone suggest that mentioning a github issue in a commit message for another project is rude behavior. seems like something github should make configurable in their issue tracker | 16:27 |
| clarkb | fungi: github is being far too helpful there :) | 16:28 |
| zbr | haha, yeah. that is weird at best. i think that also has something to do with pip maintainers willing to allow a PR. | 16:29 |
| zbr | i would not use "rude" to describe it, is more of "peer pressure" ;) | 16:29 |
| fungi | but on the actual problem, it looks like pip is simply using open() and expecting python to dtrt, so the declaring a specific metadata encoding doesn't appear to be an option | 16:30 |
| zbr | yes | 16:30 |
| clarkb | I think all python is doing is looking up your local and using the encoding there | 16:30 |
| fungi | right | 16:30 |
| clarkb | if you have C then you get ascii if you have C.utf8 then utf8 and so on | 16:30 |
| clarkb | it is entirely possible for someone to try and be funny and make a package with utf16 or similar and then cause all sorts of problems I suspect | 16:31 |
| fungi | and then pep 538 codified that as c meaning c.utf-8 but only took effect in python 3.7 | 16:31 |
| clarkb | ah that is what changed | 16:31 |
| *** rpittau is now known as rpittau|afk | 16:31 | |
| fungi | right, that's why the pip maintainers seemed disinclined to fix the regression. as soon as they drop python 3.6 support it will go away | 16:32 |
| fungi | of course, i would argue this means they have effectively already dropped python 3.6 support and just don't want to come out and say that? | 16:32 |
| clarkb | it probably wouldn't be a terrible idea to stick to ascii in package metadata though | 16:32 |
| clarkb | the openstack release team should probaly consider a check for that | 16:32 |
| clarkb | fungi: yes I would agree that is the same thing | 16:33 |
| fungi | i think we implicitly check for that by testing projects with python 3.6 on a platform with no default utf-8 locale | 16:33 |
| clarkb | fungi: zbr's efforts will undo that though | 16:33 |
| fungi | tox plus recent pip is making an implicit wheel and installing it | 16:33 |
| clarkb | yes I agree toay we are fine | 16:34 |
| clarkb | related: the requirements team might want to filter packages that don't respect that rule for compat reasons as well | 16:34 |
| clarkb | or just accept that emoji in package metadata is a thing now | 16:34 |
| fungi | well, apparently not "fine" because https://launchpad.net/bugs/1938079 was filed | 16:34 |
| clarkb | fungi: ya a third party package introduced the problem. Ansible? | 16:36 |
| clarkb | I suspect that all of our packages are still clean though | 16:36 |
| fungi | more likely one of ansible's kitchen sink of dependencies | 16:36 |
| fungi | i wouldn't necessarily characterize it as "emoji in package metadata" since it's entirely legitimate for someone whose name can't be accurately represented in pure ascii to want to have a proper encoding of their name in the maintainer field or something | 16:39 |
| clarkb | thats fair. The trouble is that there is no way to know if that is latin1 encoded or utf8 or utf16 etc | 16:39 |
| fungi | its just that pip seems to be okay with breaking that on python 3.6 and earlier, so de facto phasing out support for 3.6 i guess | 16:39 |
| clarkb | and while python wants to assume everything is utf8 it doesn't even do that properly | 16:39 |
| fungi | yeah, it does seem like there should be a pep declaring explicit and implicit encoding rules for package metadata | 16:40 |
| clarkb | the only thing that python does seem to do reliably is try ascii | 16:40 |
| clarkb | which is unfortunate if you have a legit reason to use an extended character set | 16:41 |
| fungi | you can declare encoding for the long description | 16:41 |
| fungi | though even that only works reliably if it's in a separate file | 16:41 |
| fungi | and i think that's happening in setuptools when writing the metadata anyway | 16:42 |
| fungi | the encoding conversion i mean | 16:42 |
| abhishekk | zuul status is showing "Something went wrong." | 22:03 |
| fungi | abhishekk: it's mid-restart but should be back momentarily | 22:04 |
| fungi | (see conversation in #opendev) | 22:04 |
| abhishekk | fungi, ack, thank you | 22:04 |
| fungi | all the in-flight changes will get reenqueued once it's back up | 22:06 |
| fungi | so hopefully shouldn't need to recheck anything | 22:06 |
| fungi | unless you just happened to push a new change or approve one while zuul was down | 22:06 |
| abhishekk | ok | 22:07 |
| abhishekk | no later case for me, just had couple of changes already running which got terminated | 22:07 |
| fungi | yeah, they'll come back | 22:08 |
| abhishekk | cool | 22:08 |
| fungi | abhishekk: your changes should be back into the pipelines as of a few minutes ago | 22:28 |
| abhishekk | yes | 22:29 |
| fungi | cool, just making sure | 22:30 |
| abhishekk | ack, thank you | 22:31 |
Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!