Friday, 2020-11-06

*** iurygregory has quit IRC01:26
*** dansmith has quit IRC02:34
*** dansmith has joined #openstack-release02:34
*** ykarel|away has joined #openstack-release05:01
*** ykarel|away is now known as ykarel05:02
*** vishalmanchanda has joined #openstack-release05:32
*** evrardjp has quit IRC05:33
*** evrardjp has joined #openstack-release05:33
*** e0ne has joined #openstack-release06:32
*** rpittau|afk is now known as rpittau06:47
*** e0ne has quit IRC06:59
*** e0ne has joined #openstack-release07:18
*** sboyron has joined #openstack-release07:19
*** toabctl has quit IRC07:32
*** toabctl has joined #openstack-release07:32
*** e0ne has quit IRC07:41
*** sboyron has quit IRC07:43
openstackgerritDmitriy Rabotyagov (noonedeadpunk) proposed openstack/releases master: Release OpenStack-Ansible Train
openstackgerritDmitriy Rabotyagov (noonedeadpunk) proposed openstack/releases master: Release OpenStack-Ansible Ussuri
*** iurygregory_ has joined #openstack-release08:09
*** sboyron has joined #openstack-release08:10
*** iurygregory_ is now known as iurygregory08:14
*** dtantsur|afk is now known as dtantsur08:42
openstackgerritArtem Goncharov proposed openstack/releases master: Release cliff 3.5.0 for Wallaby
*** e0ne has joined #openstack-release08:53
openstackgerritArtem Goncharov proposed openstack/releases master: Release cliff 3.5.0 for Wallaby
*** sboyron has quit IRC09:54
*** sboyron has joined #openstack-release09:54
*** slaweq has joined #openstack-release09:56
*** brinzhang0 has quit IRC10:42
*** dtantsur is now known as dtantsur|bbl11:13
*** e0ne has quit IRC11:38
*** e0ne has joined #openstack-release12:25
slaweqsmcginnis: hberaud: ttx and other stable-cores: hi, about a week ago I sent email to propose Rodolfo to be neutron stable core12:26
slaweqI didn't get any negative feedback so far12:26
slaweqis it possible that You will add him to the neutron-stable-maint group? Or should I do something more for that?12:27
ttxslaweq: let me see if I can still do that12:37
ttxok done12:38
*** ykarel has quit IRC12:40
*** ykarel has joined #openstack-release12:41
ttxhberaud, fungi: regarding that ansible-role-redhat-subscription 1.1.1 release issue, it looks like PyPI has been ignoring our in that run (see, so I'm a bit at a loss. Maybe we could reenqueue the 1.1.1 ref and see if the error persists?12:49
fungii can give it a shot, sure. i tried yesterday to inspect it myself but apparently the project doesn't build packages in the usual fashion12:50
hberaud+1 to reenqueue it12:52
fungii was finally able to work out a viable tox env to build the sdist/wheel and run twine check against them, and that passes with current twine12:54
fungiit's also possible to test uploading them to test.pypi.org12:54
*** tosky has joined #openstack-release13:07
fungisudo zuul enqueue-ref --tenant=openstack --trigger=gerrit --pipeline=release --project=openstack/ansible-role-redhat-subscription --ref=refs/tags/1.1.1 --newrev=2da70e90b5fbdf225a95040903c4eb114ef3a9a413:13
fungithat's been enqueued13:13
fungiit's puzzling that the error from the attempted wheel upload indicates it tried to parse the description with the default (rst) parser when the wheel metadata explicitly includes "Description-Content-Type: text/markdown"13:16
hberaudsame error13:20
hberaudit failed13:20
fungialso pbr and setuptools support specifying the description content type in setup.cfg, it doesn't have to be embedded in setup.py13:21
fungi"You can set the description-content-type to a MIME type that may help rendering of the description; for example text/markdown or text/x-rst; charset=UTF-8."
ttx+ the fact that 1.1.0 worked ok, that's a weird error13:22
fungiyeah, it's possible there's some regression in warehouse causing it to not see the metadata and fall back to rst13:22
ttxhberaud: maybe we should file an issue, saying it used to work and now we get this weird error13:23
ttxbecause it sure looks like we are doing the right thing13:23
hberaudbefore I'll execute a test with readme-renderer13:23
ttxand error really points to somethign weird on their side13:23
hberaudto see if we can grab some info13:23
slaweqttx: thx13:24
hberaudttx: <string>:18: (WARNING/2) Line block ends without a blank line.13:26
ttxweird error message then13:26
fungireadme-renderer doesn't render markdown, just rst13:26
hberaudapparently it doesn't love the table13:27
fungii also tested it just to see13:27
hberaudfungi: the doc said that markdown could be passed too13:27
fungibut yeah, then i realized it doesn't support markdown, the underlying issue is that the markdown content type is being ignored13:27
hberaud It can handle Markdown, reStructuredText (.rst), and plain text.13:27
fungioh, weird, the --help doesn't mention markdown13:28
hberaudI confirm the table is the issue13:28
fungi"Renders a .rst README to HTML" and doesn't indicate how to tell it what content type you have if it's not rst13:28
hberaudif I remove the table rendering works well13:28
fungibut yeah, maybe readme-renderer is guessing from the file extension?13:29
hberaudI think something is malformed13:29
hberaudno idea13:29
fungii get the same warning even if i rename it to README.rst13:29
fungiso not sure what content type it's actually trying to apply13:30
hberaudwe could run a pdb session to observe13:30
hberaudwhich renderer is loaded and how it was handled13:30
hberaudlets recheck the readme's changes first13:31
fungialso i diffed the wheel metadata files between 1.1.0 and 1.1.1 and the only differences were the version and a few rows added/changed in that table in the description13:31
fungiso 1.1.0 uploaded successfully on 2020-05-21, that probably relied on warehouse using readme-renderer 26.0 from 2020-04-2313:34
fungithe only subsequent releases were last month13:34
fungichangelog says 27.0 was "Add support for align attribute rendering Markdown headers"13:35
fungiso maybe a regression crept in there?13:35
fungi28.0 was "Support Python 3.9"13:35
fungi"There's nothing to 'check' for a Markdown readme as they will gracefully fail if there's invalid syntax, unlike reStructuredText. Any markdown readme will always be accepted by PyPI."
fungiif that's still true, then the problem lies before readme-renderer. warehouse should only be rejecting uploads with malformed rst, not md13:38
hberaudI agree14:01
hberaudthen opening a bug could be worth for us14:02
hberaudI am opening the bug14:11
*** whoami-rajat__ has joined #openstack-release14:17
*** dtantsur|bbl is now known as dtantsur14:20
smcginnisThere's an explicit setting that needs to be in setup.cfg to tell it the README content is markdown. Maybe pypi expects that now?14:36
hberaudthe PKG-INFO seems correct and markdown have been successfully applied14:36
smcginnisAh, they do have it:
hberaudeven without this in setup.cfg14:37
smcginnisFor consistency, would be good if they just had rst. But looks like this should work. So yeah, maybe a warehouse bug.14:38
hberaudyes rst could be an option14:38
hberaud(move to)14:38
hberaudBy keeping the table of the description section readme_renderer output an empty file14:46
hberaudBy removing the table of the description section readme_renderer output a valid file14:46
hberaudapparently it behave similarly with 26.0 and 28.014:47
*** suryasingh has joined #openstack-release15:07
fungismcginnis: yeah, in fact i double-checked the METADATA file in the wheel (which is the upload being rejected) and it has a line precisely matching what the relevant pep and other howtos say should be there to indicate markdown for the long description15:09
fungialso i have a feeling they're using markdown because that's standard for readme files in their ecosystem (ansible galaxy?)15:10
*** vishalmanchanda has quit IRC15:12
*** jbadiapa has quit IRC15:16
*** rpittau is now known as rpittau|afk15:18
*** e0ne has quit IRC15:54
hberaudfungi: please can you upload needed info? =>
hberaudfrom job's artifacts16:08
hberaudI don't know if I can access that from my side16:08
* hberaud afk for a while16:09
fungithe job artifacts are likely insufficient, but i can build the wheel locally and extract content (i already have in fact)16:12
fungiprobably the next thing to try, as i suggested earlier, is manually using twine to upload a locally built wheel to and make sure we get the same rejection16:13
fungii'll create the project there shortly and give it a shot16:13
fungii was able to manually upload the copy i built to
fungimy current suspicions are that we're using older setuptools which doesn't know to insert the content type into the metadata, or older twine which doesn't know to look for it when uploading16:27
fungii'll see if i can work out what versions are in use and test with those specifically16:27
fungifailed build used twine 3.2.0 which is the same one i tested with, but looks like whatever setuptools version is supplied preinstalled on the node is what got utilized16:36
fungiit also looks like we run the wheel utility locally on our executors instead of on a job node, so it's wheel 0.30.0 i think? still trying to confirm that16:43
fungior 0.32.3 on some newer executors16:44
fungimy local build used 0.35.116:44
fungier, no, sorry it's twine i'm thinking of we're running on the executors not wheel. we run wheel on the job nodes16:45
fungibut no, the log shows we do pip install twine in the executor's workspace during the job so that version should be current16:48
fungii guess what we're lacking is insight into the exact versions of setuptools and wheel used for the package built on the job node16:49
fungiwhich may be what's contemporary for ubuntu 18.04 if we don't explicitly upgrade it somewhere not apparent in the job log16:50
fungiyeah, we don't do the build in a venv and the ensure setuptools role seems to be satisfied that the executable is present, it looks like16:53
fungiin which case this is using setuptools 39.0.1 and wheel 0.30.016:54
fungithis is the tox env i was able to reproduce it with:
fricklercould someone approve the neutron stable/stein release please? /me needs to build new pkgs and that would make things easier
fungia bit more fiddling suggests it's something to do with using wheel 0.30.0 (unpinning it and relying on 0.35.1 seems to work fine)17:03
fungii'll see if i can narrow it to a specific version where it starts working and see what the changelog says there17:03
*** ykarel has quit IRC17:05
*** ykarel_ has joined #openstack-release17:05
fungistarts working between 0.30.0 and 0.31.0 (there were no intermediate releases)17:05
ttxfrickler: looking17:09
fungino smoking gun in the wheel changelog to suggest why it works with 0.31.0, the METADATA file in the wheel it build definitely includes "Description-Content-Type: text/markdown" still17:10
ttxfungi: would it be possible that the line 18 warning cascades into the other error? Like if you fix that warning, does it work?17:12
fungiaha, the only difference in the METADATA files is "Metadata-Version: 2.0" vs "-Metadata-Version: 2.1" i bet that's used to determine whether to look for Description-Content-Type17:12
fungittx: i don't think so, no. the problem is that it wants to parse that markdown file as rst on upload17:13
*** dtantsur is now known as dtantsur|afk17:13
fungiand i think it's the metadata version causing it17:13
ttxok, so the only reason why we did not hit this sooner would be that 99% of our readme files are RST?17:14
fungi"New in version 2.1."
fungittx: correct17:15
fungithe "easy" solutions would be to force upgrade wheel globally on the job node before building, or use ubuntu-focal which has a new enough python3-wheel to support metadata version 2.117:16
fungias to why this worked in may and not now, i believe in the intervening timeframe we changed how we install wheel by default to using distro packages instead of from pypi17:17
ttxonce you have a grip on the issue and the options, you can follow up on the thread17:18
fungigiven that most of our other jobs have moved to focal, i wonder if the release jobs couldn't be updated to do the same?17:18
fungiand yeah, i'll follow up on the ml thread and the github issue17:19
ttxyeah that's probably the best option, we'll have to do that at some point anyway17:19
*** ykarel_ has quit IRC17:19
fungithe risk, i guess, is that we end up trying to build stable branch packages on newer distros than existed at the time? though zuul's somewhat recent tag-to-branch mapping solution could perhaps solve that if we try to do branch-specific variants for the different nodesets17:20
fungialso i have no idea if stable branch releases would fail to build on focal or not. might work just fine17:21
hberaudfungi: sorry I already submitted a local build previously, but I wasn't sure that I was exactly aligned with our job env17:28
openstackgerritMerged openstack/releases master: Release neutron 14.4.1 for stein
fungiokay, ml and gh issue both updated with findings17:35
hberaudkudos fungi for finding all these elements!17:35
fungihberaud: yep, the key was that by default we're started relying on distro packages for things like pip, setuptools, and wheel17:36
hberaudI see17:36
fungimainly because our older "preinstall stuff from pypi" solution caused issues for projects which wanted to install different versions, and using `sudo pip install ...` doesn't really produce consistent behaviors across different distros anyway17:38
fungiand we're relying on ensure-* roles from zuul's standard library, which tries to provide more generalized solutions which work across zuul deployments rather than things specific to opendev17:39
fungii'll work on the nodeset change for the release job now17:42
*** suryasingh has quit IRC17:45
fungi Run publish-openstack-artifacts on ubuntu-focal17:49
clarkbdo weknow why older wheel breaks?17:53
fungiyes, i did my best to explain both in the ml thread and on the gh issue18:01
fungiit doesn't actually break18:02
fungiit all comes down to metadata versioning18:02
fungiwe shouldn't be trying to set a content type for the long description unless we're also declaring a new enough metadata version to support that field18:02
fungiwheel sets the metadata version, so to declare a newer metadata version you need a newer release of wheel18:03
fungiclarkb: ^18:04
fungihaving personally worked on the implementation support for some of the newer metadata features in setuptools and pbr, i vaguely recall the logic... it's keyed on metadata version to decide whether to pat attention to or ignore certain fields18:07
*** dave-mccowan has quit IRC18:49
fungiwe've got +2s on the nodeset change, shall i approve it now and reenqueue that tag or save it for monday?19:57
hberaudfungi: as you prefer, I'm near of the eod for today but I'm confident20:15
hberaudmonday could be better if something goes wrong20:17
fungihow about this: let me know on monday when a release manager is available to help vet results (i should hopefully be around myself by ~14:00 utc) and i'll approve it then and test20:23
hberaudfungi: +120:26
hberaudfungi: I'll be around 10:00am utc+1 and until the eod20:27
*** e0ne has joined #openstack-release21:21
*** e0ne has quit IRC21:23
*** melwitt is now known as jgwentworth21:26
*** whoami-rajat__ has quit IRC21:26
*** dave-mccowan has joined #openstack-release23:08

Generated by 2.17.2 by Marius Gedminas - find it at!