Thursday, 2020-03-26

openstackgerritIan Wienand proposed opendev/system-config master: Remove old hosts from cacti
ianwgood idea :)00:08
openstackgerritIan Wienand proposed openstack/diskimage-builder master: centos 8 image build: fix mirror
openstackgerritIan Wienand proposed openstack/diskimage-builder master: run_functests: handle build without tar
*** tosky has quit IRC00:52
*** mlavalle has quit IRC01:22
ianwgosh darn it, something has definitely started going wrong with pip install executable files :/01:33
ianwit must be something like ... but it happens across all our images01:34
*** diablo_rojo has quit IRC02:01
ianwand here's the issue ... ... arggh02:12
ianwso this is fixed in 46.1.3 ... our wheel cache has 46.1.1 (46.1.2 was released 11 hours ago, 46.1.3 8 hours ago)02:21
ianwergo, i guess will be fixed tomorrow02:21
*** diablo_rojo has joined #opendev03:10
openstackgerritIan Wienand proposed opendev/base-jobs master: Generate download script for logs
*** diablo_rojo has quit IRC05:39
openstackgerritMerged openstack/project-config master: Revert "Revert "Revert "Disable github reporting and re-add devel ansible job"""
*** DSpider has joined #opendev05:57
*** ralonsoh has joined #opendev07:09
*** dpawlik has joined #opendev07:22
*** factor has quit IRC07:29
*** bolg has quit IRC07:59
*** vblando has quit IRC08:06
*** jaicaa has quit IRC08:06
*** jaicaa has joined #opendev08:07
*** bolg has joined #opendev08:17
*** rpittau|afk is now known as rpittau08:26
*** bolg has quit IRC08:26
*** bolg has joined #opendev08:40
*** jaicaa has quit IRC08:41
*** jaicaa has joined #opendev08:44
*** tosky has joined #opendev09:00
*** rpittau is now known as rpittau|bbl11:45
*** roman_g has joined #opendev12:08
*** lpetrut has joined #opendev12:18
openstackgerritMonty Taylor proposed opendev/system-config master: Remove files from letsencrypt group
mordredfrickler: ^^ fixed that12:20
mtreinishmordred: I tried running openstacksdk's tests locally on 3.8 but haven't been able to reproduce that failure yet. My guess though is that an attachment stream (stdout, stderr, etc) closing is racing with testtools reading it, maybe a file refcount vs gc thing.12:29
mordredmtreinish: oh joy. that's lovely12:30
* ttx waves at mtreinish12:30
fricklermordred: what about in my understand the group only needs the hosts themselves, not the aliases12:34
mtreinishmordred: I haven't seen that error before though, but the only place iter_chunks and testtools content is really used is in handling attachments in the result stream. But let me know if you see it again and I'll try to dig more into it12:34
mtreinishmight also be worth ping stephenfin about it because he just tracked down a long standing race in attachment handling to fix the issue with super large attachments12:35
* mtreinish waves back at ttx 12:35
mordredmtreinish: kk. I've seen it on and off on the 3.8 jobs (Which are still non-voting) - but I'll see what I can learn and/or pull in stephenfin12:36
ttxmtreinish: how are you doing my friend12:36
mordredalso - same question as ttx :)12:36
mtreinishI'm doing well, I've got no complaints. The quantum computing world is keeping me really busy, but I get to play with a lot of neat things.12:38
ttxwarn me before you start breaking all encryption. I've stock to sell12:39
ttxalthough not now12:39
mtreinishheh, well if you want to factor 15, I can show you how to do that today. But going much bigger than that isn't for a long time12:41
ttxfactoring 15 can come in handy12:42
fungi15 is a lovely product of two prime factors. i don't know what anyone has against it12:47
mtreinishwell it's not typically the kind of thing rsa keys are made of :)12:49
fungirsa is so last century12:50
*** rpittau|bbl is now known as rpittau13:10
openstackgerritMerged opendev/system-config master: Remove and related puppet
openstackgerritMerged opendev/system-config master: Remove static site puppet
openstackgerritMerged opendev/system-config master: Remove old hosts from cacti
*** hashar has joined #opendev13:28
openstackgerritMerged opendev/system-config master: Start making 3.8 python images
mordredmnaser: the updates to python-base landed and now disable recommends. I hit an issue on osc related to that. looks like the fix is adding libc6-dev [compile test platform:dpkg] to the bindep (symptom is an error about a missing limits.h) - before libc6-dev was a recommends for gcc13:45
mordredmnaser: just in case you run in to that in any of your images13:45
mnasermordred: oh -- that's helpful -- thank you14:03
openstackgerritMonty Taylor proposed opendev/lodgeit master: Add libc6-dev to bindep
mordredmnaser: and ... ^^ :)14:21
openstackgerritMonty Taylor proposed opendev/lodgeit master: Be explicit about python base image version
*** jkt has quit IRC14:29
*** jkt has joined #opendev14:30
*** corvus has quit IRC14:30
*** corvus has joined #opendev14:31
*** roman_g has quit IRC14:34
openstackgerritMonty Taylor proposed opendev/storyboard master: Be explicit about base container image
openstackgerritMonty Taylor proposed opendev/lodgeit master: Add libc6-dev to bindep and pin Pygments for 2.7
openstackgerritMonty Taylor proposed opendev/lodgeit master: Be explicit about python base image version
mordredclarkb, mnaser: ^^ 715219 is required to unbreak gate for lodgeit14:48
*** roman_g has joined #opendev15:04
yoctozeptomordred: I see pinning for Pygments - kolla is also hit by it, as well as a bunch of other packages - looks like something went wrong elsewhere, pygments did not release recently either15:07
yoctozeptocame here to ask you if you saw this behavior elsewhere and saw pygments immediately15:07
* yoctozepto triggered15:07
mordredyoctozepto: oh, you're right -that release was may 815:08
yoctozeptomordred: pypi had infra problems during the day, they seem to be gone now though15:09
yoctozeptoyet the failures remain15:09
mordredis the one I'm seeing15:09
mordredoh - hahahaha. I put in the wrong pin15:09
openstackgerritMonty Taylor proposed opendev/lodgeit master: Add libc6-dev to bindep and pin Pygments for 2.7
yoctozeptomordred: yeah, but bear in mind fixing pygments may just reveal another one to fix ;p15:10
openstackgerritMonty Taylor proposed opendev/lodgeit master: Be explicit about python base image version
mordredyoctozepto: indeed.15:10
mordredlet's see how that one does15:10
*** chandan_kumar has joined #opendev15:14
*** noonedeadpunk has joined #opendev15:21
yoctozeptodebugged locally15:23
yoctozeptousing official pypi the behavior is correct15:23
*** elod has joined #opendev15:23
yoctozeptousing the behavior is wrong - py2 gets py3 packages15:24
yoctozeptoinfra-root: please help, it seems the py2 mirror issues returned15:24
fungiyoctozepto: please link an example failure15:24
fungii don't know/remember what "the py2 mirror issues" were15:24
yoctozeptofungi: mordred's one is good
yoctozeptofungi: py2 "pip install Pygments"15:25
yoctozeptofungi: you probably remember how we blamed pip the last time ;p15:26
yoctozeptothis time compared both behaviors locally15:26
clarkbit should be noted that is just a proxy to pypi15:27
clarkbI would start by comparing the indexes for pygments15:27
clarkbya that15:27
clarkbbut they should be the same with a ~10 minute cache expiry15:27
fungiyoctozepto: that looks like the same "problem" we worked around for other packages by adding environment markers in the constraints file15:27
corvusdid we suspect that our proxies were getting different data from different pypi cdn endpoints?15:28
yoctozeptofungi, clarkb: at some point we got constraints back in indices15:28
yoctozeptoit could be cached issue from pypi15:28
yoctozeptothey had problems today15:28
clarkbyoctozepto: they were always there in the simple index asits just a proxy15:28
clarkbonly the wheel mirror which us not a proxy had the previous issue15:29
yoctozeptoclarkb: yeah, that's why it worked15:29
yoctozeptoclarkb: but now it doesn't :D15:29
clarkbcorvus: in the past it wasdue to building wheels for everything. This sounds like wheel mirror isnt involved and could possibly be related to cdn deltas now15:30
clarkblooking at headers on that index request I don't see the old 600 second ttl15:34
clarkbthe direct request to pypi has the max-age set15:35
clarkboh and now there it is15:36
clarkbyoctozepto: on my first request through the proxy I got a file from march 10 with no cache-control header15:36
clarkbyoctozepto: on my latest request I get a file from march 26 with a cache-control header15:37
clarkbI think that supports corvus suspicion that cdn might be doing weird things15:37
clarkb(depending on where backend request falls we get a different result?)15:37
yoctozeptoclarkb, corvus: yeah, pypi had problems today15:37
yoctozeptoserious ones15:37
clarkband that should flip no faster than every 10 minutes if we hit the cache control because it is 600 seconds15:37
yoctozeptowell, I only see rax still feeds me with no constraints info15:39
yoctozeptowhich is really weird15:39
yoctozeptoas it has pygments 2.6.115:39
yoctozeptoso must be recent enough15:39
clarkbyoctozepto: thats the joy of cdn, it is often localized and that leads to problems like that15:39
yoctozeptoyet missing the tags15:39
yoctozeptoclarkb: ah, you mean rax is still in the window where it got it from bad cdn face15:39
clarkbnote the version I get with cache-control has the python version info15:39
clarkbyoctozepto: more likely that the rax region is local to a bad cdn node and good cdn node(s). Some of our requests get bad results some get good basedon my testing15:40
clarkbright now the data looks good15:40
clarkbwe'll just have to wait and see if that changes in 10 minutes or even further out15:41
yoctozeptoclarkb: ack, confirmed, rax looks good now15:41
clarkbif it does continue to flip back and forth we can probably ping pypi folks to see if they can prod the unhappy cdn15:42
*** njohnston has joined #opendev15:42
yoctozeptoclarkb: yeah, I already pinged them before about the general situation15:43
*** chandan_kumar is now known as chandankumar15:51
*** mlavalle has joined #opendev16:00
mordredmnaser: wanna +A ?16:08
mnasermordred: done16:18
mordredmnaser: woot16:20
*** dpawlik has quit IRC16:26
openstackgerritMerged opendev/lodgeit master: Add libc6-dev to bindep and pin Pygments for 2.7
openstackgerritMerged opendev/lodgeit master: Be explicit about python base image version
openstackgerritClark Boylan proposed opendev/system-config master: Add links to dev docs on gitea
clarkbinfra-root ^ thats soom noodling on how we can make the jump from to pushing changes a bit easier for people. I think thats about as good as we can integrate with gitea without writing a bunch of golang to update routers and set flags on the html16:42
*** amotoki has joined #opendev16:44
AJaegerclarkb: good idea,, thanks16:46
AJaegerclarkb: can we remove the gitea link from the page as well? I mean the "Help" link at the top16:47
clarkbAJaeger: the next step there will be tweaking the develop doc to be a good landing spot for someone clicking from (or maybe adding a new page if necessary)16:47
clarkbAJaeger: we can, I had left it there because it is a useful link for gitea specific help, but maybe that is confusing?16:47
AJaegersince we disable most of gitea, I consider it more confusing than helpful16:48
clarkbthats a good point16:48
clarkbits probably more useful if people arem anaging their own accounts and stuff directly there16:48
clarkblet me push that as a followup so we can consider both chagnes independently16:48
AJaegeryes, exactly16:48
AJaegersure, go for it. thanks!16:48
openstackgerritClark Boylan proposed opendev/system-config master: Remove gitea doc links
clarkbfungi: corvus aprice sent that email about community meetings to the foundation list this morning16:57
clarkb(if you've not got a local copy I can get you al ink)16:58
fricklerclarkb: I saw that and thought it would be great if we could get ppl to use meetpad for that in the future instead of zoom17:02
clarkbfrickler: yup! I think ocne we've got it running we should have as many people use it as possible17:02
*** bolg has quit IRC17:05
fungithe "community meeting" might be somewhat disjoint from our usual ptg participants, so featureset they need could differ somewhat17:11
*** bolg has joined #opendev17:11
fungi(for example, dial-in trunks, session recordings auto-uploaded to a video streaming service, et cetera)17:12
*** rpittau is now known as rpittau|afk17:12
fungii wouldn't want to overload the current meetpad plan with more than we need for the ptg at first17:12
mordredclarkb: frickler has some good feedback on 715264 - otherwise that stack looks god17:13
clarkbfrickler: thanks for the review on that gitea chnge, can you check my comments and indicate if you want me to remove the signed in portion?17:13
clarkb(I'll fix the quotes once we decide if I should keep or removed the signed in bits)17:13
mordredI think it's fine for them to be there - in case we ever wind up having an authenticated version of gitea, putting this there now will make sure we don't forget to add it later :)17:15
clarkbya that was my thought, better to be complete now and not forget later17:16
openstackgerritClark Boylan proposed opendev/system-config master: Add links to dev docs on gitea
openstackgerritClark Boylan proposed opendev/system-config master: Remove gitea doc links
clarkbthat fixes the quotes17:16
*** lpetrut has quit IRC17:21
openstackgerritDavid Shrewsbury proposed opendev/system-config master: Remove shrews from infra-root
openstackgerritDavid Shrewsbury proposed openstack/project-config master: Remove shrews from infra-root
openstackgerritDavid Shrewsbury proposed opendev/system-config master: Remove shrews from infra-root
clarkbmordred: Shrews can you check my comment on I think the openstack ssh key resource doesn't get managed by ansible in the way we expect there (I've noted we can leave it as is in this cahnge and then rotate an update separately)17:32
Shrewsclarkb: oh. fine by me if you just want to do it in the global rotation then17:33
mordredclarkb: yeah - I think you're right - but I also think we can do it later17:33
clarkbShrews: ya I'm fine with that if you want to update the change to leave the key there for now17:34
openstackgerritDavid Shrewsbury proposed opendev/system-config master: Remove shrews from infra-root
Shrewsclarkb: that should do it17:49
*** openstack has quit IRC17:49
*** openstack has joined #opendev17:51
*** ChanServ sets mode: +o openstack17:51
fricklerclarkb: o.k., in that case I have the next question why the class is called octicon-repo-push , is that some predefined thing? maybe using the class name from the to-be-removed help button would match better?18:00
clarkbfrickler: ya its defined by the glyph set here
clarkbfrickler: basically I picked one that I thought would be appropriate for getting started to push code18:01
clarkbrepo-push is in the bottom third18:01
*** roman_g has quit IRC18:14
*** hashar has quit IRC18:21
openstackgerritMerged opendev/system-config master: Remove shrews from infra-root
*** diablo_rojo has joined #opendev18:47
openstackgerritsebastian marcet proposed opendev/puppet-openstackid master: added www. cname on vhost file
*** ralonsoh has quit IRC18:51
corvusclarkb: what are your thoughts on meetpad scheduling?19:39
clarkbcorvus: of merging changes? I was planning to land the specs today after a bike ride, then I think we can start to land the implementation changes then spin up a server as soon as enough of those are ready19:39
fungii should get reviewing the implementation stack for it onto my evening agenda19:41
corvusclarkb: sounds good!  the stack is green; i'll chase down a couple more reviews so you can just approve them all together :)19:41
corvusmordred: can you look at and
corvusfungi: thanks!19:42
* clarkb eats lunch then will ride around on the bike and return to click approvals19:42
noonedeadpunkmordred: once you have some time for reviews can you kindly check one more time?:)19:56
mordrednoonedeadpunk: brilliant20:08
mordredcorvus: I, for one, welcome our new meetpad overlords20:09
mordredclarkb, corvus: gerrit humans (such as paladox) have suggested upgrading to 2.14 as part of our upgrade process - so I added jobs to build it which is green:
corvusmordred: i thought we could go straight to 2.16?20:51
* corvus revisits
corvusoh, apparently that says we need to upgrade to 2.1520:52
mordredcorvus: yeah - and people keep saying that we should do a stop at 2.14 on the way from 13 to 1520:54
corvuscan we record that and why?  it'd be really good to know that20:54
corvusand what does "stop" mean?20:54
corvusbecause i'm really hesitant to put a different unsupported version into production20:54
mordredcorvus: these are all great questions20:54
corvusif we have to do 2.14 for some sort of migration reason, i'd prefer we do that, then continue the outage to 2.15, then continue the outage to 2.16, then put 2.16 into prod20:55
corvusbut putting anything < 2.16 into prod seems scary to me20:55
mordredcorvus: yeah - that's what I'm hoping for as well20:56
corvuslike, spending weeks fixing issues with a version that's out of support and we don't want to run anyway sounds :(20:56
*** DSpider has quit IRC21:05
*** DSpider has joined #opendev21:05
ianwspeaking of wheels in scrollback, did we get setuptools 46.1.3 built?21:10
* ianw goes to check how many other brown bag releases setuptools might have made overnight21:10
ianwapropos nothing; does anyone have experience with  their promise of SRS for rewriting spf on forwards sounds like what i want21:21
mordredcorvus: I'm chatting with folks in #gerrit - and it seems the story is that we can do online reindexes/upgrades if we go one minor at a time - but if we bounce through them (like just running the init on each version) to get all the way to 2.16 - then we'll need to do an offline reindex21:23
fungikeeping in mind that reindexing requires somewhere north of 4 hours21:24
mordredcorvus: so we could do 2.13->2.14 let online reindex run to completion then 2.14->2.15 online reindex then 2.15->2.16 online reindex - but it might mean running for a few days at each of the intermediaries21:24
mordredfungi: oh is it only 4? why did I think it was more like 18?21:25
corvusmordred: thanks, i'll read backscroll21:25
fungiit used to be longer, may still be longer than 421:25
fungii can probably extract last week's online reindex time from the log if it hasn't rotated out yet21:25
mordredoh - because we did an online reindex due to the rename21:26
corvusmordred: i understand what paladox is saying there now, thanks :)21:27
mordredso if it really is only like 4 hours - then we coudl conceivably do a weekend upgrade where we do run each version during the reindex - but it would only be for the 4 hours or so that the reindex takes - then we do the next upgrade21:27
corvusi excerpted the relevant bit in
mordred(we'll obviously test this with a copy of real data on a whole new thing ... but I think given the time frames limiting the test scenarios we want to check out and validate is important)21:29
corvusmordred: ++21:29
fungi2020-03-20 14:33:18,213] [Reindex v32-v32] INFO : Starting online reindex from schema version 32 to 3221:30
fungi[2020-03-20 18:27:56,198] [Reindex v32-v32] INFO : Reindex to version 32 complete21:30
fungijust shy of 4 hours21:30
corvusfungi: well remembered! :)21:30
corvusmordred: should be feasible to get a similar sized server, make a db copy and filesystem copy, then manually step through the container images21:30
mordredcorvus: yeah - that's what I'm thinking21:30
fungikeep in mind, that's reindex under the same schema. these upgrades we're talking about not only incorporate schema migrations but also moving content to entirely different databases21:31
mordredcorvus: and we can try the stepwise-online version - and then maybe the skip-to-2.16 with an offline reindex version21:31
mordredfungi: so - doing nodedb can be done as its own activity21:31
fungicool, just making sure we're keeping that transition in mind21:31
mordredso I think the idea is to upgrade to 2.16 without notedb and get up and running ... then trigger the online nodedb migration21:31
mordredwhich will take longer than a normal migration21:32
fungii guess notedb is the reason to pause at 2.16?21:32
fungisounds good then21:32
corvusthe word on the street is: do not touch notedb before 2.16.21:32
fungii trust the streetwords21:32
mordredthat and also 2.16 is the first version with polygerrit and the last version with gwt - so it's not a bad idea to maybe pause there for a minute to let people start getting used to pg before the flag day21:32
mordrednot a ton of time - but since we've got the notedb migration anyway21:33
paladoxwe unfortunately migrated under 2.15 :)21:33
mordredthen once we're ready - 2.16 to 3.0 is basically just the removal of gwt - and then 3.1 is a normal minor release which should be a quick upgrade21:34
fungii could imagine pausing at 2.16 for as much as a few weeks to give folks an opportunity to toggle between old and new webui21:34
fungithe switch from the original change screen to the "new" change screen was fairly disruptive, or at least i would think it was if i could still remember it. that was many beers ago21:34
mordredso many beers21:35
* fungi makes it one more21:35
mordredfungi: related to this: if you have a sec21:35
paladoxcorvus thats even documented on the RN :)21:35
fungihuh, your jobs are passing on that change21:38
fungii've been trying to work out why this system-config change failed two jobs:
ianwi'm still seeing : /home/zuul/dib-venv/lib/python3.6/site-packages/diskimage_builder/lib/dib-run-parts: Permission denied ... that is disappointing21:39
mordredfungi: maybe the world has fixed itself since your change?21:40
fungiyeah, i just hate rechecking when i don't know what the cause was21:40
mordredfungi: there's a whole 2 hours different21:40
fungitime is an illusion21:40
mordredianw: that is disappointing21:40
ianwi wonder if it's our image builds not picking up the new version, not wheels21:41
ianwfungi: do you have any recollection of setuptools and wheel builds from your recent adventures there21:41
fungii've probably pickled those brain cells, but will see if i can spot something21:42
fungithis is on ubuntu-bionic i suppose?21:43
ianw2020-03-26 09:26:40.915 |   Downloading setuptools-46.1.3-py3-none-any.whl (582 kB)21:43
ianwyep, bionic21:44
ianwactually, xenial and centos7 fails too ... so ... yeah21:45
fungiso we did at least fetch 46.1.3 which is supposedly fixed (was fixed in 46.1.2 and later right?)21:47
ianw# /usr/local/lib/python3.6/dist-packages/setuptools21:47
ianwsetuptools/                  setuptools-46.1.3.dist-info/21:47
ianw suggests it is fixed ... but i have not verified as such (i was hoping things would just go green!)21:48
* clarkb back from biking catching up on reviews now21:50
*** DSpider has quit IRC21:53
ianw ... passed : rax-iad21:55
ianw ... failed: rax-dfw21:55
mordredianw: :(21:55
fungiianw: image upload issues?21:57
ianwhrm, images seem fresh ...
ianwboth have the same build date in zuul-info/zuul-info.ubuntu-bionic.txt ; so same image22:02
clarkbinfra-root I'm going ot approve which is a base job change that has been tested via base-test to add the log download script artifact. Let me know if there is additioanl testing or whatever you think we should do and I'll hold off (but I think we've followed the general process there and its ready to go)22:03
fungiianw: did you have a failing ubuntu-bionic example? you provided a working ubuntu-bionic link and a failing centos-722:03
ianwfungi: so that's both running dib and building on bionic ... but one is building bionic to test and the other centos22:06
fungiahh, okay22:06
fungiand both jobs ran from the same nodepool image, as best we can tell22:07
clarkbre gerrit iirc we also want to stop on 2.something before 3.whatever in order to populate the notedb right?22:08
ianwyep; they must be the same really, they have the same build date22:08
fungiclarkb: yeah, we clarified that, 2.16 to be precie22:09
fungiprecise as a pangolin22:10
mordredclarkb: yah22:12
mordredinfra-root: <-- got another hostkey changed error22:18
clarkbmordred: that one is in inap22:19
clarkbmordred: that is beginning to make me wonder if either Neutron has a widespread bug or maybe a bug on our side?22:19
mordredclarkb: yeah. but I'm not sure what the bug on our side would be22:20
clarkbmordred: losing the known hosts file or overwriting it somehow? I don't think that is likely but it is theoretically possible22:20
*** Shrews has left #opendev22:21
mordredI'd think we'd see the issue much more frequently if that was a thing22:21
clarkbthis was definitely a rax specific issue in the past22:22
clarkbbut whatever neutron thing was happening there could be happenign elsewhere now22:22
clarkbwow these pypi issues are far reaching /me tries another recheck on the gitea docs changes22:23
corvusclarkb, ianw: oh, i thought i asked somewhere about using the manifest file instead of generating a script?22:27
corvusi don't see a comment on the change, sorry about that22:28
corvusmaybe i asked in irc?  or maybe i didn't ask22:28
corvusanyway, the swift log upload script is one of 2, going on 3 log upload scripts; i'll grant that it's annoying that there's no code reuse between them, but my initial thought for dealing with that was to avoid changing them.22:29
corvusand it seems that now that we have the zuul manifest, you could add a log download script that's completely independent of the upload process22:30
corvusthat is, after all, exactly how zuul's log viewer handles it22:30
clarkbcorvus: how are you thinking we distribute that script?22:31
corvusianw, clarkb: does that sound feasible?  would you consider reverting the swift change in favor of that so we can try to keep the upload roles simpler?22:31
corvus(after all, we don't even use the upload roles to generate the manifest!)22:31
clarkbI think keeping the upload roles simpler is a good goal22:31
clarkb(though maybe we'd still be uploading a script someway it will just be a common script)22:32
mordredit seems like a script that one could download once and use for any zuul ... kind of like encrypt_secret22:32
corvusclarkb: any way we want? :)  we could put it in the infra manual; put it in a central location on opendev, or, if we want a copy of it in every upload, stick it in the base job so it gets uploaded with the logs.22:32
corvusit could take an argument, so it's generic, but we could put the command in our logs in our base job22:32
ianwmordred: so that was pretty much what i wanted to avoid ... the idea is for it to be a absolue zero-overhead way to grab the logs if you know nothing22:32
mordredyeah - I think that would potentially go with the "upload the script to the logs server" idea22:33
corvusso the console log would output "run 'download-logs https://....'"22:33
corvusor if you want to template that into a script, we could even do that22:34
corvusjust make it its own role22:34
corvus(i guess that's what the current patch does?)22:34
corvusbut if it's its own role, then we've got better separation of concerns22:34
corvus(to summarize, i think the proposal that meets the most number of requirements would be: a new role which templates a semi-generic script into the log directory before upload and embeds the build uuid into it; the script uses the zuul rest api to query the log location for that build, fetch the manifest, and download all the files)22:35
openstackgerritMerged opendev/infra-specs master: Add a spec for meetpad
clarkbcorvus: ya I think that would work22:36
corvus(it does not address fungi's concern; i think that could only be addressed by distributing a completely generic script and only printing the instructions to run it in the logs)22:37
corvus(that seems to be a conflicting requirement with ianw's "no-install" requirement)22:37
clarkbmaybe we could distribute it with zuul and the webserver could serve it rather than the logs server22:38
clarkbjust build it right into the web ui server22:38
clarkb(I'm not sure the distinction of server would be apparenty to end users though)22:38
corvusyeah, maybe?22:38
ianwi guess i should write a spec22:39
corvusalso, i'm pretty sure it could be done completely with the standard library, so installation shouldn't be a big deal (wget the script once, and you're set).  but i understand that still may not meet the ease of use level ianw is going for22:40
fungimordred: i spoke too soon, your choked on openstackci-beaker-puppet-4 in the gate as well22:40
fungilooking now to see if it's similar22:40
ianwcorvus: yeah, i was trying to even avoid a python script; and parsing json in pure bash is a bit of a pain22:41
fungimordred: yeah, looks like the same error condition i was seeing on my changes too22:41
corvusianw: i think the days where debian users would say "i don't have python installed, only perl" are probably past :)22:42
ianwwhile i totally understand the anti-pattern discussion we've had, i feel there is a reason why projects find it useful to give people a "curl https://... | bash - "22:42
corvusi think it's an entirely reasonable thing to assume that 99% of people debugging a CI job will have *some* version of python22:42
corvus(and yes, i would write the script to work in py27 because we can, easily, and it's less of a speed bump.  same as encrypt_secret)22:43
ianwcorvus: how does the script embed it's log url in it, but still also get uploaded?22:44
clarkbfungi: those beaker job failures I'ev seen haev been due to pypi issues22:45
corvus[i don't feel as strongly as fungi about the merits of curl|bash (or, in the case i'm suggesting, curl|python); my main concern is maximum reuse of code and minimal complexity of the log uploaders.]22:45
corvusianw: it embeds the uuid instead of the url, then uses the rest api to find the url22:45
fungimordred: is this the problem, do you think? Error: Facter: error while resolving custom fact "virtualenv_version": undefined method `[]' for nil:NilClass22:46
clarkbfungi: no that is not the problem22:46
fungiokay, benign?22:46
clarkbfungi: ya basically puppet ignores when factor fails and just takes whatever facts it can get22:46
clarkb is the issue22:47
clarkbwhich I've thought was pypi but looking at timestamps that is super recent, possible we just need to fix a dep in nodepool?22:47
mordrednope - that's an sdk thing22:47
fungioh neat, that's buried in there22:47
mordredthere is a fix already landed - just waiting on a release22:47
fungiokay, thanks! will wait22:48
mordredI pinged smcginnis about it - but I think he was EOD before the patch landed22:48
clarkbmordred: the sdk thing tweaks the futurist dep?22:49
mordredalso - fwiw - we realized in fixing the sdk issue that we could drop the dependency entirely22:49
mordredwell - there's 2 sdk issues - butonly one impacts nodepool here22:49
mordredthe first was futurist not being 3.5 compat - which we missed - which is solved in the patch by removing futurist as a direct depend22:49
mordredthe second was that we didn't add a requires-python stanza (if we had, I probably would have caught that 3.6 > 3.5) - but that also meant it broke py2 users in a way that didn't allow pip to get the "latest that will work with python2"22:50
mordredonce 0.45 is tagged, we'll be doing a followup change that adds requires-python >= 3.5 for completeness22:50
fungiyeah, i caught some of that discussion in the qa channel22:51
mordredbut the 0.45 release will fix that issue above22:51
fungijust didn't notice it was what was also blocking system-config changes22:51
mordredyeah. me either22:51
corvusianw: what do you think?  would you be okay with a revert of 592341 while we rethink it?  if you're open to it, i'd like to revert soon so we don't have to worry about supporting that.  as pennance for not reviewing in time, i'll help write the python script if we decide to go that way.  i'm very sorry for not catching that earlier.22:51
ianwcorvus: ok, we can revert if there's not consensus.  there's infact an abanonded revert of it already.  i think maybe a spec ... although  am i the only one that actually thinks this is useful?22:52
ianwit's something i've wanted for many years debugging devstack, where you really have to correlate across multiple logs to understand what's going on22:53
fungiianw: i think it's useful in the same way that i actually miss being able to tell an ftp server that i want to get a directory but tack a .tar.gz onto the name and have it send me a tarball of the entire subtree22:53
clarkbianw: I had that problem with devstack until we started publishing the journal in one file, that covers a huge chunk of the use case for me22:54
corvusfungi, ianw, clarkb: maybe *that* is what we should do?22:54
ianwthat's true -- i certainly started this well before the journal22:54
fungiianw: unfortunately i think that's something http daemons missed out on22:54
ianwcorvus: i had a *really* old change that grabbed everything and sent it down as a .tar.gz ...22:54
clarkbdiablo_rojo: are you able to push along?22:55
corvusianw: to directly answer your question: if >1 person thinks this is useful, i'm happy to do it.  i would not be surprised if other folks (especially tripleo/rdo) would find it useful.  personally, i try to use the web as much as possible.  but i don't like judging or dictating how people read log files :)22:56
ianw ... os_loganalyze ... remember that22:56
mordredfungi: we _can_ pin sdk to <0.44 if we need to while we wait on the release - that will do the same result - but worst-case we should have release by morningish22:56
corvusianw: (probably if you are literally the only person, then you could do something much simpler; but i bet there are others)22:56
fungimordred: i'm not in a huge hurry, no emergency fixes to merge at least22:57
fungifor me it was blocking addition of meetbot to a couple of channels, and it's no big loss if that waits another day22:57
* mordred is going to go for a walk.22:59
clarkbya for me it was the opendev docs link changes22:59
clarkbthose can also land tomorrow happily22:59
openstackgerritJames E. Blair proposed zuul/zuul-jobs master: Revert "upload-logs-swift: Create a download script"
ianwinfra-root: reverts the log downloader23:00
ianwok, 715325 also reverts the unicode stuff23:01
corvusi thought that was necessary...23:01
corvusdoes 714000 just revert the log downloader without reverting unicode?23:02
corvuseither way; i'm happy to keep the unicode thing if that's what the other one does23:04
* corvus is slightly confused :)23:04
ianwcorvus: so i am; it's been a long time.  i think the unicode change can stand alone, if the test-suite bits for the log downloader are removed23:05
ianwi think the unicode test is still valid for the uploader, in general23:05
ianwit's -1'd so might need untangling23:13
ianw... back to the other issue ...23:16
ianw>>> print(setuptools.version.__version__)23:16
ianwthe setuptools version on our hosts is 46.1.3 ... HOWEVER23:16
ianw$ ./dib-venv/bin/python323:17
ianw>>> print(setuptools.version.__version__)23:17
ianwthe version installed, somehow, via zuul running "pip:" into a virtualenv with is 46.1.123:18
clarkbis 46.1.3 version excluded?23:18
clarkb46.1.1 and 46.1.3 have the same python requires >= 3.5 restriction23:19
clarkbthat probably isn't it23:19
ianwi can't see that it is, although this *is* using upper-constraints i think23:19
ianw has made the "bad" virtualenv23:20
corvusianw:   Downloading (29 kB)23:21
corvusERROR: Package 'bashate' requires a different Python: 2.7.17 not in '>=3.5'23:21
corvusianw: ^ sorry, second line is what i wanted to draw your attention to23:21
clarkbsetuptools is not constrained in upper constraints23:21
clarkbcorvus: oh interesting odd that it would use 46.1.1 in that case consider that also requires >= 3.523:21
clarkbmaybe we are still getting incomplete data from ypi?23:22
corvusclarkb: wait i think i'm talking about a different issue23:22
corvusi'm talking about how we can't merge anything to zuul-jobs because apparently we use bashate23:22
clarkbgot it (I did mix streams)23:23
openstackgerritJames E. Blair proposed zuul/zuul-jobs master: Remove bashate from test-requirements
openstackgerritJames E. Blair proposed zuul/zuul-jobs master: Revert "upload-logs-swift: Create a download script"
ianwhrm, bashate released ages ago23:24
ianwwhy did it just fail now?23:24
corvusianw: i dunno -- here's what ran on your log downloader change:
corvusit sure did get an old version23:25
clarkbcorvus: ianw that could be the pypi issue. There is no python requires annotation on the index ffor the link you linked23:25
clarkb (view source)23:26
clarkbif you pull it up from the requirements are there23:26
corvusso today we get 2.0.0 because no annotation; on days when we get annotations, we get 0.6.023:26
ianwsigh, ok.  that annotation thing is a pita23:27
corvusdid we learn anything more about what's going on there?23:27
clarkbya its a great idea if it worked reliably23:27
corvusor are we just assuming that something is messed up with the cdn?23:27
clarkbcorvus: no, but I did observe it flip to working after being not working23:27
clarkbcorvus: so I assumed bad cdn nodes and good cdn nodes with luck of the draw23:27
ianwi still have to finish spec for us to publish them23:27
corvusianw: will that spec affect the annotation issue?23:28
ianwcorvus: for us to publish annotations for our wheels, which has been an issue before, but not this problem i don't think23:29
ianwle sigh ...23:29
clarkbya this is different23:29
ianw  seeder FromAppData(download=False, pip=latest, setuptools=latest, wheel=latest, via=copy, app_data_dir=/home/zuul/.local/share/virtualenv/seed-app-data/v1.0.1)23:30
clarkbthis is from pypi proper23:30
ianwwtf is seed-app-data23:30
ianwunsurprisingly, that has setuptools 46.1.1 in it.  i do not know how that got there23:30
ianwbut "setuptools=latest" is ... not true23:30
clarkbthats the new virtualenv thing where it populated the venv23:31
clarkbit uses the bundled version by default23:31
clarkbwe may need virtualenv to update now or force it to update when creating venvs23:31
clarkbby update virthalenv I mean a newrelease with bundled newer setuptools23:31
ianw"virtualenv -p python3 --seeder pip test" doesn't help23:32
clarkbhrm pip shouldve fetched latest I thought23:33
ianw$ virtualenv -p python3 --seeder pip --download  test23:33
ianwdoes work!23:33
fungiso without --download it's just grabbing the vendored copy of setuptools when something asks for it23:34
ianwi guess?  if I say "--seeder pip" why would it not use pip, without "--download"23:35
ianwso, i wonder if we should put these options in a virtualenv config file?23:35
ianwor, i bet venv gets it right23:35
fungi"--seeder pip" says you want to include pip in the set of preinstalled packages in the env23:36
fungioh, no i'm wrong, that's the other seed option23:36
fungiadded at the same time23:36
clarkbya I think its pip realizing it can use the local versions23:36
ianwhaha >>> print(setuptools.version.__version__)23:36
ianwclarkb: as in the versions from .local/share/virtualenv/seed-app-data/ ... so maybe that uses "pip" to install from there, rather than just "cp" the tree23:37
ianwwhy you would want that ... i don't know ...23:38
ianwUpgrade embeded setuptools for Python 3.5+ to 46.1.1, for Python 2.7 to 44.1.0 - by @gaborbernat. (#1745)23:38
clarkbianw: ya23:39
ianwso, as usual we have a broken setuptools version blocking CI and have to work around it23:39
ianwthis is what leads to things like pip-and-virtualenv :/23:39
ianwwe could i think set "--seeder pip --download" in a global config -- although that exposes us the other way, when setuptools then releases a broken version and we upgrade into it constantly23:40
fungi--download: pass to enable download of the latest pip/setuptools/wheel from PyPI (default: False)23:40
fungiso does look necessary23:41
fungi--clear-app-data may also solve it for the default --seeder, not sure23:42
ianwi've filed
ianwit looks like they commit wheels into the tree ... i'm not going to go there23:46
clarkbinfra-root the meetpad stack is ready to go but if I approve it now it wil lfail on the futurist problem23:52
*** tosky has quit IRC23:53
clarkbthat will be fixed by the sdk release23:53
clarkbwe might consider updating that beaker job to only run on .pp changes?23:54
ianwfungi / clarkb: so ... do you have any opinions?  should we try to work around this, or just remain broken until virtualenv gets fixed?23:54
corvusfungi: assuming the futurist problem is fixed, if you want to approve the meetpad changes tomorrow, feel free; or i can when i wake up. (cc clarkb)23:56
ianw"You don't need another seeder, just pass in the --download flag will suffice +1 or the --setuptools 46.1.3 flag"
clarkbianw: can we control that via tox?23:57
clarkbI don't know that we are using tox in this case but whatever we do we may want to be consistent with our usage of tox23:57
ianwclarkb: yeah, in this case it's a pip: call with virtualenv specified in ansible ... i guess that's some variant of what tripelo is breaking on too23:58
clarkbhrm ya so similar problem where we don't directly call virtualenv23:58

Generated by 2.15.3 by Marius Gedminas - find it at!