| openstackgerrit | Ian Wienand proposed opendev/system-config master: Remove old hosts from cacti https://review.opendev.org/715097 | 00:06 |
|---|---|---|
| ianw | good idea :) | 00:08 |
| openstackgerrit | Ian Wienand proposed openstack/diskimage-builder master: centos 8 image build: fix mirror https://review.opendev.org/714836 | 00:22 |
| openstackgerrit | Ian Wienand proposed openstack/diskimage-builder master: run_functests: handle build without tar https://review.opendev.org/715098 | 00:22 |
| *** tosky has quit IRC | 00:52 | |
| *** mlavalle has quit IRC | 01:22 | |
| ianw | gosh darn it, something has definitely started going wrong with pip install executable files :/ | 01:33 |
| ianw | it must be something like https://github.com/pypa/pip/issues/6364 ... but it happens across all our images | 01:34 |
| *** diablo_rojo has quit IRC | 02:01 | |
| ianw | and here's the issue ... https://github.com/pypa/setuptools/issues/2041 ... arggh | 02:12 |
| ianw | so this is fixed in 46.1.3 ... our wheel cache has 46.1.1 (46.1.2 was released 11 hours ago, 46.1.3 8 hours ago) | 02:21 |
| ianw | ergo, i guess will be fixed tomorrow | 02:21 |
| *** diablo_rojo has joined #opendev | 03:10 | |
| openstackgerrit | Ian Wienand proposed opendev/base-jobs master: Generate download script for logs https://review.opendev.org/715118 | 03:59 |
| *** diablo_rojo has quit IRC | 05:39 | |
| openstackgerrit | Merged openstack/project-config master: Revert "Revert "Revert "Disable github reporting and re-add devel ansible job""" https://review.opendev.org/715081 | 05:44 |
| *** DSpider has joined #opendev | 05:57 | |
| *** ralonsoh has joined #opendev | 07:09 | |
| *** dpawlik has joined #opendev | 07:22 | |
| *** factor has quit IRC | 07:29 | |
| *** bolg has quit IRC | 07:59 | |
| *** vblando has quit IRC | 08:06 | |
| *** jaicaa has quit IRC | 08:06 | |
| *** jaicaa has joined #opendev | 08:07 | |
| *** bolg has joined #opendev | 08:17 | |
| *** rpittau|afk is now known as rpittau | 08:26 | |
| *** bolg has quit IRC | 08:26 | |
| *** bolg has joined #opendev | 08:40 | |
| *** jaicaa has quit IRC | 08:41 | |
| *** jaicaa has joined #opendev | 08:44 | |
| *** tosky has joined #opendev | 09:00 | |
| *** rpittau is now known as rpittau|bbl | 11:45 | |
| *** roman_g has joined #opendev | 12:08 | |
| *** lpetrut has joined #opendev | 12:18 | |
| openstackgerrit | Monty Taylor proposed opendev/system-config master: Remove files from letsencrypt group https://review.opendev.org/715185 | 12:20 |
| mordred | frickler: ^^ fixed that | 12:20 |
| mtreinish | mordred: I tried running openstacksdk's tests locally on 3.8 but haven't been able to reproduce that failure yet. My guess though is that an attachment stream (stdout, stderr, etc) closing is racing with testtools reading it, maybe a file refcount vs gc thing. | 12:29 |
| mordred | mtreinish: oh joy. that's lovely | 12:30 |
| * ttx waves at mtreinish | 12:30 | |
| frickler | mordred: what about static.openstack.org? in my understand the group only needs the hosts themselves, not the aliases | 12:34 |
| mtreinish | mordred: I haven't seen that error before though, but the only place iter_chunks and testtools content is really used is in handling attachments in the result stream. But let me know if you see it again and I'll try to dig more into it | 12:34 |
| mtreinish | might also be worth ping stephenfin about it because he just tracked down a long standing race in attachment handling to fix the issue with super large attachments | 12:35 |
| * mtreinish waves back at ttx | 12:35 | |
| mordred | mtreinish: kk. I've seen it on and off on the 3.8 jobs (Which are still non-voting) - but I'll see what I can learn and/or pull in stephenfin | 12:36 |
| ttx | mtreinish: how are you doing my friend | 12:36 |
| mordred | also - same question as ttx :) | 12:36 |
| mtreinish | I'm doing well, I've got no complaints. The quantum computing world is keeping me really busy, but I get to play with a lot of neat things. | 12:38 |
| ttx | warn me before you start breaking all encryption. I've stock to sell | 12:39 |
| ttx | although not now | 12:39 |
| mtreinish | heh, well if you want to factor 15, I can show you how to do that today. But going much bigger than that isn't for a long time | 12:41 |
| ttx | factoring 15 can come in handy | 12:42 |
| fungi | 15 is a lovely product of two prime factors. i don't know what anyone has against it | 12:47 |
| mtreinish | well it's not typically the kind of thing rsa keys are made of :) | 12:49 |
| fungi | rsa is so last century | 12:50 |
| *** rpittau|bbl is now known as rpittau | 13:10 | |
| openstackgerrit | Merged opendev/system-config master: Remove files02.openstack.org and related puppet https://review.opendev.org/709639 | 13:11 |
| openstackgerrit | Merged opendev/system-config master: Remove static site puppet https://review.opendev.org/710388 | 13:11 |
| openstackgerrit | Merged opendev/system-config master: Remove old hosts from cacti https://review.opendev.org/715097 | 13:11 |
| *** hashar has joined #opendev | 13:28 | |
| openstackgerrit | Merged opendev/system-config master: Start making 3.8 python images https://review.opendev.org/714532 | 13:43 |
| mordred | mnaser: the updates to python-base landed and now disable recommends. I hit an issue on osc related to that. looks like the fix is adding libc6-dev [compile test platform:dpkg] to the bindep (symptom is an error about a missing limits.h) - before libc6-dev was a recommends for gcc | 13:45 |
| mordred | mnaser: just in case you run in to that in any of your images | 13:45 |
| mnaser | mordred: oh -- that's helpful -- thank you | 14:03 |
| openstackgerrit | Monty Taylor proposed opendev/lodgeit master: Add libc6-dev to bindep https://review.opendev.org/715219 | 14:21 |
| mordred | mnaser: and ... ^^ :) | 14:21 |
| openstackgerrit | Monty Taylor proposed opendev/lodgeit master: Be explicit about python base image version https://review.opendev.org/715222 | 14:22 |
| *** jkt has quit IRC | 14:29 | |
| *** jkt has joined #opendev | 14:30 | |
| *** corvus has quit IRC | 14:30 | |
| *** corvus has joined #opendev | 14:31 | |
| *** roman_g has quit IRC | 14:34 | |
| openstackgerrit | Monty Taylor proposed opendev/storyboard master: Be explicit about base container image https://review.opendev.org/715229 | 14:40 |
| openstackgerrit | Monty Taylor proposed opendev/lodgeit master: Add libc6-dev to bindep and pin Pygments for 2.7 https://review.opendev.org/715219 | 14:48 |
| openstackgerrit | Monty Taylor proposed opendev/lodgeit master: Be explicit about python base image version https://review.opendev.org/715222 | 14:48 |
| mordred | clarkb, mnaser: ^^ 715219 is required to unbreak gate for lodgeit | 14:48 |
| *** roman_g has joined #opendev | 15:04 | |
| yoctozepto | mordred: I see pinning for Pygments - kolla is also hit by it, as well as a bunch of other packages - looks like something went wrong elsewhere, pygments did not release recently either | 15:07 |
| yoctozepto | came here to ask you if you saw this behavior elsewhere and saw pygments immediately | 15:07 |
| * yoctozepto triggered | 15:07 | |
| mordred | yoctozepto: oh, you're right -that release was may 8 | 15:08 |
| mordred | s/may/march/ | 15:08 |
| yoctozepto | mordred: pypi had infra problems during the day, they seem to be gone now though | 15:09 |
| yoctozepto | yet the failures remain | 15:09 |
| mordred | yoctozepto: https://zuul.opendev.org/t/opendev/build/b873ebadfe78421c868c4bc2009b855a/log/job-output.txt#435 | 15:09 |
| mordred | is the one I'm seeing | 15:09 |
| mordred | oh - hahahaha. I put in the wrong pin | 15:09 |
| openstackgerrit | Monty Taylor proposed opendev/lodgeit master: Add libc6-dev to bindep and pin Pygments for 2.7 https://review.opendev.org/715219 | 15:10 |
| yoctozepto | mordred: yeah, but bear in mind fixing pygments may just reveal another one to fix ;p | 15:10 |
| openstackgerrit | Monty Taylor proposed opendev/lodgeit master: Be explicit about python base image version https://review.opendev.org/715222 | 15:10 |
| mordred | yoctozepto: indeed. | 15:10 |
| mordred | let's see how that one does | 15:10 |
| *** chandan_kumar has joined #opendev | 15:14 | |
| *** noonedeadpunk has joined #opendev | 15:21 | |
| yoctozepto | debugged locally | 15:23 |
| yoctozepto | using official pypi the behavior is correct | 15:23 |
| *** elod has joined #opendev | 15:23 | |
| yoctozepto | using http://mirror.dfw.rax.opendev.org/pypi/simple the behavior is wrong - py2 gets py3 packages | 15:24 |
| yoctozepto | infra-root: please help, it seems the py2 mirror issues returned | 15:24 |
| fungi | yoctozepto: please link an example failure | 15:24 |
| fungi | i don't know/remember what "the py2 mirror issues" were | 15:24 |
| yoctozepto | fungi: mordred's one is good https://zuul.opendev.org/t/opendev/build/b873ebadfe78421c868c4bc2009b855a/log/job-output.txt#435 | 15:25 |
| yoctozepto | fungi: py2 "pip install Pygments" | 15:25 |
| yoctozepto | fungi: you probably remember how we blamed pip the last time ;p | 15:26 |
| yoctozepto | this time compared both behaviors locally | 15:26 |
| clarkb | it should be noted that is just a proxy to pypi | 15:27 |
| yoctozepto | view-source:https://pypi.org/simple/pygments/ | 15:27 |
| clarkb | I would start by comparing the indexes for pygments | 15:27 |
| clarkb | ya that | 15:27 |
| yoctozepto | view-source:http://mirror.dfw.rax.opendev.org/pypi/simple/pygments/ | 15:27 |
| clarkb | but they should be the same with a ~10 minute cache expiry | 15:27 |
| fungi | yoctozepto: that looks like the same "problem" we worked around for other packages by adding environment markers in the constraints file | 15:27 |
| corvus | did we suspect that our proxies were getting different data from different pypi cdn endpoints? | 15:28 |
| yoctozepto | fungi, clarkb: at some point we got constraints back in indices | 15:28 |
| yoctozepto | it could be cached issue from pypi | 15:28 |
| yoctozepto | they had problems today | 15:28 |
| clarkb | yoctozepto: they were always there in the simple index asits just a proxy | 15:28 |
| clarkb | only the wheel mirror which us not a proxy had the previous issue | 15:29 |
| yoctozepto | clarkb: yeah, that's why it worked | 15:29 |
| yoctozepto | clarkb: but now it doesn't :D | 15:29 |
| clarkb | corvus: in the past it wasdue to building wheels for everything. This sounds like wheel mirror isnt involved and could possibly be related to cdn deltas now | 15:30 |
| clarkb | looking at headers on that index request I don't see the old 600 second ttl | 15:34 |
| clarkb | the direct request to pypi has the max-age set | 15:35 |
| clarkb | oh and now there it is | 15:36 |
| clarkb | yoctozepto: on my first request through the proxy I got a file from march 10 with no cache-control header | 15:36 |
| clarkb | yoctozepto: on my latest request I get a file from march 26 with a cache-control header | 15:37 |
| clarkb | I think that supports corvus suspicion that cdn might be doing weird things | 15:37 |
| clarkb | (depending on where backend request falls we get a different result?) | 15:37 |
| yoctozepto | clarkb, corvus: yeah, pypi had problems today | 15:37 |
| yoctozepto | serious ones | 15:37 |
| clarkb | and that should flip no faster than every 10 minutes if we hit the cache control because it is 600 seconds | 15:37 |
| yoctozepto | well, I only see rax still feeds me with no constraints info | 15:39 |
| yoctozepto | which is really weird | 15:39 |
| yoctozepto | as it has pygments 2.6.1 | 15:39 |
| yoctozepto | so must be recent enough | 15:39 |
| clarkb | yoctozepto: thats the joy of cdn, it is often localized and that leads to problems like that | 15:39 |
| yoctozepto | yet missing the tags | 15:39 |
| yoctozepto | clarkb: ah, you mean rax is still in the window where it got it from bad cdn face | 15:39 |
| clarkb | note the version I get with cache-control has the python version info | 15:39 |
| clarkb | yoctozepto: more likely that the rax region is local to a bad cdn node and good cdn node(s). Some of our requests get bad results some get good basedon my testing | 15:40 |
| clarkb | right now the data looks good | 15:40 |
| clarkb | we'll just have to wait and see if that changes in 10 minutes or even further out | 15:41 |
| yoctozepto | clarkb: ack, confirmed, rax looks good now | 15:41 |
| clarkb | if it does continue to flip back and forth we can probably ping pypi folks to see if they can prod the unhappy cdn | 15:42 |
| *** njohnston has joined #opendev | 15:42 | |
| yoctozepto | clarkb: yeah, I already pinged them before about the general situation | 15:43 |
| *** chandan_kumar is now known as chandankumar | 15:51 | |
| *** mlavalle has joined #opendev | 16:00 | |
| mordred | mnaser: wanna +A https://review.opendev.org/#/c/715219 ? | 16:08 |
| mnaser | mordred: done | 16:18 |
| mordred | mnaser: woot | 16:20 |
| *** dpawlik has quit IRC | 16:26 | |
| openstackgerrit | Merged opendev/lodgeit master: Add libc6-dev to bindep and pin Pygments for 2.7 https://review.opendev.org/715219 | 16:29 |
| openstackgerrit | Merged opendev/lodgeit master: Be explicit about python base image version https://review.opendev.org/715222 | 16:30 |
| openstackgerrit | Clark Boylan proposed opendev/system-config master: Add links to dev docs on gitea https://review.opendev.org/715264 | 16:42 |
| clarkb | infra-root ^ thats soom noodling on how we can make the jump from opendev.org to pushing changes a bit easier for people. I think thats about as good as we can integrate with gitea without writing a bunch of golang to update routers and set flags on the html | 16:42 |
| *** amotoki has joined #opendev | 16:44 | |
| AJaeger | clarkb: good idea,, thanks | 16:46 |
| AJaeger | clarkb: can we remove the gitea link from the page as well? I mean the "Help" link at the top | 16:47 |
| clarkb | AJaeger: the next step there will be tweaking the develop doc to be a good landing spot for someone clicking from opendev.org (or maybe adding a new page if necessary) | 16:47 |
| clarkb | AJaeger: we can, I had left it there because it is a useful link for gitea specific help, but maybe that is confusing? | 16:47 |
| AJaeger | since we disable most of gitea, I consider it more confusing than helpful | 16:48 |
| clarkb | thats a good point | 16:48 |
| clarkb | its probably more useful if people arem anaging their own accounts and stuff directly there | 16:48 |
| clarkb | let me push that as a followup so we can consider both chagnes independently | 16:48 |
| AJaeger | yes, exactly | 16:48 |
| AJaeger | sure, go for it. thanks! | 16:48 |
| openstackgerrit | Clark Boylan proposed opendev/system-config master: Remove gitea doc links https://review.opendev.org/715267 | 16:49 |
| clarkb | fungi: corvus aprice sent that email about community meetings to the foundation list this morning | 16:57 |
| clarkb | (if you've not got a local copy I can get you al ink) | 16:58 |
| frickler | clarkb: I saw that and thought it would be great if we could get ppl to use meetpad for that in the future instead of zoom | 17:02 |
| clarkb | frickler: yup! I think ocne we've got it running we should have as many people use it as possible | 17:02 |
| *** bolg has quit IRC | 17:05 | |
| fungi | the "community meeting" might be somewhat disjoint from our usual ptg participants, so featureset they need could differ somewhat | 17:11 |
| *** bolg has joined #opendev | 17:11 | |
| fungi | (for example, dial-in trunks, session recordings auto-uploaded to a video streaming service, et cetera) | 17:12 |
| *** rpittau is now known as rpittau|afk | 17:12 | |
| fungi | i wouldn't want to overload the current meetpad plan with more than we need for the ptg at first | 17:12 |
| mordred | clarkb: frickler has some good feedback on 715264 - otherwise that stack looks god | 17:13 |
| clarkb | frickler: thanks for the review on that gitea chnge, can you check my comments and indicate if you want me to remove the signed in portion? | 17:13 |
| clarkb | (I'll fix the quotes once we decide if I should keep or removed the signed in bits) | 17:13 |
| mordred | I think it's fine for them to be there - in case we ever wind up having an authenticated version of gitea, putting this there now will make sure we don't forget to add it later :) | 17:15 |
| clarkb | ya that was my thought, better to be complete now and not forget later | 17:16 |
| openstackgerrit | Clark Boylan proposed opendev/system-config master: Add links to dev docs on gitea https://review.opendev.org/715264 | 17:16 |
| openstackgerrit | Clark Boylan proposed opendev/system-config master: Remove gitea doc links https://review.opendev.org/715267 | 17:16 |
| clarkb | that fixes the quotes | 17:16 |
| mordred | +2 | 17:17 |
| *** lpetrut has quit IRC | 17:21 | |
| openstackgerrit | David Shrewsbury proposed opendev/system-config master: Remove shrews from infra-root https://review.opendev.org/715277 | 17:22 |
| openstackgerrit | David Shrewsbury proposed openstack/project-config master: Remove shrews from infra-root https://review.opendev.org/715278 | 17:22 |
| openstackgerrit | David Shrewsbury proposed opendev/system-config master: Remove shrews from infra-root https://review.opendev.org/715277 | 17:27 |
| clarkb | mordred: Shrews can you check my comment on https://review.opendev.org/#/c/715277/2 I think the openstack ssh key resource doesn't get managed by ansible in the way we expect there (I've noted we can leave it as is in this cahnge and then rotate an update separately) | 17:32 |
| Shrews | clarkb: oh. fine by me if you just want to do it in the global rotation then | 17:33 |
| mordred | clarkb: yeah - I think you're right - but I also think we can do it later | 17:33 |
| clarkb | Shrews: ya I'm fine with that if you want to update the change to leave the key there for now | 17:34 |
| openstackgerrit | David Shrewsbury proposed opendev/system-config master: Remove shrews from infra-root https://review.opendev.org/715277 | 17:49 |
| Shrews | clarkb: that should do it | 17:49 |
| *** openstack has quit IRC | 17:49 | |
| *** openstack has joined #opendev | 17:51 | |
| *** ChanServ sets mode: +o openstack | 17:51 | |
| frickler | clarkb: o.k., in that case I have the next question why the class is called octicon-repo-push , is that some predefined thing? maybe using the class name from the to-be-removed help button would match better? | 18:00 |
| clarkb | frickler: ya its defined by the glyph set here https://octicons.github.com/ | 18:01 |
| clarkb | frickler: basically I picked one that I thought would be appropriate for getting started to push code | 18:01 |
| clarkb | repo-push is in the bottom third | 18:01 |
| *** roman_g has quit IRC | 18:14 | |
| *** hashar has quit IRC | 18:21 | |
| openstackgerrit | Merged opendev/system-config master: Remove shrews from infra-root https://review.opendev.org/715277 | 18:31 |
| *** diablo_rojo has joined #opendev | 18:47 | |
| openstackgerrit | sebastian marcet proposed opendev/puppet-openstackid master: added www. cname on vhost file https://review.opendev.org/715293 | 18:49 |
| *** ralonsoh has quit IRC | 18:51 | |
| corvus | clarkb: what are your thoughts on meetpad scheduling? | 19:39 |
| clarkb | corvus: of merging changes? I was planning to land the specs today after a bike ride, then I think we can start to land the implementation changes then spin up a server as soon as enough of those are ready | 19:39 |
| fungi | i should get reviewing the implementation stack for it onto my evening agenda | 19:41 |
| corvus | clarkb: sounds good! the stack is green; i'll chase down a couple more reviews so you can just approve them all together :) | 19:41 |
| corvus | mordred: can you look at https://review.opendev.org/714494 and https://review.opendev.org/714510 | 19:42 |
| corvus | fungi: thanks! | 19:42 |
| * clarkb eats lunch then will ride around on the bike and return to click approvals | 19:42 | |
| noonedeadpunk | mordred: once you have some time for reviews can you kindly check https://review.opendev.org/#/c/713692/ one more time?:) | 19:56 |
| mordred | noonedeadpunk: brilliant | 20:08 |
| mordred | corvus: I, for one, welcome our new meetpad overlords | 20:09 |
| mordred | clarkb, corvus: gerrit humans (such as paladox) have suggested upgrading to 2.14 as part of our upgrade process - so I added jobs to build it which is green: https://review.opendev.org/#/c/715284/ | 20:49 |
| corvus | mordred: i thought we could go straight to 2.16? | 20:51 |
| * corvus revisits https://etherpad.openstack.org/p/gerrit-upgrade | 20:52 | |
| corvus | oh, apparently that says we need to upgrade to 2.15 | 20:52 |
| mordred | corvus: yeah - and people keep saying that we should do a stop at 2.14 on the way from 13 to 15 | 20:54 |
| corvus | can we record that and why? it'd be really good to know that | 20:54 |
| corvus | and what does "stop" mean? | 20:54 |
| corvus | because i'm really hesitant to put a different unsupported version into production | 20:54 |
| mordred | corvus: these are all great questions | 20:54 |
| corvus | if we have to do 2.14 for some sort of migration reason, i'd prefer we do that, then continue the outage to 2.15, then continue the outage to 2.16, then put 2.16 into prod | 20:55 |
| corvus | but putting anything < 2.16 into prod seems scary to me | 20:55 |
| mordred | corvus: yeah - that's what I'm hoping for as well | 20:56 |
| corvus | like, spending weeks fixing issues with a version that's out of support and we don't want to run anyway sounds :( | 20:56 |
| *** DSpider has quit IRC | 21:05 | |
| *** DSpider has joined #opendev | 21:05 | |
| ianw | speaking of wheels in scrollback, did we get setuptools 46.1.3 built? | 21:10 |
| * ianw goes to check how many other brown bag releases setuptools might have made overnight | 21:10 | |
| ianw | apropos nothing; does anyone have experience with https://forwardemail.net/en? their promise of SRS for rewriting spf on forwards sounds like what i want | 21:21 |
| mordred | corvus: I'm chatting with folks in #gerrit - and it seems the story is that we can do online reindexes/upgrades if we go one minor at a time - but if we bounce through them (like just running the init on each version) to get all the way to 2.16 - then we'll need to do an offline reindex | 21:23 |
| fungi | keeping in mind that reindexing requires somewhere north of 4 hours | 21:24 |
| mordred | corvus: so we could do 2.13->2.14 let online reindex run to completion then 2.14->2.15 online reindex then 2.15->2.16 online reindex - but it might mean running for a few days at each of the intermediaries | 21:24 |
| mordred | fungi: oh is it only 4? why did I think it was more like 18? | 21:25 |
| corvus | mordred: thanks, i'll read backscroll | 21:25 |
| fungi | it used to be longer, may still be longer than 4 | 21:25 |
| fungi | i can probably extract last week's online reindex time from the log if it hasn't rotated out yet | 21:25 |
| mordred | oh - because we did an online reindex due to the rename | 21:26 |
| corvus | mordred: i understand what paladox is saying there now, thanks :) | 21:27 |
| paladox | :) | 21:27 |
| mordred | so if it really is only like 4 hours - then we coudl conceivably do a weekend upgrade where we do run each version during the reindex - but it would only be for the 4 hours or so that the reindex takes - then we do the next upgrade | 21:27 |
| corvus | i excerpted the relevant bit in https://etherpad.openstack.org/p/gerrit-upgrade | 21:28 |
| mordred | (we'll obviously test this with a copy of real data on a whole new thing ... but I think given the time frames limiting the test scenarios we want to check out and validate is important) | 21:29 |
| corvus | mordred: ++ | 21:29 |
| fungi | 2020-03-20 14:33:18,213] [Reindex v32-v32] INFO com.google.gerrit.server.index.OnlineReindexer : Starting online reindex from schema version 32 to 32 | 21:30 |
| fungi | [2020-03-20 18:27:56,198] [Reindex v32-v32] INFO com.google.gerrit.server.index.OnlineReindexer : Reindex to version 32 complete | 21:30 |
| fungi | just shy of 4 hours | 21:30 |
| corvus | fungi: well remembered! :) | 21:30 |
| corvus | mordred: should be feasible to get a similar sized server, make a db copy and filesystem copy, then manually step through the container images | 21:30 |
| mordred | corvus: yeah - that's what I'm thinking | 21:30 |
| fungi | keep in mind, that's reindex under the same schema. these upgrades we're talking about not only incorporate schema migrations but also moving content to entirely different databases | 21:31 |
| mordred | corvus: and we can try the stepwise-online version - and then maybe the skip-to-2.16 with an offline reindex version | 21:31 |
| mordred | fungi: so - doing nodedb can be done as its own activity | 21:31 |
| fungi | cool, just making sure we're keeping that transition in mind | 21:31 |
| mordred | so I think the idea is to upgrade to 2.16 without notedb and get up and running ... then trigger the online nodedb migration | 21:31 |
| mordred | which will take longer than a normal migration | 21:32 |
| fungi | i guess notedb is the reason to pause at 2.16? | 21:32 |
| mordred | yeah | 21:32 |
| fungi | sounds good then | 21:32 |
| corvus | the word on the street is: do not touch notedb before 2.16. | 21:32 |
| paladox | Yup | 21:32 |
| fungi | i trust the streetwords | 21:32 |
| mordred | that and also 2.16 is the first version with polygerrit and the last version with gwt - so it's not a bad idea to maybe pause there for a minute to let people start getting used to pg before the flag day | 21:32 |
| mordred | not a ton of time - but since we've got the notedb migration anyway | 21:33 |
| paladox | we unfortunately migrated under 2.15 :) | 21:33 |
| mordred | then once we're ready - 2.16 to 3.0 is basically just the removal of gwt - and then 3.1 is a normal minor release which should be a quick upgrade | 21:34 |
| fungi | i could imagine pausing at 2.16 for as much as a few weeks to give folks an opportunity to toggle between old and new webui | 21:34 |
| mordred | yeah | 21:34 |
| fungi | the switch from the original change screen to the "new" change screen was fairly disruptive, or at least i would think it was if i could still remember it. that was many beers ago | 21:34 |
| mordred | so many beers | 21:35 |
| * fungi makes it one more | 21:35 | |
| mordred | fungi: related to this: https://review.opendev.org/#/c/715284/ if you have a sec | 21:35 |
| paladox | corvus thats even documented on the RN :) | 21:35 |
| fungi | huh, your jobs are passing on that change | 21:38 |
| fungi | i've been trying to work out why this system-config change failed two jobs: https://review.opendev.org/712588 | 21:39 |
| ianw | i'm still seeing : /home/zuul/dib-venv/lib/python3.6/site-packages/diskimage_builder/lib/dib-run-parts: Permission denied ... that is disappointing | 21:39 |
| mordred | fungi: maybe the world has fixed itself since your change? | 21:40 |
| fungi | yeah, i just hate rechecking when i don't know what the cause was | 21:40 |
| mordred | fungi: there's a whole 2 hours different | 21:40 |
| mordred | right? | 21:40 |
| fungi | time is an illusion | 21:40 |
| mordred | ianw: that is disappointing | 21:40 |
| ianw | i wonder if it's our image builds not picking up the new version, not wheels | 21:41 |
| ianw | fungi: do you have any recollection of setuptools and wheel builds from your recent adventures there | 21:41 |
| fungi | i've probably pickled those brain cells, but will see if i can spot something | 21:42 |
| fungi | this is on ubuntu-bionic i suppose? | 21:43 |
| ianw | 2020-03-26 09:26:40.915 | Downloading setuptools-46.1.3-py3-none-any.whl (582 kB) | 21:43 |
| ianw | https://nb02.openstack.org/ubuntu-bionic-0000104233.log | 21:43 |
| ianw | yep, bionic | 21:44 |
| ianw | actually, xenial and centos7 fails too ... so ... yeah | 21:45 |
| fungi | so we did at least fetch 46.1.3 which is supposedly fixed (was fixed in 46.1.2 and later right?) | 21:47 |
| ianw | # /usr/local/lib/python3.6/dist-packages/setuptools | 21:47 |
| ianw | setuptools/ setuptools-46.1.3.dist-info/ | 21:47 |
| ianw | https://github.com/pypa/setuptools/issues/2041 suggests it is fixed ... but i have not verified as such (i was hoping things would just go green!) | 21:48 |
| * clarkb back from biking catching up on reviews now | 21:50 | |
| *** DSpider has quit IRC | 21:53 | |
| ianw | https://4f1c7548b35d159f1fcf-f6d9d58f0487401586e66f66be7f6106.ssl.cf5.rackcdn.com/708416/9/check/dib-nodepool-functional-openstack-ubuntu-bionic/1f283ca/ ... passed : rax-iad | 21:55 |
| ianw | https://storage.gra.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_572/708416/9/check/dib-nodepool-functional-openstack-centos-7-src/572caff/ ... failed: rax-dfw | 21:55 |
| mordred | ianw: :( | 21:55 |
| fungi | ianw: image upload issues? | 21:57 |
| ianw | hrm, images seem fresh ... http://paste.openstack.org/show/791215/ | 21:58 |
| ianw | both have the same build date in zuul-info/zuul-info.ubuntu-bionic.txt ; so same image | 22:02 |
| clarkb | infra-root I'm going ot approve https://review.opendev.org/#/c/715118/1 which is a base job change that has been tested via base-test to add the log download script artifact. Let me know if there is additioanl testing or whatever you think we should do and I'll hold off (but I think we've followed the general process there and its ready to go) | 22:03 |
| fungi | ianw: did you have a failing ubuntu-bionic example? you provided a working ubuntu-bionic link and a failing centos-7 | 22:03 |
| ianw | fungi: so that's both running dib and building on bionic ... but one is building bionic to test and the other centos | 22:06 |
| fungi | ahh, okay | 22:06 |
| fungi | and both jobs ran from the same nodepool image, as best we can tell | 22:07 |
| clarkb | re gerrit iirc we also want to stop on 2.something before 3.whatever in order to populate the notedb right? | 22:08 |
| ianw | yep; they must be the same really, they have the same build date | 22:08 |
| fungi | clarkb: yeah, we clarified that, 2.16 to be precie | 22:09 |
| fungi | precise | 22:09 |
| fungi | precise as a pangolin | 22:10 |
| mordred | clarkb: yah | 22:12 |
| mordred | infra-root: https://zuul.opendev.org/t/openstack/build/a443909f2aa842b28e58bc649c321abc <-- got another hostkey changed error | 22:18 |
| clarkb | mordred: that one is in inap | 22:19 |
| clarkb | mordred: that is beginning to make me wonder if either Neutron has a widespread bug or maybe a bug on our side? | 22:19 |
| mordred | clarkb: yeah. but I'm not sure what the bug on our side would be | 22:20 |
| clarkb | mordred: losing the known hosts file or overwriting it somehow? I don't think that is likely but it is theoretically possible | 22:20 |
| *** Shrews has left #opendev | 22:21 | |
| mordred | I'd think we'd see the issue much more frequently if that was a thing | 22:21 |
| clarkb | this was definitely a rax specific issue in the past | 22:22 |
| mordred | yeah. | 22:22 |
| clarkb | but whatever neutron thing was happening there could be happenign elsewhere now | 22:22 |
| clarkb | wow these pypi issues are far reaching /me tries another recheck on the gitea docs changes | 22:23 |
| corvus | clarkb, ianw: oh, i thought i asked somewhere about using the manifest file instead of generating a script? | 22:27 |
| corvus | i don't see a comment on the change, sorry about that | 22:28 |
| corvus | maybe i asked in irc? or maybe i didn't ask | 22:28 |
| corvus | anyway, the swift log upload script is one of 2, going on 3 log upload scripts; i'll grant that it's annoying that there's no code reuse between them, but my initial thought for dealing with that was to avoid changing them. | 22:29 |
| corvus | and it seems that now that we have the zuul manifest, you could add a log download script that's completely independent of the upload process | 22:30 |
| corvus | that is, after all, exactly how zuul's log viewer handles it | 22:30 |
| clarkb | corvus: how are you thinking we distribute that script? | 22:31 |
| corvus | ianw, clarkb: does that sound feasible? would you consider reverting the swift change in favor of that so we can try to keep the upload roles simpler? | 22:31 |
| corvus | (after all, we don't even use the upload roles to generate the manifest!) | 22:31 |
| clarkb | I think keeping the upload roles simpler is a good goal | 22:31 |
| clarkb | (though maybe we'd still be uploading a script someway it will just be a common script) | 22:32 |
| mordred | it seems like a script that one could download once and use for any zuul ... kind of like encrypt_secret | 22:32 |
| corvus | clarkb: any way we want? :) we could put it in the infra manual; put it in a central location on opendev, or, if we want a copy of it in every upload, stick it in the base job so it gets uploaded with the logs. | 22:32 |
| corvus | it could take an argument, so it's generic, but we could put the command in our logs in our base job | 22:32 |
| ianw | mordred: so that was pretty much what i wanted to avoid ... the idea is for it to be a absolue zero-overhead way to grab the logs if you know nothing | 22:32 |
| mordred | yeah - I think that would potentially go with the "upload the script to the logs server" idea | 22:33 |
| corvus | so the console log would output "run 'download-logs https://....'" | 22:33 |
| corvus | or if you want to template that into a script, we could even do that | 22:34 |
| corvus | just make it its own role | 22:34 |
| corvus | (i guess that's what the current patch does?) | 22:34 |
| corvus | but if it's its own role, then we've got better separation of concerns | 22:34 |
| corvus | (to summarize, i think the proposal that meets the most number of requirements would be: a new role which templates a semi-generic script into the log directory before upload and embeds the build uuid into it; the script uses the zuul rest api to query the log location for that build, fetch the manifest, and download all the files) | 22:35 |
| openstackgerrit | Merged opendev/infra-specs master: Add a spec for meetpad https://review.opendev.org/714189 | 22:35 |
| clarkb | corvus: ya I think that would work | 22:36 |
| corvus | (it does not address fungi's concern; i think that could only be addressed by distributing a completely generic script and only printing the instructions to run it in the logs) | 22:37 |
| corvus | (that seems to be a conflicting requirement with ianw's "no-install" requirement) | 22:37 |
| clarkb | maybe we could distribute it with zuul and the webserver could serve it rather than the logs server | 22:38 |
| clarkb | just build it right into the web ui server | 22:38 |
| clarkb | (I'm not sure the distinction of server would be apparenty to end users though) | 22:38 |
| corvus | yeah, maybe? | 22:38 |
| ianw | i guess i should write a spec | 22:39 |
| corvus | also, i'm pretty sure it could be done completely with the standard library, so installation shouldn't be a big deal (wget the script once, and you're set). but i understand that still may not meet the ease of use level ianw is going for | 22:40 |
| fungi | mordred: i spoke too soon, your https://review.opendev.org/715284 choked on openstackci-beaker-puppet-4 in the gate as well | 22:40 |
| fungi | looking now to see if it's similar | 22:40 |
| ianw | corvus: yeah, i was trying to even avoid a python script; and parsing json in pure bash is a bit of a pain | 22:41 |
| fungi | mordred: yeah, looks like the same error condition i was seeing on my changes too | 22:41 |
| corvus | ianw: i think the days where debian users would say "i don't have python installed, only perl" are probably past :) | 22:42 |
| ianw | while i totally understand the anti-pattern discussion we've had, i feel there is a reason why projects find it useful to give people a "curl https://... | bash - " | 22:42 |
| corvus | i think it's an entirely reasonable thing to assume that 99% of people debugging a CI job will have *some* version of python | 22:42 |
| corvus | (and yes, i would write the script to work in py27 because we can, easily, and it's less of a speed bump. same as encrypt_secret) | 22:43 |
| ianw | corvus: how does the script embed it's log url in it, but still also get uploaded? | 22:44 |
| clarkb | fungi: those beaker job failures I'ev seen haev been due to pypi issues | 22:45 |
| corvus | [i don't feel as strongly as fungi about the merits of curl|bash (or, in the case i'm suggesting, curl|python); my main concern is maximum reuse of code and minimal complexity of the log uploaders.] | 22:45 |
| corvus | ianw: it embeds the uuid instead of the url, then uses the rest api to find the url | 22:45 |
| fungi | mordred: is this the problem, do you think? Error: Facter: error while resolving custom fact "virtualenv_version": undefined method `[]' for nil:NilClass | 22:46 |
| clarkb | fungi: no that is not the problem | 22:46 |
| fungi | okay, benign? | 22:46 |
| clarkb | fungi: ya basically puppet ignores when factor fails and just takes whatever facts it can get | 22:46 |
| clarkb | https://zuul.opendev.org/t/openstack/build/c5d31c1f03904136b937762d8fa8deb4/log/job-output.txt#5041 is the issue | 22:47 |
| clarkb | which I've thought was pypi but looking at timestamps that is super recent, possible we just need to fix a dep in nodepool? | 22:47 |
| mordred | nope - that's an sdk thing | 22:47 |
| fungi | oh neat, that's buried in there | 22:47 |
| mordred | there is a fix already landed - just waiting on a release | 22:47 |
| mordred | https://review.opendev.org/#/c/715317/ | 22:48 |
| fungi | okay, thanks! will wait | 22:48 |
| mordred | I pinged smcginnis about it - but I think he was EOD before the patch landed | 22:48 |
| clarkb | mordred: the sdk thing tweaks the futurist dep? | 22:49 |
| mordred | also - fwiw - we realized in fixing the sdk issue that we could drop the dependency entirely | 22:49 |
| mordred | well - there's 2 sdk issues - butonly one impacts nodepool here | 22:49 |
| mordred | the first was futurist not being 3.5 compat - which we missed - which is solved in the patch by removing futurist as a direct depend | 22:49 |
| mordred | the second was that we didn't add a requires-python stanza (if we had, I probably would have caught that 3.6 > 3.5) - but that also meant it broke py2 users in a way that didn't allow pip to get the "latest that will work with python2" | 22:50 |
| mordred | once 0.45 is tagged, we'll be doing a followup change that adds requires-python >= 3.5 for completeness | 22:50 |
| fungi | yeah, i caught some of that discussion in the qa channel | 22:51 |
| mordred | but the 0.45 release will fix that issue above | 22:51 |
| fungi | just didn't notice it was what was also blocking system-config changes | 22:51 |
| mordred | yeah. me either | 22:51 |
| corvus | ianw: what do you think? would you be okay with a revert of 592341 while we rethink it? if you're open to it, i'd like to revert soon so we don't have to worry about supporting that. as pennance for not reviewing in time, i'll help write the python script if we decide to go that way. i'm very sorry for not catching that earlier. | 22:51 |
| ianw | corvus: ok, we can revert if there's not consensus. there's infact an abanonded revert of it already. i think maybe a spec ... although am i the only one that actually thinks this is useful? | 22:52 |
| ianw | it's something i've wanted for many years debugging devstack, where you really have to correlate across multiple logs to understand what's going on | 22:53 |
| fungi | ianw: i think it's useful in the same way that i actually miss being able to tell an ftp server that i want to get a directory but tack a .tar.gz onto the name and have it send me a tarball of the entire subtree | 22:53 |
| clarkb | ianw: I had that problem with devstack until we started publishing the journal in one file, that covers a huge chunk of the use case for me | 22:54 |
| corvus | fungi, ianw, clarkb: maybe *that* is what we should do? | 22:54 |
| ianw | that's true -- i certainly started this well before the journal | 22:54 |
| fungi | ianw: unfortunately i think that's something http daemons missed out on | 22:54 |
| ianw | corvus: i had a *really* old change that grabbed everything and sent it down as a .tar.gz ... | 22:54 |
| clarkb | diablo_rojo: are you able to push https://review.opendev.org/#/c/715317/ along? | 22:55 |
| corvus | ianw: to directly answer your question: if >1 person thinks this is useful, i'm happy to do it. i would not be surprised if other folks (especially tripleo/rdo) would find it useful. personally, i try to use the web as much as possible. but i don't like judging or dictating how people read log files :) | 22:56 |
| ianw | https://review.opendev.org/#/c/122615/ ... os_loganalyze ... remember that | 22:56 |
| mordred | fungi: we _can_ pin sdk to <0.44 if we need to while we wait on the release - that will do the same result - but worst-case we should have release by morningish | 22:56 |
| corvus | ianw: (probably if you are literally the only person, then you could do something much simpler; but i bet there are others) | 22:56 |
| fungi | mordred: i'm not in a huge hurry, no emergency fixes to merge at least | 22:57 |
| mordred | kk | 22:57 |
| fungi | for me it was blocking addition of meetbot to a couple of channels, and it's no big loss if that waits another day | 22:57 |
| mordred | cool | 22:58 |
| * mordred is going to go for a walk. | 22:59 | |
| clarkb | ya for me it was the opendev docs link changes | 22:59 |
| clarkb | those can also land tomorrow happily | 22:59 |
| openstackgerrit | James E. Blair proposed zuul/zuul-jobs master: Revert "upload-logs-swift: Create a download script" https://review.opendev.org/715325 | 23:00 |
| ianw | infra-root: https://review.opendev.org/#/c/714000 reverts the log downloader | 23:00 |
| ianw | ok, 715325 also reverts the unicode stuff | 23:01 |
| corvus | i thought that was necessary... | 23:01 |
| corvus | does 714000 just revert the log downloader without reverting unicode? | 23:02 |
| corvus | either way; i'm happy to keep the unicode thing if that's what the other one does | 23:04 |
| * corvus is slightly confused :) | 23:04 | |
| ianw | corvus: so i am; it's been a long time. i think the unicode change can stand alone, if the test-suite bits for the log downloader are removed | 23:05 |
| ianw | i think the unicode test is still valid for the uploader, in general | 23:05 |
| ianw | it's -1'd so might need untangling | 23:13 |
| ianw | ... back to the other issue ... | 23:16 |
| ianw | >>> print(setuptools.version.__version__) | 23:16 |
| ianw | 46.1.3 | 23:16 |
| ianw | the setuptools version on our hosts is 46.1.3 ... HOWEVER | 23:16 |
| ianw | $ ./dib-venv/bin/python3 | 23:17 |
| ianw | >>> print(setuptools.version.__version__) | 23:17 |
| ianw | 46.1.1 | 23:17 |
| ianw | the version installed, somehow, via zuul running "pip:" into a virtualenv with is 46.1.1 | 23:18 |
| clarkb | is 46.1.3 version excluded? | 23:18 |
| clarkb | 46.1.1 and 46.1.3 have the same python requires >= 3.5 restriction | 23:19 |
| clarkb | that probably isn't it | 23:19 |
| ianw | i can't see that it is, although this *is* using upper-constraints i think | 23:19 |
| ianw | https://opendev.org/openstack/diskimage-builder/src/branch/master/roles/dib-functests/tasks/main.yaml#L24 has made the "bad" virtualenv | 23:20 |
| corvus | ianw: Downloading http://mirror.sjc1.vexxhost.openstack.org/pypifiles/packages/30/81/87a64fd890fa416232abb219210a48afa040a4e85436c35c7eee077c0f45/bashate-2.0.0.tar.gz (29 kB) | 23:21 |
| corvus | ERROR: Package 'bashate' requires a different Python: 2.7.17 not in '>=3.5' | 23:21 |
| corvus | ianw: ^ sorry, second line is what i wanted to draw your attention to | 23:21 |
| clarkb | setuptools is not constrained in upper constraints | 23:21 |
| clarkb | corvus: oh interesting odd that it would use 46.1.1 in that case consider that also requires >= 3.5 | 23:21 |
| clarkb | maybe we are still getting incomplete data from ypi? | 23:22 |
| corvus | clarkb: wait i think i'm talking about a different issue | 23:22 |
| corvus | i'm talking about how we can't merge anything to zuul-jobs because apparently we use bashate | 23:22 |
| clarkb | got it (I did mix streams) | 23:23 |
| openstackgerrit | James E. Blair proposed zuul/zuul-jobs master: Remove bashate from test-requirements https://review.opendev.org/715328 | 23:23 |
| openstackgerrit | James E. Blair proposed zuul/zuul-jobs master: Revert "upload-logs-swift: Create a download script" https://review.opendev.org/715325 | 23:24 |
| ianw | hrm, bashate released ages ago | 23:24 |
| ianw | why did it just fail now? | 23:24 |
| corvus | ianw: i dunno -- here's what ran on your log downloader change: https://zuul.opendev.org/t/zuul/build/2ed70ace09904d929dd5245d9d30d682/log/tox/py27-1.log#13 | 23:25 |
| corvus | it sure did get an old version | 23:25 |
| clarkb | corvus: ianw that could be the pypi issue. There is no python requires annotation on the index ffor the link you linked | 23:25 |
| clarkb | http://mirror.sjc1.vexxhost.openstack.org/pypi/simple/bashate/ (view source) | 23:26 |
| corvus | ah | 23:26 |
| clarkb | if you pull it up from pypi.org/simple/bashate the requirements are there | 23:26 |
| corvus | so today we get 2.0.0 because no annotation; on days when we get annotations, we get 0.6.0 | 23:26 |
| clarkb | ya | 23:26 |
| ianw | sigh, ok. that annotation thing is a pita | 23:27 |
| corvus | did we learn anything more about what's going on there? | 23:27 |
| clarkb | ya its a great idea if it worked reliably | 23:27 |
| corvus | or are we just assuming that something is messed up with the cdn? | 23:27 |
| clarkb | corvus: no, but I did observe it flip to working after being not working | 23:27 |
| clarkb | corvus: so I assumed bad cdn nodes and good cdn nodes with luck of the draw | 23:27 |
| ianw | i still have to finish spec https://review.opendev.org/#/c/703916/ for us to publish them | 23:27 |
| corvus | ianw: will that spec affect the annotation issue? | 23:28 |
| ianw | corvus: for us to publish annotations for our wheels, which has been an issue before, but not this problem i don't think | 23:29 |
| ianw | le sigh ... | 23:29 |
| clarkb | ya this is different | 23:29 |
| ianw | seeder FromAppData(download=False, pip=latest, setuptools=latest, wheel=latest, via=copy, app_data_dir=/home/zuul/.local/share/virtualenv/seed-app-data/v1.0.1) | 23:30 |
| clarkb | this is from pypi proper | 23:30 |
| ianw | wtf is seed-app-data | 23:30 |
| ianw | unsurprisingly, that has setuptools 46.1.1 in it. i do not know how that got there | 23:30 |
| ianw | but "setuptools=latest" is ... not true | 23:30 |
| clarkb | thats the new virtualenv thing where it populated the venv | 23:31 |
| clarkb | it uses the bundled version by default | 23:31 |
| clarkb | we may need virtualenv to update now or force it to update when creating venvs | 23:31 |
| clarkb | by update virthalenv I mean a newrelease with bundled newer setuptools | 23:31 |
| ianw | "virtualenv -p python3 --seeder pip test" doesn't help | 23:32 |
| clarkb | hrm pip shouldve fetched latest I thought | 23:33 |
| ianw | $ virtualenv -p python3 --seeder pip --download test | 23:33 |
| ianw | does work! | 23:33 |
| clarkb | ah | 23:33 |
| fungi | so without --download it's just grabbing the vendored copy of setuptools when something asks for it | 23:34 |
| ianw | i guess? if I say "--seeder pip" why would it not use pip, without "--download" | 23:35 |
| ianw | so, i wonder if we should put these options in a virtualenv config file? | 23:35 |
| ianw | or, i bet venv gets it right | 23:35 |
| fungi | "--seeder pip" says you want to include pip in the set of preinstalled packages in the env | 23:36 |
| fungi | oh, no i'm wrong, that's the other seed option | 23:36 |
| fungi | added at the same time | 23:36 |
| clarkb | ya I think its pip realizing it can use the local versions | 23:36 |
| ianw | haha >>> print(setuptools.version.__version__) | 23:36 |
| ianw | 39.0.1 | 23:36 |
| fungi | yeah | 23:36 |
| ianw | clarkb: as in the versions from .local/share/virtualenv/seed-app-data/ ... so maybe that uses "pip" to install from there, rather than just "cp" the tree | 23:37 |
| ianw | why you would want that ... i don't know ... | 23:38 |
| ianw | https://virtualenv.pypa.io/en/latest/changelog.html#features-20-0-14 | 23:38 |
| ianw | Upgrade embeded setuptools for Python 3.5+ to 46.1.1, for Python 2.7 to 44.1.0 - by @gaborbernat. (#1745) | 23:38 |
| clarkb | ianw: ya | 23:39 |
| ianw | so, as usual we have a broken setuptools version blocking CI and have to work around it | 23:39 |
| ianw | this is what leads to things like pip-and-virtualenv :/ | 23:39 |
| ianw | we could i think set "--seeder pip --download" in a global config -- although that exposes us the other way, when setuptools then releases a broken version and we upgrade into it constantly | 23:40 |
| fungi | --download: pass to enable download of the latest pip/setuptools/wheel from PyPI (default: False) | 23:40 |
| fungi | so does look necessary | 23:41 |
| fungi | --clear-app-data may also solve it for the default --seeder, not sure | 23:42 |
| ianw | i've filed https://github.com/pypa/virtualenv/issues/1753 | 23:45 |
| ianw | it looks like they commit wheels into the tree ... i'm not going to go there | 23:46 |
| ianw | https://github.com/pypa/virtualenv/commit/02c58c134631a2753ef2c7b2128a5ad6059ff882 | 23:46 |
| clarkb | infra-root the meetpad stack is ready to go but if I approve it now it wil lfail on the futurist problem | 23:52 |
| *** tosky has quit IRC | 23:53 | |
| clarkb | that will be fixed by the sdk release | 23:53 |
| clarkb | we might consider updating that beaker job to only run on .pp changes? | 23:54 |
| ianw | fungi / clarkb: so ... do you have any opinions? should we try to work around this, or just remain broken until virtualenv gets fixed? | 23:54 |
| corvus | fungi: assuming the futurist problem is fixed, if you want to approve the meetpad changes tomorrow, feel free; or i can when i wake up. (cc clarkb) | 23:56 |
| ianw | "You don't need another seeder, just pass in the --download flag will suffice +1 or the --setuptools 46.1.3 flag" https://github.com/pypa/virtualenv/issues/1752#issuecomment-604745201 | 23:56 |
| clarkb | ianw: can we control that via tox? | 23:57 |
| clarkb | I don't know that we are using tox in this case but whatever we do we may want to be consistent with our usage of tox | 23:57 |
| ianw | clarkb: yeah, in this case it's a pip: call with virtualenv specified in ansible ... i guess that's some variant of what tripelo is breaking on too | 23:58 |
| clarkb | hrm ya so similar problem where we don't directly call virtualenv | 23:58 |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!