Tuesday, 2020-09-08

ianwnp, i know it's a holiday :)00:02
fungiwell, also well into the evening here00:25
fungiso i'm even less responsive than usual00:26
*** elod has quit IRC00:35
*** elod has joined #opendev00:36
*** ysandeep|away is now known as ysandeep01:19
*** noonedeadpunk has quit IRC01:47
*** noonedeadpunk has joined #opendev01:50
fricklerianw: can you take a look at clarkb's comment on https://review.opendev.org/#/c/749604/1 ? that seems to be the only thing keeping the whole stack from getting merged05:29
ianwfrickler: i'm pretty sure that in all contexts that runs bridge==localhost ... if you agree we can probably merge05:42
*** DSpider has joined #opendev05:59
*** xiaolin has joined #opendev06:01
*** qchris has quit IRC06:20
*** xiaolin has quit IRC06:21
*** xiaolin has joined #opendev06:24
*** qchris has joined #opendev06:33
*** hashar has joined #opendev06:53
*** openstackgerrit has joined #opendev06:55
openstackgerritIan Wienand proposed openstack/diskimage-builder master: bootloader: remove dangling grubenv links  https://review.opendev.org/75027906:55
*** andrewbonney has joined #opendev07:28
openstackgerritIan Wienand proposed openstack/diskimage-builder master: bootloader: remove dangling grubenv links  https://review.opendev.org/75027907:31
*** tosky has joined #opendev07:35
*** dtantsur|afk is now known as dtantsur07:56
*** moppy has quit IRC08:01
*** moppy has joined #opendev08:02
openstackgerritIan Wienand proposed openstack/diskimage-builder master: bootloader: remove dangling grubenv links  https://review.opendev.org/75027908:05
*** tosky has quit IRC08:05
fricklerianw: I don't feel confident enough to state that, better wait for someone else to double check08:09
*** tosky has joined #opendev08:10
*** tosky has quit IRC08:14
*** tosky has joined #opendev08:14
*** priteau has joined #opendev08:35
*** priteau has quit IRC09:12
*** priteau has joined #opendev09:14
*** priteau has quit IRC09:17
*** priteau has joined #opendev09:17
openstackgerritSorin Sbarnea (zbr) proposed opendev/elastic-recheck master: Drop py27 and add py38 jobs  https://review.opendev.org/72933009:21
*** xiaolin has quit IRC09:23
*** tosky has quit IRC09:39
*** tosky has joined #opendev09:54
openstackgerritSorin Sbarnea (zbr) proposed opendev/elastic-recheck master: Enable pylint  https://review.opendev.org/75031210:25
openstackgerritMerged opendev/elastic-recheck master: Drop py27 and add py38 jobs  https://review.opendev.org/72933010:35
openstackgerritSorin Sbarnea (zbr) proposed opendev/elastic-recheck master: Avoid pip breakage due to keyring  https://review.opendev.org/75031810:53
openstackgerritSorin Sbarnea (zbr) proposed opendev/elastic-recheck master: Avoid pip breakage due to keyring  https://review.opendev.org/75031810:55
openstackgerritSorin Sbarnea (zbr) proposed opendev/elastic-recheck master: Enable pylint  https://review.opendev.org/75031211:01
*** ysandeep is now known as ysandeep|brb11:50
openstackgerritSorin Sbarnea (zbr) proposed opendev/elastic-recheck master: Enable pylint  https://review.opendev.org/75031212:12
*** redrobot has joined #opendev12:18
donnydwhenever you are around fungi can you check the MTU on for the OE mirror. Should be 1480, but I just wanted to check12:21
fungidonnyd: i think it may be broken. i can't seem to ssh into it at all. getting "/bin/bash: Input/output error" and then the connection closes. i wonder if it's got a problem with its disk12:27
donnydwell that is probably not a good thing12:32
*** ysandeep|brb is now known as ysandeep12:42
donnydfungi:  I know why12:49
donnydMy core switch flaps ports whenever I change the speed on just one12:49
donnydso its likely it just needs a rebootr12:50
zbrthere is something wrong with http://logstash.openstack.org/ -- it does fail to load12:51
zbrbasically 50312:51
zbrsadly its gui is not even able to render a proper 503 page12:52
donnydfungi: I rebooted the mirror12:53
donnydI think we should probably try to move the OE mirror to use cinder as a backing store, now that I have the issues with it resolved12:54
fungizbr: yep, sorry, responded to your comment in #openstack-infra about it just now. i think we lost both elasticsearch03 and 04, and the cluster can only handle losing one node at a time12:56
fungiianw rebooted 03 when i noticed it wasn't responding to ssh around utc midnight12:57
fungii'm about to do the same to 04 now12:57
fungidonnyd: i can ssh into the oe mirror again, thanks!12:58
fungiens3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1480 qdisc fq_codel state UP group default qlen 100012:58
fungiso not 140012:58
fungiis it supposed to pick that up from dhcp or configdrive?12:58
fungi#status log elasticsearch03 was rebooted 2020-09-08 23:48z after it was found to be hung13:00
openstackstatusfungi: finished logging13:00
openstackgerritMerged opendev/elastic-recheck master: Avoid pip breakage due to keyring  https://review.opendev.org/75031813:01
openstackgerritMerged opendev/elastic-recheck master: Enable pylint  https://review.opendev.org/75031213:02
fungiyeah, confirmed it's the same situation with 04, "INFO task blah-blah:blah blocked for more than 120 seconds"13:04
fungihard rebooting it via nova api now13:04
donnyd1480 is what it's supposed to be13:05
fungi#status log elasticsearch04 rebooted after it was found to be hung13:05
openstackstatusfungi: finished logging13:05
donnydthank you for checking fungi13:05
fungidonnyd: oh, cool, yep now i see you said 1480 earlier not 1400. time for glasses i guess13:06
fungizbr: even after rebooting elasticsearch04 and confirming all 6 are reachable now, i suspect that it will need the index repaired or wiped since it lost two out of the six cluster nodes13:17
fungii'll try to dig out our documentation on how to do that here shortly13:17
zbrfungi: thanks. ping when ready so I can resume my e-r work (that is how i found it)13:18
fungii figured, and yep, will do13:18
* fungi mutters something about clouds13:18
*** AnonKapes has joined #opendev13:28
*** AnonKapes has quit IRC13:30
fungiokay, so turns out part of the problem was that elasticsearch did not start successfully on 03 and 04 when they were rebooted. i manually started them with systemctl (after stopping them just to make sure the status was cleaned up correctly) and they seem to be running again now13:42
fungithough the cluster status is currently "red" http://paste.openstack.org/show/797577/13:42
fungizbr: kibana is working again, and returning log events, though i don't know how much data might be missing13:43
*** hashar is now known as hasharAway13:51
tristanCdear openstack/opendev operator, would it be possible to take over the pynotedb package on pypi? it seems like the project was aborted13:52
fungitristanC: yeah, i forget exactly what zara was wanting to use that for, maybe SotK remembers13:54
fungiare you wanting to resume development of https://opendev.org/opendev/pynotedb or starting from scratch elsewhere and just reusing the package name?13:55
fungi(if the latter, then we should properly retire the git repository too)13:55
tristanCi happen to have created another implementation from scratch named pynotedb, without checking if the name was available13:57
fungitristanC: for the same purpose (python library to interface with a gerrit notedb)?13:58
fungitristanC: where are you hosting the source code for it? if we retire the repository on opendev we should probably make sure the readme links to where your source code lives so as to avoid confusion13:59
* fungi guesses it's in softwarefactory13:59
*** mlavalle has joined #opendev14:00
tristanCthe purpose of my implementation is mostly to manage the All-Users external-ids reference, it's: https://softwarefactory-project.io/r/#/q/project:software-factory/pynotedb14:00
fungiyep, just found it14:01
fungitristanC: ooh, that rings a bell. yes i think zara may have been developing hers for similar reasons, to make it possible to deploy gerrit from scratch and inject initial accounts/config into notedb14:01
fungianyway, what we're hosting in opendev is obviously defunct. looks like there was just a cookiecutter template pushed three years ago, so i don't see any problem relinquishing the name on pypi to someone who is actively making the thing we never got around to building14:03
fungii'm just one voice though. we can try to discuss it in the opendev meeting at 19:00 today, at least during open discussion. you don't need to be there, i'm happy to present the case on your behalf if that's an inconvenient time for you14:05
fungii expect everyone else will agree, but i don't want to be making that call without conferring with others14:06
tristanCfungi: thank you very much, otherwise no big deal, the library usage changes are not yet merged and it still easy to pick another name14:06
fungiclarkb: i don't recall seeing a meeting agenda go out yet. are we still meeting today?14:11
SotKI'm in favour of handing the name over14:11
SotKand retiring the defunct repo14:12
fungithanks SotK. if you still need a python library for interfacing with notedb then maybe work with tristanC to see if the design for theirs can be made to meets your needs14:12
fungier, s/meets/meet/14:13
tristanCSotK: fungi: thank you, AJaeger: it seems like https://review.opendev.org/#/c/597402/1 could now be restored ^14:14
SotKfungi: I think that is the approach that makes the most sense, yeah14:15
AJaegertristanC: I can restore - will you update if needed and propose the changes to remove it completely from opendev, please?14:17
tristanCAJaeger: sure, but i can't click restore14:19
clarkbfungi: yes, I need to semd one this morning. I didnt do it yesterday14:22
*** priteau has quit IRC14:23
fungino worries, just double checking. thanks!14:29
dtantsurhey folks! can I make zuul checkout difference branches of the same project, i.e. similar to how grenade works?14:31
*** ysandeep is now known as ysandeep|afk14:37
tristanCdtantsur: iiuc that is not supported, but that is likely a question for the #zuul channel14:37
clarkbdtantsur: tristanC well zuul ensures that every branch in a repo has its head set to the correct speculative state14:41
dtantsuryeah, but for a grenade-alike job I need two sets of repos: old and new14:41
clarkbdtantsur: that is all done in the job14:41
clarkbit clones the source repo set up by zuul then checks out the appropriate branches for old and new iirc14:42
clarkbzuul itself doesn't need to do anything to make that happen, it has already handed up the appropriate repo state14:42
AJaegertristanC: thanks!14:42
dtantsurthis way depends-on will only work for "new" branches, right?14:42
*** tkajinam has quit IRC14:43
dtantsurso I cannot Depends-On say stable/ussuri ironic change?14:43
clarkbdtantsur: no, zuul configues all branches with the speculative states14:43
dtantsurah! so `git checkout stable/ussuri` will give me the change?14:43
dtantsurand `master` will mean the change in question?14:43
clarkbdtantsur: it depends if the proposed change was to master, but ys14:43
dtantsuryeah, right, assuming ussuri->master14:44
dtantsurthanks clarkb!14:44
clarkb(if anyone is wondering this is why we combine all branches into the same zuul pipeline queue in the gate)14:48
*** priteau has joined #opendev14:49
*** Gyuseok_Jung has quit IRC14:49
*** diablo_rojo has joined #opendev14:49
*** Gyuseok_Jung has joined #opendev14:50
*** sgw has joined #opendev15:01
*** mlavalle has quit IRC15:04
*** mlavalle has joined #opendev15:07
fungidtantsur: also the "origin" remote is set to have the parent change for each branch15:13
fungiif you end up needing that for anything15:13
clarkbusually that comes up in the context of git diffs or when you want to examine specifically the unmerged state15:13
*** hasharAway has quit IRC15:19
openstackgerritSorin Sbarnea (zbr) proposed opendev/elastic-recheck master: WIP: Use pytest for queries  https://review.opendev.org/75044515:36
*** admin0 has joined #opendev15:47
*** hasharAway has joined #opendev15:50
clarkbI've approved the dns change for nb03.opendev.org. https://review.opendev.org/#/c/750039/ is the next thing thatneeds to go in and that is the potentially dangerious one if nodepools fix for using server uuids instead of hostnames doesnt work as expected15:51
*** hasharAway is now known as hashar15:51
clarkb(I expect it will be fine but if there is a change to watch closely in this stack for nb03 it is that one)15:51
clarkb*reapproved the dns change. It was already approved but due to not sharing a gate pipeline with its depends on didn't auto enqueue when theparent merged15:52
clarkbhttps://review.opendev.org/#/c/744821/ would be a useful change ot land for the management of the sshfp records15:52
openstackgerritMerged opendev/zone-opendev.org master: Add nb03.opendev.org to DNS  https://review.opendev.org/75003715:58
clarkbI'm also double checking setuptools changelogs and I think we're ok but I'm rechecking https://review.opendev.org/#/c/749766/2 now to double check16:11
clarkbif that is still a +1 from zuul I think we should also land https://review.opendev.org/749777 so we don't forget16:11
fungiyeah, my confusion yesterday was that a couple of jobs which had been previously been broken by setuptools failures landed in vexxhost ca-ymq-1 when the mirror there had broken ipv6 routes16:18
clarkbfungi: do you have time to review https://review.opendev.org/#/c/750039/ to enroll nb03.opendev.org into ansible? And if so any thoughts on trying to land that before or after the meeting?16:22
fungii thought i already had reviewed it. checking16:24
*** ysandeep|afk is now known as ysandeep16:24
fungiaha, yep, i had missed that one. let's land it now16:24
fungiclarkb: out of curiosity, hadn't we blanked out the ipv6 address for the old builder there for a reason? was it just not reachable, or were there intermittent connectivity issues over v6 there?16:26
clarkbdns seems to resolve for me after the previous change merged16:26
clarkbI haven't accepted the ssh host keys because I think we're relying on sshfp for that now16:26
clarkbfungi: do you know if ^ is the case? If not I can manually accept the host key16:26
clarkbI don't recall where the make ssh and ansible honor sshfp discussion got to16:27
fungii thought we added the -owhatever to the ssh invocations, but yeah i don't entirely recall. checking16:28
fungihrm, codesearch isn't turning up uses of VerifyHostKeyDNS for us16:30
clarkbI guess I'll manually accept the ssh host key for now and can sync up with ianw later today on sshfp usage16:31
clarkbthats done16:31
fungithis was our last discussion about it in here: http://eavesdrop.openstack.org/irclogs/%23opendev/%23opendev.2020-07-14.log.html16:33
fungiand then a few days later we covered that as soon as you update to glibc 2.31 your local resolution will stop trusting verified assertions from your configured recursive resolver, and so dnssec will be broken for you unless you override it with the trust-ad option in your /etc/resolv.conf16:38
fungithat's a fun one16:38
*** lpetrut has joined #opendev16:46
*** dtantsur is now known as dtantsur|afk16:47
clarkbhttps://review.opendev.org/#/c/749766/ is still happy with newer setuptools17:06
clarkbcan probably land that one and then once nb03.opendev.org is sorted we can do the image side change17:06
clarkbless than a minute to the nb03.opendev.org change merging17:21
openstackgerritMerged opendev/system-config master: Add nb03.opendev.org  https://review.opendev.org/75003917:25
*** hashar has quit IRC17:27
clarkbthats enqueued a ton of things (I think due to the inventory file changing) we'll have to wait for it to get to nodepool17:29
*** ysandeep is now known as ysandeep|away17:30
*** lpetrut has quit IRC17:51
clarkbis there a way to tell ansible to summarize the failure in its logging? bsae failed and I'm having a hard time finding what actually broke18:05
clarkbok I think its because nb03.opendev.org was unreachable18:06
clarkbbut I can ssh as root to it from bridge18:06
clarkbok it is a hostkey issue18:07
clarkbI think because we've reused the ip address in that cloud? sshing to the name works but ansible actually sshes to the ip18:08
clarkbI removed the old ip specific ssh host key in known_hosts and added in the new one.18:11
clarkbI think that service-nodepool will be happy when it gets there18:11
clarkboh except that failure kicked out the run for the change? the hourly run for nodepool has just started but not sure if I updated known hosts quickly enough18:11
clarkbit seems to be doing stuff so maybe I got it early enough18:13
openstackgerritClark Boylan proposed opendev/system-config master: Add dev packages on arm64 for docker-compose installation  https://review.opendev.org/75047218:28
clarkbinfra-root ^ I think that is the next thing we need for nb03.opendev.org18:28
clarkbI guess we can test this if we set up an arm64 only job. I'll look into that after our meeting too but I need to get ready for that now18:29
*** hashar has joined #opendev18:43
*** priteau has quit IRC19:20
ianwhttps://tools.ietf.org/html/rfc6104 is another one of these ipv6 rfc's that spends a long time basically telling you why a bunch of things don't work19:43
ianwthe multi-homed host one is similar19:43
ianwit really seems that "manual configuration" is about the only answer19:44
ianwpossibly firewalling, but as the rfc points out, can be unreliable if upstream routers change mac addresses, etc.19:44
clarkbhttps://etherpad.opendev.org/p/vexxhost-mirror-netplan found the netplan investigating I did19:49
clarkbI think one question about ^ I had outstanding was we actually have multiple valid ipv6 gateways on that host right now and I wasn't sure how to express that with netplan19:49
clarkbI think we'll be ok using a single gateway as long as it doesn't have an outage19:49
clarkbmnaser: ^ can you confirm that? if it is an issue to use a single gateway we can probably dig into netplan further19:52
fungii think the idea is that ipv6 provides route redundancy so you can rely on that instead of vrrp/hsrp/carp for your gateways and not need to worry about careful vip migrations during router maintenances19:53
clarkboh that was the other failsafe, we should still get working ipv4 with the netplan stuff I put up there as we aren't touching that19:59
clarkbso worst case ipv6 will not work and we can undo ? maybe we should disable vexxhost nowish in nodepool then we can iterate on that?19:59
openstackgerritClark Boylan proposed opendev/system-config master: Add dev packages on arm64 for docker-compose installation  https://review.opendev.org/75047220:01
ianwclarkb: https://review.opendev.org/#/c/749604/ -- are you happy that localhost == bridge in all contexts so we can merge that?20:02
clarkbianw: yes I think so20:02
clarkbI couldn't come up with a counter example20:02
clarkbinfra-root I had bad quoting  in https://review.opendev.org/750472 which should be fixed now20:02
fungisomething i would have brought up during the meeting if we'd had more time, for generating historical mailing list usage statistics it would help to publish a list of the mailman sites we host and the public archives we have in pipermail for them (whether or not the lists themselves still exist or were retired). i have a 7-line shell script which generates this: but am wondering what the best mechanism would20:04
fungibe to orchestrate it (ansible via our deploy jobs? cron entry in the existing puppet-mailman module?) http://lists.opendev.org/archives.yaml20:04
openstackgerritClark Boylan proposed openstack/project-config master: Disable vexxhost for mirror work  https://review.opendev.org/75048420:04
ianwclarkb: ^ we have the same thing in the install ansible roles too.  perhaps we should just move it into base for arm64 nodes?20:04
clarkbmnaser: ^ fyi that will impact your vexxhots specific flavors20:04
clarkbianw: looking20:04
mnaserclarkb: sorry, been a little all over the place20:05
mnaserthe single gateway _should_ be ok20:05
clarkbianw: I think install-ansible only runs on bridge though20:05
clarkbianw: thats there for the test jobs that runs an arm64 bridge iirc and won't apply to the nodepool-builder?20:06
ianwclarkb: yeah, that's right, i'm saying maybe we should make it apply to all aarch64 servers :)20:06
clarkbianw: pull it out of install-ansible into base maybe?20:09
ianwclarkb: it's an option, anyway ... but given that we try to do everything in containers, having docker-compose installed probably means we've covered most things pip installs anyway20:10
clarkbalso looking at base the way its currently split up there isn't a clear place to put that without making a new role which seems heavyweight for this20:10
clarkbhttps://netplan.io/reference/ is the netplan docs. We may have to set up manual routes to use multiple gateways20:32
clarkbthere is a v4 example showing that and I expect converting to ipv6 would work20:33
clarkbI'll edit the etherpad with that20:33
*** hashar has quit IRC20:38
clarkbdoesn't seem like I can say it is the default route though?20:39
clarkbdoes that mean I should set the metric higher so that the other routes will win over the catch all :: dest there?20:39
clarkbmaybe we should do it with the gateway6 option to keep it simple20:39
*** andrewbonney has quit IRC20:39
openstackgerritMerged opendev/system-config master: install-ansible: move install_modules.sh to puppet-setup-ansible  https://review.opendev.org/74960420:42
openstackgerritMerged opendev/system-config master: launch: move old scripts out of top-level  https://review.opendev.org/74960520:42
openstackgerritMerged opendev/system-config master: Update README.rst  https://review.opendev.org/74961520:42
openstackgerritMerged opendev/system-config master: docs: Update some of sysadmin details  https://review.opendev.org/74963020:42
fungiclarkb: yeah, "default" is usually just an alias for the wildcard/full cidr route20:58
fungiso creating a route for :: should be handled the same way20:58
clarkbah ok in that case I think the etherpad is correct if people want to review it and if that looks good we can land https://review.opendev.org/750484 and try it20:58
fungithough as you noted, ipv6 also has route priorities, those are taken into account after shortest prefix20:58
clarkbI tried to match what is on the mirror already20:59
fungier, longest prefix20:59
fungiif you have a route for a::/8 and a route for a:b::/16 then the second is preferred for destinations where it applies, even if the first has a higher priority21:00
clarkbgot it21:00
fungiconsider it like other traditional routing protocols. most specific match/longest prefix is preferred, followed by your path length and local priority rules21:07
clarkbfungi: does that netplan config in the etherpad look good to you then?21:09
clarkbI mean as much as we grok netplan :)21:09
fungii'm very green to netplan as well (did we really need *yet another* way to configure networking?!?), but will take a look21:13
fungii guess interface hardware addresses shouldn't change just from something like live migration right?21:14
fungiwe'll probably have to adjust that any time we replace the instance though21:15
fungigranted, we'd need to fix the static v6 address when that happened too21:16
clarkbyes and yes21:16
fungiyeah, i think what's there looks reasonable21:17
clarkbneutron fixes the l2 address with the port and that follows the host on migration21:17
clarkbalso I was sort of thinking we'd manually apply that for now21:17
fungikeeping in mind that this is a workaround until the neutron bug or whatever it is can be identified and eliminated in that provider21:18
fungiso hopefully not something we have to keep in place for years21:18
clarkbI probably won't touch that until nb03 is happy (fix for that is in the gate)21:22
clarkbbut I'm happy to help drive the netplan change along too21:23
corvusfungi: you have a lot of notes on the opendev.org change, along with a +221:29
corvusfungi: how do you want to proceed?21:29
fungicorvus: i can push those notes as a followup change, feel free to go ahead and approve21:30
fungiadding to my to do list for tomorrow21:30
corvusk.  seems like iterate forward is good here21:30
openstackgerritMerged opendev/system-config master: Add dev packages on arm64 for docker-compose installation  https://review.opendev.org/75047222:09
openstackgerritMohammed Naser proposed opendev/system-config master: Add ceph octopus mirrors  https://review.opendev.org/75051922:35
mnaser^ hopefully i didnt miss something there22:35
*** auristor has quit IRC22:38
*** auristor has joined #opendev22:41
clarkb2020-09-08 22:53:29,838 INFO nodepool.builder.BuildWorker.0: Building image ubuntu-xenial-arm6422:54
clarkbthat is what nb03.opendev.org says :)22:54
ianwwooo!  thanks clarkb22:54
*** tosky has quit IRC22:55
clarkbthat took longer than I had hoped, but ya super excited that we've now got a fully multi arch container pipeline that we consume ourselves22:55
openstackgerritMerged opendev/system-config master: Explain "why opendev" on opendev.org index page  https://review.opendev.org/74826322:56
*** tkajinam has joined #opendev22:57
*** tkajinam has quit IRC22:57
*** tkajinam has joined #opendev22:58
*** mlavalle has quit IRC22:58
clarkbnb03's image build is pulling in git repos now, so far looks like I would expect it23:00
clarkbhrm apache2 failed to restart on nb03.opendev.org (which is how we load in the LE certs?)23:10
clarkblooking into that now. The nodepool-builder side of things looks finethough23:10
clarkbSep 08 22:53:29 nb03 apachectl[32731]: SSLCertificateFile: file '/etc/letsencrypt-certs/nb03.opendev.org/nb03.opendev.org.cer' does not exist or is empty23:11
clarkbianw: I think the issue is that the LE playbook hasn't run against nb03.opendev.org23:12
clarkbpossibly due to the earlier ssh key issue?23:12
clarkbwe run that job in periodic. I'm inclined to just let it run tonight (my time) and have it correct itself23:14
clarkbianw: ^ any concerns with that?23:14
clarkbmostly asking you as I think you may be around when that runs? if not thats fine too and I'll check it tomorrow morning23:17
clarkberror: invalid command 'bdist_wheel' <- we got that from the image build trying to intsall os-testr in a venv in the image23:34
clarkbit used setuptools 49.6.0 and pip 8.someting (xenials pip)23:35
clarkbis that familiar to anyone?23:35
clarkbdo we need to install the python-wheel package in the image first so that its present when making the venv?23:36
clarkbnote this is all within the new VM image chroot context I don't think the nodepool-builder docker image makes a difference here23:36
clarkbI think it must not show up on x86 because there are x86 wheelsfor those packages already?23:36
clarkbfuture, voluptious, pyyaml, pyperclip, and prettytable23:37
clarkbhrm no thoes pacakges only have sdists (or at least the first two are sdist only)23:37
clarkbya the same error occurs on nb03.openstack.org so not a docker regression23:39
clarkbianw: I think we fix this by adding python3-wheel to ensure-venv's package list in dib?23:41
ianwclarkb: hrm, you can probably just manually run the LE playbook to get the certs23:43
ianwor i can23:43
ianwi'm pretty sure python3-wheel is in ensure-venv23:44
clarkbok that wheel error happens on x86 too I think its just noise23:44
ianwthis is *in the chroot* though?23:44
clarkb2020-09-08 23:33:18.951 | DEBUG diskimage_builder.block_device.utils [-] exec_sudo: sudo: sgdisk: command not found exec_sudo /usr/local/lib/python3.7/site-packages/diskimage_builder/block_device/utils.py:135 <- that is the actual reason the build fails23:44
clarkbthe os-testr installation failures don't seem to cause the builds to fail normally (we should still fix that and I think python3-wheel in the ensure-venv element is how we can do that)23:45
clarkbsgdisk will be a nodepool-builder imgae issue though I expect23:45
clarkbwe only hit the sgdisk path on the uefi and gpt partitioning branch23:46
clarkbwhich we only do for arm64 is my current theory23:46
fungivenvs don't get wheel added to them by default, you have to explicitly install wheel into them if you want to do bdist_wheel23:46
fungiand pip will fall back to directly installing files instead of its default of making wheels and installing those23:47
clarkbfungi: this is with venv not virtualenv on ubuntu (so has debians split up packaging)23:47
clarkbwell pip isn't falling back there23:47
fungimm, may also depend on the package being installed23:47
clarkbanyway the sgdisk issue is the real cause of the failure I think so trying to figure out how to intsall that on the container image now23:48
clarkbits the gdisk package looks like23:48
clarkbfungi: if I create a virtualenv using virtualenv 20.something locally it has wheel in it23:53
ianwclarkb: ahhh, yes we will need that when making gpt partitions23:53
ianwi actually hit that trying the x86 efi testing and didn't fix it :/23:53
clarkbif I create it with python3 -m venv I don't get a wheel23:53
fungiyep, that's what i was remembering23:54
clarkbianw: no worries, change is prposed to nodepool now23:54
ianwsorry, i'm getting periodic internet dropouts here :/23:55
clarkbI'm now installing python3-wheel to see if venv behavior changes but this is on tumbleweed not ubuntu23:56
clarkbstill no wheel23:57
ianwclarkb: did you just hand-install sgdisk to see if that works too?23:57
ianw(in the container)23:57
clarkbianw: I did not. Should I?23:57
ianwit might be worth it to see if any other issue lurks quickly23:58
ianwif you have a window open23:58
clarkbok I'll do that now, you may want to check logs later today to see if there are new issues23:58
ianwok, i hope not but you never know :)23:59

Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!