| clarkb | and I've got three different cables in my yard I have to be careful not to dig up | 00:00 |
|---|---|---|
| clarkb | where my parents live they have a single fiber cable with I think 10Gbe allocated to the entire country. So they get dsl at 1.2Mbps | 00:03 |
| clarkb | and the provider is the government | 00:04 |
| ianw | yeah, basically impossible to get more than 100/40 in .au ; that does beat 1.2 though :) | 00:06 |
| fungi | most of the broadband in the usa is monopolies who have built (or more often purchased out of bankruptcy) the lines and then resell (because they're forced to by law) to other companies who give them a cut. it's sort of like mafia internet | 00:07 |
| clarkb | fungi: I think the reselling only applies to phone infrastructure | 00:08 |
| clarkb | fungi: the fiber and cable stuff doesn't fall under the same rules iirc | 00:08 |
| fungi | yup, true, it's even worse | 00:09 |
| fungi | i pay the cable company here for broadband internet service (100mbps up, 25mbps down currently, i think?), no television or voip or anything else. us$100/mo | 00:10 |
| fungi | but there are rural areas which are faaar worse | 00:10 |
| paladox | :O | 00:11 |
| clarkb | fungi: in https://review.opendev.org/#/c/702571/2 the commit message says we are migrating legacy jobs to zuulv3 but it seems we are just copying the existing legacy jobs into the repo? | 00:11 |
| fungi | my parents in the mountains struggle to get unreliable ~5mbps adsl at learly the same price | 00:11 |
| paladox | ADSL is becomming less of a thing now here | 00:12 |
| *** lathiat has joined #openstack-infra | 00:12 | |
| fungi | clarkb: yup, smarcet said in a comment he was going to fix the commit message | 00:12 |
| paladox | expecially as BT have removed it from there website and will only show it in areas that have not got fibre. | 00:12 |
| clarkb | fungi: ok I wont approve it then | 00:12 |
| fungi | i removed my +2 for now | 00:13 |
| clarkb | the other two changes related to that look good. So should be able to roll that all into their repos as soon as the commit message reflects reality | 00:14 |
| paladox | fungi BT's ultra fast recently moved into my area :P | 00:14 |
| paladox | and not a bad price either! | 00:14 |
| fungi | congrats! | 00:14 |
| paladox | so i can get upto 150Mbps | 00:14 |
| paladox | (haven't upgraded to them yet) | 00:15 |
| clarkb | speedtest just showed 325/345 for me. Thats better than it has been since updating | 00:15 |
| fungi | clarkb: yup, i was planning to approve them as well once that one merges | 00:15 |
| clarkb | really you stop noticing the speed improvements once it gets faster than your wifi network | 00:15 |
| clarkb | I'm too lazy to run cable in my walls | 00:15 |
| fungi | right, ours is theoretically "up to" 100mbps downloads but i've never seen it get close even directly plugging into the transceiver | 00:16 |
| paladox | heh | 00:16 |
| paladox | fungi they will also give you £20 if speeds drop below the gurenteed speeds | 00:16 |
| *** lathiat has quit IRC | 00:16 | |
| paladox | of i think 100mbps | 00:16 |
| *** ahosam has quit IRC | 00:17 | |
| fungi | oh, nobody guarantees anything here | 00:17 |
| paladox | heh | 00:17 |
| *** jamespage has quit IRC | 00:18 | |
| *** thedac has quit IRC | 00:18 | |
| *** TheJulia has quit IRC | 00:18 | |
| *** kevinz has quit IRC | 00:18 | |
| *** cyberpear has quit IRC | 00:18 | |
| *** donnyd has quit IRC | 00:18 | |
| *** tdasilva has quit IRC | 00:18 | |
| *** rajinir has quit IRC | 00:18 | |
| *** dave-mccowan has quit IRC | 00:19 | |
| *** rajinir has joined #openstack-infra | 00:19 | |
| *** tdasilva has joined #openstack-infra | 00:19 | |
| *** thedac has joined #openstack-infra | 00:19 | |
| *** donnyd has joined #openstack-infra | 00:19 | |
| *** kevinz has joined #openstack-infra | 00:20 | |
| *** jamespage has joined #openstack-infra | 00:20 | |
| *** lathiat has joined #openstack-infra | 00:20 | |
| openstackgerrit | Tristan Cacqueray proposed zuul/zuul-operator master: Add Openshift SCC and functional test https://review.opendev.org/702758 | 00:20 |
| *** cyberpear has joined #openstack-infra | 00:20 | |
| paladox | fungi bt also have a backup service that uses mobile network, so if the internet cannot be fixed quickly, they'll send out a modem | 00:20 |
| fungi | i mean, there are guarantees in negotiated business-to-business contracts with an sla or whatever, but finding an isp which is willing to make service level guarantees for residential service is rare | 00:21 |
| *** lathiat has quit IRC | 00:21 | |
| *** lathiat has joined #openstack-infra | 00:21 | |
| *** TheJulia has joined #openstack-infra | 00:22 | |
| *** csatari has joined #openstack-infra | 00:24 | |
| paladox | yeh | 00:26 |
| *** crodriguez has joined #openstack-infra | 00:27 | |
| paladox | https://www.ispreview.co.uk/index.php/2019/10/isp-bt-sharply-cuts-uk-ultrafast-broadband-and-phone-prices.html | 00:27 |
| paladox | heh | 00:27 |
| paladox | didn't know that! | 00:27 |
| fungi | i don't think service providers here ever cut prices, because they've pretty much all got monopolies in their respective service areas | 00:29 |
| clarkb | fungi: I upgraded my service because they offered the faster service at the same price as my old service | 00:29 |
| paladox | fungi each year i recontract the cost of my sim has gone down substatially! (e.g 35 -> 27 -> £10/6m then £20 for the rest (24 months contract) | 00:29 |
| clarkb | so they don't really discount things as much as offer more to be competitive :/ | 00:29 |
| paladox | i got it for £10 for 6 months then the rest for £20 only in december last month | 00:30 |
| fungi | true, mine has upped the speeds they provide, but also seem to like to increase rates steadily every year | 00:30 |
| fungi | s/rates/prices/ | 00:30 |
| *** zxiiro has joined #openstack-infra | 00:30 | |
| fungi | yeah, cell phone prices are one of the few places here where there's sufficient competition to drive prices lower | 00:32 |
| fungi | er, cell service prices that is | 00:32 |
| *** rpittau|afk has joined #openstack-infra | 00:34 | |
| SpamapS | Spectrum never lowers the price, but they have bumped the bandwidth several times. | 00:35 |
| *** bauzas has quit IRC | 00:36 | |
| paladox | BT is forced to make competition due to ofcom. | 00:37 |
| *** bauzas has joined #openstack-infra | 00:38 | |
| *** rpittau|afk has quit IRC | 00:42 | |
| *** tdasilva has quit IRC | 00:42 | |
| *** kevinz has quit IRC | 00:42 | |
| *** knikolla has quit IRC | 00:42 | |
| *** csatari has quit IRC | 00:42 | |
| *** clayg has quit IRC | 00:42 | |
| *** rm_work has quit IRC | 00:43 | |
| *** donnyd has quit IRC | 00:43 | |
| *** ildikov has quit IRC | 00:43 | |
| *** gmann has quit IRC | 00:43 | |
| *** zxiiro has quit IRC | 00:43 | |
| *** TheJulia has quit IRC | 00:44 | |
| *** davidlenwell has quit IRC | 00:44 | |
| *** vdrok has quit IRC | 00:44 | |
| *** Ng has quit IRC | 00:44 | |
| *** seongsoocho has quit IRC | 00:44 | |
| *** rajinir has quit IRC | 00:44 | |
| *** tdasilva has joined #openstack-infra | 00:44 | |
| *** arne_wiebalck has quit IRC | 00:45 | |
| *** setuid has quit IRC | 00:45 | |
| *** nickv1985 has quit IRC | 00:45 | |
| *** sparkycollier has quit IRC | 00:45 | |
| *** cjohnston has quit IRC | 00:45 | |
| *** jdelaros1 has quit IRC | 00:45 | |
| *** davecore has quit IRC | 00:45 | |
| *** crodriguez has quit IRC | 00:45 | |
| *** mwhahaha has quit IRC | 00:45 | |
| *** beisner has quit IRC | 00:45 | |
| *** ofosos has quit IRC | 00:46 | |
| *** mnasiadka has quit IRC | 00:46 | |
| *** abelur has quit IRC | 00:46 | |
| *** lsell has quit IRC | 00:46 | |
| *** jungleboyj has quit IRC | 00:46 | |
| *** dustinc has quit IRC | 00:46 | |
| *** portdirect has quit IRC | 00:46 | |
| *** lathiat has quit IRC | 00:46 | |
| *** jamespage has quit IRC | 00:46 | |
| *** coreycb has quit IRC | 00:46 | |
| ianw | ok, generating wheel indexes has turned out to be a lot harder than i first thought | 00:46 |
| *** Shrews has quit IRC | 00:47 | |
| *** guilhermesp has quit IRC | 00:47 | |
| *** dougwig has quit IRC | 00:47 | |
| *** tonyb[m] has quit IRC | 00:47 | |
| *** rpioso has quit IRC | 00:47 | |
| *** hrybacki has quit IRC | 00:47 | |
| *** jrosser has quit IRC | 00:48 | |
| *** philroche has quit IRC | 00:48 | |
| fungi | not that i want to discourage you if you're having fun, but i would be fine if we just focused on scaling down the wheel cache and didn't worry about generating metadata-rich indices for it | 00:48 |
| *** masayukig has quit IRC | 00:49 | |
| *** mnaser has quit IRC | 00:49 | |
| *** thedac has quit IRC | 00:49 | |
| ianw | i don't know if fun is the word, but i came this far ... | 00:50 |
| fungi | there's almost nothing besides packaging bootstrap packages where this issue will crop up that it can't be mitigated by requirements versionspecs and environment markers | 00:50 |
| *** smarcet has joined #openstack-infra | 00:51 | |
| *** cyberpear has quit IRC | 00:51 | |
| *** petevg has quit IRC | 00:52 | |
| *** johnsom has quit IRC | 00:52 | |
| ianw | i do agree, but for projects that are not as requirements focused as openstack it would be nice to be maximally compatible | 00:52 |
| ianw | the hang up is the time it takes to generate the indexes on AFS when it's checksumming | 00:52 |
| *** rpioso has joined #openstack-infra | 00:53 | |
| fungi | keep in mind that the current wheel cache is also built *from* openstack's requirements repository | 00:53 |
| fungi | so it's already inherently openstacky | 00:53 |
| clarkb | ianw: can we make the index without checksums? | 00:53 |
| ianw | true, but then we put the mirror config on every host | 00:53 |
| *** sdoran has quit IRC | 00:54 | |
| clarkb | our current indexes don't have checksums iirc | 00:54 |
| *** petevg has joined #openstack-infra | 00:54 | |
| *** srwilkers has quit IRC | 00:54 | |
| ianw | we build the wheels every time, but then we basically do "if ! -f /afs/.../<thefile.whl>; then copy it" ... i.e. don't recopy | 00:54 |
| *** gsantomaggio has quit IRC | 00:55 | |
| ianw | which makes sense. but makes it hard to know what indexes to generate | 00:55 |
| fungi | if we delete all the unnecessary wheels (those which were simply fetched from pypi) does traversing the remainder in afs become more tractable? | 00:56 |
| ianw | possibly; i mean one solution is to just up the job timeout | 00:56 |
| fungi | fair, it doesn't need to run *that* often | 00:56 |
| ianw | otherwise it's more like "did we copy a new wheel into this directory, if so, regenerate index" | 00:57 |
| ianw | which is possible, just more logic | 00:57 |
| fungi | one maybe terrible idea: we don't need to look at what's in the directory, all we need is the previous index | 00:57 |
| fungi | (and then parse it) | 00:58 |
| ianw | yes, true. it has a bootstrap problem though that the first run does all indexes, which is the same problem as making all the indexes every time | 00:58 |
| fungi | so it does become fetching a file from each directory where we're going to put a new wheel, but it only needs to be the one for that directory, right? | 00:59 |
| fungi | do we actually need to bootstrap it across the whole mirror? | 00:59 |
| fungi | can't we just add an index if we're adding a new wheel and there's no index present yet? | 01:00 |
| fungi | basically assume all the wheels we've already cached don't have this problem (yet) | 01:00 |
| ianw | yes, but my senses are tingling that, say we had to blow a wheel partition away and recreate it, in that stressful situation it would go back to making everything | 01:00 |
| ianw | so now someone would possibly be debugging two issues | 01:01 |
| fungi | that's a good point. it turns the indices into something we need to back up | 01:01 |
| *** MarkMaglana has quit IRC | 01:01 | |
| *** npochet has quit IRC | 01:01 | |
| fungi | they become precious | 01:01 |
| fungi | or at least painful to replace | 01:01 |
| ianw | it gets up to about "p" with the current timeout http://zuul.openstack.org/build/e3db5d9b8a0e414a94bf5b83fd129338/log/job-output.txt#51962 | 01:03 |
| *** rpioso has quit IRC | 01:03 | |
| *** petevg has quit IRC | 01:06 | |
| clarkb | which is 2/3rds? | 01:08 |
| *** zhurong has joined #openstack-infra | 01:08 | |
| ianw | i think this is doomed; i doubt whl's have any sort of build impotence. while the filename might be the same, the checksum will be different build to build | 01:09 |
| ianw | so i can't create an index from the build host, and then selectively copy over only those whl's that changed ... all the checksums will be wrong | 01:10 |
| fungi | yeah, at least for a lot of wheels this will be the case anyway | 01:11 |
| ianw | i think though, if our wheel jobs started fresh today, they probably would timeout anyway. they might have worked on a fresh partition when they only had one branch, etc, but i don't think they would be able to copy out everything today | 01:13 |
| *** jamesmcarthur has joined #openstack-infra | 01:14 | |
| ianw | bionic, that has py2&3 builds, only got to "b" http://zuul.openstack.org/build/f7d290c5d4454831b4b84a0e4bc24c59/log/job-output.txt#26138 | 01:15 |
| *** jamesmcarthur_ has joined #openstack-infra | 01:16 | |
| *** csatari has joined #openstack-infra | 01:16 | |
| fungi | also worth thinking about, we don't prune the contents there, so we have years of accumulated cruft which jobs will no longer reference | 01:16 |
| *** jamesmcarthur has quit IRC | 01:17 | |
| ianw | very true, it's constantly appending | 01:18 |
| *** zhurong has quit IRC | 01:18 | |
| *** mnasiadka has joined #openstack-infra | 01:18 | |
| ianw | there might be an argument to recreate it fully every time. that way we know we don't have this bootstrap problem should we ever drop a wheel afs partition | 01:19 |
| ianw | it would cost the publish step of the job being longer, but we only do that once a day in periodic, so ... | 01:19 |
| fungi | though if we do that, slimming down the scope of what we copy to just wheels which aren't already on pypi becomes more important | 01:20 |
| *** smarcet has quit IRC | 01:20 | |
| *** mattw4 has joined #openstack-infra | 01:20 | |
| ianw | that could be phase 2; and as a bonus it would flush itself | 01:21 |
| openstackgerrit | Merged openstack/diskimage-builder master: Fix Yum repositories and GPG keys for CentOS 8.1 https://review.opendev.org/702721 | 01:21 |
| fungi | yeah, just thinking in terms of the afs write (and vos release) bottleneck | 01:21 |
| fungi | the less we commit to afs, the faster that job will finish | 01:22 |
| *** csatari has quit IRC | 01:22 | |
| ianw | fungi / clarkb: i think this has got sufficient out of hand that a small spec is in order | 01:22 |
| fungi | i have to concur | 01:22 |
| ianw | i'll revert the extant changes and write something up | 01:22 |
| ianw | it would be good to have a plan how to do the "if it's on pypi skip it" bits too | 01:23 |
| fungi | sure, there's always the "check pypi" option, which since this job has to download sdists from there anyway, probably isn't a big additional stability risk | 01:23 |
| fungi | though if we're logging what gets retrieved from pypi, we could optionally just parse that log | 01:24 |
| fungi | and save some network calls | 01:24 |
| *** csatari has joined #openstack-infra | 01:24 | |
| fungi | basically "if we fetched it from pypi, don't save it to afs" | 01:24 |
| fungi | right now we build the cache on an untrusted node, pull it onto the executor and then selectively write from it into afs | 01:26 |
| fungi | so if we delete those files on the untrusted node we also save some network i/o between the node and the executor | 01:27 |
| *** zhurong has joined #openstack-infra | 01:28 | |
| *** mnasiadka has quit IRC | 01:29 | |
| *** gyee has quit IRC | 01:30 | |
| openstackgerrit | Ian Wienand proposed openstack/project-config master: copy-wheels: disable PEP503 indexing https://review.opendev.org/702780 | 01:30 |
| clarkb | ianw: k | 01:31 |
| *** csatari has quit IRC | 01:31 | |
| ianw | i can do a dib release now | 01:32 |
| fungi | what kind of checksum is pypi using? | 01:38 |
| fungi | sha256? | 01:38 |
| fungi | a.k.a. sha2-256 | 01:38 |
| fungi | the one used in my local wheel cache seems to be 56 hex digits in length | 01:40 |
| *** jamesmcarthur_ has quit IRC | 01:40 | |
| fungi | if you concatenate the parent directory digits anyway | 01:41 |
| *** mattw4 has quit IRC | 01:41 | |
| fungi | so shorter than sha2-256, but longer than md5 or sha1 | 01:41 |
| *** igordc has quit IRC | 01:42 | |
| ianw | might be blake2? | 01:42 |
| *** jamesmcarthur has joined #openstack-infra | 01:42 | |
| ianw | sha256 was the recommendation in PEP503 for the <a> tags | 01:43 |
| fungi | ahh | 01:44 |
| clarkb | ya that us what pypi upplies. I thinj pip can do md5 too | 01:47 |
| *** sparkycollier has joined #openstack-infra | 01:47 | |
| *** jamesmcarthur has quit IRC | 01:47 | |
| fungi | aha, this may be sha2-224 (224 bits is 56 hex digits) | 01:49 |
| *** sparkycollier has quit IRC | 01:52 | |
| fungi | ianw: also, i've confirmed, rerunning the command to build a platform-specific wheel (i did pyyaml for my test) produces files with varying checksums | 01:58 |
| fungi | wheel building is not (even trivially) reproducible | 01:58 |
| fungi | ran the same command seconds apart, got totally different checksums on the results | 01:59 |
| fungi | probably embeds a build timestamp somewhere | 02:01 |
| *** smarcet has joined #openstack-infra | 02:06 | |
| *** jamesmcarthur has joined #openstack-infra | 02:12 | |
| fungi | because of that i'm slightly skeptical of the "build indices on the node" approach, as it'll mean a lot more load on afs vs just adding and deleting files incrementally (outside of bootstrapping events where we need to write the full fileset regardless) | 02:12 |
| *** jamesmcarthur_ has joined #openstack-infra | 02:16 | |
| *** jamesmcarthur has quit IRC | 02:16 | |
| *** setuid has joined #openstack-infra | 02:24 | |
| *** jamesmcarthur_ has quit IRC | 02:26 | |
| *** setuid has quit IRC | 02:29 | |
| *** jamesmcarthur has joined #openstack-infra | 02:34 | |
| *** jamesdenton has quit IRC | 02:40 | |
| *** jamesdenton has joined #openstack-infra | 02:43 | |
| *** ricolin has quit IRC | 02:49 | |
| *** dougwig has joined #openstack-infra | 02:50 | |
| *** dougwig has quit IRC | 02:55 | |
| ianw | yeah, that was what i meant by the builds not being idempotent | 02:56 |
| ianw | pypi might want a resuable build effort | 02:56 |
| fungi | reproducible but yeah | 02:58 |
| fungi | i'm tempted to post on the distutils-sig ml asking for suggestions on reproducible wheel builds, but that's probably a tomorrow thing | 02:58 |
| fungi | the compression is zip which is invariant | 02:59 |
| fungi | i tested that much | 02:59 |
| fungi | i probably just need to unzip two wheels and diff the tree | 02:59 |
| fungi | at least that'll tell us what we're dealing with | 02:59 |
| fungi | obviously wheels will change if they're linked against/embed c libs which get updated on the build system but tracking that's a feature in my opinion | 03:00 |
| openstackgerrit | Ian Wienand proposed opendev/system-config master: Add roles for a basic static server https://review.opendev.org/697587 | 03:10 |
| *** dougwig has joined #openstack-infra | 03:10 | |
| ianw | yes, that was something i thought about just comparing the filename for the copy. if something embeds a static library, you probably want a new copy of even the old version wheels | 03:13 |
| ianw | iirc things commonly embedded such as gzip libraries have had security issues in the past | 03:13 |
| *** sparkycollier has joined #openstack-infra | 03:17 | |
| fungi | you rebased as part of your changes in the latest revision of 697587 i guess? | 03:18 |
| *** knikolla has joined #openstack-infra | 03:18 | |
| fungi | makes it hard to diff against the previous revision. can you provide a short summary of what you adjusted? | 03:19 |
| fungi | was it just the "done"s? | 03:19 |
| *** dougwig has quit IRC | 03:20 | |
| *** dougwig has joined #openstack-infra | 03:21 | |
| *** sparkycollier has quit IRC | 03:22 | |
| ianw | fungi: yes sorry it needed a merge; i made the restart handler a reload, and change the name of the afs_mirror variable to afs_root (checking afs was mounted i copied from the mirror role) | 03:23 |
| *** knikolla has quit IRC | 03:24 | |
| fungi | ahh, cool. thanks! | 03:24 |
| fungi | sometimes it helps to make adjustments and rebase as separate revisions, but it's a judgment call | 03:24 |
| fungi | i know it's extra work, and also not easy to remember to think about | 03:24 |
| ianw | ++ good point | 03:27 |
| *** dougwig has quit IRC | 03:29 | |
| ianw | i can ping other things, but not review.openstack.org | 03:30 |
| ianw | false alarm, must be that great .au internet we were talking about | 03:34 |
| *** psachin has joined #openstack-infra | 03:36 | |
| fungi | turn the icmp packets upside-down? ;) | 03:37 |
| *** knikolla has joined #openstack-infra | 03:39 | |
| *** jamesmcarthur has quit IRC | 03:41 | |
| *** jamesmcarthur has joined #openstack-infra | 03:42 | |
| *** dougwig has joined #openstack-infra | 03:44 | |
| *** knikolla has quit IRC | 03:44 | |
| *** petevg has joined #openstack-infra | 03:45 | |
| *** jamesmcarthur has quit IRC | 03:48 | |
| *** dougwig has quit IRC | 03:49 | |
| *** jamesmcarthur has joined #openstack-infra | 03:56 | |
| *** jamesmcarthur_ has joined #openstack-infra | 03:59 | |
| *** jamesmcarthur has quit IRC | 03:59 | |
| *** petevg has quit IRC | 04:01 | |
| *** knikolla has joined #openstack-infra | 04:03 | |
| openstackgerrit | Merged openstack/project-config master: Retire x/zmq-event-publisher https://review.opendev.org/702749 | 04:04 |
| *** udesale has joined #openstack-infra | 04:05 | |
| openstackgerrit | Merged openstack/project-config master: Update compute-hyperv test jobs https://review.opendev.org/702673 | 04:05 |
| *** rlandy has quit IRC | 04:07 | |
| *** knikolla has quit IRC | 04:09 | |
| *** openstackgerrit has quit IRC | 04:12 | |
| *** jungleboyj has joined #openstack-infra | 04:14 | |
| *** jungleboyj has quit IRC | 04:20 | |
| *** jdelaros1 has joined #openstack-infra | 04:21 | |
| *** masayukig has joined #openstack-infra | 04:21 | |
| *** jungleboyj has joined #openstack-infra | 04:24 | |
| *** masayukig has quit IRC | 04:26 | |
| *** jdelaros1 has quit IRC | 04:26 | |
| *** jungleboyj has quit IRC | 04:29 | |
| *** ykarel|away has joined #openstack-infra | 04:35 | |
| *** jdelaros1 has joined #openstack-infra | 04:37 | |
| *** ykarel|away is now known as ykarel | 04:37 | |
| *** tkajinam has quit IRC | 04:39 | |
| *** tkajinam has joined #openstack-infra | 04:41 | |
| *** jdelaros1 has quit IRC | 04:41 | |
| *** openstackgerrit has joined #openstack-infra | 04:44 | |
| openstackgerrit | Merged openstack/project-config master: Removed OpenstackId legacy jobs https://review.opendev.org/702572 | 04:44 |
| *** jamesmcarthur_ has quit IRC | 04:55 | |
| *** jamesmcarthur has joined #openstack-infra | 04:57 | |
| *** jamesmcarthur has quit IRC | 05:02 | |
| *** tkajinam_ has joined #openstack-infra | 05:03 | |
| *** MarkMaglana has joined #openstack-infra | 05:05 | |
| *** npochet has joined #openstack-infra | 05:06 | |
| *** tkajinam has quit IRC | 05:07 | |
| *** npochet has quit IRC | 05:11 | |
| *** davidlenwell has joined #openstack-infra | 05:12 | |
| *** MarkMaglana has quit IRC | 05:14 | |
| *** smarcet has quit IRC | 05:18 | |
| *** davidlenwell has quit IRC | 05:20 | |
| *** prometheanfire has quit IRC | 05:26 | |
| *** jamesmcarthur has joined #openstack-infra | 05:26 | |
| *** surpatil has joined #openstack-infra | 05:27 | |
| *** prometheanfire has joined #openstack-infra | 05:28 | |
| *** ociuhandu has joined #openstack-infra | 05:31 | |
| *** jamesmcarthur has quit IRC | 05:32 | |
| *** rpioso has joined #openstack-infra | 05:32 | |
| *** evrardjp has quit IRC | 05:34 | |
| *** evrardjp has joined #openstack-infra | 05:34 | |
| *** ociuhandu has quit IRC | 05:35 | |
| *** davidlenwell has joined #openstack-infra | 05:42 | |
| *** davidlenwell has quit IRC | 05:52 | |
| openstackgerrit | Merged openstack/project-config master: copy-wheels: disable PEP503 indexing https://review.opendev.org/702780 | 05:54 |
| openstackgerrit | Merged openstack/project-config master: Remove infra trusty jobs https://review.opendev.org/702764 | 05:56 |
| openstackgerrit | Merged openstack/openstack-zuul-jobs master: Removed OpenstackId Legacy Jobs https://review.opendev.org/702573 | 05:57 |
| *** jamesmcarthur has joined #openstack-infra | 06:00 | |
| *** davidlenwell has joined #openstack-infra | 06:05 | |
| *** jamesmcarthur has quit IRC | 06:07 | |
| openstackgerrit | Tobias Henkel proposed zuul/zuul master: Add --validate-tenants option to zuul scheduler https://review.opendev.org/542160 | 06:11 |
| *** setuid has joined #openstack-infra | 06:11 | |
| *** davidlenwell has quit IRC | 06:11 | |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/project-config master: Normalize projects.yaml https://review.opendev.org/702801 | 06:13 |
| *** setuid has quit IRC | 06:16 | |
| *** rpioso has quit IRC | 06:21 | |
| *** rajinir has joined #openstack-infra | 06:22 | |
| *** tdasilva has quit IRC | 06:23 | |
| *** jtomasek has joined #openstack-infra | 06:25 | |
| *** rajinir has quit IRC | 06:30 | |
| *** jaicaa_ has quit IRC | 06:41 | |
| *** jaicaa has joined #openstack-infra | 06:44 | |
| *** roman_g has joined #openstack-infra | 06:45 | |
| *** davidlenwell has joined #openstack-infra | 06:48 | |
| *** davecore has joined #openstack-infra | 06:48 | |
| *** cjohnston has joined #openstack-infra | 06:48 | |
| *** gsantomaggio has joined #openstack-infra | 06:49 | |
| *** gmann has joined #openstack-infra | 06:49 | |
| *** abelur has joined #openstack-infra | 06:49 | |
| *** lsell has joined #openstack-infra | 06:49 | |
| *** philroche has joined #openstack-infra | 06:49 | |
| *** mnaser has joined #openstack-infra | 06:49 | |
| *** crodriguez has joined #openstack-infra | 06:50 | |
| *** clayg has joined #openstack-infra | 06:50 | |
| *** mwhahaha has joined #openstack-infra | 06:50 | |
| *** sparkycollier has joined #openstack-infra | 06:51 | |
| *** donnyd has joined #openstack-infra | 06:52 | |
| *** beisner has joined #openstack-infra | 06:52 | |
| *** rpioso has joined #openstack-infra | 06:52 | |
| *** rajinir has joined #openstack-infra | 06:52 | |
| *** thedac has joined #openstack-infra | 06:53 | |
| *** setuid has joined #openstack-infra | 06:53 | |
| *** dougwig has joined #openstack-infra | 06:54 | |
| *** lmiccini has joined #openstack-infra | 06:55 | |
| *** srwilkers has joined #openstack-infra | 06:57 | |
| *** seongsoocho has joined #openstack-infra | 06:58 | |
| *** Ng has joined #openstack-infra | 06:59 | |
| *** kevinz has joined #openstack-infra | 06:59 | |
| *** lathiat has joined #openstack-infra | 06:59 | |
| *** portdirect has joined #openstack-infra | 07:00 | |
| *** vdrok has joined #openstack-infra | 07:00 | |
| *** csatari has joined #openstack-infra | 07:00 | |
| *** pcaruana has joined #openstack-infra | 07:00 | |
| *** jamesmcarthur has joined #openstack-infra | 07:02 | |
| *** johnsom has joined #openstack-infra | 07:02 | |
| *** coreycb has joined #openstack-infra | 07:03 | |
| *** jrosser has joined #openstack-infra | 07:03 | |
| *** TheJulia has joined #openstack-infra | 07:03 | |
| *** dustinc has joined #openstack-infra | 07:03 | |
| *** nickv1985 has joined #openstack-infra | 07:03 | |
| *** masayukig has joined #openstack-infra | 07:04 | |
| AJaeger | ianw: thanks for taking care of trusty! | 07:05 |
| *** MarkMaglana has joined #openstack-infra | 07:06 | |
| *** Shrews has joined #openstack-infra | 07:06 | |
| *** jdelaros1 has joined #openstack-infra | 07:07 | |
| *** jamesmcarthur has quit IRC | 07:09 | |
| *** jamespage has joined #openstack-infra | 07:10 | |
| *** knikolla has joined #openstack-infra | 07:10 | |
| *** jungleboyj has joined #openstack-infra | 07:11 | |
| *** npochet has joined #openstack-infra | 07:13 | |
| *** petevg has joined #openstack-infra | 07:13 | |
| *** sdoran has joined #openstack-infra | 07:14 | |
| *** mnasiadka has joined #openstack-infra | 07:14 | |
| openstackgerrit | Merged openstack/project-config master: Normalize projects.yaml https://review.opendev.org/702801 | 07:15 |
| *** pgaxatte has joined #openstack-infra | 07:18 | |
| *** rpittau|afk has joined #openstack-infra | 07:19 | |
| *** arne_wiebalck has joined #openstack-infra | 07:22 | |
| openstackgerrit | Merged openstack/openstack-zuul-jobs master: Remove legacy-puppet-openstack-infra-spec-helper-unit-ubuntu-trusty https://review.opendev.org/702765 | 07:23 |
| openstackgerrit | Merged openstack/openstack-zuul-jobs master: Move legacy-gearman-plugin-mavin-build away from Trusty https://review.opendev.org/702768 | 07:23 |
| *** ahosam has joined #openstack-infra | 07:24 | |
| *** rcernin has quit IRC | 07:25 | |
| *** ildikov has joined #openstack-infra | 07:25 | |
| *** soumplis has joined #openstack-infra | 07:27 | |
| *** evgenyl has joined #openstack-infra | 07:28 | |
| *** tonyb[m] has joined #openstack-infra | 07:31 | |
| openstackgerrit | Merged openstack/project-config master: Remove legacy-gearman-plugin-mavin-build-ubuntu-trusty https://review.opendev.org/702769 | 07:41 |
| *** tdasilva has joined #openstack-infra | 07:47 | |
| *** guilhermesp has joined #openstack-infra | 07:48 | |
| *** rm_work has joined #openstack-infra | 07:49 | |
| *** rm_work has quit IRC | 07:49 | |
| *** rm_work has joined #openstack-infra | 07:49 | |
| *** ykarel is now known as ykarel|lunch | 07:56 | |
| *** roman_g has quit IRC | 07:58 | |
| *** jpena|off is now known as jpena | 08:01 | |
| AJaeger | ianw: dib is still using trusty, see https://opendev.org/openstack/diskimage-builder/src/branch/master/.zuul.d/jobs.yaml | 08:07 |
| *** slaweq has joined #openstack-infra | 08:08 | |
| AJaeger | ianw: and system-config etc. | 08:10 |
| *** dchen has quit IRC | 08:10 | |
| *** tesseract has joined #openstack-infra | 08:10 | |
| openstackgerrit | Tobias Henkel proposed opendev/gear master: Ignore keepalive on unsupported platforms https://review.opendev.org/702816 | 08:12 |
| openstackgerrit | Andreas Jaeger proposed opendev/glean master: Remove trusty job https://review.opendev.org/702817 | 08:13 |
| AJaeger | ianw: one place ^ | 08:13 |
| *** hrybacki has joined #openstack-infra | 08:14 | |
| openstackgerrit | Andreas Jaeger proposed opendev/base-jobs master: Remove ubuntu-trusty nodeset https://review.opendev.org/702818 | 08:17 |
| AJaeger | clarkb, ianw that one ^ will show us what still needs to be done as well | 08:18 |
| *** surpatil has quit IRC | 08:22 | |
| *** udesale_ has joined #openstack-infra | 08:25 | |
| openstackgerrit | Matthieu Huin proposed zuul/zuul master: OIDCAuthenticator: add capabilities, scope option https://review.opendev.org/702275 | 08:25 |
| *** udesale has quit IRC | 08:28 | |
| *** rpittau|afk is now known as rpittau | 08:31 | |
| *** tosky has joined #openstack-infra | 08:32 | |
| *** pkopec has joined #openstack-infra | 08:32 | |
| *** ociuhandu has joined #openstack-infra | 08:33 | |
| *** ociuhandu has quit IRC | 08:33 | |
| *** ociuhandu has joined #openstack-infra | 08:34 | |
| *** surpatil has joined #openstack-infra | 08:34 | |
| *** dmellado has quit IRC | 08:35 | |
| *** iurygregory has joined #openstack-infra | 08:36 | |
| *** dmellado has joined #openstack-infra | 08:37 | |
| *** ykarel|lunch is now known as ykarel | 08:38 | |
| openstackgerrit | Tobias Henkel proposed zuul/zuul master: DNM: Try out test sharding for tox-py35 https://review.opendev.org/702473 | 08:44 |
| *** aedc has quit IRC | 08:47 | |
| *** ociuhandu_ has joined #openstack-infra | 08:48 | |
| *** soumplis has left #openstack-infra | 08:49 | |
| *** ociuhandu has quit IRC | 08:51 | |
| *** ralonsoh has joined #openstack-infra | 08:52 | |
| *** ociuhandu_ has quit IRC | 08:58 | |
| *** ociuhandu has joined #openstack-infra | 08:59 | |
| *** ykarel is now known as ykarel|afk | 09:03 | |
| *** udesale_ has quit IRC | 09:04 | |
| *** jamesmcarthur has joined #openstack-infra | 09:06 | |
| openstackgerrit | Jan Kubovy proposed zuul/zuul master: Add spec for scale out scheduler https://review.opendev.org/621479 | 09:09 |
| *** kjackal has joined #openstack-infra | 09:10 | |
| *** jamesmcarthur has quit IRC | 09:10 | |
| openstackgerrit | Jan Kubovy proposed zuul/zuul master: Add spec for scale out scheduler https://review.opendev.org/621479 | 09:10 |
| *** xek_ has joined #openstack-infra | 09:11 | |
| *** lucasagomes has joined #openstack-infra | 09:14 | |
| *** kjackal has quit IRC | 09:27 | |
| *** elod has quit IRC | 09:28 | |
| openstackgerrit | Matthieu Huin proposed zuul/zuul master: OIDCAuthenticator: add capabilities, scope option https://review.opendev.org/702275 | 09:28 |
| *** kjackal has joined #openstack-infra | 09:29 | |
| *** ccamacho has joined #openstack-infra | 09:29 | |
| openstackgerrit | Simon Westphahl proposed zuul/nodepool master: Cleanup exception logging in static provider https://review.opendev.org/702828 | 09:37 |
| *** apetrich has joined #openstack-infra | 09:39 | |
| openstackgerrit | Simon Westphahl proposed zuul/nodepool master: Cleanup exception logging in static provider https://review.opendev.org/702828 | 09:39 |
| *** elod has joined #openstack-infra | 09:53 | |
| *** gfidente has joined #openstack-infra | 10:00 | |
| *** zbr is now known as zbr|rover | 10:09 | |
| *** rpittau is now known as rpittau|bbl | 10:21 | |
| openstackgerrit | Simon Westphahl proposed zuul/zuul master: Report buildset result in MQTT reporter https://review.opendev.org/702838 | 10:27 |
| *** ykarel|afk is now known as ykarel | 10:36 | |
| *** udesale has joined #openstack-infra | 10:38 | |
| *** gfidente has quit IRC | 10:45 | |
| *** lxkong has joined #openstack-infra | 11:00 | |
| *** gfidente has joined #openstack-infra | 11:09 | |
| *** udesale has quit IRC | 11:14 | |
| *** Lucas_Gray has joined #openstack-infra | 11:22 | |
| donnyd | sean-k-mooney: yea the power issue was a voltage spike that burn both of my UPS units. Everything is back online, just haven't had cycles to re enable it | 11:26 |
| *** surpatil has quit IRC | 11:31 | |
| *** lourot has quit IRC | 11:34 | |
| *** chandankumar has joined #openstack-infra | 11:41 | |
| *** Lucas_Gray has quit IRC | 11:41 | |
| sean-k-mooney | ouch | 11:45 |
| *** panda is now known as panda|drappt | 11:46 | |
| sean-k-mooney | ok no worries | 11:46 |
| openstackgerrit | Benjamin Schanzel proposed zuul/zuul master: Handle Erroneous Cron Strings in TimerDriver https://review.opendev.org/702237 | 11:46 |
| openstackgerrit | Benjamin Schanzel proposed zuul/zuul master: Handle Erroneous Cron Strings in TimerDriver https://review.opendev.org/702237 | 11:55 |
| *** jpena is now known as jpena|lunch | 12:05 | |
| *** dtantsur|afk is now known as dtantsur | 12:07 | |
| openstackgerrit | Benjamin Schanzel proposed zuul/zuul master: Allow Passing of Jitter Values in TimerDriver https://review.opendev.org/702854 | 12:08 |
| openstackgerrit | Simon Westphahl proposed zuul/zuul master: Report buildset result in MQTT reporter https://review.opendev.org/702838 | 12:08 |
| *** xek__ has joined #openstack-infra | 12:13 | |
| *** xek_ has quit IRC | 12:16 | |
| *** rpittau|bbl is now known as rpittau | 12:25 | |
| openstackgerrit | Simon Westphahl proposed zuul/zuul master: Report buildset result in MQTT reporter https://review.opendev.org/702838 | 12:31 |
| *** ykarel has quit IRC | 12:43 | |
| *** ykarel has joined #openstack-infra | 12:43 | |
| *** tetsuro has quit IRC | 12:45 | |
| *** ykarel is now known as ykarel|afk | 12:52 | |
| *** ykarel|afk has quit IRC | 12:54 | |
| *** ykarel_ has joined #openstack-infra | 12:54 | |
| *** ykarel_ is now known as ykarel | 12:54 | |
| *** smarcet has joined #openstack-infra | 12:56 | |
| yoctozepto | cirros-cloud.net seemingly down - does infra have a mirror for cirros? | 12:58 |
| *** ykarel is now known as ykarel|afk | 12:58 | |
| *** rlandy has joined #openstack-infra | 12:59 | |
| *** smarcet has quit IRC | 13:00 | |
| openstackgerrit | Miklos Vajna proposed opendev/git-review master: Make it possible to specify who is notified https://review.opendev.org/702861 | 13:03 |
| *** jpena|lunch is now known as jpena | 13:10 | |
| *** jamesmcarthur has joined #openstack-infra | 13:15 | |
| *** ramishra_ has quit IRC | 13:18 | |
| *** lxkong has quit IRC | 13:20 | |
| openstackgerrit | Benjamin Schanzel proposed zuul/zuul master: Allow Passing of Jitter Values in TimerDriver https://review.opendev.org/702854 | 13:21 |
| *** jamesmcarthur has quit IRC | 13:24 | |
| *** jamesmcarthur has joined #openstack-infra | 13:25 | |
| *** panda|drappt is now known as panda | 13:25 | |
| *** jamesmcarthur has quit IRC | 13:33 | |
| *** electrofelix has joined #openstack-infra | 13:35 | |
| *** vesper11 has quit IRC | 13:42 | |
| *** vesper11 has joined #openstack-infra | 13:42 | |
| *** ianychoi_ has quit IRC | 13:44 | |
| *** jamesmcarthur has joined #openstack-infra | 13:44 | |
| *** Lucas_Gray has joined #openstack-infra | 13:45 | |
| openstackgerrit | Tristan Cacqueray proposed zuul/zuul-operator master: Import dhall-operator https://review.opendev.org/702104 | 13:45 |
| openstackgerrit | Tristan Cacqueray proposed zuul/zuul-operator master: Import zuul application from dhall-zuul https://review.opendev.org/702105 | 13:46 |
| openstackgerrit | Tristan Cacqueray proposed zuul/zuul-operator master: Replace existing operator tasks with the new application https://review.opendev.org/702106 | 13:46 |
| openstackgerrit | Tristan Cacqueray proposed zuul/zuul-operator master: Add OpenShift SCC and functional test https://review.opendev.org/702758 | 13:46 |
| openstackgerrit | Tristan Cacqueray proposed zuul/zuul-operator master: Add custom image override https://review.opendev.org/702866 | 13:46 |
| *** eharney has quit IRC | 13:47 | |
| *** ricolin has joined #openstack-infra | 13:48 | |
| jrosser | yoctozepto: there is a cirros image inside the CI vm | 13:51 |
| *** udesale has joined #openstack-infra | 13:52 | |
| *** smarcet has joined #openstack-infra | 13:59 | |
| *** zxiiro has joined #openstack-infra | 14:01 | |
| *** diablo_rojo_phon has joined #openstack-infra | 14:02 | |
| *** ociuhandu has quit IRC | 14:02 | |
| openstackgerrit | Benjamin Schanzel proposed zuul/zuul master: Allow Passing of Jitter Values in TimerDriver https://review.opendev.org/702854 | 14:03 |
| *** dtantsur is now known as dtantsur|brb | 14:03 | |
| *** diablo_rojo has joined #openstack-infra | 14:04 | |
| *** jamesmcarthur has quit IRC | 14:04 | |
| frickler | yoctozepto: I do have a private copy which is about half a year old, if that helps you, I could put it up for you somewhere. maybe we (infra) could also setup a mirror, but otoh hrw and smoser were target to move things to github, so that should also get rid of the dreamhost issues | 14:05 |
| frickler | targetting | 14:05 |
| *** ociuhandu has joined #openstack-infra | 14:06 | |
| *** cyberpear has joined #openstack-infra | 14:08 | |
| *** ramishra has joined #openstack-infra | 14:12 | |
| yoctozepto | frickler: thanks | 14:13 |
| yoctozepto | jrosser, frickler: hmm, where are they in VM? It seems quite a few CI jobs try downloading it instead | 14:14 |
| jrosser | we did this change to pick it up https://review.opendev.org/#/c/689843/1/tasks/tempest_resources.yml | 14:14 |
| jrosser | using this var https://opendev.org/openstack/openstack-ansible-os_tempest/src/branch/master/defaults/main.yml#L254 | 14:15 |
| frickler | fungi: you were looking at mailing lists earlier, looks like http://lists.openstack.org/pipermail/openstack-sos/ hasn't been used in a while, too | 14:15 |
| *** diablo_rojo has quit IRC | 14:15 | |
| *** Lucas_Gray has quit IRC | 14:15 | |
| *** ociuhandu has quit IRC | 14:15 | |
| *** diablo_rojo has joined #openstack-infra | 14:16 | |
| *** ociuhandu has joined #openstack-infra | 14:17 | |
| frickler | yoctozepto: not sure whether that comment is still up to date, but I'd look here first https://opendev.org/openstack/devstack/src/branch/master/tools/image_list.sh#L5 | 14:17 |
| fungi | frickler: yeah, looks like dansmith is the listed owner for the openstack-sos ml, i can ask him if he's interested in having that one retired. the other ones i closed down last week were cases where list owners had approached me saying they wanted that | 14:19 |
| *** ociuhandu has quit IRC | 14:24 | |
| *** ianychoi has joined #openstack-infra | 14:25 | |
| *** psachin has quit IRC | 14:26 | |
| *** jamesmcarthur has joined #openstack-infra | 14:28 | |
| *** derekh has joined #openstack-infra | 14:31 | |
| *** eharney has joined #openstack-infra | 14:32 | |
| *** pcaruana has quit IRC | 14:33 | |
| *** jamesmcarthur has quit IRC | 14:44 | |
| *** gfidente has quit IRC | 14:45 | |
| *** lseki has joined #openstack-infra | 14:58 | |
| *** gfidente has joined #openstack-infra | 15:04 | |
| *** jamesmcarthur has joined #openstack-infra | 15:04 | |
| dansmith | fungi: that one totally never panned out. don't think there has been a message since 2013 | 15:08 |
| dansmith | fungi: delete with no notice | 15:08 |
| openstackgerrit | Sorin Sbarnea proposed zuul/zuul-website master: Add verification file https://review.opendev.org/702888 | 15:12 |
| openstackgerrit | Stephen Finucane proposed opendev/git-review master: trivial: Update to hacking 2.x https://review.opendev.org/702889 | 15:15 |
| openstackgerrit | Stephen Finucane proposed opendev/git-review master: gitignore: Ignore reno artefacts https://review.opendev.org/702890 | 15:16 |
| openstackgerrit | Stephen Finucane proposed opendev/git-review master: tox: Use python3 for all testenvs https://review.opendev.org/702891 | 15:16 |
| fungi | dansmith: thanks for confirming! the message archive will still be preserved of course, i'll merely be deleting the ml configuration for it | 15:16 |
| dansmith | there's no value in the archive, but whatever you want :) | 15:17 |
| fungi | it's no harm to preserve, unlike the rest of the ml bits which cause unchecked messages to pile up in moderation and stuff | 15:17 |
| jrosser | yoctozepto: heres the cirros download shown to be using the image from in the CI VM https://storage.bhs.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_4ce/702714/3/check/openstack-ansible-deploy-aio_metal-ubuntu-bionic/4cefcc2/logs/ara-report/result/92ff67ac-72d8-4650-aa67-cde55428c057/ | 15:19 |
| jrosser | if you set everything up right then the get_url module will just check in the location that the file is present / sha is correct and not download again | 15:19 |
| jrosser | outside CI it does the right thing and downloads the image | 15:20 |
| *** jamesmcarthur has quit IRC | 15:21 | |
| openstackgerrit | Andreas Jaeger proposed openstack/project-config master: Remove retired x/js-* repos from gerritbot https://review.opendev.org/702028 | 15:24 |
| *** ociuhandu has joined #openstack-infra | 15:27 | |
| *** ociuhandu has quit IRC | 15:28 | |
| openstackgerrit | Stephen Finucane proposed opendev/git-review master: Make it possible to specify who is notified https://review.opendev.org/702861 | 15:33 |
| *** gyee has joined #openstack-infra | 15:34 | |
| openstackgerrit | Tristan Cacqueray proposed zuul/zuul-operator master: Replace existing operator tasks with the new application https://review.opendev.org/702106 | 15:35 |
| openstackgerrit | Tristan Cacqueray proposed zuul/zuul-operator master: Add custom image override https://review.opendev.org/702866 | 15:35 |
| openstackgerrit | Tristan Cacqueray proposed zuul/zuul-operator master: Add OpenShift SCC and functional test https://review.opendev.org/702758 | 15:35 |
| *** pgaxatte has quit IRC | 15:36 | |
| *** ociuhandu has joined #openstack-infra | 15:37 | |
| AJaeger | config-core, a small gerritbot change for review, please: https://review.opendev.org/702028 | 15:39 |
| *** ahosam has quit IRC | 15:42 | |
| *** ociuhandu has quit IRC | 15:45 | |
| *** ociuhandu has joined #openstack-infra | 15:45 | |
| *** ricolin has quit IRC | 15:46 | |
| *** rfolco is now known as rfolco|doctor | 15:48 | |
| openstackgerrit | Miklos Vajna proposed opendev/git-review master: Make it possible to specify who is notified https://review.opendev.org/702861 | 15:58 |
| openstackgerrit | Merged opendev/git-review master: trivial: Update to hacking 2.x https://review.opendev.org/702889 | 15:59 |
| *** dtantsur|brb is now known as dtantsur | 16:02 | |
| openstackgerrit | Jeremy Stanley proposed zuul/zuul-website master: Add and update redirects for Zuul docs reorg https://review.opendev.org/702904 | 16:02 |
| *** michael-beaver has joined #openstack-infra | 16:04 | |
| *** ianychoi_ has joined #openstack-infra | 16:05 | |
| *** jpena is now known as jpena|off | 16:07 | |
| openstackgerrit | Merged opendev/git-review master: gitignore: Ignore reno artefacts https://review.opendev.org/702890 | 16:07 |
| openstackgerrit | Merged opendev/git-review master: tox: Use python3 for all testenvs https://review.opendev.org/702891 | 16:07 |
| *** ianychoi has quit IRC | 16:08 | |
| *** kjackal has quit IRC | 16:12 | |
| *** kjackal has joined #openstack-infra | 16:17 | |
| openstackgerrit | Miklos Vajna proposed opendev/git-review master: Make it possible to specify who is notified https://review.opendev.org/702861 | 16:17 |
| openstackgerrit | Jeremy Stanley proposed zuul/zuul-website master: Add and update redirects for Zuul docs reorg https://review.opendev.org/702904 | 16:17 |
| *** gfidente has quit IRC | 16:23 | |
| *** rpittau is now known as rpittau|afk | 16:27 | |
| *** electrofelix has quit IRC | 16:29 | |
| *** jackedin has joined #openstack-infra | 16:29 | |
| *** roman_g has joined #openstack-infra | 16:34 | |
| *** tesseract has quit IRC | 16:39 | |
| yoctozepto | jrosser: nice, thanks; it surely is nice the image is cached in there, will try this out for kolla, devstack tempest jobs could use that too | 16:42 |
| *** lmiccini has quit IRC | 16:42 | |
| jrosser | yoctozepto: yes it's super neat if you can wire it up so it works transparently in and out of CI, basically keep the image dir the same does the trick | 16:43 |
| yoctozepto | jrosser: indeed, though I worry more about CI, locally I just have the image around when I need it :-) | 16:44 |
| *** mattw4 has joined #openstack-infra | 16:45 | |
| openstackgerrit | Merged zuul/zuul-website master: Add and update redirects for Zuul docs reorg https://review.opendev.org/702904 | 16:45 |
| *** gfidente has joined #openstack-infra | 16:48 | |
| *** chandankumar is now known as raukadah | 16:49 | |
| *** dtroyer has quit IRC | 16:52 | |
| *** psachin has joined #openstack-infra | 16:53 | |
| *** slaweq has quit IRC | 17:01 | |
| *** lucasagomes has quit IRC | 17:02 | |
| *** slaweq has joined #openstack-infra | 17:04 | |
| openstackgerrit | Merged zuul/zuul master: Extract project config YAML into ref docs https://review.opendev.org/701977 | 17:06 |
| *** mattw4 has quit IRC | 17:07 | |
| *** slaweq has quit IRC | 17:09 | |
| *** mattw4 has joined #openstack-infra | 17:10 | |
| openstackgerrit | Miklos Vajna proposed opendev/git-review master: Make it possible to specify who is notified https://review.opendev.org/702861 | 17:10 |
| openstackgerrit | Clark Boylan proposed zuul/zuul-website master: Add client-user redirect https://review.opendev.org/702926 | 17:11 |
| *** iurygregory has quit IRC | 17:14 | |
| *** slaweq has joined #openstack-infra | 17:20 | |
| *** slaweq has quit IRC | 17:25 | |
| clarkb | I think the disk images and git repos are basically the last things we cache straight onto the images | 17:28 |
| *** jpena|off is now known as jpena | 17:29 | |
| *** udesale has quit IRC | 17:31 | |
| *** evrardjp has quit IRC | 17:34 | |
| *** evrardjp has joined #openstack-infra | 17:34 | |
| openstackgerrit | Merged openstack/project-config master: Remove retired x/js-* repos from gerritbot https://review.opendev.org/702028 | 17:39 |
| yoctozepto | jrosser: seems we were already using this particular trick, it's those devstack tempest jobs that did not, let me see if I can mitigate that particular failure... | 17:39 |
| clarkb | yoctozepto: the devstack jobs should, its devstack that we write the files for | 17:40 |
| yoctozepto | clarkb: those were devstack tempest | 17:40 |
| yoctozepto | let me find it again | 17:40 |
| clarkb | (like we write the files to the place devstack look for them in iirc) | 17:40 |
| yoctozepto | https://zuul.opendev.org/t/openstack/build/d9e2cebcaa674474ad1cc97b1fc7379a neutron-tempest-plugin doh | 17:41 |
| *** rfolco|doctor is now known as rfolco | 17:42 | |
| clarkb | yoctozepto: is it because that is cirros 3.4? and we use 3.5 elsewhere? | 17:43 |
| *** anteaya has joined #openstack-infra | 17:43 | |
| clarkb | we might not cache the older version | 17:43 |
| clarkb | oh we use 0.4.0 elsewhere now | 17:43 |
| yoctozepto | clarkb: checking script | 17:43 |
| clarkb | so ya I bet that is what it is | 17:43 |
| yoctozepto | yeah, 0.4.0 | 17:43 |
| clarkb | that job needs to use an up to date cirros to get the cached version | 17:44 |
| openstackgerrit | Merged zuul/zuul-website master: Add client-user redirect https://review.opendev.org/702926 | 17:44 |
| yoctozepto | clarkb: mhm, now I see | 17:45 |
| yoctozepto | it just feeds this oldie into the vars | 17:45 |
| clarkb | really not sure why they don't use the existing default cirros upload | 17:46 |
| clarkb | devstack already provides that by default, we shouldn't do an end around it | 17:46 |
| yoctozepto | they seemingly use ADVANCED_IMAGE_NAME with ubuntu 16.04 | 17:49 |
| yoctozepto | wonder if that is the cause they overrode images list | 17:49 |
| clarkb | oh maybe a bug in that where it falls back to old cirros for some reason? | 17:50 |
| clarkb | could be | 17:50 |
| *** smarcet has quit IRC | 17:50 | |
| *** smarcet has joined #openstack-infra | 17:53 | |
| *** anteaya has quit IRC | 17:54 | |
| yoctozepto | clarkb: I posted an update to ml so that others can react | 17:55 |
| yoctozepto | I'll see that little plugin if it passes with current cirros | 17:56 |
| clarkb | yoctozepto: thanks | 17:56 |
| fungi | though worth noting, if it remains down our image builds will all fail | 17:57 |
| fungi | since we'll stop being able to cache it | 17:57 |
| clarkb | ya though for a day or two thats probably fine | 17:57 |
| yoctozepto | fungi: indeed, but it seems it's not down per se, just tends to be overloaded | 17:57 |
| fungi | ahh | 17:57 |
| yoctozepto | so this way we minimize this problem | 17:57 |
| fungi | may still result in random image update failures for us | 17:57 |
| yoctozepto | or possibly remove it | 17:57 |
| *** pkopec has quit IRC | 17:57 | |
| clarkb | fungi: yoctozepto also dib should HEAD the url and if the file it already has is current not redownload | 17:58 |
| yoctozepto | yeah, sure, could use permacopy | 17:58 |
| yoctozepto | cirros does not change | 17:58 |
| clarkb | so if it is just overloading of the upstream then I think we'll be fine | 17:58 |
| fungi | maybe, as long as http head requests are not affected by the overloading | 17:58 |
| clarkb | ya | 17:58 |
| openstackgerrit | Stephen Finucane proposed opendev/git-review master: Make it possible to specify who is notified https://review.opendev.org/702861 | 17:59 |
| yoctozepto | fungi, clarkb: https://github.com/cirros-dev/cirros/issues/12#issuecomment-575224457 | 17:59 |
| *** derekh has quit IRC | 18:01 | |
| yoctozepto | by the looks of it neutron-tempest-plugin was left with 0.3.4 by accident | 18:08 |
| yoctozepto | https://review.opendev.org/609762 | 18:08 |
| clarkb | yoctozepto: yup looks like the job config should be updated | 18:14 |
| clarkb | maybe we can drop the default value as well and rely on the actual default in devstack rather than specifying a new default | 18:14 |
| yoctozepto | clarkb: you mean like this: https://review.opendev.org/702946 ? :D | 18:15 |
| clarkb | yup! | 18:15 |
| yoctozepto | moreover: | 18:17 |
| yoctozepto | http://codesearch.openstack.org/?q=download.cirros-cloud.net&i=nope&files=zuul.*%5C.yaml&repos= | 18:17 |
| yoctozepto | looks like one and only | 18:17 |
| yoctozepto | with this particular issue | 18:17 |
| yoctozepto | well, that sorts it out | 18:17 |
| clarkb | yoctozepto: we can also use logstash to search for cirros download failures | 18:18 |
| yoctozepto | clarkb: I think I need some guidance with that tough ;D | 18:18 |
| yoctozepto | though* | 18:18 |
| fungi | also codesearch won't turn up instances on other branches besides master | 18:18 |
| clarkb | yoctozepto: the first step is to find an identifying string in the logs | 18:18 |
| *** ociuhandu_ has joined #openstack-infra | 18:19 | |
| yoctozepto | fungi: yeah, that's true | 18:19 |
| clarkb | yoctozepto: Connecting to download.cirros-cloud.net (download.cirros-cloud.net)|64.90.42.85|:80... failed: Connection timed out. is probably a good one | 18:19 |
| yoctozepto | though master always gets hit the most | 18:19 |
| clarkb | yoctozepto: then we go to http://logstash.openstack.org/#/dashboard/file/logstash.json | 18:19 |
| clarkb | we want to search for the above in the message field in job-output.txt | 18:20 |
| clarkb | that gives us a query like this: message:"Connecting to download.cirros-cloud.net (download.cirros-cloud.net)|64.90.42.85|:80... failed: Connection timed out." AND filename:"job-output.txt" | 18:20 |
| clarkb | the problem with that query is that the ip address may be different if they load balance | 18:20 |
| yoctozepto | clarkb: and no regexp? :D | 18:21 |
| clarkb | we can search for two different matching message sections with separate message fields AND'd together | 18:21 |
| clarkb | yoctozepto: correct it is lucene | 18:21 |
| yoctozepto | clarkb: yeah, though for now I get like 0 hits | 18:21 |
| yoctozepto | :D | 18:21 |
| fungi | yeah, you could split it into separate message tokens around the ip address | 18:21 |
| clarkb | I think we want message:"Connecting to download.cirros-cloud.net" AND message:"failed: Connection timed out." AND filename:"job-output.txt" | 18:21 |
| clarkb | yoctozepto: you probably have to extend the timeframe (note I'm not testing thsi in reality just going from memory so there may be minor bugs ;) | 18:22 |
| fungi | yoctozepto: changed the timeframe from kibana's default of "last 15 minutes"? | 18:22 |
| yoctozepto | clarkb, fungi: yeah, forgot | 18:22 |
| yoctozepto | I still forget locally too | 18:22 |
| *** ociuhandu has quit IRC | 18:22 | |
| clarkb | I get 77 hits | 18:23 |
| clarkb | now on the left you can ask for job names | 18:23 |
| clarkb | "build_name" is the field | 18:23 |
| *** ociuhandu_ has quit IRC | 18:23 | |
| clarkb | looks like the vast majority are from the neutron plugin tests | 18:23 |
| yoctozepto | clarkb: yeah, got it | 18:23 |
| yoctozepto | mostly neutron | 18:23 |
| yoctozepto | yeah | 18:23 |
| yoctozepto | so is going to be fixed | 18:23 |
| clarkb | one is a dib build which intentionally doesn't use the cache in order to test the build iirc | 18:23 |
| clarkb | and ya overall should be fixed | 18:24 |
| yoctozepto | neutron-tempest-plugin-scenario-openvswitch | 18:25 |
| yoctozepto | 25 | 18:25 |
| yoctozepto | 2. neutron-tempest-plugin-dvr-multinode-scenario | 18:25 |
| yoctozepto | 22 | 18:25 |
| yoctozepto | 3. neutron-tempest-plugin-scenario-linuxbridge | 18:25 |
| yoctozepto | 18 | 18:25 |
| yoctozepto | 4. neutron-tempest-plugin-scenario-openvswitch-iptables_hybrid | 18:25 |
| yoctozepto | 17 | 18:25 |
| yoctozepto | 5. neutron-tempest-plugin-dvr-multinode-scenario-stein | 18:25 |
| yoctozepto | 16 | 18:25 |
| yoctozepto | 6. neutron-tempest-plugin-designate-scenario-stein | 18:25 |
| yoctozepto | 11 | 18:25 |
| yoctozepto | 7. neutron-tempest-plugin-dvr-multinode-scenario-rocky | 18:25 |
| yoctozepto | 7 | 18:25 |
| yoctozepto | sorry for the spam | 18:25 |
| yoctozepto | the even lines are counts there | 18:25 |
| yoctozepto | so neutron-tempest to merge and backport | 18:26 |
| yoctozepto | clarkb, fungi: thanks for the enlightenment :-) feeling more confident with infra now | 18:28 |
| clarkb | yoctozepto: great! as a side note we only have ~10 days of logs idnexed there because the db gets too large otherwise. And if you want to learn about doing queries in there the lucene docs are a good start (because ya its not as flexible as regexes) | 18:28 |
| clarkb | our elasticsearch cluster is 6 nodes, each has a 1TB volume for data attached to it and we need to be resilient to the failure of a single node which means we get 5TB of effective storage | 18:29 |
| clarkb | and we still fill that up within about 12 days | 18:29 |
| yoctozepto | wow, nice | 18:30 |
| fungi | there's some details at https://docs.openstack.org/infra/system-config/logstash.html though it may be outdated since the zuul v3 implementation changed a few bits | 18:30 |
| *** dtantsur is now known as dtantsur|afk | 18:30 | |
| *** slaweq has joined #openstack-infra | 18:31 | |
| *** liuyulong has quit IRC | 18:34 | |
| *** slaweq has quit IRC | 18:36 | |
| *** armax has quit IRC | 18:37 | |
| yoctozepto | clarkb, fungi: http://paste.openstack.org/show/788497/ | 18:37 |
| yoctozepto | record has two statuses, two builds | 18:38 |
| yoctozepto | two branches | 18:38 |
| yoctozepto | what the | 18:38 |
| yoctozepto | <insert ur fav word in here> | 18:39 |
| *** ociuhandu has joined #openstack-infra | 18:40 | |
| clarkb | yoctozepto: bug in the file parsing | 18:40 |
| yoctozepto | (and note kolla could not produce this message) | 18:40 |
| yoctozepto | mhm, odd | 18:40 |
| clarkb | usually indicates we have concatenated two events into one and those came from different sources | 18:40 |
| yoctozepto | yeah, but kolla has no such entry in its logs :D | 18:41 |
| yoctozepto | the merger is really mysterious | 18:41 |
| *** jpena is now known as jpena|off | 18:41 | |
| clarkb | ya Im sure it came from the neutron side | 18:41 |
| clarkb | then for some reason in the stream it added a kolla log to it | 18:42 |
| clarkb | thestuff after the newline in the message | 18:42 |
| *** ociuhandu has quit IRC | 18:45 | |
| yoctozepto | yeah, it looks kollish | 18:45 |
| yoctozepto | 2020-01-16 09:58:37.780868 | primary | INFO:kolla.common.utils.monasca-thresh:496/634 KB 4/359 KB 68/407 KB 432/3862 KB | 18:46 |
| yoctozepto | 496/634 KB 4/359 KB 68/407 KB 436/3862 KB | 18:46 |
| yoctozepto | 496/634 KB 4/359 KB 68/407 KB 440/3862 KB | 18:46 |
| *** psachin has quit IRC | 18:46 | |
| clarkb | ya the reason it tries to concatenate there is we have a multiline event. The 496 prefixed lines should all be concatenated to the 2020 prefixed line | 18:47 |
| clarkb | for some reason it ends up associated to the other file event | 18:47 |
| clarkb | in theory this should never happen because we process files serially | 18:47 |
| clarkb | or rather in separate serial pipelines | 18:48 |
| clarkb | I think this ends up being a logstash bug because it must be crossing those pipelines | 18:48 |
| clarkb | I looked into it a while back and decided it happened infrequently enough to not worry too much | 18:48 |
| clarkb | when processing a billion results a day figuring out why .001% are weird is not always easy | 18:49 |
| *** smarcet has quit IRC | 18:50 | |
| *** gfidente is now known as gfidente|afk | 18:51 | |
| *** ralonsoh has quit IRC | 18:51 | |
| *** rlandy is now known as rlandy|afk | 18:54 | |
| *** openstackgerrit has quit IRC | 18:58 | |
| *** openstackgerrit has joined #openstack-infra | 18:59 | |
| openstackgerrit | Matthieu Huin proposed zuul/zuul master: JWT drivers: Deprecate RS256withJWKS, introduce OpenIDConnect https://review.opendev.org/701972 | 18:59 |
| *** kjackal has quit IRC | 18:59 | |
| openstackgerrit | Matthieu Huin proposed zuul/zuul master: JWT drivers: Deprecate RS256withJWKS, introduce OpenIDConnect https://review.opendev.org/701972 | 19:02 |
| *** jackedin has quit IRC | 19:02 | |
| openstackgerrit | James E. Blair proposed zuul/zuul master: Re-order reference index https://review.opendev.org/702962 | 19:05 |
| *** aedc has joined #openstack-infra | 19:08 | |
| openstackgerrit | Matthieu Huin proposed zuul/zuul master: JWT drivers: Deprecate RS256withJWKS, introduce OpenIDConnect https://review.opendev.org/701972 | 19:10 |
| openstackgerrit | Mohammed Naser proposed zuul/zuul-helm master: Added support for configuring disk_limit_per_job https://review.opendev.org/702963 | 19:17 |
| *** ccamacho has quit IRC | 19:18 | |
| *** smarcet has joined #openstack-infra | 19:18 | |
| yoctozepto | 19:49:13 <clarkb> when processing a billion results a day figuring out why .001% are weird is not always easy | 19:19 |
| yoctozepto | always nice to keep some sanity to ourselves isn't it? | 19:19 |
| openstackgerrit | Tristan Cacqueray proposed zuul/zuul-operator master: Add OpenShift SCC and functional test https://review.opendev.org/702758 | 19:20 |
| *** armax has joined #openstack-infra | 19:23 | |
| openstackgerrit | Sorin Sbarnea proposed zuul/zuul-jobs master: tox: change default calling as a module https://review.opendev.org/690057 | 19:28 |
| openstackgerrit | Mohammed Naser proposed zuul/zuul-helm master: Add extra files for secret https://review.opendev.org/702965 | 19:29 |
| openstackgerrit | Merged openstack/cookiecutter master: Update CONTRIBUTING.rst template https://review.opendev.org/696001 | 19:31 |
| *** smarcet has quit IRC | 19:32 | |
| clarkb | yoctozepto: indeed | 19:35 |
| johnsom | Did we deploy the new zuul recently? I'm getting blank pages where the logs list should be.... https://zuul.opendev.org/t/openstack/build/eacbccd51e5f4125aab3eb329c976f3a/log/controller/logs/ | 19:42 |
| clarkb | johnsom: we did deploy a new zuul on tuesday iirc | 19:43 |
| clarkb | https://storage.gra.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_eac/702921/1/check/octavia-v2-dsvm-scenario/eacbccd/controller/logs/ the logs are there at least | 19:44 |
| clarkb | and navigating the tree in the web dashboard shows them too | 19:44 |
| clarkb | I'm not sure we do deep links in the dashboard though? | 19:44 |
| johnsom | I guess direct nav in the file paths doesn't work? | 19:44 |
| clarkb | ya doing the navigating from the root it doesn't update the url | 19:44 |
| johnsom | It used to. I use it regularly to jump around in the folders | 19:45 |
| clarkb | its valid when you open a file that way | 19:45 |
| clarkb | but not the intermediate dirs looks like | 19:45 |
| clarkb | I don't think the js has updated around that | 19:45 |
| clarkb | corvus: ^ would probably know for sure though | 19:45 |
| smcginnis | johnsom: Looks like if you step back, you can get there from the summary - https://zuul.opendev.org/t/openstack/build/eacbccd51e5f4125aab3eb329c976f3a/ | 19:45 |
| johnsom | Yeah, looks like you have to go back to the root then down again. | 19:46 |
| fungi | did it previously auto-expand the location you indicated in the log browser if you provided a deep link to a directory under the dashboard url? | 19:46 |
| fungi | i too had never tried doing that so didn't notice what it did | 19:47 |
| *** cyberpear has quit IRC | 19:47 | |
| johnsom | I *think* I got a non-pretty view of the directory. I.e. it didn't have that (raw) or expander links, just the old view | 19:47 |
| clarkb | johnsom: thats if you used the raw urls | 19:48 |
| clarkb | https://storage.gra.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_eac/702921/1/check/octavia-v2-dsvm-scenario/eacbccd/controller/logs/ is the raw url for this particular example | 19:48 |
| johnsom | Yeah, maybe it just directed me there | 19:48 |
| johnsom | I can open a story, should it be zuul/zuul or zuul/zuul-website? Any advice? | 19:50 |
| clarkb | johnsom: zuul/zuul. zuul-website is for zuul-ci.org content | 19:50 |
| johnsom | +1 Thanks | 19:50 |
| clarkb | I would specify that deep links to directories do not work but they do work for files | 19:51 |
| johnsom | Thanks, opened: https://storyboard.openstack.org/#!/story/2007104 | 19:56 |
| corvus | clarkb: i don't think any of that js has changed | 19:57 |
| fungi | ianw: eureka! | 20:02 |
| corvus | clarkb, johnsom: i don't think we ever had js rendering of directories. for a moment, we tried putting the tree on the top of every page, but there were complexities with that. | 20:02 |
| fungi | ianw: research into reproducible wheels has paid off | 20:03 |
| *** kjackal has joined #openstack-infra | 20:03 | |
| fungi | ianw: the short answer is we want to `export SOURCE_DATE_EPOCH=1540393768;export CFLAGS=-g0` | 20:03 |
| fungi | (pick whatever epoch time we want, just keep it consistent between builds) | 20:04 |
| clarkb | fungi: that still depends on the package itself right? | 20:04 |
| clarkb | fungi: I know pbr a long time ago produced non reproducable builds (we fixed that though) | 20:04 |
| *** tosky has quit IRC | 20:04 | |
| fungi | the problems i found were: 1. file last modified times recorded by zip, 2. c debug info compiling timestamps into .so files | 20:05 |
| clarkb | in the pbr case it was serialization of dicts which isn't deterministic in python2 | 20:05 |
| clarkb | as order of keys can change | 20:06 |
| fungi | yeah, also some python projects may simply do terrible things like altering the output of autogenerated files from one build to the next like not sorting outputs or embedding timestamps in things | 20:06 |
| fungi | i'm sure across a larger test set we'll find more | 20:07 |
| fungi | but also the number of projects we cache which don't already publish usable wheels is probably vanishingly small | 20:07 |
| JayF | I don't have meaningful technical input, but it's really awesome to hear that you all are trying to publish reproducable wheels. Super exciting. | 20:08 |
| fungi | JayF: it's not so much that (i suspect our own wheels are already trivially reproducible) | 20:08 |
| fungi | it's about the platform-specific wheels we cache for dependencies | 20:08 |
| JayF | Oh wow, so even going a layer up. That's even more intense! | 20:09 |
| fungi | and trying to find ways to reliably index them by checksum (a feature pip relies on) without replacing them all on every update | 20:09 |
| *** sshnaidm is now known as sshnaidm|afk | 20:14 | |
| openstackgerrit | Sorin Sbarnea proposed zuul/zuul-jobs master: ensure-tox: use portable pip call https://review.opendev.org/702978 | 20:26 |
| openstackgerrit | Sorin Sbarnea proposed zuul/zuul-jobs master: ensure-tox: use portable pip call https://review.opendev.org/702978 | 20:27 |
| openstackgerrit | Tristan Cacqueray proposed zuul/zuul-operator master: DNM: test arbritary uid support for zuul image https://review.opendev.org/702980 | 20:34 |
| openstackgerrit | Tristan Cacqueray proposed zuul/zuul-operator master: DNM: test arbritary uid support for zuul image https://review.opendev.org/702980 | 20:37 |
| *** slaweq has joined #openstack-infra | 20:38 | |
| openstackgerrit | Merged opendev/system-config master: Switch nodes to xenial https://review.opendev.org/698343 | 20:43 |
| *** gfidente|afk has quit IRC | 20:46 | |
| openstackgerrit | Sorin Sbarnea proposed zuul/zuul-jobs master: ensure-tox: improve pip detection https://review.opendev.org/702978 | 20:47 |
| openstackgerrit | Sorin Sbarnea proposed zuul/zuul-jobs master: ensure-tox: improve pip detection https://review.opendev.org/702978 | 20:48 |
| openstackgerrit | Sorin Sbarnea proposed zuul/zuul-jobs master: ensure-tox: improve pip detection https://review.opendev.org/702978 | 20:50 |
| *** aedc has quit IRC | 20:51 | |
| *** cyberpear has joined #openstack-infra | 20:53 | |
| *** smarcet has joined #openstack-infra | 21:00 | |
| mnaser | does infra run a "docker cache" for quay.io ? | 21:01 |
| *** rlandy|afk is now known as rlandy | 21:03 | |
| openstackgerrit | Sorin Sbarnea proposed zuul/zuul-jobs master: ensure-tox: improve pip detection https://review.opendev.org/702978 | 21:04 |
| openstackgerrit | Sorin Sbarnea proposed zuul/zuul-jobs master: ensure-tox: improve pip detection https://review.opendev.org/702978 | 21:06 |
| clarkb | mnaser: we do | 21:07 |
| clarkb | cloudnull: added it. I would have to check the vhost to know the location | 21:07 |
| mnaser | ok ill search up systemc-onfig | 21:07 |
| cloudnull | o/ | 21:08 |
| mordred | it's a wild cloudnull | 21:08 |
| mnaser | https://opendev.org/opendev/system-config/src/branch/master/playbooks/roles/mirror/templates/mirror.vhost.j2#L467-L513 | 21:08 |
| mnaser | looks like its this guy | 21:08 |
| * cloudnull hides | 21:08 | |
| cloudnull | anything I can help with ? | 21:08 |
| donnyd | fungi: or clarkb FN should be good to go if you have time to turn it back on. If not I can get to it later tonight | 21:08 |
| mnaser | cloudnull: looks like you already did help out :D | 21:08 |
| cloudnull | woot! | 21:09 |
| mnaser | i guess there isnt a trivial way to use this registry without changing actual test jobs though | 21:09 |
| cloudnull | nope :'( | 21:09 |
| clarkb | donnyd: I'll get it thanks | 21:09 |
| *** slaweq has quit IRC | 21:09 | |
| fungi | ianw: one other building block. we can indeed parse the pip log to find out what wheels we downloaded from pypi: sed -n 's,.*Downloading from URL | 21:09 |
| fungi | .*/\([^/]*\.whl\)#.*,\1,p' pip.log | 21:09 |
| cloudnull | mnaser we're tinkering with it in the land of tripleo | 21:09 |
| * fungi curses stray newlines | 21:09 | |
| cloudnull | weshay|ruck - might have some insight into using it ? | 21:09 |
| fungi | ianw: though might want to extend the pip wheel command to something like `pip --log pip.log wheel ...` so we can more directly control where the appending log resides | 21:10 |
| mnaser | i wonder if its just a matter of s/quay.io/<foo>/ | 21:10 |
| * weshay|ruck scrolls | 21:11 | |
| weshay|ruck | mnaser, I think there was a config added for quay.io | 21:11 |
| mnaser | ooo | 21:11 |
| *** jamesmcarthur has joined #openstack-infra | 21:11 | |
| mnaser | maybe that might be a podman thing which llets you do multiple mirrors | 21:12 |
| weshay|ruck | a while ago.. we got blocked on using quay though because of manifests | 21:12 |
| fungi | ianw: anyway, with that sed pattern and reusing the same appending pip.log between all invocations, i think we can do a delete pass from the built wheelhouse just once at the end so we don't lose out on local caching of dependencies between invocations | 21:12 |
| mnaser | weshay|ruck: i'd appreciate if you can share that, trying to speed up some jobs here that hit up quay.io | 21:13 |
| openstackgerrit | Tristan Cacqueray proposed zuul/zuul-operator master: Replace existing operator tasks with the new application https://review.opendev.org/702106 | 21:13 |
| openstackgerrit | Tristan Cacqueray proposed zuul/zuul-operator master: Add custom image override https://review.opendev.org/702866 | 21:13 |
| openstackgerrit | Tristan Cacqueray proposed zuul/zuul-operator master: Add OpenShift SCC and functional test https://review.opendev.org/702758 | 21:13 |
| openstackgerrit | Tristan Cacqueray proposed zuul/zuul-operator master: DNM: test arbritary uid support for zuul image https://review.opendev.org/702980 | 21:13 |
| weshay|ruck | mnaser, let me go email diving | 21:14 |
| weshay|ruck | mnaser, one thing is clear.. I have a better memory than cloudnull https://review.opendev.org/#/c/683240/ | 21:17 |
| cloudnull | ^ ++ | 21:17 |
| weshay|ruck | cloudnull, :) | 21:17 |
| *** jamesmcarthur has quit IRC | 21:17 | |
| weshay|ruck | cloudnull, I think it's due to the fact you no longer have to drive yourself | 21:17 |
| *** jamesmcarthur has joined #openstack-infra | 21:18 | |
| cloudnull | this is true, I'm gradually becoming more energy efficient (lazy) | 21:18 |
| weshay|ruck | this awesomeness.. thanks for the patch cloudnull :) | 21:19 |
| cloudnull | weshay|ruck do we have any jobs running with quay at this time ? | 21:20 |
| weshay|ruck | cloudnull, we've been in contact w/ their support.. we're just starting to support mutiarch manifests.. and pushing those to docker.io | 21:21 |
| weshay|ruck | and quay doesn't have that yet.. | 21:21 |
| weshay|ruck | so we're in touch w/ them | 21:21 |
| weshay|ruck | but can't start using it | 21:21 |
| weshay|ruck | but we have the code in to push to quay and docker.io and hoping to turn off docker.io at some point | 21:22 |
| cloudnull | ++ | 21:23 |
| *** jamesmcarthur has quit IRC | 21:25 | |
| mordred | mnaser: the podman config supports multiple mirrors - we use it in setting up the buildset registry | 21:25 |
| mnaser | mordred: neat. i guess that might have to push these jobs to use them instead of docker | 21:26 |
| mordred | mnaser: at this point, I *highly* recommend switching to podman for at least image building | 21:26 |
| mnaser | ya this is runtime unfortunately :< | 21:26 |
| mordred | you can still use docker for running of course | 21:26 |
| mordred | nod | 21:26 |
| *** eharney has quit IRC | 21:26 | |
| mordred | well- I *also* recommend switching to podman for those - but that's a bigger thing | 21:26 |
| mordred | mnaser: but the buildset registry stuff has code that sets up the config file to point at multiple locations for mirrors | 21:27 |
| *** rfolco has quit IRC | 21:29 | |
| mordred | mnaser: https://opendev.org/zuul/zuul-jobs/src/branch/master/roles/use-buildset-registry/tasks/main.yaml#L88-L94 | 21:30 |
| mordred | mnaser: corvus had to write a python module to deal with it :) | 21:31 |
| paladox | mordred apparently maven has enforced https breaking all tagged gerrit releases :( (i presume) | 21:32 |
| paladox | based on https://gerrit-review.googlesource.com/c/bazlets/+/250532 | 21:32 |
| mordred | paladox: oh goodie | 21:33 |
| mordred | paladox: that change has been merged - so it should be fixed again for rebuilds, yes? | 21:34 |
| paladox | yup | 21:34 |
| paladox | but only on the branch | 21:34 |
| paladox | no tagged releases have been done with that fix | 21:34 |
| mordred | paladox: ok by us - we build from branches anyway :) | 21:36 |
| paladox | :P | 21:36 |
| openstackgerrit | Clark Boylan proposed openstack/project-config master: Revert "Temporarily disable fortnebula" https://review.opendev.org/702989 | 21:37 |
| clarkb | infra-root ^ I've chcked that the mirror is available and the pypi cache as well as ubuntu afs content seem to be working | 21:37 |
| fungi | excellent | 21:38 |
| fungi | thanks! | 21:38 |
| paladox | mordred We're setting up a test server to test the schema upgrade on real data (gerrit.w.org). | 21:38 |
| paladox | our upgrade is part of this quater https://phabricator.wikimedia.org/T242237 :) | 21:42 |
| clarkb | mordred: are we going to add webserver management to review-dev next? | 21:42 |
| mordred | clarkb: yup | 21:42 |
| mordred | clarkb: that's next on my list | 21:43 |
| openstackgerrit | James E. Blair proposed zuul/zuul master: Docs: move project config docs to user reference https://review.opendev.org/702992 | 21:48 |
| clarkb | I'm trying to be a good reviewer and get through project-config and system-config and related changes today. If you've got changes to point out to me feel free | 21:48 |
| clarkb | I'll try to do zuul and nodepool after too if I get time | 21:48 |
| openstackgerrit | Merged openstack/project-config master: Add <build|publish>-wheel-mirror-centos-8 jobs https://review.opendev.org/702350 | 21:53 |
| openstackgerrit | James E. Blair proposed zuul/zuul master: Move overview section to reference https://review.opendev.org/702995 | 21:55 |
| *** smarcet has quit IRC | 21:56 | |
| donnyd | thank you clarkb :) | 21:56 |
| openstackgerrit | Merged openstack/project-config master: Revert "Temporarily disable fortnebula" https://review.opendev.org/702989 | 21:59 |
| openstackgerrit | Merged openstack/project-config master: Add ovs and dpdk source to available repos https://review.opendev.org/701539 | 21:59 |
| clarkb | infra-root https://review.opendev.org/#/c/698089/2 is a nice change I've noticed in clearing my backlog. steve baker has fixed the comment filtering in gerritbot | 21:59 |
| donnyd | sometime in the next couple months I will be rebuilding my DC to be more efficient in cooling - so there will be a couple days to a week FN is down. I will let everyone know well in advance | 22:03 |
| donnyd | It should be far ahead of crunch time at the end of the cycle | 22:04 |
| fungi | donnyd: that sounds exciting! looking forward to it | 22:04 |
| fungi | curious to hear what you wind up implementing | 22:04 |
| clarkb | corvus: any idea why https://review.opendev.org/#/c/660541/3 doesn't get enqueued to the gate even though the depends on there has merged? | 22:04 |
| donnyd | This time around I am going to have some shareable blueprints | 22:05 |
| clarkb | or maybe this is the github cache being stale and I just have to wait for 5 minutes? | 22:05 |
| corvus | clarkb: not immediately. are you asking me to read the debug log? | 22:06 |
| clarkb | corvus: no, just asking if github depends on like that have known problems with our zuul | 22:06 |
| clarkb | if not I'll start looking in the debug log | 22:06 |
| corvus | clarkb: not that i'm aware of | 22:06 |
| *** Lucas_Gray has joined #openstack-infra | 22:06 | |
| donnyd | can everyone else hit this http://mirror.regionone.fortnebula.opendev.org/ | 22:06 |
| clarkb | donnyd: I can | 22:06 |
| clarkb | via ipv4 | 22:07 |
| donnyd | I checked from my phone and here.. just wanted to check before workloads start coming in | 22:07 |
| donnyd | thanks clarkb | 22:07 |
| *** rcernin has joined #openstack-infra | 22:07 | |
| fungi | i'm getting to it fine over ipv6 | 22:07 |
| *** Lucas_Gray has quit IRC | 22:08 | |
| mordred | works for me | 22:09 |
| donnyd | thank you all for checking - things should be a go | 22:09 |
| *** smarcet has joined #openstack-infra | 22:12 | |
| openstackgerrit | James E. Blair proposed zuul/zuul master: Docs: re-order reference index https://review.opendev.org/702962 | 22:13 |
| openstackgerrit | James E. Blair proposed zuul/zuul master: Docs: move project config docs to user reference https://review.opendev.org/702992 | 22:13 |
| openstackgerrit | James E. Blair proposed zuul/zuul master: Docs: move overview section to reference https://review.opendev.org/702995 | 22:13 |
| openstackgerrit | James E. Blair proposed zuul/zuul master: Docs: add admin reference section https://review.opendev.org/702997 | 22:13 |
| *** jamesmcarthur has joined #openstack-infra | 22:13 | |
| openstackgerrit | Merged opendev/gerritbot master: Fix event comment-added https://review.opendev.org/698089 | 22:16 |
| *** smarcet has quit IRC | 22:16 | |
| *** xek__ has quit IRC | 22:17 | |
| *** jamesmcarthur has quit IRC | 22:19 | |
| *** smarcet has joined #openstack-infra | 22:22 | |
| *** kjackal has quit IRC | 22:22 | |
| donnyd | first set of jobs are in and all looks good from here | 22:26 |
| *** jamesmcarthur has joined #openstack-infra | 22:28 | |
| openstackgerrit | Clark Boylan proposed opendev/system-config master: Replace skip with errors=ignore https://review.opendev.org/660541 | 22:34 |
| *** jamesmcarthur has quit IRC | 22:35 | |
| clarkb | infra-root I had to update the depends-on in https://review.opendev.org/#/c/660541/4 to work around a bug in github3 that zuul is hitting. Any chance yall can give it a quick rereview? | 22:36 |
| clarkb | https://review.opendev.org/#/c/526979/ is another one that would be good to land (it makes puppet module cloning more sane) | 22:37 |
| Shrews | clarkb: I remember that change. looking | 22:38 |
| clarkb | thanks! | 22:38 |
| Shrews | clarkb: both lgtm | 22:40 |
| clarkb | Shrews: I also confirmed that ansible 2.9.1 which is what we run on bridge includes ianw's fix upstream | 22:40 |
| Shrews | ++ | 22:41 |
| *** jamesmcarthur has joined #openstack-infra | 22:47 | |
| clarkb | mordred: comments on https://review.opendev.org/#/c/690505/2 I think it may cargo cult an existing bug | 22:47 |
| *** jamesmcarthur has quit IRC | 22:48 | |
| clarkb | https://review.opendev.org/#/c/701870/1 is another change that I think is an easy review that we can just land. cc infra-root | 22:49 |
| *** smarcet has quit IRC | 22:50 | |
| clarkb | https://review.opendev.org/#/c/702053/1 too | 22:51 |
| openstackgerrit | Monty Taylor proposed opendev/system-config master: Add job dependencies to haproxy-statsd https://review.opendev.org/690505 | 22:51 |
| openstackgerrit | Monty Taylor proposed opendev/system-config master: Update python-base image upload job depends https://review.opendev.org/703001 | 22:51 |
| mordred | clarkb: ^^ there's an update to the haproxy one - and a followup to fix the latent bug | 22:51 |
| *** jamesmcarthur has joined #openstack-infra | 22:52 | |
| *** smcginnis is now known as smcginnis|PTO | 22:52 | |
| clarkb | +2 thanks! | 22:52 |
| openstackgerrit | Merged opendev/system-config master: Add notes on manual host configuration runs https://review.opendev.org/516510 | 22:53 |
| mordred | corvus: got a sec to re-review 690505? (and the followup from the review from clarkb?) | 22:53 |
| *** smarcet has joined #openstack-infra | 22:53 | |
| *** smarcet has quit IRC | 22:53 | |
| corvus | +3s all around | 22:54 |
| clarkb | thanks all looks like we will flush a good number of changes out | 22:54 |
| *** jamesmcarthur has quit IRC | 22:55 | |
| fungi | corvus: was slightly faster on the draw than i was | 22:56 |
| fungi | good thing this wasn't the o.k. corral | 22:57 |
| *** smarcet has joined #openstack-infra | 22:57 | |
| corvus | fungi: imagine if gertty were a fps? | 22:58 |
| fungi | now i'm getting flashbacks to the old doom mod that let you shoot system processes to send them kill signals | 23:01 |
| openstackgerrit | Merged opendev/system-config master: Better checking for tags when cloning puppet modules https://review.opendev.org/526979 | 23:01 |
| mordred | corvus: gerrty isn't a FPS? | 23:01 |
| * fungi imagines a corridor full of ravenous pink munchers with change ids tattooed on what passes for their foreheads | 23:02 | |
| fungi | though maybe it would be more like a duck hunt clone, with diffie the cuckoo instead of ducks | 23:03 |
| * fungi starts polishing his nes zapper | 23:04 | |
| clarkb | poor diffie | 23:06 |
| openstackgerrit | Merged opendev/system-config master: Switch to collect-container-logs https://review.opendev.org/701870 | 23:13 |
| openstackgerrit | Merged opendev/system-config master: Add quick script for cleaning boot from volume leaks https://review.opendev.org/702053 | 23:16 |
| melwitt | could someone please give me a hint as to why one job might be ok with a 'no such file or directory' when exporting logs, example: https://zuul.opendev.org/t/openstack/build/f4eba417964042598702f0f17213393b/log/job-output.txt#30940 | 23:26 |
| melwitt | but another might not be ok with it and fail the job in post? https://zuul.opendev.org/t/openstack/build/ee415b226ce444c6af0d4dcb0eaf7a34/log/job-output.txt#31051 | 23:27 |
| *** armax has quit IRC | 23:28 | |
| melwitt | the first one is a zuul v3 grenade job proposed at https://review.opendev.org/548936 and the second one is Depends-On that change in nova https://review.opendev.org/701478 I don't understand how the same job can behave differently (same job, 'grenade') | 23:28 |
| *** dchen has joined #openstack-infra | 23:29 | |
| clarkb | melwitt: this is a fun one | 23:30 |
| clarkb | melwitt: the console gives us better insight. https://zuul.opendev.org/t/openstack/build/f4eba417964042598702f0f17213393b/console#4/0/6/controller vs https://zuul.opendev.org/t/openstack/build/ee415b226ce444c6af0d4dcb0eaf7a34/console#4/0/6/controller | 23:30 |
| melwitt | and the first grenade job tolerates the 'cat: /opt/stack/log-start-timestamp.txt: No such file or directory' and the Depends-On one does not. I just don't know where to understand how or why that is | 23:30 |
| clarkb | melwitt: what that shows us is that when it fails the return code is 1 and when it succeeds the return code is 0 (which is what we'd expect) | 23:30 |
| clarkb | the reason for the different return codes is that the job that passes uses a pipe to write out the file contents and the one that fails uses a redirect | 23:31 |
| clarkb | if that task had set the pipefail option then they would both fail | 23:31 |
| melwitt | ohhh gosh I didn't catch the | vs > | 23:32 |
| melwitt | sorry | 23:32 |
| clarkb | We should probably fix the file being gone, then ensure we catch those failures in either case | 23:32 |
| melwitt | and thank you so much for showing the console thing. I'm trying to learn how to troubleshoot jobs. and also want to help with converting our jobs to zuul v3 ... I was trying to see if when that grenade zuul v3 change lands if we could get a v3 grenade for free | 23:33 |
| clarkb | you're welcome. Happy to help | 23:34 |
| clarkb | and ya I thinkn once the base grenade stuff is v3'd it should be pretty easy to run a nova job for it (the regular grenade setup is probably good for nova?) | 23:35 |
| melwitt | oh geez, the redirect is because of the recent change from sean-k-mooney https://opendev.org/openstack/devstack/commit/d02fa6f856ac5951b8a879c23b57d5a752f28918 and I was looking at an older result on that proposed grenade patch | 23:35 |
| melwitt | that result was from back when it was a pipe | 23:35 |
| melwitt | so even that change would fail today with the newer change. so that explains the puzzling difference | 23:36 |
| melwitt | s/newer change/redirect in the command/ | 23:36 |
| melwitt | ok. so need to focus on the file being gone. I'll see what I can find out about it | 23:38 |
| clarkb | ya I think fixing that should make everything happy | 23:39 |
| clarkb | melwitt: iirc that file is written so that we can dump the journalctl logs from that point (in the file timestamp) forawrd | 23:40 |
| clarkb | because devstack might be run multiple times | 23:40 |
| clarkb | or server might come up and sit for hours before a job runs on it | 23:40 |
| melwitt | ah I see | 23:40 |
| melwitt | https://opendev.org/openstack/devstack/src/branch/master/roles/start-fresh-logging/tasks/main.yaml#L14 | 23:42 |
| melwitt | and the output of running that shows ... nothing really https://zuul.opendev.org/t/openstack/build/ee415b226ce444c6af0d4dcb0eaf7a34/log/job-output.txt#1304 | 23:46 |
| openstackgerrit | Merged opendev/system-config master: Replace skip with errors=ignore https://review.opendev.org/660541 | 23:46 |
| clarkb | melwitt: https://zuul.opendev.org/t/openstack/build/ee415b226ce444c6af0d4dcb0eaf7a34/console#2/0/50/controller that shows it writing to /opt/stack/old/log-start-timestamp.txt | 23:48 |
| clarkb | so I think the paths are mismatched | 23:48 |
| clarkb | (and that happens because with grenade we have two frames of reference instead of one | 23:48 |
| melwitt | right ok. (I really need to figure out console asap :P) | 23:48 |
| clarkb | melwitt: the console is a rendering of ansible things | 23:49 |
| clarkb | the job output is nice because its kind of the simplified version and when your issues are in tox output or similar good place to go | 23:49 |
| clarkb | but if the problems are related to the ansible tasks it is helpful to go there typically | 23:49 |
| melwitt | gotcha | 23:50 |
| openstackgerrit | Merged opendev/system-config master: Add roles for a basic static server https://review.opendev.org/697587 | 23:54 |
| melwitt | I see, so it's writing to /opt/stack/old/log-start-timestamp.txt and trying to read from /opt/stack/log-start-timestamp.txt. that's interesting that during a grenade job the same variable {{ devstack_base_dir }} can be either /opt/stack/old/ or /opt/stack/ | 23:55 |
| clarkb | melwitt: ya I think the mechanism that allows that to happen is we run ansible-playbook for each pre-run and post-run playbook separately | 23:55 |
| clarkb | which means they effectively get a clean slate for var defs | 23:55 |
| melwitt | I would not have expected that devstack_base_dir would not be /opt/stack/ only | 23:55 |
| clarkb | is it possible the post-run playbook is setting it improperly? | 23:56 |
| melwitt | I'll look | 23:56 |
| zbr|rover | mordred: fungi: if you could workflow the ensure-tox today https://review.opendev.org/#/c/702978/ it would be really helpul, thanks. | 23:57 |
| clarkb | melwitt: looks like pre.yaml sets devstack_base_dir explicitly but post does not (so post falls back to the default | 23:58 |
| clarkb | melwitt: maybe try setting it in post.yaml too ? https://review.opendev.org/#/c/548936/106/playbooks/pre.yaml line 23 is where pre sets it | 23:58 |
| melwitt | in the example I'm looking at, I'm seeing it set to ansible_user_dir https://review.opendev.org/#/c/548936/106/playbooks/post.yaml@26 | 23:58 |
| melwitt | oh wait that's a different role | 23:59 |
| melwitt | I see, ok | 23:59 |
| clarkb | melwitt: yup https://review.opendev.org/#/c/548936/106/playbooks/post.yaml@15 is where we need to set it I think | 23:59 |
| melwitt | yeah makes sense | 23:59 |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!