| *** mriedem has quit IRC | 00:06 | |
| clarkb | corvus: fungi the usb security device I talked about in china has been delayed to january due to a part sourcing shortage (I still have fingers crossed they will ship) | 00:07 |
|---|---|---|
| *** ociuhandu has joined #openstack-infra | 00:07 | |
| fungi | too bad | 00:07 |
| jlvillal | clarkb, Is that an alternative to a Yubikey or FIOD2 keys? | 00:09 |
| clarkb | jlvillal: ya let me get a link. Basically it does many of the things a yubikey does including FIDO2 but is all open source and has encrypted storage for general use too | 00:10 |
| * jlvillal likes his Yubikey and sites that support U2F | 00:10 | |
| *** slaweq has joined #openstack-infra | 00:10 | |
| jlvillal | clarkb, Ah cool. Some people like Nitrokey. | 00:10 |
| clarkb | jlvillal: me too. The biggest issue is that they stopped being open source. And all the open source options all do u2f and nothing else or everything else and not u2f | 00:11 |
| corvus | iirc it was also open hardware; could maybe go ahead and make one assuming the parts are available in small quantities :) | 00:11 |
| clarkb | nitrokey ^ has that problem | 00:11 |
| *** ociuhandu has quit IRC | 00:11 | |
| clarkb | https://www.crowdsupply.com/nth-dimension/signet-high-capacity | 00:11 |
| clarkb | corvus: yup they open sourced the hardware design too | 00:11 |
| jlvillal | clarkb, Thanks | 00:11 |
| clarkb | jlvillal: to me the two killer features of this one in addition to the openess are the u2f/fido + everything else as well as ability to write encrypted blocks arbitrarily | 00:12 |
| clarkb | of course they haven't shipped yet :/ | 00:12 |
| jlvillal | clarkb, Sounds good. Even though not opensource I do like the Yubikey 5 versions. I like that I can store TOTP tokens on them. | 00:13 |
| jlvillal | Plus the U2F. | 00:13 |
| clarkb | jlvillal: yup exactly. That is what nitrokey lacks too. But these will do both and more | 00:13 |
| jlvillal | clarkb, One thing I recommend is attempt to login to your email provider and see if you can do it without your token. | 00:14 |
| *** slaweq has quit IRC | 00:15 | |
| clarkb | jlvillal: ya, the worst one about that is actually my bank. They use some proprietary otp system and then also never ask me to use it because they remember some browser fingerprint (its not a cookie as using incognito mode exhibits this too) | 00:16 |
| jlvillal | clarkb, :( | 00:16 |
| clarkb | github is actually the best I think | 00:17 |
| jlvillal | I was surprised at first with my Gmail account as I thought I had got rid of my cell phone, but it was also in another part of my account so they offered to text a code to it :( | 00:17 |
| clarkb | they actually disable all other 2fa options if you enable a hard token device | 00:17 |
| clarkb | well except the copy list of otps | 00:17 |
| corvus | the cad files are from kicad -- so they used free software to design the hardware too :) | 00:17 |
| corvus | https://github.com/nthdimtech/signet-cad | 00:17 |
| *** ociuhandu has joined #openstack-infra | 00:18 | |
| jlvillal | And of course have multiple tokens as it would suck if only had one and it died... | 00:18 |
| corvus | or maybe https://github.com/nthdimtech/signet-hc-pcb is the new one | 00:18 |
| clarkb | jlvillal: yup, I buy in paris | 00:18 |
| clarkb | er not in france. Pairs | 00:19 |
| jlvillal | clarkb, Long way to go ;) | 00:19 |
| jlvillal | heh | 00:19 |
| *** slaweq has joined #openstack-infra | 00:19 | |
| clarkb | and ya google is happy to use devices. I'm not sure you can turn that off? would have to completely remove the device from your account instead probably | 00:19 |
| clarkb | When I manage to get my hands on these I'll be sure to let people know what I think of them | 00:20 |
| *** tosky has quit IRC | 00:22 | |
| *** slaweq has quit IRC | 00:24 | |
| *** ociuhandu has quit IRC | 00:32 | |
| ianw | clarkb: arrgh, well sshd now doesn't allow the login on that vos-release thing | 00:34 |
| ianw | i guess the easiest thing to do is to make a separate user than can sudo to run "vos release" | 00:35 |
| *** jamesmcarthur has joined #openstack-infra | 00:35 | |
| ianw | i was hoping the restricted wrapper script would work out easier | 00:35 |
| *** slaweq has joined #openstack-infra | 00:37 | |
| clarkb | why is it prevented? | 00:37 |
| ianw | sshd_config only allows root logins from bridge.o.o | 00:40 |
| ianw | not mirror-update | 00:41 |
| clarkb | oh! | 00:41 |
| ianw | which is sane, i was trying to take the kiss approach | 00:41 |
| clarkb | so ya I guess another user with vos release privs is simplest? | 00:41 |
| ianw | so it has to be root to run as -localauth, which is kind of the point :) | 00:42 |
| *** slaweq has quit IRC | 00:42 | |
| ianw | but can pass it through sudo | 00:42 |
| *** slaweq has joined #openstack-infra | 00:44 | |
| ianw | can you have nologin as a shell, if you're only calling a script via ssh? | 00:45 |
| clarkb | typically ssh would fork the shell, but does it fork the script directly or via the shell if you call it with a script? I guess that is the question | 00:46 |
| clarkb | ianw: reading sshd manpage I think the command is run without regard to the shell so its probably ok? | 00:48 |
| *** slaweq has quit IRC | 00:48 | |
| *** slaweq has joined #openstack-infra | 00:51 | |
| *** slaweq has quit IRC | 00:55 | |
| openstackgerrit | Ian Wienand proposed opendev/system-config master: vos-release: have separate user https://review.opendev.org/695331 | 00:55 |
| ianw | clarkb: ok... try again | 00:55 |
| *** slaweq has joined #openstack-infra | 00:57 | |
| clarkb | ianw: did you forget to git add the sudo config file? | 00:57 |
| ianw | yes :) | 01:01 |
| *** slaweq has quit IRC | 01:01 | |
| openstackgerrit | Ian Wienand proposed opendev/system-config master: vos-release: have separate user https://review.opendev.org/695331 | 01:01 |
| ianw | that's not quite right is it, the sudo has to run "vos" | 01:02 |
| *** slaweq has joined #openstack-infra | 01:03 | |
| openstackgerrit | Ian Wienand proposed opendev/system-config master: vos-release: have separate user https://review.opendev.org/695331 | 01:03 |
| *** ociuhandu has joined #openstack-infra | 01:04 | |
| *** ociuhandu has quit IRC | 01:08 | |
| *** slaweq has quit IRC | 01:13 | |
| *** slaweq has joined #openstack-infra | 01:17 | |
| *** roman_g has quit IRC | 01:17 | |
| *** slaweq has quit IRC | 01:21 | |
| fungi | okay, mirror.ubuntu-ports is now in sync and caught up. doing mirror.ubuntu-cloud next | 01:30 |
| openstackgerrit | Ian Wienand proposed openstack/diskimage-builder master: Introduce manual setting of DIB_INIT_SYSTEM https://review.opendev.org/414179 | 01:38 |
| openstackgerrit | Ian Wienand proposed openstack/diskimage-builder master: Add a Dockerfile and related jobs https://review.opendev.org/693971 | 01:38 |
| openstackgerrit | Ian Wienand proposed openstack/diskimage-builder master: Update bindep.txt for some missing dependencies https://review.opendev.org/693975 | 01:38 |
| openstackgerrit | Ian Wienand proposed openstack/diskimage-builder master: Dockerfile: install vhd-util https://review.opendev.org/693976 | 01:38 |
| openstackgerrit | Ian Wienand proposed openstack/diskimage-builder master: Start a buildset registry with docker image creation https://review.opendev.org/694175 | 01:38 |
| openstackgerrit | Ian Wienand proposed openstack/diskimage-builder master: Only wait for checksum processes https://review.opendev.org/695338 | 01:38 |
| ianw | mnaser: ^ ... worth a go | 01:38 |
| ianw | https://review.opendev.org/#/c/693464/20 will test it | 01:39 |
| fungi | mirror.ubuntu-cloud finished quickly, now doing mirror.debian | 01:39 |
| mnaser | thats "a solution" i guess :) | 01:39 |
| ianw | fungi: if you agree with https://review.opendev.org/#/c/695331/ hopefully that's the last word on vos-release and i get some of the mirror-update.opendev.org scripts using it to avoid the need for this | 01:43 |
| openstackgerrit | Brian Rosmaita proposed opendev/irc-meetings master: Update Cinder meeting information https://review.opendev.org/695339 | 01:48 |
| *** armax has quit IRC | 01:48 | |
| *** slaweq has joined #openstack-infra | 01:51 | |
| *** gyee has quit IRC | 01:52 | |
| *** ricolin has joined #openstack-infra | 01:55 | |
| *** slaweq has quit IRC | 01:55 | |
| *** slaweq has joined #openstack-infra | 01:58 | |
| *** larainema has joined #openstack-infra | 01:59 | |
| *** slaweq has quit IRC | 02:10 | |
| *** ociuhandu has joined #openstack-infra | 02:11 | |
| openstackgerrit | Merged opendev/system-config master: vos-release: have separate user https://review.opendev.org/695331 | 02:14 |
| *** ociuhandu has quit IRC | 02:16 | |
| *** slaweq has joined #openstack-infra | 02:23 | |
| *** slaweq has quit IRC | 02:30 | |
| *** slaweq has joined #openstack-infra | 02:31 | |
| *** slaweq has quit IRC | 02:35 | |
| openstackgerrit | Sundar Nadathur proposed opendev/irc-meetings master: Update Cyborg weekly IRC meeting info. https://review.opendev.org/695343 | 02:42 |
| *** xinranwang has joined #openstack-infra | 02:45 | |
| *** slaweq has joined #openstack-infra | 02:48 | |
| *** bdodd has quit IRC | 02:48 | |
| *** bdodd has joined #openstack-infra | 02:50 | |
| *** slaweq has quit IRC | 02:52 | |
| *** Lucas_Gray has joined #openstack-infra | 03:03 | |
| openstackgerrit | Ian Wienand proposed openstack/diskimage-builder master: Only wait for checksum processes https://review.opendev.org/695338 | 03:04 |
| openstackgerrit | Ian Wienand proposed openstack/diskimage-builder master: Add a Dockerfile and related jobs https://review.opendev.org/693971 | 03:04 |
| openstackgerrit | Ian Wienand proposed openstack/diskimage-builder master: Update bindep.txt for some missing dependencies https://review.opendev.org/693975 | 03:04 |
| openstackgerrit | Ian Wienand proposed openstack/diskimage-builder master: Dockerfile: install vhd-util https://review.opendev.org/693976 | 03:04 |
| openstackgerrit | Ian Wienand proposed openstack/diskimage-builder master: Start a buildset registry with docker image creation https://review.opendev.org/694175 | 03:04 |
| *** apetrich has quit IRC | 03:08 | |
| *** jamesmcarthur has quit IRC | 03:09 | |
| *** jamesmcarthur has joined #openstack-infra | 03:10 | |
| openstackgerrit | Ian Wienand proposed opendev/system-config master: vos-release: fix sudo https://review.opendev.org/695347 | 03:15 |
| *** jamesmcarthur has quit IRC | 03:15 | |
| *** Wryhder has joined #openstack-infra | 03:17 | |
| *** Lucas_Gray has quit IRC | 03:19 | |
| *** Wryhder is now known as Lucas_Gray | 03:19 | |
| *** rlandy|rover|bbl has quit IRC | 03:20 | |
| *** ociuhandu has joined #openstack-infra | 03:28 | |
| *** ociuhandu has quit IRC | 03:33 | |
| *** jamesmcarthur has joined #openstack-infra | 03:40 | |
| *** jamesmcarthur has quit IRC | 03:45 | |
| *** tonyb[m] has joined #openstack-infra | 03:50 | |
| *** udesale has joined #openstack-infra | 03:53 | |
| *** armax has joined #openstack-infra | 03:59 | |
| *** jamesmcarthur has joined #openstack-infra | 04:01 | |
| *** armax has quit IRC | 04:10 | |
| *** jamesmcarthur has quit IRC | 04:14 | |
| *** jamesmcarthur has joined #openstack-infra | 04:15 | |
| *** jamesmcarthur has quit IRC | 04:21 | |
| *** ramishra has joined #openstack-infra | 04:37 | |
| *** jamesmcarthur has joined #openstack-infra | 04:46 | |
| *** kjackal has joined #openstack-infra | 04:47 | |
| *** jamesmcarthur has quit IRC | 04:52 | |
| *** factor has joined #openstack-infra | 04:52 | |
| *** surpatil has joined #openstack-infra | 05:00 | |
| *** raukadah is now known as chandankumar | 05:09 | |
| *** igordc has quit IRC | 05:11 | |
| *** Lucas_Gray has quit IRC | 05:22 | |
| *** ociuhandu has joined #openstack-infra | 05:30 | |
| *** bdodd has quit IRC | 05:34 | |
| *** Tergel has joined #openstack-infra | 05:35 | |
| *** ociuhandu has quit IRC | 05:35 | |
| *** bdodd has joined #openstack-infra | 05:37 | |
| fungi | mirror.debian is done, mirror.debian-security underway | 05:58 |
| Tergel | Hello everyone, I am from Mongolia. We have faced an issue related to Ceph. First, let me briefly explain our issue. We have deployed Openstack by kolla-ansible, and we use ceph cluster which consist of 3 nodes. One node has 12x2TB drives and 32GB RAM. According to the Ceph's site, it is enough RAM for one node. However, the memory usage is constantly increasing until the 32GB RAM is completely used. After that, the node is failed and removed | 06:05 |
| Tergel | temporarily from the cluster due to the memory shortage, then it is recovered automatically and join to the cluster. It seems they cannot clear old caches of memory. Is there anyone who faced it before? Why do these nodes not clear the old caches? If you help me to solve it, it would be great. :) | 06:05 |
| *** xinranwang has quit IRC | 06:15 | |
| *** udesale has quit IRC | 06:15 | |
| *** udesale has joined #openstack-infra | 06:16 | |
| fungi | Tergel: you seem to have found the wrong channel. this is where we coordinate running code review and ci systems, mailing list servers, and similar sorts of collaboration tools for the community | 06:19 |
| fungi | Tergel: https://wiki.openstack.org/wiki/IRC has a list of known project channels, you may be able to find a more appropriate one there | 06:20 |
| fungi | maybe start with #openstack-kolla | 06:20 |
| *** lpetrut has quit IRC | 06:21 | |
| *** ociuhandu has joined #openstack-infra | 06:25 | |
| *** slaweq has joined #openstack-infra | 06:39 | |
| *** udesale has quit IRC | 06:39 | |
| *** Tengu has quit IRC | 06:39 | |
| *** udesale has joined #openstack-infra | 06:39 | |
| *** dpawlik has joined #openstack-infra | 06:41 | |
| *** Tengu has joined #openstack-infra | 06:41 | |
| *** slaweq has quit IRC | 06:44 | |
| *** Tergel has quit IRC | 06:45 | |
| *** Tergel has joined #openstack-infra | 06:46 | |
| *** ociuhandu has quit IRC | 06:46 | |
| *** jamesmcarthur has joined #openstack-infra | 06:48 | |
| openstackgerrit | Ian Wienand proposed opendev/system-config master: vos-release: fix sudo https://review.opendev.org/695347 | 06:51 |
| *** jamesmcarthur has quit IRC | 06:52 | |
| *** factor has quit IRC | 07:01 | |
| *** ociuhandu has joined #openstack-infra | 07:05 | |
| *** xek_ has joined #openstack-infra | 07:10 | |
| *** rcernin has quit IRC | 07:12 | |
| Tergel | Thanks @fungi | 07:21 |
| *** chandankumar is now known as chkumar|rover | 07:22 | |
| *** pcaruana has joined #openstack-infra | 07:27 | |
| *** jamesmcarthur has joined #openstack-infra | 07:28 | |
| *** tosky has joined #openstack-infra | 07:31 | |
| *** jaicaa has quit IRC | 07:34 | |
| *** apetrich has joined #openstack-infra | 07:34 | |
| *** jamesmcarthur has quit IRC | 07:34 | |
| *** jaicaa has joined #openstack-infra | 07:37 | |
| *** ociuhandu has quit IRC | 07:39 | |
| *** pgaxatte has joined #openstack-infra | 07:44 | |
| *** kopecmartin|off is now known as kopecmartin | 07:49 | |
| *** surpatil has quit IRC | 07:55 | |
| *** jklare has quit IRC | 07:57 | |
| *** kjackal has quit IRC | 07:59 | |
| *** jklare has joined #openstack-infra | 07:59 | |
| *** ociuhandu has joined #openstack-infra | 08:00 | |
| *** ociuhandu has quit IRC | 08:05 | |
| *** surpatil has joined #openstack-infra | 08:11 | |
| *** jamesmcarthur has joined #openstack-infra | 08:15 | |
| *** iurygregory has joined #openstack-infra | 08:16 | |
| *** dchen has quit IRC | 08:20 | |
| *** jamesmcarthur has quit IRC | 08:20 | |
| *** rpittau|afk is now known as rpittau | 08:20 | |
| *** Tergel has quit IRC | 08:27 | |
| *** tkajinam has quit IRC | 08:28 | |
| *** jpena|off is now known as jpena | 08:35 | |
| *** gfidente has joined #openstack-infra | 08:36 | |
| *** iurygregory has quit IRC | 08:40 | |
| *** ociuhandu has joined #openstack-infra | 08:44 | |
| *** slaweq has joined #openstack-infra | 08:44 | |
| *** ralonsoh has joined #openstack-infra | 08:45 | |
| *** ociuhandu has quit IRC | 08:46 | |
| *** ociuhandu has joined #openstack-infra | 08:47 | |
| *** jamesmcarthur has joined #openstack-infra | 08:47 | |
| *** mkolesni has joined #openstack-infra | 08:49 | |
| *** ociuhandu has quit IRC | 08:50 | |
| mkolesni | mnaser, ping | 08:50 |
| *** ociuhandu has joined #openstack-infra | 08:50 | |
| dulek | Hi! Apparently Zuul haven't picked up patches on kuryr and kuryr-libnetwork: https://review.opendev.org/#/c/694449 and https://review.opendev.org/#/c/694443/. | 08:51 |
| AJaeger | mkolesni: please don't a naked ping - instead leave a message and somebody might read backscroll and help. | 08:51 |
| AJaeger | dulek: let me check the first quickly | 08:51 |
| AJaeger | dulek: the first has a depends-on. Is that one merged and in the same queue? | 08:51 |
| dulek | AJaeger: Oh my. Sorry for the trouble. | 08:52 |
| AJaeger | ;) | 08:52 |
| * dulek needs another coffee, it seems. | 08:52 | |
| mkolesni | AJaeger, sure, just reaching out to mnaser since he offered his help on the IRC for adding submariner to opendev.org | 08:52 |
| AJaeger | good idea - me grabs one now as well... | 08:52 |
| *** jamesmcarthur has quit IRC | 08:52 | |
| AJaeger | mkolesni: https://docs.openstack.org/infra/manual/creators.html explains the setup, please read it first - and mnaser can answer questions later. NOte he's in CAnada, so hopefully asleep right now ;) | 08:53 |
| mkolesni | I read the project creator's guide which still is quite OpenStack centric and wanted to make sure i understand the process correctly | 08:54 |
| mkolesni | Yeah I figured he's probably not in my timezone anyway :) | 08:54 |
| mkolesni | So we're thinking of moving to opendev.org with submariner since, IMHO (and agreed by everyone that knows Gerrit), GitHub sucks | 08:55 |
| mkolesni | Seems like opendev.org is a good place with friendly people, and many of us already have prior experience from OpenStack, so it all seems quite natural | 08:56 |
| mkolesni | What's unnatural is that Submariner is not related to OpenStack in any way (well unless you run your K8s on top of OpenStack I guess) so we'd like to know we're welcome here :) | 08:57 |
| *** ociuhandu has quit IRC | 08:57 | |
| *** iurygregory has joined #openstack-infra | 08:57 | |
| mkolesni | Also we currently have a quite elaborate CI with Travis on top of GitHub and wanted to know would it be possible to have it run and vote on the patches on opendev.org (with a possible Zuul migration in the future) | 08:58 |
| *** lourot has joined #openstack-infra | 08:58 | |
| AJaeger | mkolesni: yes, you're welcome ;). I don't think CI with travis will work but others might know more here. | 09:01 |
| mkolesni | AJaeger, i'm no expert on CI but our CI guy told me that if gerrit can trigger it with a hook it should work, question is if that's even possible to set up | 09:02 |
| mkolesni | I know 3rd party CI is supported but not sure what's the mechanism there as I never needed to study it, but I'm guessing there's a wiki for it? | 09:03 |
| *** priteau has joined #openstack-infra | 09:03 | |
| AJaeger | mkolesni: the rest of the team need to help out on that question. The 3rd party CI information is in the infra manual as well, but outdated ;( | 09:03 |
| mkolesni | OK then, would it be possible to evaluate (i.e some staging server) or is the way forward to request the project creation and roll with that? | 09:05 |
| *** lucasagomes has joined #openstack-infra | 09:08 | |
| AJaeger | mkolesni: I suggest you discuss with rest of team during US hours later, figure out CI, and decide whether the setup will work for you. Then request project creation according to the Creator's Guide. | 09:08 |
| mkolesni | OK sure, thanks | 09:09 |
| *** pkopec has joined #openstack-infra | 09:11 | |
| *** sshnaidm|ruck is now known as sshnaidm | 09:19 | |
| *** ricolin has quit IRC | 09:22 | |
| *** dpawlik has quit IRC | 09:30 | |
| *** dpawlik has joined #openstack-infra | 09:35 | |
| openstackgerrit | Merged opendev/irc-meetings master: Update Cinder meeting information https://review.opendev.org/695339 | 09:46 |
| openstackgerrit | Merged opendev/irc-meetings master: Update Cyborg weekly IRC meeting info. https://review.opendev.org/695343 | 09:46 |
| *** derekh has joined #openstack-infra | 09:53 | |
| *** ociuhandu has joined #openstack-infra | 09:54 | |
| *** ociuhandu has quit IRC | 09:57 | |
| *** ociuhandu has joined #openstack-infra | 09:58 | |
| *** sshnaidm has quit IRC | 10:01 | |
| *** ociuhandu has quit IRC | 10:03 | |
| *** derekh has quit IRC | 10:04 | |
| *** sshnaidm has joined #openstack-infra | 10:05 | |
| *** liuyulong has quit IRC | 10:06 | |
| *** priteau has quit IRC | 10:07 | |
| *** derekh has joined #openstack-infra | 10:09 | |
| *** priteau has joined #openstack-infra | 10:12 | |
| *** roman_g has joined #openstack-infra | 10:14 | |
| *** jamesmcarthur has joined #openstack-infra | 10:49 | |
| *** dtantsur|afk is now known as dtantsur | 10:49 | |
| iurygregory | morning all o/ | 10:51 |
| *** derekh has quit IRC | 10:52 | |
| iurygregory | quick question, we have started the work to drop py27 in ironic, but we are facing an issue with the check-requirements job that we don't match the lines on openstack/requirements https://review.opendev.org/#/c/695213/2 | 10:53 |
| iurygregory | I saw other projects dropping things from doc/requirements ant the `requirements-check` job was green for example https://review.opendev.org/#/c/694443/2 | 10:53 |
| *** jamesmcarthur has quit IRC | 10:53 | |
| AJaeger | iurygregory: that change includes "Depends-On: https://review.opendev.org/#/c/693631/" - you need that change first | 10:57 |
| AJaeger | oh, that merged - let's check timing... | 10:57 |
| AJaeger | 631 merged 2 hours ago, so a "recheck" on your "ancient" change will magically make it work - I hope ;) | 10:58 |
| iurygregory | AJaeger, tks! | 10:58 |
| sshnaidm | mordred, hi | 10:59 |
| sshnaidm | mordred, wanted to sync with you and dtantsur wrt https://review.opendev.org/#/c/694751/ and OS ansible modules, as I understood there will be some changes | 11:00 |
| dtantsur | sshnaidm: assuming he's on kubecon, an email might work better | 11:02 |
| sshnaidm | dtantsur, ack | 11:05 |
| *** rpittau is now known as rpittau|bbl | 11:11 | |
| *** dpawlik has quit IRC | 11:13 | |
| *** udesale has quit IRC | 11:14 | |
| *** rfolco has joined #openstack-infra | 11:18 | |
| *** rfolco is now known as rfolco|doctor | 11:18 | |
| *** ociuhandu has joined #openstack-infra | 11:18 | |
| *** ociuhandu has quit IRC | 11:23 | |
| *** jamesmcarthur has joined #openstack-infra | 11:25 | |
| *** apetrich has quit IRC | 11:29 | |
| *** jamesmcarthur has quit IRC | 11:31 | |
| *** ramishra has quit IRC | 11:38 | |
| *** dpawlik has joined #openstack-infra | 11:48 | |
| *** Lucas_Gray has joined #openstack-infra | 11:52 | |
| *** surpatil has quit IRC | 11:52 | |
| *** dpawlik has quit IRC | 11:52 | |
| openstackgerrit | Frode Nordahl proposed openstack/project-config master: Add gerritbot trigger for microstack https://review.opendev.org/695401 | 11:56 |
| *** apetrich has joined #openstack-infra | 11:59 | |
| *** Lucas_Gray has quit IRC | 12:00 | |
| *** jamesmcarthur has joined #openstack-infra | 12:01 | |
| *** Lucas_Gray has joined #openstack-infra | 12:04 | |
| *** ociuhandu has joined #openstack-infra | 12:05 | |
| *** ociuhandu has quit IRC | 12:11 | |
| *** derekh has joined #openstack-infra | 12:12 | |
| *** jpena is now known as jpena|lunch | 12:14 | |
| *** mriedem has joined #openstack-infra | 12:21 | |
| openstackgerrit | Frode Nordahl proposed openstack/project-config master: Add gerritbot trigger for microstack https://review.opendev.org/695401 | 12:21 |
| *** ricolin has joined #openstack-infra | 12:27 | |
| *** slaweq has quit IRC | 12:39 | |
| openstackgerrit | Witold Bedyk proposed opendev/irc-meetings master: Update Monasca Team Meeting time https://review.opendev.org/695418 | 12:44 |
| *** dpawlik has joined #openstack-infra | 12:48 | |
| *** jamesmcarthur has quit IRC | 12:50 | |
| *** jamesmcarthur_ has joined #openstack-infra | 12:50 | |
| *** efried has joined #openstack-infra | 12:52 | |
| efried | o/ infra | 12:52 |
| efried | Lately, any time I try to open zuul.o.o/jobs, the page freezes my browser. If I leave it long enough it actually hard hangs my whole OS (bionic). Is this a known problem? | 12:53 |
| efried | I run chrome, and when it doesn't work there I try firefox, so it's not clear to me which of them is actually hanging the OS. | 12:54 |
| efried | and then of course when I reboot I forget to bring it up here, and then enough time passes that I forget that it's so severe and I try it again :( | 12:55 |
| *** olivierbourdon38 has joined #openstack-infra | 12:56 | |
| *** olivierbourdon38 has quit IRC | 12:58 | |
| *** jklare has quit IRC | 12:58 | |
| *** rlandy has joined #openstack-infra | 12:58 | |
| *** jklare has joined #openstack-infra | 13:02 | |
| openstackgerrit | Matt Riedemann proposed opendev/elastic-recheck master: Add query for test_shelve_volume_backed_instance bug 1853453 https://review.opendev.org/695423 | 13:02 |
| openstack | bug 1853453 in tempest "test_shelve_volume_backed_instance intermittently fails guest ssh with dhcp lease fail" [Undecided,New] https://launchpad.net/bugs/1853453 | 13:02 |
| AJaeger | efried: at least reproduceable - for me. Let's tell corvus about it... | 13:03 |
| efried | AJaeger: phew, at least it's not just me | 13:04 |
| *** rpittau|bbl is now known as rpittau | 13:12 | |
| *** slaweq has joined #openstack-infra | 13:17 | |
| fungi | AJaeger: efried: that happened a few days ago too, prognosis uncertain, it seemed to clear up on its own at some point. i'm going to see if i can grab a snapshot of the status api output | 13:18 |
| *** jpena|lunch is now known as jpena | 13:20 | |
| efried | fungi: "seemed to clear up on its own" isn't super comforting if it's like a client-side memory leak... | 13:20 |
| efried | Anyway, thanks for looking into it. | 13:20 |
| AJaeger | efried: come here for group conselling sessions, we share the same trouble ;) | 13:20 |
| AJaeger | counselling? or how do I spell that? | 13:21 |
| fungi | efried: yeah, or some sort of infinite loop in the javascript triggered by a particular pattern in the data | 13:21 |
| efried | "Hi, my name is efried, and I have a client-side memory leak" | 13:21 |
| efried | AJaeger: the latter, though I think either one L or two is acceptable :) | 13:21 |
| * AJaeger hugs efried | 13:21 | |
| fungi | anyway, i snagged the content of https://zuul.opendev.org/api/tenant/openstack/status | 13:22 |
| fungi | it's ~1.5mb right now | 13:22 |
| efried | fungi: my repro was day before yesterday. Not sure when AJaeger's was. | 13:22 |
| efried | (see, by the time I reboot I always forget to report it) | 13:23 |
| fungi | oof, 1214 items in the gate pipeline | 13:23 |
| fungi | i wonder if that's related to the problem | 13:23 |
| *** rfolco|doctor is now known as rfolco | 13:23 | |
| *** armax has joined #openstack-infra | 13:23 | |
| openstackgerrit | Merged opendev/elastic-recheck master: Add query for test_shelve_volume_backed_instance bug 1853453 https://review.opendev.org/695423 | 13:24 |
| openstack | bug 1853453 in tempest "test_shelve_volume_backed_instance intermittently fails guest ssh with dhcp lease fail" [Undecided,New] https://launchpad.net/bugs/1853453 | 13:24 |
| fungi | a vast majority of the entries in there are headless too | 13:25 |
| *** hashar has joined #openstack-infra | 13:28 | |
| *** pgaxatte has quit IRC | 13:30 | |
| AJaeger | https://zuul.opendev.org/api/tenant/openstack/status or https://zuul.opendev.org/api/tenant/openstack/jobs ? | 13:30 |
| AJaeger | wow, /jobs has 3204 entries | 13:31 |
| fungi | ahh, yep, i misunderstood and thought you said /status | 13:31 |
| AJaeger | and /jobs is "only" 730 k... | 13:31 |
| *** hashar has quit IRC | 13:32 | |
| AJaeger | fungi: and I thought you were one step ahead in debugging already ;) | 13:32 |
| *** yolanda has quit IRC | 13:33 | |
| fungi | okay, my mistake. the status dashboard does not seem to be involved in this (though having 1214 entries in the gate pipeline's json does seem strange) | 13:33 |
| fungi | i should read more carefully | 13:33 |
| fungi | so this is /jobs again | 13:33 |
| tristanC | fungi: iirc gates has one entry per queue, even if there are no jobs | 13:34 |
| fungi | tristanC: thanks, that explains my confusion there (though does seem inefficient, there's probably a reason for it) | 13:34 |
| fungi | so yeah, maybe it's just a matter of the dashboard's javascript having trouble rendering all 3k+ entries from https://zuul.opendev.org/api/tenant/openstack/jobs | 13:35 |
| *** yolanda has joined #openstack-infra | 13:36 | |
| AJaeger | let's kill some more jobs ;) | 13:36 |
| tristanC | fungi: it shouldn't though. Perhaps there is an issue with the data, e.g. circular job parenting. though the js should already detect and prevent that | 13:36 |
| fungi | yeah, it doesn't seem like an insane number of entries, you're right | 13:37 |
| fungi | unfortunately my broswer won't let me fire up the js debugger on it until the page loads completely | 13:37 |
| *** udesale has joined #openstack-infra | 13:37 | |
| fungi | (which it won't since my computer will fall over before it ever completes) | 13:37 |
| openstackgerrit | Andreas Jaeger proposed openstack/openstack-zuul-jobs master: Remove legacy-devstack-dsvm-updown job https://review.opendev.org/695434 | 13:38 |
| openstackgerrit | Matt Riedemann proposed opendev/elastic-recheck master: Update query for bug 1853453 https://review.opendev.org/695435 | 13:38 |
| openstack | bug 1853453 in tempest "test_shelve_volume_backed_instance intermittently fails guest ssh with dhcp lease fail" [Undecided,New] https://launchpad.net/bugs/1853453 | 13:38 |
| AJaeger | fungi: only one job - frickler, FYI ^ | 13:38 |
| *** aarents has joined #openstack-infra | 13:39 | |
| fungi | so with a fresh start and no content loaded, firefox 70 has allocated 2819580 virtual/289624 resident memory (i'll set aside for now how insane that is on its own) | 13:41 |
| fungi | now to load https://zuul.opendev.org/t/openstack/jobs and see what happens | 13:41 |
| frickler | AJaeger: ah, thanks, I wanted to look into doing a replacement, but got distracted. I'm wondering why that ran on heat+sahara, maybe tosky can tell | 13:41 |
| AJaeger | frickler: might have been removed from heat+sahara some time ago already... | 13:42 |
| fungi | definitely memory, i can now see a "Web content" subprocess eating 2gb resident memory and climbing | 13:43 |
| fungi | consuming 100% of cpu too | 13:43 |
| fungi | now up to 3gb resident | 13:44 |
| frickler | AJaeger: I meant "with heat+sahara enabled", not specifically against those projects, sorry for being unclear | 13:44 |
| fungi | i don't think this is a memory "leak" more of a runaway process/allocation | 13:44 |
| tosky | frickler, AJaeger : what, where? | 13:44 |
| frickler | tosky: do you know why the (now-to-be-removed) legacy-devstack-dsvm-updown job enables heat+sahara? | 13:45 |
| tosky | (that zuul page kills my browser too, I think I mentioned it here some time ago) | 13:45 |
| frickler | tosky: also, is that something that you'd want to keep seeing tested in a replacement zuulv3 job? | 13:46 |
| tosky | frickler: uhm, from which repository? I don't know - maybe it was part of the "second tier" of tests? I remember a job which enabled sahara and trove in tempest long ago | 13:46 |
| tosky | back when it was important for interoperability of graduated projects | 13:46 |
| frickler | tosky: it was defined in openstack-zuul-jobs but running only in devstack | 13:46 |
| frickler | tosky: see https://review.opendev.org/695434 | 13:47 |
| *** pgaxatte has joined #openstack-infra | 13:47 | |
| *** kjackal has joined #openstack-infra | 13:48 | |
| fungi | the browser subprocess consumed all 8gb of my physical memory on this system and started eating into swap, paged out several gb more showing no signs of slowing so i finally killed it | 13:49 |
| openstackgerrit | Matthieu Huin proposed zuul/zuul master: authentication config: add optional token_expiry https://review.opendev.org/642408 | 13:51 |
| tosky | I was trying to check the story of that job (and its ancestor gate-devstack-dsvm-updown) | 13:51 |
| openstackgerrit | Matt Riedemann proposed opendev/elastic-recheck master: Update query for bug 1853453 https://review.opendev.org/695435 | 13:54 |
| openstack | bug 1853453 in tempest "test_shelve_volume_backed_instance intermittently fails guest ssh with dhcp lease fail" [Undecided,New] https://launchpad.net/bugs/1853453 | 13:54 |
| efried | fungi: that seems consistent with what I observed, though anecdotally (I didn't get a chance to watch phys/swap, but the behavior of my fan and disk light were what I would expect for that) | 13:55 |
| *** Jeremy_H has joined #openstack-infra | 14:00 | |
| *** kjackal has quit IRC | 14:00 | |
| Jeremy_H | Hello, got a wonderful problem thats been plaguing me since last night in my zuul pipeline - If anyone could lend a hand in figuring out why zuul cant install the same requirements its been installing for several months now, I would greatly appreciate it | 14:01 |
| Jeremy_H | https://d2ba61b055381755abe6-7d59354a7b453d58914b817150958c54.ssl.cf5.rackcdn.com/695311/6/check/openstack-tox-py36/9206d00/job-output.txt | 14:01 |
| Jeremy_H | grep for line 2019-11-21 13:58:03.578295 | ubuntu-bionic | py36 create: /home/zuul/src/opendev.org/x/ranger/.tox/py36 | 14:01 |
| AJaeger | Jeremy_H: you can link to that line ;) | 14:01 |
| Jeremy_H | I dont know how to link to the line, sorry | 14:02 |
| AJaeger | Jeremy_H: if you use the proper URL, let me show you... | 14:02 |
| Jeremy_H | dope I love to learn | 14:02 |
| AJaeger | Jeremy_H: go to https://zuul.opendev.org/t/openstack/build/c119f53468cf4ecb8829d78d55dc06d6 and scroll down - and see the error already | 14:03 |
| AJaeger | "/usr/bin/ld: cannot find -lssl" | 14:03 |
| AJaeger | You need a bindep file to add the missing libs - that happens if there are no prebuild wheels for one python lib | 14:03 |
| AJaeger | Jeremy_H: I guess you wanted to link to https://zuul.opendev.org/t/openstack/build/c119f53468cf4ecb8829d78d55dc06d6/log/job-output.txt#875 , didn't you? | 14:04 |
| Jeremy_H | Yep! | 14:04 |
| AJaeger | from the URL I gave you, access the file itself - not the raw link | 14:04 |
| fungi | odds are whatever is being compiled there (i haven't looked at the log) used to get installed from a prebuilt wheel and is now being installed via an sdist instead | 14:04 |
| AJaeger | Jeremy_H: so, add a bindep file that includes the missing packages, see what others have in bindep.txt | 14:04 |
| fungi | either because of switching to a different python interpreter version or because a new release of it was pushed to pypi | 14:04 |
| Jeremy_H | that makes total sense. I will see what I can do. Thank you both so much for the prompt assistance! | 14:05 |
| *** Lucas_Gray has quit IRC | 14:06 | |
| *** Goneri has quit IRC | 14:09 | |
| *** factor has joined #openstack-infra | 14:09 | |
| *** jamesmcarthur_ has quit IRC | 14:10 | |
| *** ramishra has joined #openstack-infra | 14:13 | |
| *** dpawlik has quit IRC | 14:15 | |
| Jeremy_H | worked like a charm! | 14:19 |
| *** jamesmcarthur has joined #openstack-infra | 14:20 | |
| openstackgerrit | Matt Riedemann proposed opendev/elastic-recheck master: Update query for bug 1853453 https://review.opendev.org/695435 | 14:20 |
| openstack | bug 1853453 in tempest "test_shelve_volume_backed_instance intermittently fails guest ssh with dhcp lease fail" [Undecided,New] https://launchpad.net/bugs/1853453 | 14:20 |
| Jeremy_H | added libssl-dev and openssl-devel to bindep.txt | 14:20 |
| *** Jeremy_H has quit IRC | 14:27 | |
| openstackgerrit | Matthieu Huin proposed zuul/zuul master: enqueue: make trigger optional https://review.opendev.org/695446 | 14:28 |
| fungi | Jeremy_H: i suspect the reason it cropped up is related to the new mysqlclient sdist uploaded to pypi roughly two hours ago; they don't provide any linux wheels, but it's so far unclear to me why you didn't see it with earlier mysqlclient versions | 14:28 |
| fungi | looks like earlier builds of mysqlclient (for the older 1.4.0 release) didn't complain about needing to link libssl: https://zuul.opendev.org/t/openstack/build/48d150ff49454ef59a05024ad90894bf/log/tox/py36-1.log#391-394 | 14:31 |
| *** tosky_ has joined #openstack-infra | 14:31 | |
| *** tosky has quit IRC | 14:32 | |
| *** jtomasek has quit IRC | 14:32 | |
| *** tosky_ is now known as tosky | 14:33 | |
| *** jtomasek has joined #openstack-infra | 14:35 | |
| fungi | er, for 1.4.5 i mean. 1.4.6 is what was just released | 14:36 |
| fungi | #status log etherpad.openstack.org was offline between 01:55 and 03:00 utc due to a hypervisor host issue, provider incident id CSHD-53e26fd0 (no trouble ticket reference was provided this time) | 14:42 |
| openstackstatus | fungi: finished logging | 14:42 |
| openstackgerrit | Merged opendev/elastic-recheck master: Update query for bug 1853453 https://review.opendev.org/695435 | 14:42 |
| openstack | bug 1853453 in tempest "test_shelve_volume_backed_instance intermittently fails guest ssh with dhcp lease fail" [Undecided,New] https://launchpad.net/bugs/1853453 | 14:42 |
| *** fdegir has quit IRC | 14:42 | |
| *** fdegir has joined #openstack-infra | 14:43 | |
| *** goldyfruit has joined #openstack-infra | 14:46 | |
| mnaser | mkolesni: oh darn, timezones are gonna be hard for this one, im up late usually so that might be a much better timeline | 14:46 |
| mnaser | also, it looks like the new jobs seem to have merged -- https://zuul.opendev.org/t/openstack/build/7c6a2ba514894becbdcfba13e3f616e9 | 14:47 |
| mnaser | how can we test this? | 14:47 |
| mkolesni | mnaser, no prob | 14:47 |
| openstackgerrit | James E. Blair proposed zuul/zuul-jobs master: WIP: use-buildset-registry: add podman support https://review.opendev.org/695051 | 14:48 |
| mnaser | mkolesni: its probably pretty late for you but we can chat later tonight (for me) and we can start with the initial proposal of getting those repos/etc :) | 14:48 |
| mkolesni | so now that there's more people here i wanted to know if it would be possible to trigger our existing Travis CI jobs from the opendev gerrit? | 14:48 |
| mkolesni | mnaser, i just wanted to clarify a few things :) | 14:49 |
| openstackgerrit | Tristan Cacqueray proposed zuul/zuul master: web: only display the first parent in the jobs tree https://review.opendev.org/695450 | 14:49 |
| jrosser | ianw: re the arm stuff on the ML, are there any prebuilt arm wheels? | 14:50 |
| mnaser | mkolesni: i think the hard part is figuring out how travis ci could report back to gerrit | 14:50 |
| mkolesni | mnaser, i understood its possible but maybe im being given false info :) | 14:51 |
| mnaser | mkolesni: it would probably be good to instead start by migrating to zuul (while still being on github), then migrating to gerrit -- that might be a more native (but a bit more involving process) | 14:51 |
| openstackgerrit | Matthieu Huin proposed zuul/zuul master: enqueue: make trigger optional https://review.opendev.org/695446 | 14:51 |
| *** rosmaita has quit IRC | 14:51 | |
| mnaser | mkolesni: it might be, how did you see the possibility of it happening? | 14:52 |
| mkolesni | we were thinking of doing a POC to assess the possibility of moving.. if there's some staging env where we can try it first would be awesome, otherwise i guess we can request the repo creation and work from there? | 14:53 |
| mkolesni | mnaser, well our CI guy told me that travis could also post it's results to gerrit, but im no CI expert so further than that i can't say.. hence a POC would be a good idea | 14:54 |
| *** tosky has quit IRC | 14:54 | |
| mkolesni | another possibility is some sort of self hosted gerrit for POC but that could be non-trivial to set up to act similar to opendev, i reckon | 14:55 |
| mkolesni | if we go the zuul migration route first, then how would we get zuul to post it's results on github? | 14:56 |
| mnaser | mkolesni: we already have that infrastructure in place, a few OSF projects like kata containers do get github reporting :) | 14:56 |
| *** tosky has joined #openstack-infra | 14:57 | |
| mnaser | and we do third party testing about ansible too | 14:57 |
| mkolesni | ah ok interesting so we can look at their settings to get an idea of how to set that up? | 14:58 |
| mnaser | yep, step 1 would be getting the app installed in your github repo, and then step 2 would be getting zuul config change up | 14:58 |
| mkolesni | which app would that be? | 14:59 |
| *** igordc has joined #openstack-infra | 15:01 | |
| *** pcaruana has quit IRC | 15:02 | |
| mnaser | mkolesni: https://docs.openstack.org/infra/system-config/zuul.html#github-projects | 15:02 |
| corvus | mnaser, mkolesni: we don't really support opendev doing first-party ci on github | 15:03 |
| *** rosmaita has joined #openstack-infra | 15:03 | |
| mnaser | corvus: i think the plan is to start doing zuul migration (to be able to move to gerrit afterwards) | 15:04 |
| mnaser | the project currently uses travis ci and moving to gerrit first would mean they would have no CI | 15:04 |
| mkolesni | the CI migration is really my biggest technical concern | 15:04 |
| *** ociuhandu has joined #openstack-infra | 15:04 | |
| mnaser | i would agree with you however, if the project doesn't intend to move to gerrit eventually, this is a different conversation | 15:04 |
| mnaser | but it seems like the intention is to fully migrate to opendev | 15:05 |
| *** igordc has quit IRC | 15:05 | |
| fungi | mnaser: the opendev position on integrating with github is outlined at http://lists.openstack.org/pipermail/openstack-infra/2019-January/006269.html | 15:05 |
| fungi | just for clarificaiton | 15:06 |
| *** igordc has joined #openstack-infra | 15:06 | |
| mkolesni | the intent is to migrate but first assess the required work and also demonstrate the value to the community | 15:06 |
| fungi | opendev provides an integrated code review and project gating solution, via a combination of gerrit and zuul. those two components aren't really separable in opendev, though anyone could run them standalone as they're both free software | 15:07 |
| mkolesni | there is no intent of switching just the ci | 15:07 |
| mnaser | so what do we do in the case of someone like mkolesni who's trying to on-board? | 15:07 |
| fungi | it's a great question, i think we need to discuss the challenge and identify solutions | 15:07 |
| mkolesni | fungi, yeah we're quite farmiliar with opendev before the rename :) | 15:07 |
| corvus | what kind of ci jobs are we talking about? how much of a challenge would it be to prepare them for a flag-day move? | 15:08 |
| mnaser | my plan of thought was: setup github app against that project, get both travis and zuul working (so you know you have working jobs), then migrate to gerrit | 15:08 |
| mkolesni | theres a few but i think its better if we can schedule a discussion when our ci guy can be present and answer these questions | 15:09 |
| mkolesni | or possibly raise it on the irc weekly? (idk what time you have it) | 15:09 |
| corvus | another option we could consider would be to import the repos, get the ci working on gerrit while continuing work on github, then force-push an update to the repos in gerrit from github when we're ready | 15:09 |
| *** ociuhandu has quit IRC | 15:09 | |
| mnaser | ah yes, i like that idea too corvus , i was trying to think of the options that didnt involve $superpowers :) | 15:10 |
| fungi | as far as the idea of having travis-ci report on changes in opendev's gerrit, it seems like the hardest part technically would be hooking travis up to gerrit's event stream (which is how our other third-party ci systems trigger their jobs) | 15:10 |
| mnaser | fungi: i had a terrible silly idea that a zuul job which calls the travis ci hook and waits for it to finish and report back but i will pretend i never ever wrote that | 15:10 |
| corvus | mnaser: the prep-on-github idea is a reasonable option, and i think we should consider it. i brought up the opendev github support issue mostly because it would be an exception to our current practice/policy, so i didn't want to presume that would be free of issues. but of course, we can consider exceptions. | 15:11 |
| fungi | well, i suppose a zuul job could call a web hook somewhere, we have plenty like that already. i wouldn't want folks to have those jobs busywait and then rely on that as primary gating though | 15:11 |
| corvus | so i think we've got at least 2 good ideas, and 2 less good ones :) | 15:11 |
| mnaser | hit a webhook and poll it is the good one, right? :) | 15:12 |
| corvus | mnaser: that is totally the best one | 15:12 |
| fungi | but i do find the suggestion of having lightweight zuul jobs actively notify third-party ci systems intriguing | 15:12 |
| mnaser | i mean, its not far of what zuul used to be a while back :p | 15:13 |
| mnaser | technically speaking, it wouldnt be that bad IMHO, if we manage to do it inside the executor directly without touching a node | 15:13 |
| *** igordc has quit IRC | 15:14 | |
| *** igordc has joined #openstack-infra | 15:14 | |
| fungi | sort of? zuul added requests into a gearman queue and then waited for a jenkins plugin to pick up those requests and act on them, so doesn't seem like the same model to me | 15:14 |
| corvus | mnaser: it's incapable of actually testing the repo correctly like zuul, but other than that, yes, it will totally work and produce a green or red circle. :) | 15:14 |
| mkolesni | is it a problem that the thing is in golang (wrt zuul)? | 15:14 |
| mkolesni | i know most if not all openstack projects are python | 15:15 |
| corvus | mkolesni: not at all -- in fact, some folks from volvo have been working on a library of go jobs for zuul: https://review.opendev.org/691114 | 15:15 |
| mkolesni | huh interesting stuff | 15:15 |
| fungi | zuul doesn't care what you run. anything you can run from ansible (which includes arbitrary shell commands) should be usable in a zuul job | 15:16 |
| mkolesni | so then doing a zuul first migration is something feasible or not? | 15:16 |
| corvus | i'm sure they'd love folks to collaborate with on that | 15:16 |
| mnaser | mkolesni: infra does have a weekly meeting fyi if you'd like to have someone hop into that and discuss some of this stuff (on a more planned basis) | 15:16 |
| fungi | no idea if the 19:00 utc meeting time is reasonable for mkolesni's timezone though | 15:17 |
| mkolesni | yeah i found 1900 i guess its UTC | 15:17 |
| fungi | and honestly we're happy to have these discussions any ol' time, just there might be more people on hand to discuss things at meeting time | 15:17 |
| mkolesni | should be fine for a one off :) | 15:17 |
| mkolesni | ill talk to our ci guy see if he can make it for the next week | 15:19 |
| mkolesni | should i propose a topic if so, or should it be part of open discussion? | 15:19 |
| *** pcaruana has joined #openstack-infra | 15:20 | |
| corvus | yeah, add a topic to the agenda in the wiki please | 15:20 |
| mkolesni | ok sure ill ping the CI guy next week (it's end of week for us) and if he can make it I'll add a topic | 15:21 |
| mkolesni | ok one other question to make sure i understood, to do the initial import would it be necessary to specify the "upstream:" field in the projects.yaml? i only see it used to track upstream gerrit and nothing else | 15:22 |
| corvus | mkolesni: yes, we remove it after the import happens | 15:22 |
| mkolesni | ah ok cool | 15:22 |
| mkolesni | makes sense then :) | 15:22 |
| mkolesni | OK thanks for all the info, I'll touch base with the team and we'll continue on the weekly meeting to understand the way forward WRT CI | 15:23 |
| *** priteau has quit IRC | 15:23 | |
| corvus | mkolesni: thanks! i'm looking forward to it. i'm sure we'll figure out the best way | 15:24 |
| mkolesni | GTG, have a nice weekend | 15:24 |
| *** pgaxatte has quit IRC | 15:25 | |
| *** larainema has quit IRC | 15:27 | |
| *** mkolesni has quit IRC | 15:29 | |
| corvus | where's the podman stuff mordred was working on? | 15:35 |
| corvus | for gerrit | 15:35 |
| corvus | or something.. it wasn't gerrit was it | 15:36 |
| fungi | corvus: https://review.opendev.org/689890 | 15:36 |
| fungi | maybe? | 15:36 |
| corvus | ah it's https://review.opendev.org/630406 | 15:37 |
| fungi | yep, looks like it | 15:37 |
| *** jtomasek has quit IRC | 15:38 | |
| corvus | i'm trying to figure out why i get this error running a container with podman: https://zuul.opendev.org/t/zuul/build/95180ea0c29a430ebc1a484568cc9aa2 | 15:39 |
| corvus | but apparently the podman-compose stuff in that change works | 15:39 |
| fungi | infra-root: i'm back to cleaning up the stuck afs volume releases, from mirror-update.openstack.org it's now down to just the ceph mirrors which i hope to knock out over the course of the next hour or so | 15:40 |
| *** ociuhandu has joined #openstack-infra | 15:40 | |
| *** jpena is now known as jpena|off | 15:41 | |
| fungi | but also additional reviews of https://review.opendev.org/695347 would be appreciated to get some automation for avoiding this working on mirror-update.opendev.org | 15:41 |
| *** ociuhandu has quit IRC | 15:41 | |
| fungi | there are a bunch of stale vldb locks for the volumes that server's updating, which we might be able to just clear if the remote release automation is completed | 15:42 |
| *** ociuhandu has joined #openstack-infra | 15:42 | |
| *** ociuhandu has quit IRC | 15:47 | |
| openstackgerrit | James E. Blair proposed zuul/zuul-jobs master: WIP: use-buildset-registry: add podman support https://review.opendev.org/695051 | 15:47 |
| *** priteau has joined #openstack-infra | 15:50 | |
| openstackgerrit | Merged opendev/irc-meetings master: Update Monasca Team Meeting time https://review.opendev.org/695418 | 15:52 |
| openstackgerrit | Tristan Cacqueray proposed zuul/zuul master: web: handle jobs that have multiple parent https://review.opendev.org/695450 | 15:55 |
| *** pgaxatte has joined #openstack-infra | 15:57 | |
| *** derekh has quit IRC | 16:03 | |
| *** udesale has quit IRC | 16:04 | |
| *** chkumar|rover is now known as raukadah | 16:08 | |
| *** ociuhandu has joined #openstack-infra | 16:18 | |
| openstackgerrit | Matthieu Huin proposed zuul/zuul master: [WIP] admin REST API: zuul-web integration https://review.opendev.org/643536 | 16:20 |
| openstackgerrit | James E. Blair proposed zuul/zuul-jobs master: WIP: use-buildset-registry: add podman support https://review.opendev.org/695051 | 16:26 |
| *** ociuhandu has quit IRC | 16:27 | |
| *** jaosorior has joined #openstack-infra | 16:33 | |
| *** jaosorior has quit IRC | 16:38 | |
| *** jaosorior has joined #openstack-infra | 16:39 | |
| *** ricolin has quit IRC | 16:39 | |
| clarkb | infra-root https://review.opendev.org/#/c/694185/ is one of the last bits of the opendev.org LE certification. Adds the dns records for cert issuer verification. | 16:43 |
| clarkb | I'll also be taking the "hiera" lock shortly to clean out the old ssl key and cert from the gitea.yaml group file | 16:43 |
| clarkb | I've cleaned up gitea_tls_cert and gitea_tls_key in the gitea.yaml vars file. | 16:52 |
| clarkb | mordred: ^ you have uncommitted changes to review-dev's host vars which I did not commit | 16:52 |
| *** armstrong_ has joined #openstack-infra | 16:54 | |
| openstackgerrit | James E. Blair proposed zuul/zuul-jobs master: WIP: use-buildset-registry: add podman support https://review.opendev.org/695051 | 16:54 |
| clarkb | inap mirror disk usage continues to look good | 16:56 |
| openstackgerrit | Merged opendev/zone-opendev.org master: Add cert issuer validation to opendev.org https://review.opendev.org/694185 | 16:57 |
| *** jaosorior has quit IRC | 17:00 | |
| fungi | okay, i've finished up the ceph mirrors, that just leaves these stuck according to vos listvldb: mirror.epel mirror.fedora mirror.opensuse mirror.yum-puppetlabs | 17:01 |
| fungi | those are all managed by mirror-update.opendev.org | 17:01 |
| *** rpittau is now known as rpittau|afk | 17:06 | |
| *** ociuhandu has joined #openstack-infra | 17:07 | |
| clarkb | fungi: we should be able to iterate through those in a similar manner just holding the locks on the other host? Let me know if I can help | 17:08 |
| fungi | yeah, but i was hoping instead to land 695347 so we could use that to do it | 17:09 |
| *** dtantsur is now known as dtantsur|afk | 17:10 | |
| clarkb | mriedem: I see shelve and unshelve continue to be problematic :/ is it possible that the dhcp anti spoofing rules neutron adds is blocking the necessary dhcp packets from hitting on the new compute host? | 17:10 |
| clarkb | fungi: gotcha | 17:10 |
| mriedem | clarkb: see my just posted ML thread | 17:10 |
| mriedem | bbiab | 17:10 |
| *** mriedem has quit IRC | 17:10 | |
| clarkb | mriedem: ya that is what prompted my question. I'll respond there. | 17:10 |
| fungi | so basically just release the vldb lock on those volumes and then let the next mirror pulse take care of the rest | 17:11 |
| clarkb | fungi: looks like you can approve that change now if you want. I haven't managed to review it yet myself so won't approve (at least until I can review it) | 17:11 |
| fungi | ahh, yep, looks like frickler also left a +2 | 17:11 |
| fungi | in it goes | 17:12 |
| fungi | once i confirm it's deployed and no mirror pulses are in progress i'll clear those locks | 17:12 |
| *** lucasagomes has quit IRC | 17:12 | |
| *** goldyfruit_ has joined #openstack-infra | 17:14 | |
| *** pgaxatte has quit IRC | 17:14 | |
| *** gyee has joined #openstack-infra | 17:17 | |
| *** ociuhandu has quit IRC | 17:17 | |
| *** goldyfruit has quit IRC | 17:17 | |
| *** priteau has quit IRC | 17:20 | |
| *** priteau has joined #openstack-infra | 17:21 | |
| *** priteau has quit IRC | 17:22 | |
| *** jamesmcarthur has quit IRC | 17:27 | |
| *** jpena|off is now known as jpena | 17:27 | |
| *** jamesmcarthur has joined #openstack-infra | 17:28 | |
| openstackgerrit | Merged opendev/system-config master: vos-release: fix sudo https://review.opendev.org/695347 | 17:31 |
| fungi | that should hopefully be in place by the time the sb meeting ends | 17:33 |
| *** ociuhandu has joined #openstack-infra | 17:36 | |
| *** ociuhandu has quit IRC | 17:36 | |
| mordred | clarkb: whoops. those are actually ok to comit - I'll do that now | 17:36 |
| *** ociuhandu has joined #openstack-infra | 17:37 | |
| *** ociuhandu has quit IRC | 17:37 | |
| *** ociuhandu has joined #openstack-infra | 17:38 | |
| mordred | infra-root: I finally slept somewhat reasonably and woke up similarly somewhat usefully so I might actually be somewhat useful again | 17:39 |
| openstackgerrit | James E. Blair proposed zuul/zuul-jobs master: WIP: use-buildset-registry: add podman support. https://review.opendev.org/695051 | 17:41 |
| *** ociuhandu has quit IRC | 17:44 | |
| *** michael-beaver has joined #openstack-infra | 17:50 | |
| *** pcaruana has quit IRC | 17:51 | |
| *** mattw4 has joined #openstack-infra | 17:51 | |
| Shrews | mordred: be glad you did not spend all of yesterday either in bed or hugging porcelain, as i did :( | 17:58 |
| clarkb | Shrews: oof. We've been dealing with a bug here too. Thankfully i'm feeling much better since about monday | 17:58 |
| Shrews | ick. luckily, i seem to be on the mend now | 18:00 |
| *** aedc has joined #openstack-infra | 18:02 | |
| mordred | Shrews: that sounds terrible | 18:06 |
| mordred | Shrews: I suggest not doing that | 18:06 |
| clarkb | what is crazy to me is I managed to go to a conference in a city of like 30 million people and be fine | 18:06 |
| clarkb | but come home and several days later I'm sick with something unfun | 18:06 |
| openstackgerrit | Tristan Cacqueray proposed zuul/zuul master: config: add tenant.toDict() method and REST endpoint https://review.opendev.org/621344 | 18:07 |
| *** weshay is now known as weshay|ruck | 18:08 | |
| openstackgerrit | James E. Blair proposed zuul/zuul-jobs master: install-podman: also install uidmap https://review.opendev.org/695545 | 18:09 |
| openstackgerrit | James E. Blair proposed zuul/zuul-jobs master: WIP: use-buildset-registry: add podman support https://review.opendev.org/695051 | 18:10 |
| *** pcaruana has joined #openstack-infra | 18:14 | |
| fungi | my money's still on the toddler who sneezed on you at the airport... lengthy incubation time | 18:16 |
| *** ociuhandu has joined #openstack-infra | 18:16 | |
| clarkb | fungi: ya could be | 18:17 |
| *** iurygregory is now known as iurygregory_afk | 18:19 | |
| mordred | clarkb: maybe next time don't get sneezed on by a toddler at the airport | 18:20 |
| *** armax has quit IRC | 18:20 | |
| clarkb | that airport experience was interesting too. I'm sad to have missed the guy angry about losing his screwdriver | 18:20 |
| fungi | if any system-config reviewers have a moment for https://review.opendev.org/691440 that will help me be able to take storyboard-dev back out of the emergency disable list | 18:22 |
| clarkb | fungi: I have a moment | 18:22 |
| fungi | it's a very small change, and already manually applied on storyboard-dev since weeks | 18:22 |
| fungi | turns on the wildcard host filter features so we can have working storyboard-webclient draft previews | 18:23 |
| clarkb | sorry, I've done a bad job of getting back itno the swing of things recently as I've been sick and dealing with house trouble. opendev.org cert was top of list due to time sensitivity and now thats done. Need to find next thing on the list (I think opendev governance email? I need to clean up the draft so that it is ready to go) | 18:24 |
| fungi | sounds great, thanks clarkb! glad you're back on your feet again | 18:24 |
| clarkb | Also, next week is a major USA holiday | 18:25 |
| clarkb | I expect those of us in the USA will be afk at least Thursday. | 18:26 |
| fungi | my plans for the coming week are up in the air at the moment due to health-related things going on with my inlaws. i may end up travelling even though i hadn't originally planned to go anywhere | 18:27 |
| *** iurygregory_afk has quit IRC | 18:28 | |
| *** tosky has quit IRC | 18:29 | |
| clarkb | I'll be out thursday and friday and back monday or tuesdayish | 18:29 |
| clarkb | we are actually venturing east for the long weekend and the forecast just got a bit nastier than I had hoped. An adventure in the snow/ice/slush/mud | 18:30 |
| Zara | :( ❄ ❄ ❄ (totally unrelated: my foundation membership is gonna expire because I haven't been voting. I get the impression things have changed and I no longer need to be a foundation member in order to commit code as an individual contributor, but wanted to check... I may be out of date by like 4 years on information around this) | 18:33 |
| clarkb | Zara: that is basically correct. Membership is only required if you want to vote on things (including TC and PTL) | 18:33 |
| fungi | Zara: you no longer need to be a foundation member in order to agree to the openstack foundation individual contributor license agreement | 18:33 |
| Zara | neat, thanks | 18:34 |
| fungi | clarkb: well, or if you want to run for office in the openstack project (and maybe also starlingx) | 18:34 |
| clarkb | oh right to be a candidate it is required as well | 18:34 |
| Zara | :D I like the idea of running as a candidate while not even knowing if I need to be a member | 18:35 |
| fungi | that's the best way to do it | 18:36 |
| Zara | 'I would like to be PTL of nova because it has a cool name, idk what it does but plz vote for me' | 18:36 |
| Zara | anyway thanks | 18:37 |
| *** gmann is now known as gmann_afk | 18:38 | |
| *** pcaruana has quit IRC | 18:42 | |
| *** gfidente has quit IRC | 18:44 | |
| clarkb | infra-root are we ready to try https://review.opendev.org/#/c/694894/ and upgrade gitea to 1.9.6? | 18:50 |
| clarkb | mordred: ^ I think you may have a 1.10.x change too | 18:50 |
| openstackgerrit | James E. Blair proposed zuul/zuul-jobs master: WIP: use-buildset-registry: add podman support https://review.opendev.org/695051 | 18:51 |
| fungi | clarkb: i'm around to help if it goes sideways | 18:51 |
| clarkb | I'm not sure what the preferred order is on that but we think it may help address the go-git issues that tonyb has seen | 18:51 |
| *** mloza has joined #openstack-infra | 18:51 | |
| *** mloza has left #openstack-infra | 18:52 | |
| fungi | or at least it will aid the gitea maintainers in helping us troubleshoot | 18:52 |
| corvus | clarkb: i think we're ready in general but i'm in-and-out so unable to commit to fixing ops issues | 18:52 |
| corvus | (ready for 1.9.6; the 1.10.0 change isn't ready) | 18:53 |
| clarkb | rgr. I think it is fairly low risk to do 1.9.6 and the revert is to go back to 1.9.5 which is fairly straightforward to do | 18:53 |
| corvus | yeah, as long as someone can push the revert button i'd say go | 18:53 |
| clarkb | I pushed the +A button. Can also push revert later if necessary | 18:54 |
| *** ralonsoh has quit IRC | 18:55 | |
| mordred | fungi: done | 18:59 |
| mordred | fungi: (the storyboard one) | 18:59 |
| fungi | thanks mordred! | 18:59 |
| mordred | clarkb: I do have a 1.10 change - it's not ready for raesons I don't yet understand (there's a breakage) | 19:00 |
| corvus | clarkb: i'm likely to be out all next week | 19:00 |
| *** armax has joined #openstack-infra | 19:01 | |
| tonyb[m] | So 1.9.6 has merged? | 19:04 |
| clarkb | tonyb[m]: it has been approved. yet to be merged and deployed | 19:04 |
| tonyb[m] | Ahh okay | 19:05 |
| *** pcaruana has joined #openstack-infra | 19:06 | |
| openstackgerrit | Jeremy Stanley proposed opendev/system-config master: Add missing newline in vos_release.sudo https://review.opendev.org/695554 | 19:09 |
| fungi | brown-bag fix to un-break sudo on some servers ^ | 19:09 |
| fungi | a quick review would be appreciated | 19:10 |
| corvus | done | 19:10 |
| clarkb | why does gerrit show 2 lines on the old side of the diff if there is no newline there? | 19:10 |
| clarkb | I wonder if that is a gerrit bug | 19:10 |
| openstackgerrit | Monty Taylor proposed opendev/system-config master: Update gitea to v1.10.0 https://review.opendev.org/690873 | 19:11 |
| fungi | thanks! | 19:11 |
| corvus | the view from gertty: http://paste.openstack.org/show/786507/ | 19:11 |
| corvus | i hope that doesn't mean i'm in charge of checking newlines now :) | 19:12 |
| fungi | i didn't point it out in the original patch because i feel like a pedant complaining about whatever crazy editor people are using these days that wants to save a byte by not terminating the last line of files | 19:13 |
| fungi | but clearly such pedantry is justified in at least some cases | 19:13 |
| mordred | I think that newline should get a trailing period | 19:14 |
| corvus | mordred: apparently sudo is also really picky about periods in files or something | 19:14 |
| fungi | i mostly dislike missing final newlines because if i cat a file on my terminal i wind up with my shell prompt appended to the last line | 19:14 |
| corvus | so, like, they're all coming home to roost here | 19:15 |
| corvus | (er periods in filenames) | 19:15 |
| fungi | sudo: where your unpicked nits come home to roost | 19:15 |
| corvus | fungi: you should get those looked at | 19:15 |
| *** mriedem has joined #openstack-infra | 19:15 | |
| fungi | it also causes me no end of finger memory conflict now that openbsd has replaced sudo with doas | 19:15 |
| * corvus starts googling | 19:16 | |
| fungi | https://man.openbsd.org/doas | 19:16 |
| fungi | the configuration syntax is written more like firewall rules | 19:17 |
| fungi | https://man.openbsd.org/doas.conf.5#EXAMPLES | 19:17 |
| corvus | okay that's nice | 19:18 |
| fungi | all in all i like the switch, but it will take me ages to get used to typing doas instead of sudo on my openbsd systems | 19:18 |
| fungi | but also in typical openbsd fashion doas is rewritten from the ground up to take advantage of all manner of memory protections and validation | 19:18 |
| fungi | not that sudo is rife with new vulnerabilities in recent history, but it did have rather a lot in years gone by and is apparently a bit of a mess to audit | 19:19 |
| clarkb | sudo did just have one though | 19:20 |
| fungi | yep | 19:20 |
| fungi | though that was a really corner corner case | 19:20 |
| fungi | not something anyone with a sane system would be likely to configure | 19:20 |
| corvus | i suspect a certain measure of insanity may be required in order to understand the sudo config syntax though | 19:22 |
| fungi | yep | 19:24 |
| *** jpena is now known as jpena|off | 19:36 | |
| openstackgerrit | James E. Blair proposed zuul/zuul-jobs master: WIP: use-buildset-registry: add podman support https://review.opendev.org/695051 | 19:37 |
| *** armstrong_ has quit IRC | 19:41 | |
| openstackgerrit | Merged opendev/system-config master: gitea: Use 1.9.6 https://review.opendev.org/694894 | 19:43 |
| openstackgerrit | Merged opendev/system-config master: Use a wildcard regex for storyboard-dev OAuth/CORS https://review.opendev.org/691440 | 19:43 |
| *** openstackstatus has quit IRC | 19:50 | |
| *** Buggys has quit IRC | 19:50 | |
| openstackgerrit | Merged zuul/zuul master: Zuul Web: add /api/user/authorizations endpoint https://review.opendev.org/641099 | 19:50 |
| *** dtroyer has quit IRC | 19:50 | |
| *** dtroyer has joined #openstack-infra | 19:51 | |
| *** openstackstatus has joined #openstack-infra | 19:51 | |
| *** ChanServ sets mode: +v openstackstatus | 19:51 | |
| *** mattw4 has quit IRC | 19:54 | |
| *** mattw4 has joined #openstack-infra | 19:54 | |
| ianw | fungi: sorry, catching up ... urgh after our great gitops with the letsencrypt stuff this has not been my shining hour | 19:57 |
| clarkb | gitea will be updating shortly I belieev | 20:01 |
| clarkb | https://gitea02.opendev.org:3000/ has upgraded | 20:02 |
| clarkb | browsing the webui works for me | 20:03 |
| clarkb | all 8 gitea backends are running 1.9.6 now | 20:09 |
| clarkb | tonyb[m]: ^ fyi | 20:09 |
| tonyb[m] | Oh cool. After the kids are at school I'll see if my repo is fixed | 20:10 |
| *** ociuhandu has quit IRC | 20:13 | |
| *** admcleod has quit IRC | 20:14 | |
| *** ociuhandu has joined #openstack-infra | 20:15 | |
| *** admcleod has joined #openstack-infra | 20:17 | |
| *** ociuhandu has quit IRC | 20:19 | |
| *** ianw has quit IRC | 20:20 | |
| *** ssbarnea has quit IRC | 20:26 | |
| clarkb | : vos_release : parse error in near line 1 ; | 20:27 |
| clarkb | more bugs to find and fix? | 20:27 |
| *** ianw has joined #openstack-infra | 20:27 | |
| clarkb | oh https://review.opendev.org/#/c/695554/1 failed to merge so maybe related to that still not being in | 20:28 |
| fungi | ianw: assuming sudo is working there now (i haven't checked to see if the newline got deployed yet) i can clear the stale vldb locks for the volumes managed from mirror-update.opendev.org if you want to retry | 20:29 |
| ianw | clarkb: looks like those errors are still there :/ | 20:30 |
| ianw | fungi: newline not deployed yet | 20:30 |
| fungi | k | 20:31 |
| fungi | also, what editor are you using which doesn't put newlines on the final lines of files? | 20:31 |
| fungi | (mainly so that i know not to try it) | 20:31 |
| ianw | heh, emacs | 20:32 |
| fungi | wow! | 20:32 |
| ianw | maybe i deleted it? i know i was wondering if the final component needed to be /usr/sbin/vos or just "vos" if the script was just calling "vos" | 20:33 |
| fungi | internets say you want "(setq require-final-newline t)" in your ~/.emacs | 20:33 |
| fungi | but as not an emacs user i can neither confirm nor deny the accuracy of that recommendation | 20:34 |
| ianw | i think that's usually something that the major modes set | 20:36 |
| openstackgerrit | Merged openstack/openstack-zuul-jobs master: Remove legacy-devstack-dsvm-updown job https://review.opendev.org/695434 | 20:37 |
| ianw | which i guess sudoers doesn't have | 20:37 |
| openstackgerrit | Merged zuul/zuul-jobs master: install-podman: also install uidmap https://review.opendev.org/695545 | 20:39 |
| openstackgerrit | Ian Wienand proposed opendev/system-config master: Fedora mirror update: use localauth release https://review.opendev.org/695572 | 20:39 |
| openstackgerrit | James E. Blair proposed zuul/zuul-jobs master: WIP: use-buildset-registry: add podman support https://review.opendev.org/695051 | 20:39 |
| openstackgerrit | Ian Wienand proposed opendev/system-config master: Fedora mirror update: use localauth release https://review.opendev.org/695572 | 20:39 |
| ianw | fungi: ^ i'm thinking something like that as a first test | 20:40 |
| ianw | then we can convert the other update scripts | 20:40 |
| ianw | then we can write reprepro support for the opendev.org updater | 20:40 |
| ianw | then we can convert them | 20:40 |
| ianw | easy :) | 20:40 |
| *** ociuhandu has joined #openstack-infra | 20:42 | |
| *** gmann_afk is now known as gmann | 20:45 | |
| fungi | the reprepro mirrors are cleaned back up and in sync again for now at least | 20:45 |
| ianw | fungi: thanks! | 20:47 |
| corvus | ianw, fungi, clarkb: do you remember why we didn't move dib to opendev/ during the great renaming? | 20:48 |
| fungi | i think we considered it fairly openstack-specific, or at least it had a number of dependency stakeholders there | 20:49 |
| ianw | i'm not sure it was ever explicitly discussed, but i imagine because it's used by openstack projects it was considered probably at home there | 20:49 |
| fungi | i agree it's a grey area | 20:49 |
| clarkb | We also didnt move it to opemstack-infra when it got adopted | 20:49 |
| ianw | octavia, trove (maybe?), tripleo | 20:49 |
| fungi | ironic too i think | 20:50 |
| ianw | ah yes, of course | 20:50 |
| *** rfolco has quit IRC | 20:50 | |
| johnsom | sahara and manila too | 20:55 |
| ianw | to be concrete, this is about where the dib dockerfile should put it's image | 20:58 |
| ianw | https://review.opendev.org/#/c/693971/ | 20:58 |
| ianw | i think we're down to opendev/diskimage-builder or openstack/diskimage-builder | 20:58 |
| corvus | well, my question was actually about the git repo | 20:58 |
| corvus | since there is the related question of whether the repository prefix and git repo prefix should differ | 20:59 |
| johnsom | Yeah, I have no issue where the image goes. I was just inquiring what value renaming the openstack/diskimage-builder would really bring. | 20:59 |
| corvus | i thought it would be helpful to understand the background in answering that question | 20:59 |
| clarkb | right I think the reason it was never renamed in the past is the same reason it hasn't been renamed with the opendev/ transition. Its a tool built in openstack predominantly used by openstack projects and changing that would require updating things that while doable would be annoying. | 21:00 |
| corvus | (however, you will find that i have not compunction about moving repos if warranted. we tried really hard to make a system where we didn't have to move repos based on project lifecycle considerations. the openstack tc has rejected that with the latest policy change, so more moves are inevitable.) | 21:00 |
| corvus | so maybe uploading it to the openstack/ namespace in dockerhub would be warranted? | 21:01 |
| * mordred has no strong feelings about the dib git repo prefix | 21:01 | |
| corvus | (aside from the fact that the tc hasn't issued any guidance about how that should be done; only something along the lines of "if you're going to do it, do it right") | 21:02 |
| mordred | corvus: I think it might be - but I think that namespace starts to have complicated optics | 21:02 |
| mordred | yeah | 21:02 |
| corvus | well, it's either an openstack project or it's not | 21:02 |
| corvus | seems uncomplicated to me :) | 21:02 |
| mordred | that - most of the openstack projects producing docker images put them into some other docker namspace | 21:02 |
| corvus | mordred: afaict that's because they don't want to talk to the tc about it | 21:02 |
| mordred | well - also because there are different things producing similar images - like loci/nova vs kolla/nova or something | 21:03 |
| fungi | assuming we're talking about https://governance.openstack.org/tc/resolutions/20170530-binary-artifacts.html for guidelines (or lack thereof) | 21:03 |
| corvus | openstack/loci-nova? | 21:03 |
| corvus | fungi: yeah, that's the thing i'm thinking of | 21:03 |
| *** openstack has joined #openstack-infra | 21:15 | |
| *** ChanServ sets mode: +o openstack | 21:15 | |
| ianw | but, if that functional test can produce bootable images, i'm pretty sure production deployments of the containers will be able to as well | 21:16 |
| corvus | we might be at the point where a write-up would be useful; since i can see like 3 small parts of what must be a bigger plan :) | 21:16 |
| clarkb | I think the easiest to understand version of that for testing would be to have dib manage the dockerfile directly and publish directly. I think the ideal dockerhub namespace for dib in that case is openstack/ to avoid confusion and question asking | 21:16 |
| corvus | clarkb: that seems like a straightforward mini-goal which stands on its own | 21:17 |
| ianw | really have to run out for a bit, i can cleanup the changes and write something up. at least we've narrowed the options | 21:19 |
| ianw | we *don't* want it in zuul/ :) | 21:19 |
| corvus | that's the one thing i'm sure of :) | 21:19 |
| *** pkopec has quit IRC | 21:21 | |
| *** sshnaidm has quit IRC | 21:21 | |
| *** dave-mccowan has joined #openstack-infra | 21:22 | |
| *** pcaruana has quit IRC | 21:25 | |
| *** mriedem has quit IRC | 21:41 | |
| clarkb | for some reason we didn't get a message from gerritbot that the sudo newline fix emrged, but it did merge | 21:44 |
| clarkb | I'm going to guess the bot is talking on some partitioned subset of freenode right now | 21:45 |
| *** ramishra has quit IRC | 21:57 | |
| ianw | clarkb: look like openstackstatus is no longer here too | 22:00 |
| tonyb | ianw, clarkb: Looks like my git update still hangs | 22:06 |
| ianw | tonyb: yeah, i see the same messages on 1.9.6 | 22:07 |
| ianw | on gitea06 | 22:07 |
| tonyb | so far I've only left it for about a minute but I'm pretty sure it's still b0rked | 22:07 |
| tonyb | ianw: I'm hitting gitea01 but | 22:07 |
| *** mriedem has joined #openstack-infra | 22:08 | |
| tonyb | ianw: Not sure what else to try now, I guess I need to understand the socket/client model of gitea to guess at what else to try :/ | 22:08 |
| ianw | tonyb: could you run it via a https proxy to watch the traffic? | 22:09 |
| ianw | maybe it says something before it dies? | 22:09 |
| *** rcernin has joined #openstack-infra | 22:09 | |
| tonyb | ianw: The answer to that is almost certinaly yes, but you'd need to provide a few more details, as I'm not sure what the proxy will tell me that git doesn't | 22:11 |
| ianw | hrm, i dunno, clutching at straws. it seems like the remote end dies, but your end doesn't notice | 22:11 |
| ianw | in fact you can probably just hit it at http:// ? | 22:12 |
| tonyb | Yeah | 22:12 |
| tonyb | ianw: Oh that's a thing? | 22:12 |
| ianw | run a tcpdump and see the discussion between yourself and the server? | 22:12 |
| tonyb | ianw: I figured that was off | 22:12 |
| tonyb | ianw: What port? | 22:12 |
| ianw | oh, no i don't think you can | 22:13 |
| ianw | thing i've used before is mitmproxy | 22:13 |
| clarkb | ya http redirects to https | 22:15 |
| *** sshnaidm has joined #openstack-infra | 22:17 | |
| * tonyb reads about mitmproxy | 22:21 | |
| johnsom | Yeah, gerritbot is missing from the octavia channel as well | 22:23 |
| *** kaisers1 has joined #openstack-infra | 22:23 | |
| *** harlowja has quit IRC | 22:23 | |
| *** kaisers has quit IRC | 22:24 | |
| ianw | tonyb: mitmproxy in a venv, run mitmproxy, then "https_proxy=localhost:8080 git clone -c http.sslVerify=false https://opendev.org/openstack/diskimage-builder" | 22:25 |
| ianw | works for my | 22:25 |
| ianw | but i guess it doesn't show much | 22:25 |
| tonyb | ianw: Oh that's waaaaaay better than what I was thinking | 22:26 |
| ianw | there's really a GET and a POST | 22:27 |
| *** ccamacho has quit IRC | 22:29 | |
| tonyb | Yeah, nothing really that git doesn't show | 22:29 |
| * tonyb could try with a differnt git and possibly even OS? | 22:30 | |
| ianw | tonyb: so does it show you getting anything at all back from the POST? | 22:30 |
| tonyb | ianw: Yeah my client looks to do 9 POSTs and get 9 200's back and then just stops | 22:33 |
| tonyb | ianw: runnign a second time the results look the same | 22:34 |
| ianw | hrm ... that's interesting maybe | 22:34 |
| *** tosky has joined #openstack-infra | 22:35 | |
| ianw | when cloning nova, my client does exactly one GET to ref?service=git-upload-pack and one POST to git-upload-pack | 22:35 |
| tonyb | ianw: same when I clone | 22:35 |
| ianw | and when you look at the POST response, how much data was there? | 22:37 |
| ianw | did it even start? | 22:37 |
| tonyb | The last one is like 116.1k | 22:38 |
| tonyb | It looks like the client sends a bunch of 'I have <objects>' and the server replies to only some if them? | 22:39 |
| ianw | wait, so is there only one POST, or multiple? | 22:39 |
| tonyb | ianw: for a clone the is one, for an update there are multiple | 22:39 |
| ianw | ah | 22:40 |
| *** lennyb has joined #openstack-infra | 22:41 | |
| ianw | does it always stop at the same point? | 22:41 |
| tonyb | Seems to | 22:42 |
| ianw | can you post your replication script again? | 22:43 |
| tonyb | ianw: I wasn't able to create a successful repo script, I posted what I was trying. | 22:44 |
| tonyb | ianw: I can tar up my 'currently' b0rked repo and give that to you | 22:44 |
| tonyb | ianw: and see if it continues to fail for you | 22:44 |
| ianw | worth a try i guess | 22:45 |
| ianw | on the server side, git seems to say the other end hung up. the other end doesn't seem to think it has hung up | 22:46 |
| ianw | ... shrug ...? | 22:46 |
| tonyb | yeah pretty much | 22:46 |
| clarkb | ianw: should we tcpdump our side and look for a fin? | 22:54 |
| ianw | clarkb: we could ... but we do have the strace that shows the git process dies | 22:58 |
| ianw | so i imagine it closes itself down | 22:59 |
| clarkb | its git so probably, but it not doing so might explain this behavior? similar to what we've seen wtih gerrit ssh and some clients/ | 22:59 |
| ianw | i would say at this point, any idea is a good idea :) | 23:01 |
| tonyb | So FWIW grabbing a newwer git locally doesn't change anything | 23:02 |
| fungi | has anyone else tried to reproduce this with a local copy of tonyb's repo content? | 23:03 |
| tonyb | fungi: not yet | 23:03 |
| clarkb | I have not but can try to do so | 23:03 |
| tonyb | that's next | 23:03 |
| clarkb | git version 2.24.0 is wghat I have | 23:04 |
| fungi | yeah, like if you can tar it up one or more of us can attempt to reproduce with it on other platforms and/or git versions, and maybe that will turn up some new evidence | 23:04 |
| fungi | granted that's a rather scattershot approach | 23:04 |
| fungi | grasping at straws now really | 23:04 |
| clarkb | I assume process is roughyl add remote for gitea06:3000 then git remote update ? | 23:05 |
| tonyb | clarkb: Yeah pretty much | 23:05 |
| ianw | fungi: so the fix isn't rolling out because sudo is broken? | 23:05 |
| ianw | fungi: i can log in as root and fix the files | 23:05 |
| fungi | ianw: oh, yep, i bet so | 23:05 |
| *** xek_ has quit IRC | 23:05 | |
| fungi | manually applying the fix to sudoers via root is also the only idea i have | 23:05 |
| fungi | you can ssh from root on bridge.o.o at least, as that's how i diagnosed the issue earlier | 23:06 |
| ianw | ok done (and old file with . removed) | 23:07 |
| ianw | yay, ssh -i ~/.ssh/id_vos_release vos_release@afs01.dfw.openstack.org vos release foo fails but works | 23:07 |
| clarkb | "fails but works" | 23:07 |
| clarkb | I think I understood that | 23:08 |
| tonyb | wget https://ozlabs.org/~tony/nova_bad.tar.gz | 23:08 |
| tonyb | https_proxy=localhost:8080 GIT_TRACE=2 GIT_CURL_VERBOSE=2 GIT_TRACE_PERFORMANCE=2 GIT_TRACE_PACK_ACCESS=2 GIT_TRACE_PACKET=2 GIT_TRACE_PACKFILE=2 GIT_TRACE_SETUP=2 GIT_TRACE_SHALLOW=2 strace -o strace -f ~/projects/git/bin-wrappers/git -c http.sslVerify=false remote -v update | 23:08 |
| tonyb | is the last update command I used | 23:09 |
| * tonyb will try in a couple of vms locally | 23:09 | |
| ianw | running now ... | 23:10 |
| clarkb | note you should be able to verify ssl now :) | 23:10 |
| ianw | that's just for the mitmproxy bit right? | 23:10 |
| clarkb | oh | 23:11 |
| ianw | 10:11:24.008133 pkt-line.c:80 packet: fetch-pack> have 95586b741b628de6b45cc7c69a03e93673ea3271 | 23:11 |
| ianw | tonyb: ^ that's the last thing you see too? | 23:11 |
| tonyb | Yup tha's wheer mine stops | 23:12 |
| ianw | well, consider it replicated :) | 23:12 |
| ianw | git version 2.23.0 | 23:12 |
| tonyb | Well last week I need that without mitmproxy, but if you've fixed it since then #winning | 23:12 |
| tonyb | okay | 23:12 |
| * tonyb is booting centos7 and ubuntu16.04 | 23:12 | |
| clarkb | same here on 2.24.0 | 23:12 |
| tonyb | okay so it *probably* isn't a client thing | 23:13 |
| ianw | a centos7 client would probably be a good comparision | 23:14 |
| fungi | i have 2.24.0.rc0 on debian/sid, so probably not worth trying to reproduce here given clarkb's result | 23:14 |
| ianw | could we (someone with root access) clone the repo from inside the container via ssh? | 23:15 |
| ianw | that would be using the same git binary that gitea calls to, right? | 23:15 |
| * tonyb shrugs I thought it was under bionic | 23:16 | |
| tonyb | where it == gitea's container | 23:16 |
| clarkb | undoing the typechange does not change the behavior fwiw | 23:17 |
| *** slaweq has quit IRC | 23:17 | |
| clarkb | ianw: I don't know if the ssh server there allows for reads? it probably does so you can probably do that | 23:18 |
| ianw | fungi: i'll just take the fedora lock on opendev.org and manually test a vos release via ssh, see if it works? | 23:18 |
| fungi | ianw: i need to release the stale vldb lock first | 23:18 |
| fungi | mirror.fedora volume? | 23:19 |
| ianw | fungi: yep | 23:19 |
| fungi | running `vos unlock mirror.fedora --localauth` now | 23:19 |
| fungi | Released lock on vldb entry for volume mirror.fedora | 23:19 |
| fungi | go for it | 23:19 |
| ianw | ok, it's running in a screen | 23:20 |
| ianw | looks ok from mirror-update.opendev.org end | 23:20 |
| clarkb | I don't see the server closing the connection | 23:20 |
| tonyb | so centos7 gets an additional response but then stops | 23:20 |
| clarkb | (I tcpdumped my own tests) | 23:20 |
| tonyb | packet: fetch-pack< ACK 95586b741b628de6b45cc7c69a03e93673ea3271 common | 23:20 |
| tonyb | packet: fetch-pack> have 95586b741b628de6b45cc7c69a03e93673ea3271 | 23:20 |
| tonyb | packet: fetch-pack< ACK 6105ac902e0ba6233c9a1e86cddacb5a0c1df4f1 common | 23:20 |
| tonyb | packet: fetch-pack> have 6105ac902e0ba6233c9a1e86cddacb5a0c1df4f1 | 23:20 |
| clarkb | I think it is possible the issue here is on gitea network connection handling ? | 23:20 |
| clarkb | both sides end up acking the same bits over and over again to keep tcp connection open | 23:21 |
| tonyb | clarkb: Yeah I think so | 23:21 |
| clarkb | basically the server has failed but not told the client properly about that | 23:22 |
| clarkb | the client just waits patiently | 23:22 |
| clarkb | then when I kill the client the F goes from my client to the server and things clean up | 23:22 |
| tonyb | clarkb: Yeah that seems to be my understanding | 23:22 |
| tonyb | Oh LOL, my previously 'fixed' nova clone also fails now ;P | 23:30 |
| tonyb | AT least I know how to 'fix' it | 23:31 |
| clarkb | what is the "fix"? | 23:36 |
| tonyb | git remote update gerrit ; to pull the missing objects from there, then when updating the origin remote everythign is local and it just updates the refs to match | 23:37 |
| *** slaweq has joined #openstack-infra | 23:44 | |
| ianw | yeah, so we can't ssh into the gitea container to try cloning like that | 23:48 |
| *** slaweq has quit IRC | 23:48 | |
| ianw | it seems like it needs some git verbosity flags | 23:51 |
| *** jamesmcarthur has quit IRC | 23:52 | |
| *** jamesmcarthur has joined #openstack-infra | 23:53 | |
| *** jamesmcarthur has quit IRC | 23:57 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!