| *** rfolco has joined #openstack-infra | 00:02 | |
| *** tinwood has quit IRC | 00:03 | |
| tkajinam | :-P | 00:03 |
|---|---|---|
| *** tinwood has joined #openstack-infra | 00:04 | |
| *** jamesmcarthur has joined #openstack-infra | 00:12 | |
| *** ysandeep|away is now known as ysandeep|ruck | 00:18 | |
| *** gyee has quit IRC | 00:24 | |
| openstackgerrit | Merged openstack/pbr master: Adding pre-commit https://review.opendev.org/742160 | 00:24 |
| *** jamesmcarthur has quit IRC | 00:29 | |
| *** rfolco has quit IRC | 00:30 | |
| *** seongsoocho has quit IRC | 00:43 | |
| *** seongsoocho has joined #openstack-infra | 00:43 | |
| *** hamalq has quit IRC | 00:44 | |
| *** stevebaker has quit IRC | 01:03 | |
| *** Goneri has quit IRC | 01:04 | |
| clarkb | fwiw now fairly confident that was the source of the slowdown. We've blocked the ip and the server has calmed down | 01:10 |
| *** ldenny has quit IRC | 01:15 | |
| *** rlandy|bbl is now known as rlandy | 01:21 | |
| *** rlandy has quit IRC | 01:23 | |
| *** Linda1_00 has joined #openstack-infra | 01:28 | |
| *** Linda1_00 is now known as LindaGuo | 01:29 | |
| *** lbragstad has quit IRC | 01:30 | |
| *** LindaGuo is now known as LindaG | 02:04 | |
| *** ysandeep|ruck is now known as ysandeep|afk | 02:11 | |
| tkajinam | clarkb, yeah it works much smoothly now. thanks | 02:11 |
| *** rfolco has joined #openstack-infra | 02:14 | |
| *** rfolco has quit IRC | 02:18 | |
| *** ldenny has joined #openstack-infra | 02:27 | |
| *** larsks has quit IRC | 02:36 | |
| *** ramishra has joined #openstack-infra | 02:40 | |
| *** larsks has joined #openstack-infra | 02:50 | |
| *** rcernin has quit IRC | 02:52 | |
| *** rfolco has joined #openstack-infra | 02:54 | |
| *** rcernin has joined #openstack-infra | 02:56 | |
| *** larsks has quit IRC | 02:59 | |
| *** rfolco has quit IRC | 02:59 | |
| *** larsks has joined #openstack-infra | 03:02 | |
| *** larsks has quit IRC | 03:05 | |
| *** larsks has joined #openstack-infra | 03:08 | |
| *** larsks has quit IRC | 03:09 | |
| *** larsks has joined #openstack-infra | 03:12 | |
| *** dklyle has quit IRC | 03:15 | |
| *** zxiiro has quit IRC | 03:47 | |
| *** Topner has joined #openstack-infra | 04:27 | |
| *** Topner has quit IRC | 04:32 | |
| *** evrardjp has quit IRC | 04:33 | |
| *** evrardjp has joined #openstack-infra | 04:33 | |
| *** gfidente has joined #openstack-infra | 04:57 | |
| *** ysandeep|afk is now known as ysandeep | 05:14 | |
| *** ysandeep is now known as ysandeep|ruck | 05:14 | |
| *** matt_kosut has joined #openstack-infra | 05:19 | |
| *** artom has quit IRC | 05:33 | |
| *** artom has joined #openstack-infra | 05:33 | |
| *** vishalmanchanda has joined #openstack-infra | 05:39 | |
| *** jamesmcarthur has joined #openstack-infra | 05:46 | |
| *** sboyron has joined #openstack-infra | 06:06 | |
| *** lpetrut has joined #openstack-infra | 06:09 | |
| *** nightmare_unreal has joined #openstack-infra | 06:19 | |
| *** jtomasek has joined #openstack-infra | 06:20 | |
| *** slaweq has joined #openstack-infra | 06:21 | |
| *** yonglihe has joined #openstack-infra | 06:29 | |
| *** noonedeadpunk has quit IRC | 06:31 | |
| *** eolivare has joined #openstack-infra | 06:35 | |
| *** noonedeadpunk has joined #openstack-infra | 06:39 | |
| *** slaweq has quit IRC | 06:44 | |
| *** slaweq has joined #openstack-infra | 06:46 | |
| *** rpittau|afk is now known as rpittau | 06:54 | |
| *** ralonsoh has joined #openstack-infra | 07:05 | |
| *** jamesmcarthur has quit IRC | 07:22 | |
| *** jamesmcarthur has joined #openstack-infra | 07:22 | |
| *** gfidente is now known as gfidente|pto | 07:23 | |
| *** andrewbonney has joined #openstack-infra | 07:24 | |
| *** jamesmcarthur has quit IRC | 07:27 | |
| *** jpena|off is now known as jpena | 07:36 | |
| *** ociuhandu has joined #openstack-infra | 07:41 | |
| *** rcernin has quit IRC | 07:42 | |
| *** tosky has joined #openstack-infra | 07:44 | |
| *** grandfso has joined #openstack-infra | 07:47 | |
| *** guillaumec has quit IRC | 07:48 | |
| *** rcernin has joined #openstack-infra | 07:48 | |
| *** guillaumec has joined #openstack-infra | 07:49 | |
| *** samP has quit IRC | 07:51 | |
| *** samP has joined #openstack-infra | 07:51 | |
| *** gfidente|pto has quit IRC | 07:52 | |
| *** matbu has quit IRC | 07:53 | |
| *** matbu has joined #openstack-infra | 07:54 | |
| *** rcernin has quit IRC | 07:54 | |
| *** rcernin has joined #openstack-infra | 08:00 | |
| *** jamesmcarthur has joined #openstack-infra | 08:04 | |
| *** lucasagomes has joined #openstack-infra | 08:10 | |
| *** rcernin has quit IRC | 08:12 | |
| *** ysandeep|ruck is now known as ysandeep|rck|afk | 08:12 | |
| *** ysandeep|rck|afk is now known as ysandep|ruck|afk | 08:13 | |
| *** gfidente has joined #openstack-infra | 08:16 | |
| *** gfidente is now known as gfidente|pto | 08:17 | |
| *** jamesmcarthur has quit IRC | 08:20 | |
| *** rcernin has joined #openstack-infra | 08:21 | |
| *** rcernin has quit IRC | 08:26 | |
| *** sshnaidm|afk is now known as sshnaidm|rover | 08:29 | |
| *** derekh has joined #openstack-infra | 08:38 | |
| *** ddurst has quit IRC | 08:48 | |
| *** ddurst has joined #openstack-infra | 08:55 | |
| *** dtantsur|afk is now known as dtantsur | 08:56 | |
| *** iurygregory has quit IRC | 08:57 | |
| *** rcernin has joined #openstack-infra | 09:21 | |
| *** jaicaa has quit IRC | 09:22 | |
| *** jaicaa has joined #openstack-infra | 09:23 | |
| *** rcernin has quit IRC | 09:25 | |
| *** iurygregory has joined #openstack-infra | 09:28 | |
| *** ysandep|ruck|afk is now known as ysandep|ruck | 09:28 | |
| *** matt_kosut has quit IRC | 09:36 | |
| *** matt_kosut has joined #openstack-infra | 09:41 | |
| *** matt_kosut has quit IRC | 09:45 | |
| *** matt_kosut has joined #openstack-infra | 09:58 | |
| *** rcernin has joined #openstack-infra | 10:43 | |
| *** rcernin has quit IRC | 10:48 | |
| *** jamesmcarthur has joined #openstack-infra | 10:48 | |
| *** jamesmcarthur has quit IRC | 10:53 | |
| *** jpena is now known as jpena|lunch | 11:35 | |
| *** jcapitao has joined #openstack-infra | 11:53 | |
| *** takamatsu has joined #openstack-infra | 11:54 | |
| *** sai438 has quit IRC | 11:55 | |
| *** pmannidi has joined #openstack-infra | 11:57 | |
| *** rlandy has joined #openstack-infra | 12:03 | |
| *** Goneri has joined #openstack-infra | 12:08 | |
| *** gfidente|pto is now known as gfidente | 12:15 | |
| *** rcernin has joined #openstack-infra | 12:17 | |
| *** rfolco has joined #openstack-infra | 12:20 | |
| *** rcernin has quit IRC | 12:22 | |
| openstackgerrit | Jeremy Stanley proposed openstack/project-config master: Temporarily stop booting nodes in inap-mtl01 https://review.opendev.org/759251 | 12:24 |
| *** jpena|lunch is now known as jpena | 12:32 | |
| *** zxiiro has joined #openstack-infra | 12:33 | |
| noonedeadpunk | hey folks! Need some help/advice about how to solve issue that has recently raised with ubuntu mirrors and dunno, maybe there's an option how openstack infra can help with that | 12:47 |
| noonedeadpunk | so canonical has dropped history of their base images recently on which we're relying heavily | 12:52 |
| noonedeadpunk | http://cdimage.ubuntu.com/ubuntu-base/releases/18.04/release/ | 12:53 |
| noonedeadpunk | so the only option at the moment is to determine latest one and pick it up. | 12:53 |
| noonedeadpunk | (or use some different source) | 12:53 |
| fungi | what is "history of their base images"? | 12:54 |
| noonedeadpunk | where they can be stored more reliably and having control over them... | 12:54 |
| fungi | as in, like, old copies of their cloud images or a history of what changes were made in their images? | 12:54 |
| fungi | or some other kind of images? | 12:54 |
| noonedeadpunk | i mean 18.04.2, 18.04.3 - point releases | 12:54 |
| noonedeadpunk | same for focal | 12:54 |
| fungi | also who is "we" relying on them? | 12:55 |
| noonedeadpunk | old copies | 12:55 |
| noonedeadpunk | OSA) | 12:55 |
| fungi | got it, so osa has been using old copies of ubuntu cloud images for $reasons and now ubuntu has either deleted or relocated all but a few recent point releases? | 12:55 |
| noonedeadpunk | which means all of our current releases are broken because of this | 12:55 |
| *** matt_kosut has quit IRC | 12:55 | |
| noonedeadpunk | fungi: yeah | 12:56 |
| fungi | osa uses them in tests, or something to do with production deployments? | 12:56 |
| noonedeadpunk | for production deployments | 12:56 |
| noonedeadpunk | so we have kind of url hardocded here https://opendev.org/openstack/openstack-ansible-lxc_hosts/src/branch/master/vars/ubuntu-18.04.yml#L16 | 12:57 |
| fungi | oh, got it, running them under lxc for some sort of containerized deployment model? | 12:57 |
| noonedeadpunk | exactly | 12:58 |
| *** matt_kosut has joined #openstack-infra | 12:58 | |
| noonedeadpunk | we retrieve base image, modify it a bit and store local cache. but for ppl who don't have a cache or if it's expired... | 12:58 |
| fungi | and i guess the deployment process still upgrades them to latest packages for that series, so the end result is the same as if they'd used a more recent image point release | 12:58 |
| noonedeadpunk | kind of, yes. but the point here was that the process is reliable and fixed for release | 12:59 |
| noonedeadpunk | as we use exact same image every time | 12:59 |
| noonedeadpunk | (for osa release) | 12:59 |
| fungi | but just to be clear, osa users aren't running old versions of ubuntu packages with known security vulnerabilities (this is why ubuntu removes the old versions of the images, after all, to help dissuade anyone from deploying vulnerable software) | 13:00 |
| noonedeadpunk | we run this for the image https://opendev.org/openstack/openstack-ansible-lxc_hosts/src/branch/master/templates/prep-scripts/ubuntu_20_prep.sh.j2 | 13:01 |
| fungi | have you looked/asked around to find out if ubuntu still keeps those old image versions somewhere and has simply moved them out of the normal location? | 13:01 |
| noonedeadpunk | well, they pointed to this ML https://lists.ubuntu.com/archives/ubuntu-release/2020-October/005113.html | 13:01 |
| noonedeadpunk | and that storing these images was not planned at all, however it was since 16.04 or smth... | 13:02 |
| noonedeadpunk | which was pretty frustrating | 13:02 |
| *** matt_kosut has quit IRC | 13:02 | |
| fungi | i see, yeah i don't have any good ideas other than to issue a point release of stable branches updating the urls to more recent point releases of those various lts series | 13:03 |
| fungi | or, alternatively, trying to use ubuntu's image build process to recreate those old images | 13:04 |
| fungi | which seems like ot would probably be much more work and a lot more error-prone | 13:04 |
| noonedeadpunk | well, doing that every time is too time consuming and we already take plenty of ci time... | 13:04 |
| noonedeadpunk | well, they have daily builds, but I'm not sure how good to use them | 13:05 |
| fungi | ask ubuntu to create a "latest" symlink for their images so you can refer to those | 13:05 |
| fungi | ? | 13:05 |
| noonedeadpunk | yeah I think I should, thanks! | 13:05 |
| noonedeadpunk | is #ubuntu-server good place to ask? | 13:06 |
| openstackgerrit | Merged openstack/project-config master: Temporarily stop booting nodes in inap-mtl01 https://review.opendev.org/759251 | 13:08 |
| AJaeger | config-core, https://review.opendev.org/#/c/751993/ is ready to merge - retire monasca-analytics. Please review | 13:19 |
| * AJaeger just -1ed, wiwll fix quickly | 13:20 | |
| openstackgerrit | Andreas Jaeger proposed openstack/project-config master: Remove openstack/monasca-analytics https://review.opendev.org/751993 | 13:20 |
| AJaeger | now ready | 13:21 |
| *** lpetrut has quit IRC | 13:22 | |
| *** artom has quit IRC | 13:36 | |
| *** tdasilva has quit IRC | 13:36 | |
| *** tdasilva has joined #openstack-infra | 13:37 | |
| *** rcernin has joined #openstack-infra | 13:41 | |
| *** rcernin has quit IRC | 13:45 | |
| *** matt_kosut has joined #openstack-infra | 13:53 | |
| *** sshnaidm|rover is now known as sshnaidm|afk | 13:54 | |
| *** xek has joined #openstack-infra | 14:00 | |
| *** takamatsu is now known as mauro|call | 14:04 | |
| *** mauro|call is now known as takamatsu | 14:04 | |
| *** Qianbiao has joined #openstack-infra | 14:13 | |
| Qianbiao | Hello, when visit https://docs.openstack.org/diskimage-builder/latest/elements, it index the folder. | 14:13 |
| Qianbiao | Maybe a security problem, just a remind. | 14:14 |
| *** grandfso has quit IRC | 14:24 | |
| *** tdasilva_ has joined #openstack-infra | 14:29 | |
| *** tdasilva has quit IRC | 14:32 | |
| *** d34dh0r53 has quit IRC | 14:40 | |
| AJaeger | Qianbiao: not a security problem. There's just no index.html page... | 14:41 |
| AJaeger | Qianbiao: you should browse via https://docs.openstack.org/diskimage-builder/latest/elements.html | 14:41 |
| Qianbiao | AJaeger if no index, it should 404 i think. index folder is not a good option. | 14:42 |
| Qianbiao | index file system may cause security issue. | 14:42 |
| *** xek has quit IRC | 14:42 | |
| *** xek has joined #openstack-infra | 14:43 | |
| *** d34dh0r53 has joined #openstack-infra | 14:43 | |
| Qianbiao | And it supports folder visit: https://docs.openstack.org/diskimage-builder/latest/elements/../.. | 14:45 |
| AJaeger | Qianbiao: I still do not see any security problem here so far - if you know of a concrete one, please tell us in detail. | 14:47 |
| *** sshnaidm|afk is now known as sshnaidm|rover | 14:48 | |
| *** jamesmcarthur has joined #openstack-infra | 14:50 | |
| Qianbiao | AJaeger I am not good at this :) | 14:50 |
| *** xek has quit IRC | 14:51 | |
| Qianbiao | Just a remind. In my company index file systems is marked as security issue which is not allowed. | 14:51 |
| AJaeger | IMHO that's just hidding informatio, we have everything open anyway since we publish from open source etc. So, I see not how in our setup, this would add a security risk. | 14:54 |
| AJaeger | Qianbiao: still, thanks for pointing it out. | 14:55 |
| *** xek has joined #openstack-infra | 14:55 | |
| AJaeger | maybe others reading have more information and can enlighten me... | 14:55 |
| *** jamesmcarthur has quit IRC | 14:55 | |
| *** dklyle has joined #openstack-infra | 14:55 | |
| Qianbiao | AJaeger np, you are welcome :) | 14:56 |
| *** zxiiro has quit IRC | 14:59 | |
| *** Qianbiao is now known as Qianbiao|afk | 15:09 | |
| *** Qianbiao|afk has quit IRC | 15:09 | |
| *** auristor has quit IRC | 15:14 | |
| *** ysandep|ruck is now known as ysandep|away | 15:18 | |
| *** artom has joined #openstack-infra | 15:19 | |
| *** artom has quit IRC | 15:19 | |
| *** artom has joined #openstack-infra | 15:19 | |
| fungi | some folks attempt to hide information in files on web sites by not linking to them from other files. this is a practice commonly referred to as "security by obscurity" (which is really just another way of saying "not really secured at all" or "wishful thinking") | 15:22 |
| fungi | while most of the documentation on that site probably doesn't benefit from apache mod_index autoindices, we make the exact same filesystems completely browseable by other means, for example on the static.opendev.org site or via afs | 15:23 |
| *** auristor has joined #openstack-infra | 15:27 | |
| *** sboyron has quit IRC | 15:28 | |
| *** jamesmcarthur has joined #openstack-infra | 15:32 | |
| *** zxiiro has joined #openstack-infra | 15:38 | |
| *** rcernin has joined #openstack-infra | 15:41 | |
| *** rcernin has quit IRC | 15:46 | |
| *** lmiccini has quit IRC | 15:56 | |
| *** rpittau is now known as rpittau|afk | 15:57 | |
| *** lucasagomes has quit IRC | 16:00 | |
| *** jamesmcarthur has quit IRC | 16:06 | |
| *** hamalq has joined #openstack-infra | 16:07 | |
| *** jamesmcarthur has joined #openstack-infra | 16:07 | |
| *** jamesmcarthur has quit IRC | 16:07 | |
| *** hamalq has quit IRC | 16:10 | |
| *** hamalq has joined #openstack-infra | 16:10 | |
| *** jamesmcarthur has joined #openstack-infra | 16:26 | |
| *** ociuhandu_ has joined #openstack-infra | 16:28 | |
| *** jpena is now known as jpena|off | 16:29 | |
| *** dtantsur is now known as dtantsur|afk | 16:31 | |
| *** jamesmcarthur has quit IRC | 16:31 | |
| *** ociuhandu has quit IRC | 16:32 | |
| *** jamesmcarthur has joined #openstack-infra | 16:32 | |
| *** ociuhandu_ has quit IRC | 16:32 | |
| *** jamesmcarthur has quit IRC | 16:34 | |
| *** jamesmcarthur has joined #openstack-infra | 16:34 | |
| *** valleedelisle has joined #openstack-infra | 16:39 | |
| *** eolivare has quit IRC | 16:44 | |
| *** jcapitao has quit IRC | 16:45 | |
| *** derekh has quit IRC | 17:00 | |
| *** jtomasek has quit IRC | 17:07 | |
| *** ralonsoh has quit IRC | 17:08 | |
| *** valleedelisle has quit IRC | 17:13 | |
| *** jamesmcarthur has quit IRC | 17:17 | |
| *** jamesmcarthur has joined #openstack-infra | 17:18 | |
| *** valleedelisle has joined #openstack-infra | 17:24 | |
| *** andrewbonney has quit IRC | 17:26 | |
| *** gfidente is now known as gfidente|afk | 17:28 | |
| *** vishalmanchanda has quit IRC | 17:39 | |
| *** xek has quit IRC | 17:41 | |
| *** sshnaidm|rover is now known as sshnaidm|off | 17:45 | |
| *** ricolin has quit IRC | 17:54 | |
| *** nightmare_unreal has quit IRC | 18:02 | |
| *** dklyle has quit IRC | 18:13 | |
| *** eandersson has quit IRC | 19:10 | |
| *** eandersson has joined #openstack-infra | 19:11 | |
| *** gyee has joined #openstack-infra | 19:26 | |
| *** raissa has joined #openstack-infra | 19:27 | |
| *** raissa has quit IRC | 19:31 | |
| *** stevebaker has joined #openstack-infra | 19:36 | |
| *** lbragstad has joined #openstack-infra | 19:39 | |
| *** rcernin has joined #openstack-infra | 19:43 | |
| *** rcernin has quit IRC | 19:48 | |
| *** lbragstad has quit IRC | 19:56 | |
| *** jamesmcarthur has quit IRC | 20:03 | |
| *** lbragstad has joined #openstack-infra | 20:03 | |
| *** matt_kosut has quit IRC | 20:05 | |
| smcginnis | I've noticed the aarch64 jobs that run on the requirements repo patches has been getting NODE_FAILURE errors the last few days. | 20:06 |
| smcginnis | https://zuul.opendev.org/t/openstack/builds?job_name=requirements-tox-py38-check-uc-aarch64 | 20:06 |
| smcginnis | Anyone know what's up with those? | 20:06 |
| fungi | entirely possible that cloud has fallen offline again | 20:06 |
| clarkb | I think kevinz was doing cloud updates | 20:08 |
| clarkb | or no it was a colo facility outage? maybe things haven't come back up | 20:08 |
| fungi | right, he said something about pdu maintenance | 20:09 |
| fungi | that likely also explains the deployment failures related to nb03 | 20:09 |
| *** raissa has joined #openstack-infra | 20:14 | |
| *** jamesmcarthur has joined #openstack-infra | 20:17 | |
| *** jamesmcarthur has quit IRC | 20:18 | |
| *** jamesmcarthur has joined #openstack-infra | 20:19 | |
| *** jamesmcarthur has quit IRC | 20:22 | |
| *** yolanda has quit IRC | 20:24 | |
| *** yolanda has joined #openstack-infra | 20:24 | |
| *** jamesmcarthur has joined #openstack-infra | 20:26 | |
| *** slaweq has quit IRC | 20:26 | |
| *** jamesmcarthur has quit IRC | 20:31 | |
| *** jamesmcarthur has joined #openstack-infra | 20:45 | |
| *** jamesmcarthur has quit IRC | 20:49 | |
| *** rfolco has quit IRC | 21:03 | |
| *** dychen has joined #openstack-infra | 21:06 | |
| *** slaweq has joined #openstack-infra | 21:08 | |
| *** dchen has quit IRC | 21:10 | |
| *** raissa has quit IRC | 21:23 | |
| *** slaweq has quit IRC | 21:27 | |
| sean-k-mooney | fungi: clarkb what is the best way to approch debuging why zuul is not loading all jobs you have asked it too | 21:43 |
| fungi | can you define "loading"? | 21:44 |
| fungi | like you're browsing to the jobs page and not finding them>? | 21:44 |
| clarkb | or maybe not running them on changes? | 21:44 |
| sean-k-mooney | well that and i gett waring in the schduler | 21:44 |
| fungi | check the top row if the webui for a "bell" icon which may indicate dependency errors | 21:45 |
| sean-k-mooney | saying that require repos cant be found | 21:45 |
| clarkb | ya the bell should point you at the repos that are a problem iirc | 21:45 |
| sean-k-mooney | but in one case devstack is saying it cant find iteself | 21:45 |
| fungi | yeah, so that's a known channel for reusing externally-defined jobs. you'll need to at least include those repos in your tenant config, even if you don't load configuration from them | 21:45 |
| sean-k-mooney | yep i have | 21:46 |
| sean-k-mooney | not sure if you can see https://zuul.cloud.seanmooney.info/t/openstack/config-errors | 21:46 |
| clarkb | can't connect from here | 21:47 |
| clarkb | there it goes | 21:47 |
| sean-k-mooney | basically im seeing Unknown projects: opendev.org/openstack/devstack | 21:48 |
| clarkb | sean-k-mooney: that required project uses the fully qualified name | 21:48 |
| sean-k-mooney | from openstack/devstack - .zuul.yaml | 21:48 |
| sean-k-mooney | yep | 21:48 |
| clarkb | is it included in your tenant config from opendev.org? | 21:48 |
| sean-k-mooney | i named my driver opendev.org | 21:48 |
| *** hamalq_ has joined #openstack-infra | 21:48 | |
| sean-k-mooney | http://paste.openstack.org/show/799297/ | 21:49 |
| sean-k-mooney | thats my tenants file | 21:49 |
| clarkb | and that config adds it to the openstack tenant and your errors page is for the openstack tenant too (means we don't have different tenant issues) | 21:50 |
| *** dklyle has joined #openstack-infra | 21:50 | |
| sean-k-mooney | http://paste.openstack.org/show/799298/ | 21:50 |
| sean-k-mooney | that is my zull config - my webhook token | 21:51 |
| *** hamalq has quit IRC | 21:51 | |
| clarkb | I wonder if the canonical hostname on line 41 is the issue | 21:51 |
| sean-k-mooney | oh it could be | 21:51 |
| clarkb | it may be thinking the fully qualified project name is review.opendev.org/openstack/devstack | 21:51 |
| sean-k-mooney | im modifying an old config i had form before the openstack to opendev rename | 21:52 |
| *** sai438 has joined #openstack-infra | 21:52 | |
| sean-k-mooney | ok i can try that | 21:52 |
| clarkb | I would try changing that to opendev.org | 21:52 |
| clarkb | or maybe drop it entirely? | 21:52 |
| sean-k-mooney | sure ill drop it and see | 21:52 |
| *** pmannidi has quit IRC | 21:53 | |
| *** odyssey4me has quit IRC | 21:53 | |
| *** odyssey4me has joined #openstack-infra | 21:53 | |
| sean-k-mooney | i remember working though all of this like 2 years ago but i didnt write down all the steps | 21:54 |
| sean-k-mooney | we also dont have a thirdpart zuul ci guide as far as i know. but i have been trying to figure this out for a few hours now and not really making progress | 21:55 |
| clarkb | ya no one has written one yet | 21:56 |
| clarkb | if you take down your steps I'm sure we'd love to publish it somehow | 21:56 |
| sean-k-mooney | i may end up redeploying in a seperate vm for zuul/nodepool using the docker hub images | 21:56 |
| sean-k-mooney | if i do ill record what i have to do | 21:57 |
| sean-k-mooney | while im asking question can i drop 2 of those connections | 21:57 |
| sean-k-mooney | the zuul-git one i dont need | 21:57 |
| sean-k-mooney | because i can get that from gerrit | 21:57 |
| sean-k-mooney | but what about the github and githhub.com ones | 21:58 |
| sean-k-mooney | can the github driver connect be used to add repos that do not have my applciation? | 21:58 |
| clarkb | yes | 21:58 |
| fungi | yeah, to be clear, the third-party ci guide we had for zuul v2 was written by third-party ci operators who had set up zuul v2 and documented how they did so. the reason we have no third-party ci guide for zuul v3 is that nobody who has set up a third-party ci with zuul v3 has submitted documentation explaining how they did so | 21:58 |
| clarkb | sean-k-mooney: I think that is the difference with the two github connectons. One hooks into the app events system. the other will poll repos | 21:58 |
| sean-k-mooney | yes | 21:59 |
| clarkb | if you don't need the polling or don't need the app you could drop the one you don't need | 21:59 |
| sean-k-mooney | am well they are providing different things | 21:59 |
| sean-k-mooney | im using the app for the config repo | 21:59 |
| sean-k-mooney | and the pooling one for dpdk and ovs repos likely also libvirt and qemu in time | 22:00 |
| clarkb | in that case you probably need both? | 22:00 |
| sean-k-mooney | ya i think so | 22:00 |
| sean-k-mooney | i can drop the one on line 29 | 22:00 |
| sean-k-mooney | that was just form the docs | 22:00 |
| sean-k-mooney | it would make sense to keep it if i was not also adding opendev via gerrit | 22:01 |
| sean-k-mooney | hum | 22:01 |
| sean-k-mooney | so no that does not seam to have fixed it | 22:01 |
| clarkb | also there is a third operating mode iwht github where you can add them to your app connection and if the app isn't there zuul won't get the vents but can incorporate the repo state of those repos when other events trigger jobs | 22:02 |
| clarkb | sean-k-mooney: maybe you need the canonical name but set to opendev.org? sorry its been a long week and I dont have a lot of zuul things paged in right now | 22:02 |
| sean-k-mooney | clarkb: its ok dont stress out about it | 22:02 |
| sean-k-mooney | ya ill give that a try | 22:02 |
| clarkb | we do the app less github connection thing with stuff we want zuul to cache for reliability | 22:03 |
| clarkb | but don't get events | 22:03 |
| sean-k-mooney | ya that is basically what im trying to do with ovs and dpdk | 22:04 |
| sean-k-mooney | ill look into it once i get the basics working :) | 22:04 |
| sean-k-mooney | this was my old working config form like 2 year ago http://paste.openstack.org/show/799300/ | 22:05 |
| fungi | also maybe a good reminder that we have several tenants with similar sorts of situations configured here: https://opendev.org/openstack/project-config/src/branch/master/zuul/main.yaml | 22:05 |
| sean-k-mooney | there i had canonical_hostname=git.openstack.org | 22:05 |
| sean-k-mooney | fungi: hehe i have deffenetly stolen some of this from there at different points | 22:06 |
| sean-k-mooney | based on the docs canonical_hostname likely is the issue https://zuul-ci.org/docs/zuul/reference/drivers/gerrit.html?highlight=canonical_hostname#attr-%3Cgerrit%20connection%3E.canonical_hostname | 22:08 |
| sean-k-mooney | clarkb: fungi thank you both | 22:19 |
| fungi | yw | 22:19 |
| sean-k-mooney | it was the canonical hostname | 22:19 |
| fungi | awesome | 22:19 |
| fungi | docs ftw | 22:19 |
| clarkb | sean-k-mooney: it had to be explicitly set to opendev.org? | 22:20 |
| sean-k-mooney | also name is not a config option that was removed at some point | 22:20 |
| sean-k-mooney | clarkb: yes it did | 22:20 |
| sean-k-mooney | clarkb: i always tought that was set by [connection opendev.org] | 22:21 |
| sean-k-mooney | not by the canonical_hostname line | 22:22 |
| sean-k-mooney | i may also be doing horible horible hacks | 22:24 |
| sean-k-mooney | i redfien my own dsvm-base, dsvm-tempest-base tempest-all and tempest-full | 22:25 |
| sean-k-mooney | that way i dont need to import jobs form tempest which pulls in basically all of openstack | 22:25 |
| fungi | to be fair, running devstack jobs on your own zuul is likely to require a modicum of horribleness. so many required projects... | 22:26 |
| sean-k-mooney | i have only 23 and i dont actully need all of them | 22:27 |
| sean-k-mooney | http://paste.openstack.org/show/799297/ | 22:27 |
| sean-k-mooney | o can remove like 5 of them | 22:27 |
| sean-k-mooney | actully i dont need devstack-gate anymore either | 22:27 |
| sean-k-mooney | this is basically what i use instead of the upstream tempest jobs which still use the tempest roles https://github.com/SeanMooney/ci-sean-mooney/blob/main/zuul.d/jobs.yaml#L31-L132 | 22:29 |
| sean-k-mooney | they are basically a copy past without the require projects and a few minor tweeks | 22:29 |
| *** rlandy has quit IRC | 22:29 | |
| *** rcernin has joined #openstack-infra | 22:30 | |
| sean-k-mooney | oh right... now im hittign the default branch name change | 22:34 |
| sean-k-mooney | three is no devstack job defined for main | 22:35 |
| sean-k-mooney | this is where i go steal the branch mapping stuff form project config :) | 22:36 |
| fungi | yeah, you'll likely need an override-checkout to deal with that | 22:36 |
| sean-k-mooney | yep i was considrign defining my own intermdiary that mapped to master | 22:37 |
| sean-k-mooney | oh that in the job definetion cool | 22:37 |
| fungi | pretty much, you could make a job which triggers on branch: main and then override-checkout: master to cause it to use master branches on other projects | 22:38 |
| fungi | though that might get hairy if you're mixing multiple repos with different default branch names | 22:38 |
| clarkb | ya this is one reason the early draft we wrote on the subject suggests larger project orgs switch together | 22:39 |
| fungi | this is where hard-coding zuul to have an explicit fallback to main when there is no master might be a more user-friendly solution | 22:39 |
| clarkb | oh thats an interesting idea too | 22:40 |
| fungi | clarkb: you probably missed that discussion in #zuul earlier today | 22:40 |
| clarkb | I did! | 22:40 |
| sean-k-mooney | ya im reall tempted to just make a master branch and use that but if i can figure out how to make it work i will give it a try | 22:40 |
| * clarkb goes to look at scrollback | 22:40 | |
| sean-k-mooney | fungi: was that the disucssion where i mention it did nto load the config project with out the load-branch override to main | 22:41 |
| fungi | sean-k-mooney: yep | 22:41 |
| sean-k-mooney | ya a fallback would have been nice it was pretty obvious once that clicked but it took me ages to figure out it was because of main | 22:42 |
| fungi | but with my zuul maintainer hat on for a moment, if git upstream has already decided that "main" is the new default name for default branches, then i could see adding a search precedence to zuul's current master assumption | 22:42 |
| sean-k-mooney | fungi: well github has | 22:43 |
| sean-k-mooney | i dont know if the git project has or not | 22:43 |
| sean-k-mooney | i normally dont create github repos form github but sicne i was setting up the zuul app i did thistime | 22:44 |
| sean-k-mooney | if i pushed the repo manually it would have crated with master as the defualt branch | 22:44 |
| sean-k-mooney | still i guess it better for someone like me to hit these things and let ye know then someone coming to zuul for the first time | 22:45 |
| sean-k-mooney | even with the nameing issue its so much nice the deploy zuulv2 with jenkins or just using jenkins | 22:46 |
| sean-k-mooney | although that mgiht be the years of brainwashing talking | 22:47 |
| clarkb | its a bit different of an appraoch. Zuul wants you to manage config in git and up front. Jenkins will spin up and do absolutely nothing to start | 22:48 |
| clarkb | there is value in both method | 22:48 |
| *** rcernin has quit IRC | 22:49 | |
| sean-k-mooney | its a bit of a steaper learning curve but having everything version contoled is a big win | 22:49 |
| sean-k-mooney | try as you might jenkins always involves imperitive steps to defining things where as zuul can basically be deployed and operated declaritivly | 22:50 |
| fungi | yeah, honestly the predecessor of zuul v3's configuration was our jenkins-job-builder approach to putting jenkins configs under actual version controk | 22:51 |
| fungi | control | 22:51 |
| *** rcernin has joined #openstack-infra | 22:51 | |
| fungi | after so many times of "what changed with this job? i dunno, check the job revision history... ugh that's no help" | 22:52 |
| fungi | though honestly, my experience with using the jenkins webui to create jobs, or using jenkins at all for that matter, is years stale. i'm sure it's gotten much better in that time | 22:53 |
| sean-k-mooney | we deploy zullv2.5 in our lab with jjb configuration and it worked but i can honetly say i did not really understand why | 22:53 |
| sean-k-mooney | it helpped but still not as easy to follow as just using ansible | 22:53 |
| *** tosky has quit IRC | 22:54 | |
| fungi | yeah, for v3 we really didn't want to go the jenkins route and create a dsl for job definitions, and ansible also had its own runners and was in yaml like we had already been using with jjb, so it was an obvious solution | 22:55 |
| fungi | i know some people are turned off by the "i have to learn ansible to write jobs?" aspect, but honestly the alternative was "i have to learn some new job description language with lots of its own unique bugs?" | 22:56 |
| fungi | even if you've never seen ansible before, at least you're learning something which has applicability beyond just zuul jobs | 22:57 |
| fungi | plus, we don't have to write or maintain it, there's a vibrant existing community already doing that | 22:57 |
| sean-k-mooney | once plus to it was i could say no it support ansible im not going to write the job in puppet | 22:58 |
| * sean-k-mooney really does not like puppet/ruby | 22:58 | |
| fungi | i'll say there are things i like better about puppet than ansible, but their featureset has converged a bit over time | 23:00 |
| sean-k-mooney | it still pains me so much that downstream we hvae so much jenkins | 23:00 |
| fungi | the idempotent declarative state approach to puppet is alluring, unfortunately systems are all too often a seething foam of uncertainty underneath so it's not as useful as it seems like it should be | 23:00 |
| JayF | We use (and I think are the primary OSS contributors to?) http://screwdriver.cd/ downstream -- I don't think anything is as comprehensive as Zuul, but this is *much* better than Jenkins IME | 23:01 |
| fungi | trying to do cross-systems orchestration with puppet was double-unfun | 23:01 |
| JayF | all CI/CD configs in repo, in a machine readable/writeable format is just absolutely wonderful | 23:01 |
| JayF | although it does suffer a bit from what fungi noted -- the "I have to learn how to do X" that isn't particularly applicable outside of screwdriver | 23:02 |
| sean-k-mooney | there is a defienite split in redhat. all of the upstream engineers know and love zuul all our qe know jenkins | 23:02 |
| fungi | JayF: does it do speculative config execution? that's the obvious next-step benefit to in-repo configuration | 23:02 |
| JayF | If that means "it uses the new configuration file in PRs as a form of testing it", yes | 23:03 |
| fungi | awesome | 23:03 |
| JayF | if you don't mean that, I need a definition :) | 23:03 |
| JayF | jroll even wrote some software downstream that we use to centralize and generate those configs, and PR out changes to the repos | 23:04 |
| fungi | yep, that was one of our most compelling design elements for zuul v3, we had already hacked self-testing secondary configuration into frameworks like devstack-gate but the job definitions were a blind spot | 23:04 |
| JayF | so for the "N" openstack repo/branch combos we have that are generally the same (apply patches; run tests; publish to local pypi), we only write the config once and it gets PR'd to the repos | 23:04 |
| JayF | the visibility is the piece I think Zuul does really well, although IDK if the searchable page on status.opendev.org is "zuul" or "other cool opendev infra stuff" | 23:05 |
| clarkb | everything under zuul.opendev.org is zuul not special | 23:05 |
| fungi | status.opendev.org is increasingly less cool openstack stuff, the zuul.opendev.org webui is getting waaaay better at standing on its own now | 23:05 |
| JayF | ah it is zuul.opendev.org, not status. | 23:05 |
| clarkb | we've been replacing a lot of the special stuff with more built in functionality and it has gone well | 23:06 |
| JayF | that's one of those URLs I type in wrong everytime :D | 23:06 |
| JayF | Zuul is mostly an openstack-grown-and-run project, right? | 23:06 |
| JayF | s/stack/dev/ | 23:06 |
| fungi | yeah, the idea is that zuul knows all the configuration and build history, so make that browseable and queryable | 23:06 |
| clarkb | JayF: its gotten a fair bit of outside involvment since becoming a top level project. BMW does a lot of work on it | 23:07 |
| clarkb | volvo is getting more and more involved too | 23:07 |
| fungi | JayF: zuul grew up alongside openstack to serve openstack's use cases initially, but it's always been a bit independent and is now pretty much entirely independent for the last year or two | 23:07 |
| clarkb | and then there are smaller orgs using it as well | 23:07 |
| JayF | That's really nice that it's getting external adoption and contribution | 23:07 |
| JayF | fungi: Yeah, I know with some of the infra pieces it is "grew up with openstack" moreso than "grown by openstack" but I've never been particularly good at that distinction | 23:08 |
| JayF | BTW; thanks for the response to the incident the other day. | 23:08 |
| fungi | JayF: shameless plug for an article i wrote outlining zuul's history, if you're bored: https://opensource.com/article/20/2/zuul | 23:09 |
| JayF | I'm not but I'm also a speedreader :D | 23:09 |
| JayF | it's actually EOD for me, so I'll read that then turn into a pumpkin :) | 23:09 |
| sean-k-mooney | im in one of those case wehre i almost have my third party ci runing a devstack job | 23:11 |
| sean-k-mooney | i shoudl stop but im really close | 23:11 |
| sean-k-mooney | by the way when im using override checkout i should also use "branches: main" right | 23:15 |
| sean-k-mooney | to only override if it the job that is calling it is on main | 23:15 |
| fungi | sean-k-mooney: in that particular case, yes i believe that's what you'll need | 23:20 |
| fungi | because you won't want to override other branches, so you'd do a second version of the job to catch the stable/.* branches | 23:21 |
| sean-k-mooney | yep | 23:21 |
| fungi | ideally those would just be lightweight child jobs inheriting from a common job definition anyway | 23:21 |
| *** benj_ has quit IRC | 23:21 | |
| sean-k-mooney | http://paste.openstack.org/show/799302/ | 23:23 |
| sean-k-mooney | that is basically what im going to try | 23:23 |
| sean-k-mooney | i think zull give precidence to the closted match | 23:24 |
| sean-k-mooney | so the one with branches main shoud match before the one without it | 23:24 |
| sean-k-mooney | but if i need to i can match on stable in the second one | 23:24 |
| fungi | on the second version of the job you'll want a branch filter too i think | 23:25 |
| fungi | either negative lookahead regex excluding main, or a pattern match like stable/.* if those will be the only non-main branches | 23:26 |
| *** benj_ has joined #openstack-infra | 23:26 | |
| fungi | you may be right though, we usually just get pedantic in our configs out of an abundance of caution | 23:26 |
| sean-k-mooney | im basing that on this exmple https://opendev.org/zuul/zuul/src/branch/master/doc/source/reference/job_def.rst#L410 | 23:28 |
| sean-k-mooney | well ignore the line | 23:28 |
| sean-k-mooney | if you go to the very bottom | 23:28 |
| sean-k-mooney | the old node set example | 23:28 |
| sean-k-mooney | cool now i have a node failure but that is just because i have not added the focal lable to my nodepool yet so its now working :) | 23:33 |
Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!