| *** dchen has quit IRC | 00:09 | |
| *** slaweq has joined #openstack-infra | 00:10 | |
| *** dchen has joined #openstack-infra | 00:12 | |
| *** slaweq has quit IRC | 00:15 | |
| *** ociuhandu has joined #openstack-infra | 00:19 | |
| *** jamesmcarthur has joined #openstack-infra | 00:20 | |
| *** slaweq has joined #openstack-infra | 00:21 | |
| *** jamesmcarthur has quit IRC | 00:26 | |
| *** tosky has quit IRC | 00:26 | |
| *** ociuhandu has quit IRC | 00:26 | |
| *** armax has joined #openstack-infra | 00:36 | |
| *** armax has quit IRC | 00:40 | |
| *** jamesmcarthur has joined #openstack-infra | 00:58 | |
| *** jamesmcarthur has quit IRC | 01:17 | |
| *** ijw has quit IRC | 01:21 | |
| *** kmarc_ has quit IRC | 01:23 | |
| *** kmarc has joined #openstack-infra | 01:23 | |
| *** ijw has joined #openstack-infra | 01:25 | |
| *** armax has joined #openstack-infra | 01:28 | |
| *** spsurya has joined #openstack-infra | 01:28 | |
| *** ociuhandu has joined #openstack-infra | 01:29 | |
| *** ijw has quit IRC | 01:30 | |
| *** ijw has joined #openstack-infra | 01:30 | |
| *** kmarc has quit IRC | 01:31 | |
| *** kmarc has joined #openstack-infra | 01:33 | |
| *** kmarc has quit IRC | 01:33 | |
| *** kmarc has joined #openstack-infra | 01:33 | |
| *** ijw_ has joined #openstack-infra | 01:33 | |
| *** armax has quit IRC | 01:33 | |
| *** ociuhandu has quit IRC | 01:34 | |
| *** auristor has quit IRC | 01:36 | |
| *** ijw has quit IRC | 01:37 | |
| *** kevinz has joined #openstack-infra | 01:41 | |
| *** dave-mccowan has joined #openstack-infra | 01:46 | |
| kevinz | ping ianw | 01:46 |
|---|---|---|
| ianw | kevinz: hey :) | 01:46 |
| ianw | kevinz: https://review.opendev.org/#/c/690798/ is one we've had constant failures on | 01:47 |
| ianw | perhaps i should delete any current nodes, and start the job, and we can get the id's and watch what's happening with their boot? | 01:47 |
| kevinz | ianw: np. there is no logs. Maybe the node is died I guess.. | 01:48 |
| kevinz | The Zuul will re-create 1 new instances? | 01:48 |
| ianw | yeah, in that case, nodepool never actually got the node started | 01:49 |
| kevinz | you can re-triagger, I can check the node status in Linaro side | 01:49 |
| ianw | this job is actually a two-node job | 01:50 |
| ianw | give me a sec to pull up a console to see what nodepool thinks is happening too | 01:50 |
| kevinz | sure, are the nodes below "os-control" projects? | 01:51 |
| ianw | it will be under "os-jobs" | 01:53 |
| *** auristor has joined #openstack-infra | 01:53 | |
| *** kmarc has quit IRC | 01:53 | |
| ianw | so deb43673-030a-4c82-be93-fa80dfa1110c is active ATM | 01:54 |
| kevinz | sure, now 4 instances under os-jobs | 01:54 |
| *** kmarc has joined #openstack-infra | 01:54 | |
| ianw | i'm running a recheck, let's see if one starts building | 01:54 |
| ianw | ok, e9add7ec-167d-4ed5-aad2-40432962b869 has started to build | 01:55 |
| kevinz | 1110c has a kernel hungs | 01:56 |
| kevinz | deb43673-030a-4c82-be93-fa80dfa1110c | 01:56 |
| kevinz | but could ssh and console | 01:57 |
| kevinz | what is the bydefault user/password for the ubuntu bionic image? | 01:57 |
| ianw | hrm i didn't see anything in the conosle | 01:57 |
| ianw | there's isn't one as such; we install various keys that are deployed via cloud-init | 01:58 |
| kevinz | 1 second, I think I should disable my connection to the console | 01:58 |
| *** ociuhandu has joined #openstack-infra | 01:59 | |
| ianw | 213.146.141.78 | 01:59 |
| ianw | so i can log in there, my key (and other admins) is in the default config | 02:01 |
| ianw | i think what we want to watch is the two building ones -- 198b3967-3dfa-4203-a239-30a58abc3ec6 & e9add7ec-167d-4ed5-aad2-40432962b869 | 02:01 |
| *** slaweq has quit IRC | 02:02 | |
| ianw | the problem seems to be that nodes coming up are very unreliable -- clearly we do get nodes starting, but not reliably | 02:03 |
| *** ociuhandu has quit IRC | 02:03 | |
| *** ociuhandu has joined #openstack-infra | 02:04 | |
| *** ociuhandu has quit IRC | 02:09 | |
| ianw | hrm, did we have a quota issue with 198b3967-3dfa-4203-a239-30a58abc3ec6 ? | 02:11 |
| *** armax has joined #openstack-infra | 02:14 | |
| ianw | 2019-12-02 02:05:31,305 INFO nodepool.NodeLauncher: [node: 0013157473] Quota exceeded, invalidating quota cache | 02:16 |
| ianw | 2019-12-02 02:05:31,305 INFO nodepool.NodeLauncher: [node: 0013157473] Aborting node 0013157473 due to quota failure | 02:16 |
| ianw | kevinz: ^ that's what nodepool thinks happened to 198b ... | 02:17 |
| *** Xuchu has joined #openstack-infra | 02:17 | |
| kevinz | ianw: can I increase the quota of os-jobs? | 02:17 |
| ianw | kevinz: well, we won't say no :) let's just see why we might exceed it... | 02:20 |
| *** armax has quit IRC | 02:20 | |
| ianw | kevin: https://opendev.org/openstack/project-config/src/branch/master/nodepool/nl03.openstack.org.yaml#L411 | 02:22 |
| kevinz | ianw: it is weird. the quota for the os-jobs is 75000GB | 02:22 |
| ianw | we're setup for 8 max servers | 02:22 |
| ianw | maybe it's some other component? | 02:22 |
| ianw | openstack.exceptions.ConflictException: ConflictException: 409: Client Error for url: https://uk.linaro.cloud:9696/v2.0/floatingips.json, Quota exceeded for resources: ['floatingip']. | 02:22 |
| ianw | ahh, it's right above it in the logs :) | 02:22 |
| ianw | so are we leaking floating ips? | 02:22 |
| kevinz | I can increase | 02:23 |
| ianw | is the quota 8? | 02:25 |
| kevinz | Yes, I check it is 8 | 02:25 |
| kevinz | I increase to 10 now | 02:25 |
| ianw | i wonder why we exceeded it then? | 02:25 |
| ianw | maybe slow to clean up | 02:27 |
| kevinz | let me check the totally floating ip info | 02:28 |
| kevinz | there are 7 floating ip assigned to this projects os-jobs | 02:30 |
| kevinz | by default it will re-create the floating-ip when create instance? or can leverage the unused ip that assigned to this projects? | 02:31 |
| ianw | so e9add... became active | 02:32 |
| kevinz | I should say that 7 floating ips, 3 of them are unused | 02:32 |
| *** auristor has quit IRC | 02:34 | |
| ianw | so the job is running, we got the nodes in this case -> http://zuul.openstack.org/stream/0099edffa9be4383b06371cf92fc0449?logfile=console.log | 02:36 |
| kevinz | Good to hear | 02:37 |
| kevinz | should I remove the floating ip manually? | 02:38 |
| ianw | kevinz: no, nodepool shoul dmanage them | 02:38 |
| kevinz | OK | 02:38 |
| ianw | kevinz: although weirdly, 192.168.1.15 doesn't show up in the floating ip list, but it has one? | 02:39 |
| ianw | e9add7ec-167d-4ed5-aad2-40432962b869 | 02:39 |
| kevinz | let me check | 02:39 |
| ianw | i wonder if that's how the quota is getting out of sync | 02:42 |
| *** auristor has joined #openstack-infra | 02:43 | |
| kevinz | sorry I don't find this instance now.. | 02:44 |
| kevinz | just 4 of use | 02:44 |
| kevinz | just 4. | 02:44 |
| ianw | yeah, the job finished | 02:45 |
| *** auristor has quit IRC | 02:48 | |
| kevinz | I think the quota is getting out of sync when cleaning? | 02:48 |
| *** Xuchu has quit IRC | 02:48 | |
| *** Xuchu has joined #openstack-infra | 02:48 | |
| kevinz | From Linaro cloud side, we also offer resources to OpenLab, also use Zuul, but don't meet the problem of quota issue | 02:49 |
| *** auristor has joined #openstack-infra | 02:50 | |
| ianw | hrm; let's try the job again, see if the nodes allocate | 02:50 |
| *** armax has joined #openstack-infra | 02:51 | |
| *** ociuhandu has joined #openstack-infra | 02:51 | |
| *** ociuhandu has quit IRC | 02:53 | |
| *** ociuhandu has joined #openstack-infra | 02:55 | |
| *** armax has quit IRC | 02:56 | |
| *** armax has joined #openstack-infra | 02:58 | |
| kevinz | OK | 02:59 |
| *** ociuhandu has quit IRC | 03:03 | |
| *** armax has quit IRC | 03:05 | |
| ianw | well, running again; that's good! | 03:06 |
| ianw | kevinz: i'll keep an eye, maybe it was just quota being slow to update | 03:06 |
| kevinz | ubuntu-bionic-arm64-linaro-london-0013157624， that 1 is created | 03:06 |
| kevinz | yes I see | 03:06 |
| *** apetrich has quit IRC | 03:08 | |
| *** bstinson has quit IRC | 03:15 | |
| *** bstinson has joined #openstack-infra | 03:22 | |
| *** Xuchu_ has joined #openstack-infra | 03:23 | |
| *** Xuchu has quit IRC | 03:26 | |
| *** Xuchu_ is now known as Xuchu | 03:26 | |
| *** ramishra has joined #openstack-infra | 03:36 | |
| *** bstinson has quit IRC | 03:39 | |
| *** ricolin has joined #openstack-infra | 03:47 | |
| openstackgerrit | Ian Wienand proposed opendev/system-config master: [wip] test arm64 mirror https://review.opendev.org/690798 | 03:49 |
| *** bstinson has joined #openstack-infra | 03:51 | |
| *** ociuhandu has joined #openstack-infra | 04:03 | |
| *** ociuhandu has quit IRC | 04:08 | |
| *** Lucas_Gray has joined #openstack-infra | 04:17 | |
| *** ykarel has joined #openstack-infra | 04:20 | |
| ianw | we're pretty consistenly getting nodes for ^ now, which is great! | 04:25 |
| *** udesale has joined #openstack-infra | 04:37 | |
| *** auristor has quit IRC | 04:38 | |
| *** dave-mccowan has quit IRC | 04:40 | |
| *** auristor has joined #openstack-infra | 04:42 | |
| *** Lucas_Gray has quit IRC | 05:08 | |
| *** surpatil has joined #openstack-infra | 05:12 | |
| ianw | it even works! | 05:16 |
| *** soniya29 has joined #openstack-infra | 05:17 | |
| *** ramishra has quit IRC | 05:20 | |
| *** lbragstad_ has joined #openstack-infra | 05:21 | |
| *** lbragstad has quit IRC | 05:24 | |
| *** ociuhandu has joined #openstack-infra | 05:30 | |
| *** ociuhandu has quit IRC | 05:36 | |
| *** ramishra has joined #openstack-infra | 05:39 | |
| *** jamesmcarthur has joined #openstack-infra | 05:46 | |
| *** jamesmcarthur has quit IRC | 05:50 | |
| *** raukadah is now known as chkumar|rover | 05:54 | |
| *** ijw has joined #openstack-infra | 05:56 | |
| *** ijw_ has quit IRC | 05:59 | |
| *** jtomasek has joined #openstack-infra | 06:27 | |
| *** pcaruana has joined #openstack-infra | 06:32 | |
| *** Xuchu has quit IRC | 06:37 | |
| *** Xuchu has joined #openstack-infra | 06:37 | |
| *** chkumar|rover is now known as chkumar|ruck | 06:51 | |
| *** ralonsoh has joined #openstack-infra | 06:51 | |
| *** dpawlik has joined #openstack-infra | 07:00 | |
| *** ociuhandu has joined #openstack-infra | 07:01 | |
| *** Xuchu has quit IRC | 07:03 | |
| *** Xuchu has joined #openstack-infra | 07:03 | |
| AJaeger | \o/ | 07:04 |
| *** dpawlik has quit IRC | 07:05 | |
| *** ociuhandu has quit IRC | 07:05 | |
| *** slaweq has joined #openstack-infra | 07:08 | |
| *** tkajinam has quit IRC | 07:11 | |
| *** tkajinam has joined #openstack-infra | 07:12 | |
| *** dpawlik has joined #openstack-infra | 07:15 | |
| *** pgaxatte has joined #openstack-infra | 07:24 | |
| *** surpatil has quit IRC | 07:29 | |
| *** slaweq has quit IRC | 07:37 | |
| *** dciabrin has joined #openstack-infra | 07:38 | |
| *** slaweq has joined #openstack-infra | 07:40 | |
| *** surpatil has joined #openstack-infra | 07:40 | |
| *** ykarel is now known as ykarel|lunch | 07:43 | |
| *** ijw_ has joined #openstack-infra | 07:45 | |
| *** ijw has quit IRC | 07:47 | |
| *** apetrich has joined #openstack-infra | 07:52 | |
| openstackgerrit | Harald Jensås proposed openstack/diskimage-builder master: dnf - Don't error on missing packages https://review.opendev.org/696532 | 08:04 |
| *** dchen has quit IRC | 08:09 | |
| *** tesseract has joined #openstack-infra | 08:14 | |
| kevinz | ianw: Thanks! looks Zuul +1 now. Great! | 08:18 |
| kevinz | \o/ | 08:18 |
| *** iurygregory has joined #openstack-infra | 08:19 | |
| *** ykarel|lunch is now known as ykarel | 08:20 | |
| *** ijw_ has quit IRC | 08:24 | |
| *** tkajinam has quit IRC | 08:25 | |
| *** tkajinam has joined #openstack-infra | 08:26 | |
| *** tosky has joined #openstack-infra | 08:31 | |
| *** rpittau|afk is now known as rpittau | 08:31 | |
| *** ccamacho has joined #openstack-infra | 08:34 | |
| *** tkajinam has quit IRC | 08:39 | |
| *** hashar has joined #openstack-infra | 08:46 | |
| *** jpena|off is now known as jpena | 08:48 | |
| *** ttx has quit IRC | 08:50 | |
| *** ttx has joined #openstack-infra | 08:50 | |
| *** dtantsur|afk is now known as dtantsur | 08:52 | |
| *** lucasagomes has joined #openstack-infra | 08:53 | |
| *** ociuhandu has joined #openstack-infra | 08:55 | |
| *** ociuhandu has quit IRC | 08:56 | |
| *** ociuhandu has joined #openstack-infra | 08:57 | |
| *** pkopec has joined #openstack-infra | 08:58 | |
| *** Tengu has quit IRC | 09:01 | |
| *** Tengu has joined #openstack-infra | 09:02 | |
| *** SurajPatil has joined #openstack-infra | 09:07 | |
| *** trident has quit IRC | 09:07 | |
| *** surpatil has quit IRC | 09:09 | |
| *** trident has joined #openstack-infra | 09:09 | |
| *** derekh has joined #openstack-infra | 09:35 | |
| ianw | kevinz: i'll start looking at the wheel builds too | 09:55 |
| lucasagomes | hi, anyone else having problem logging into gerrit ? | 09:56 |
| lucasagomes | I'm being redirected to an openId login page | 09:56 |
| lucasagomes | "Provider is not supported, or was incorrectly entered." | 09:56 |
| ralonsoh | same problem here: first time I see this | 10:00 |
| ralonsoh | https://review.opendev.org/login/%23%2Fsettings%2F | 10:00 |
| lucasagomes | I think it's launchpad, even https://launchpad.net/+login is giving me: "OpenID Provider Is Unavailable at This Time" | 10:00 |
| *** ociuhandu has quit IRC | 10:04 | |
| ianw | yeah, i just noticed it too :) 503 errors from login.ubuntu.com | 10:06 |
| ianw | this seems like an upstream issue with launchpad login, i don't think there's much we can do; i'm sure they'll be on it | 10:06 |
| lucasagomes | ianw, thanks yeah | 10:08 |
| ianw | there we go, i can already log back in now :) | 10:08 |
| *** lpetrut has joined #openstack-infra | 10:13 | |
| *** gfidente has joined #openstack-infra | 10:16 | |
| *** dchen has joined #openstack-infra | 10:22 | |
| *** tosky_ has joined #openstack-infra | 10:26 | |
| *** ociuhandu has joined #openstack-infra | 10:28 | |
| *** tosky has quit IRC | 10:28 | |
| openstackgerrit | Ian Wienand proposed openstack/project-config master: Import openstacksdk and dib into zuul tenant https://review.opendev.org/696859 | 10:33 |
| *** tosky_ is now known as tosky | 10:38 | |
| *** lourot has quit IRC | 10:43 | |
| *** lourot has joined #openstack-infra | 10:44 | |
| *** pgaxatte has quit IRC | 10:49 | |
| brtknr | hi all, we cant use an etherpad we created last week: https://etherpad.openstack.org/p/magnum-ussuri-virtual-ptg-planning | 10:54 |
| brtknr | someone said it was due to an emoji??? | 10:54 |
| brtknr | is this a known issue? | 10:54 |
| *** ralonsoh has quit IRC | 11:01 | |
| kevinz | ianw: Cool, thanks. Just let me know if problem. I can also grant more resources to you if needed | 11:01 |
| *** ralonsoh has joined #openstack-infra | 11:03 | |
| *** ociuhandu has quit IRC | 11:16 | |
| *** ociuhandu has joined #openstack-infra | 11:17 | |
| *** ociuhandu has quit IRC | 11:26 | |
| *** ociuhandu has joined #openstack-infra | 11:26 | |
| *** surpatil has joined #openstack-infra | 11:29 | |
| *** SurajPatil has quit IRC | 11:31 | |
| openstackgerrit | Bogdan Dobrelya (bogdando) proposed zuul/zuul-jobs master: DNM testing custom pypi mirrors (696337) https://review.opendev.org/696867 | 11:41 |
| *** Lucas_Gray has joined #openstack-infra | 11:43 | |
| *** ociuhandu has quit IRC | 11:43 | |
| *** ociuhandu has joined #openstack-infra | 11:45 | |
| *** ociuhandu has quit IRC | 11:50 | |
| *** dtantsur is now known as dtantsur|afk | 11:54 | |
| *** jklare has quit IRC | 11:59 | |
| *** jklare has joined #openstack-infra | 12:00 | |
| *** Xuchu has quit IRC | 12:05 | |
| *** hashar has quit IRC | 12:07 | |
| *** rfolco has joined #openstack-infra | 12:09 | |
| *** hashar has joined #openstack-infra | 12:10 | |
| openstackgerrit | Hervé Rousseau proposed openstack/diskimage-builder master: Break retry loop on success in dhcp-all-interfaces https://review.opendev.org/696873 | 12:15 |
| *** ociuhandu has joined #openstack-infra | 12:24 | |
| *** jpena is now known as jpena|lunch | 12:26 | |
| *** ociuhandu has quit IRC | 12:29 | |
| *** udesale has quit IRC | 12:36 | |
| *** udesale has joined #openstack-infra | 12:37 | |
| *** SurajPatil has joined #openstack-infra | 12:46 | |
| *** surpatil has quit IRC | 12:48 | |
| *** soniya29 has quit IRC | 12:52 | |
| *** rlandy has joined #openstack-infra | 12:57 | |
| *** rh-jelabarre has joined #openstack-infra | 12:59 | |
| *** rosmaita has joined #openstack-infra | 13:02 | |
| *** rh-jelabarre has quit IRC | 13:08 | |
| *** rh-jelabarre has joined #openstack-infra | 13:08 | |
| *** ccamacho has quit IRC | 13:12 | |
| *** ccamacho has joined #openstack-infra | 13:12 | |
| *** efried_pto is now known as efried | 13:13 | |
| *** ccamacho has quit IRC | 13:13 | |
| *** ccamacho has joined #openstack-infra | 13:13 | |
| *** jamesmcarthur has joined #openstack-infra | 13:15 | |
| *** dave-mccowan has joined #openstack-infra | 13:15 | |
| *** lucasagomes_ has joined #openstack-infra | 13:16 | |
| *** lucasagomes_ has quit IRC | 13:16 | |
| *** lucasagomes has quit IRC | 13:17 | |
| *** lucasagomes has joined #openstack-infra | 13:20 | |
| *** dave-mccowan has quit IRC | 13:21 | |
| *** jpena|lunch is now known as jpena | 13:24 | |
| *** surpatil has joined #openstack-infra | 13:26 | |
| *** SurajPatil has quit IRC | 13:28 | |
| *** ociuhandu has joined #openstack-infra | 13:30 | |
| *** SurajPatil has joined #openstack-infra | 13:30 | |
| *** pgaxatte has joined #openstack-infra | 13:31 | |
| *** surpatil has quit IRC | 13:33 | |
| *** jamesmcarthur has quit IRC | 13:35 | |
| *** ociuhandu has quit IRC | 13:55 | |
| *** ociuhandu has joined #openstack-infra | 13:55 | |
| *** surpatil has joined #openstack-infra | 13:56 | |
| *** SurajPatil has quit IRC | 13:56 | |
| *** jroll has quit IRC | 13:57 | |
| *** SurajPatil has joined #openstack-infra | 13:58 | |
| openstackgerrit | Bogdan Dobrelya (bogdando) proposed zuul/zuul-jobs master: DNM testing custom pypi mirrors (696337) https://review.opendev.org/696867 | 13:58 |
| *** jroll has joined #openstack-infra | 13:59 | |
| *** surpatil has quit IRC | 14:01 | |
| openstackgerrit | Bogdan Dobrelya (bogdando) proposed zuul/zuul-jobs master: DNM testing custom pypi mirrors (696337) https://review.opendev.org/696867 | 14:02 |
| *** surpatil has joined #openstack-infra | 14:03 | |
| *** SurajPatil has quit IRC | 14:05 | |
| *** tkajinam has joined #openstack-infra | 14:06 | |
| *** Xuchu has joined #openstack-infra | 14:06 | |
| *** ociuhandu has quit IRC | 14:10 | |
| *** rlandy is now known as rlandy|mtg | 14:14 | |
| *** Xuchu has quit IRC | 14:16 | |
| *** ykarel is now known as ykarel|afk | 14:17 | |
| *** jamesmcarthur has joined #openstack-infra | 14:17 | |
| *** haleyb|away is now known as haleyb | 14:17 | |
| *** SurajPatil has joined #openstack-infra | 14:18 | |
| *** surpatil has quit IRC | 14:21 | |
| *** ociuhandu has joined #openstack-infra | 14:22 | |
| *** lbragstad_ is now known as lbragstad | 14:22 | |
| *** eharney has joined #openstack-infra | 14:23 | |
| openstackgerrit | Bogdan Dobrelya (bogdando) proposed zuul/zuul-jobs master: DNM testing custom pypi mirrors (696337) https://review.opendev.org/696867 | 14:23 |
| *** Goneri has joined #openstack-infra | 14:41 | |
| *** mriedem has joined #openstack-infra | 14:48 | |
| *** jamesmcarthur has quit IRC | 14:49 | |
| openstackgerrit | Hervé Rousseau proposed openstack/diskimage-builder master: Break retry loop on success in dhcp-all-interfaces https://review.opendev.org/696873 | 14:51 |
| *** tkajinam has quit IRC | 14:58 | |
| *** jamesmcarthur has joined #openstack-infra | 15:03 | |
| *** ociuhandu has quit IRC | 15:14 | |
| *** ociuhandu has joined #openstack-infra | 15:15 | |
| *** rlandy|mtg is now known as rlandy | 15:18 | |
| *** ociuhandu has quit IRC | 15:21 | |
| *** dpawlik has quit IRC | 15:22 | |
| *** SurajPatil has quit IRC | 15:24 | |
| *** ociuhandu has joined #openstack-infra | 15:32 | |
| openstackgerrit | Merged openstack/cookiecutter master: Remove py27 support for future projects https://review.opendev.org/694312 | 15:33 |
| openstackgerrit | David Moreau Simard proposed zuul/zuul master: DNM: Test zuul-stream-functional with ara 1.3 https://review.opendev.org/694622 | 15:35 |
| *** ociuhandu has quit IRC | 15:38 | |
| *** ociuhandu has joined #openstack-infra | 15:39 | |
| openstackgerrit | Merged zuul/nodepool master: Delete images by id https://review.opendev.org/693672 | 15:41 |
| *** eernst has joined #openstack-infra | 15:42 | |
| *** diablo_rojo has joined #openstack-infra | 15:43 | |
| *** diablo_rojo has quit IRC | 15:44 | |
| *** diablo_rojo has joined #openstack-infra | 15:44 | |
| *** ociuhandu has quit IRC | 15:44 | |
| *** ociuhandu has joined #openstack-infra | 15:48 | |
| *** Lucas_Gray has quit IRC | 15:48 | |
| *** dpawlik has joined #openstack-infra | 15:51 | |
| *** Lucas_Gray has joined #openstack-infra | 15:53 | |
| *** Lucas_Gray has quit IRC | 15:54 | |
| *** ykarel|afk is now known as ykarel|away | 15:54 | |
| *** eernst has quit IRC | 15:55 | |
| *** dpawlik has quit IRC | 15:56 | |
| *** eharney has quit IRC | 15:59 | |
| mriedem | i posted this a few days ago https://review.opendev.org/#/c/696212/ and have rechecked it but never get any results from zuul - i can't figure out if there is a problem with the change? | 15:59 |
| mriedem | there are changes in the series after that one that have results, so it's not like something to do with checking out the entire series to run tests | 16:00 |
| *** ociuhandu has quit IRC | 16:01 | |
| fungi | i haven't looked yet, but could there be a circular dependency somewhere in there? | 16:01 |
| mriedem | doesn't depends-on anything | 16:01 |
| *** ykarel|away has quit IRC | 16:02 | |
| mriedem | i do'nt see anything in the openstack-tox-docs job that would filter it out and not run any jobs on it | 16:02 |
| Shrews | mriedem: you're changing a file in the doc directory. Most of the nova jobs in .zuul.yaml seem to include that in the irrelevant-files section, so they won't run with that change. What job are you expecting to run that didn't? | 16:05 |
| *** armax has joined #openstack-infra | 16:05 | |
| Shrews | I don't see a nova job to test generating docs, which is surprising to me | 16:08 |
| fungi | usually there's a project-template which includes a tox docs job | 16:09 |
| frickler | probably unrelated to mriedem's issue, but there's also a bunch of execptions in the zuul log http://paste.openstack.org/show/786995/ | 16:11 |
| mriedem | Shrews: publish-openstack-docs-pti is the template nova uses which should run the openstack-tox-docs job | 16:12 |
| *** ociuhandu has joined #openstack-infra | 16:15 | |
| *** chkumar|ruck is now known as raukadah | 16:17 | |
| frickler | mriedem: https://review.opendev.org/656656 is below your patch in the stack and disabled most jobs | 16:18 |
| *** ociuhandu has quit IRC | 16:19 | |
| *** Lucas_Gray has joined #openstack-infra | 16:20 | |
| *** jamesmcarthur has quit IRC | 16:24 | |
| *** jamesmcarthur has joined #openstack-infra | 16:26 | |
| mriedem | oh gdi | 16:27 |
| mriedem | i knew it was going to be something dumb like that :/ sorry | 16:28 |
| *** gyee has joined #openstack-infra | 16:29 | |
| *** ykarel|away has joined #openstack-infra | 16:35 | |
| *** lbragsta_ has joined #openstack-infra | 16:36 | |
| *** gyee has quit IRC | 16:36 | |
| *** rpittau is now known as rpittau|afk | 16:38 | |
| *** udesale has quit IRC | 16:41 | |
| openstackgerrit | Bogdan Dobrelya (bogdando) proposed zuul/zuul-jobs master: DNM testing custom pypi mirrors (696337) https://review.opendev.org/696918 | 16:42 |
| openstackgerrit | Bogdan Dobrelya (bogdando) proposed zuul/zuul-jobs master: DNM testing custom pypi mirrors (696337) https://review.opendev.org/696918 | 16:43 |
| openstackgerrit | Bogdan Dobrelya (bogdando) proposed opendev/base-jobs master: DNM testing custom pypi mirrors (696337) https://review.opendev.org/696921 | 16:45 |
| *** gyee has joined #openstack-infra | 16:51 | |
| *** hashar has quit IRC | 16:51 | |
| *** trident has quit IRC | 16:51 | |
| *** trident has joined #openstack-infra | 16:53 | |
| *** lucasagomes has quit IRC | 16:55 | |
| *** df_sbr has joined #openstack-infra | 16:56 | |
| *** eharney has joined #openstack-infra | 16:57 | |
| *** pgaxatte has quit IRC | 17:01 | |
| openstackgerrit | Bogdan Dobrelya (bogdando) proposed zuul/zuul-jobs master: DNM testing custom pypi mirrors (696337) #2 https://review.opendev.org/696929 | 17:01 |
| *** Lucas_Gray has quit IRC | 17:09 | |
| *** Lucas_Gray has joined #openstack-infra | 17:13 | |
| *** sshnaidm is now known as sshnaidm|afk | 17:13 | |
| *** ijw has joined #openstack-infra | 17:15 | |
| *** KeithMnemonic has joined #openstack-infra | 17:18 | |
| *** iurygregory has quit IRC | 17:19 | |
| *** Lucas_Gray has quit IRC | 17:23 | |
| *** ykarel has joined #openstack-infra | 17:32 | |
| *** ykarel|away has quit IRC | 17:34 | |
| *** Goneri has quit IRC | 17:40 | |
| *** michael-beaver has joined #openstack-infra | 17:41 | |
| openstackgerrit | Merged zuul/zuul master: Ensure correct re-enqueue on changing gate queue https://review.opendev.org/696578 | 17:43 |
| *** lbragsta_ has quit IRC | 17:44 | |
| donnyd | infraroot FN is experiencing an outage of the mirror. | 17:44 |
| jrosser | i have seen a few of these with the fortnebula repo https://zuul.opendev.org/t/openstack/build/dfa6f3595b9f4e499db7093e24c0edca/log/job-output.txt#2075 | 17:44 |
| donnyd | Fix should be in | 17:50 |
| donnyd | My apologies | 17:50 |
| fungi | thanks donnyd! | 17:51 |
| fungi | no worries | 17:51 |
| *** dpawlik has joined #openstack-infra | 17:52 | |
| *** Goneri has joined #openstack-infra | 17:54 | |
| *** dpawlik has quit IRC | 17:57 | |
| *** jpena is now known as jpena|off | 17:59 | |
| *** derekh has quit IRC | 18:00 | |
| openstackgerrit | James E. Blair proposed zuul/zuul-jobs master: WIP: openshift speculative containers https://review.opendev.org/696939 | 18:01 |
| *** factor has quit IRC | 18:04 | |
| *** Darcidride has joined #openstack-infra | 18:07 | |
| *** Darcidride has quit IRC | 18:07 | |
| corvus | donnyd, infra-root: i just saw another fn error: https://zuul.opendev.org/t/zuul/build/8627bc1fa7a0410a82010482585210a2 | 18:12 |
| corvus | E: Failed to fetch http://mirror.regionone.fortnebula.opendev.org/ubuntu/dists/bionic/main/binary-amd64/Packages 403 Forbidden [IP: 2001:470:e045:2:f816:3eff:fee6:691d 80] | 18:12 |
| fungi | that's still giving me a 403 as well | 18:13 |
| corvus | and not answering ssh for me | 18:13 |
| donnyd | But the machine is alive? | 18:13 |
| fungi | i wonder if that ip address got mapped to something else? | 18:13 |
| fungi | i can ssh into it now | 18:14 |
| openstackgerrit | James E. Blair proposed openstack/project-config master: Disable fortnebula https://review.opendev.org/696941 | 18:14 |
| fungi | 2001:470:e0d6:0:96de:80ff:feec:f9e7 - - [2019-12-02 18:15:33.431] "GET /ubuntu/dists/bionic/main/binary-amd64/Packages HTTP/1.1" 403 199 - "-" "Mozilla/5.0 (X11; Linux x86_64; rv:70.0) Gecko/20100101 Firefox/70.0" | 18:15 |
| fungi | the request is reaching it | 18:15 |
| fungi | that's weird | 18:16 |
| corvus | agreed | 18:16 |
| corvus | fungi: are you sshing via ipv4 or 6? | 18:16 |
| fungi | v6 | 18:16 |
| corvus | i'm on v4 and still failing | 18:17 |
| fungi | last change to that apache config was in september | 18:17 |
| corvus | fungi: afs isn't going to work well if ipv4 isn't working | 18:17 |
| *** ijw has quit IRC | 18:17 | |
| corvus | fungi: maybe see if you can browse the filesystem? | 18:17 |
| fungi | aha, yep | 18:17 |
| corvus | errors there might show up as 403's | 18:17 |
| fungi | ls: cannot open directory '/afs/openstack.org/': Connection timed out | 18:18 |
| fungi | that's gotta be it | 18:18 |
| corvus | (if apache can't figure out if there's an .htaccess file, then it can't determine if the client is authorized) | 18:18 |
| corvus | donnyd: ^ suspected ipv4 issue | 18:19 |
| openstackgerrit | Merged openstack/project-config master: Disable fortnebula https://review.opendev.org/696941 | 18:20 |
| fungi | i bypassed gating on ^ | 18:21 |
| *** ricolin has quit IRC | 18:22 | |
| *** spsurya has quit IRC | 18:25 | |
| donnyd | Thanks fungi | 18:30 |
| donnyd | I can take a deeper look today | 18:31 |
| fungi | appreciated, just let us know if you need any help! | 18:41 |
| *** gfidente is now known as gfidente|adk | 18:45 | |
| *** gfidente|adk is now known as gfidente|afk | 18:45 | |
| *** igordc has joined #openstack-infra | 18:45 | |
| fungi | can i get a second reviewer on 682425 for system-config? retiring some long-abandoned mailing lists | 18:57 |
| *** Tengu has quit IRC | 19:00 | |
| *** lpetrut has quit IRC | 19:00 | |
| *** Tengu has joined #openstack-infra | 19:01 | |
| *** dpawlik has joined #openstack-infra | 19:01 | |
| *** ijw has joined #openstack-infra | 19:05 | |
| *** dpawlik has quit IRC | 19:06 | |
| *** ykarel has quit IRC | 19:07 | |
| openstackgerrit | Merged opendev/system-config master: Retire the Women-of-OpenStack list https://review.opendev.org/682417 | 19:19 |
| *** jamesmcarthur has quit IRC | 19:19 | |
| openstackgerrit | Merged opendev/system-config master: Retire elections-committee, openstack-content MLs https://review.opendev.org/682425 | 19:21 |
| *** hashar has joined #openstack-infra | 19:27 | |
| *** dpawlik has joined #openstack-infra | 19:41 | |
| *** dpawlik has quit IRC | 19:46 | |
| *** tesseract has quit IRC | 19:49 | |
| *** thedac_ has joined #openstack-infra | 19:54 | |
| *** fdegir4 has joined #openstack-infra | 19:54 | |
| *** donnyd_ has joined #openstack-infra | 19:54 | |
| *** jrosser_ has joined #openstack-infra | 19:55 | |
| *** persia_ has joined #openstack-infra | 19:56 | |
| *** tinwood_ has joined #openstack-infra | 19:56 | |
| *** mugsie_ has joined #openstack-infra | 19:56 | |
| *** ijw_ has joined #openstack-infra | 19:57 | |
| *** StevenK_ has joined #openstack-infra | 19:57 | |
| *** ijw has quit IRC | 20:02 | |
| *** rosmaita has quit IRC | 20:02 | |
| *** ralonsoh has quit IRC | 20:02 | |
| *** lourot has quit IRC | 20:02 | |
| *** pcaruana has quit IRC | 20:02 | |
| *** rascasoft has quit IRC | 20:02 | |
| *** fdegir has quit IRC | 20:02 | |
| *** Roamer` has quit IRC | 20:02 | |
| *** gibi has quit IRC | 20:02 | |
| *** Jeffrey4l_ has quit IRC | 20:02 | |
| *** thedac has quit IRC | 20:02 | |
| *** tinwood has quit IRC | 20:02 | |
| *** persia has quit IRC | 20:02 | |
| *** mugsie has quit IRC | 20:02 | |
| *** StevenK has quit IRC | 20:02 | |
| *** jrosser has quit IRC | 20:02 | |
| *** donnyd has quit IRC | 20:02 | |
| *** fnordahl has quit IRC | 20:02 | |
| *** clarkb has quit IRC | 20:02 | |
| *** cmurphy has quit IRC | 20:02 | |
| *** thedac_ is now known as thedac | 20:02 | |
| *** donnyd_ is now known as donnyd | 20:02 | |
| *** jrosser_ is now known as jrosser | 20:02 | |
| *** Jeffrey4l has joined #openstack-infra | 20:02 | |
| *** clarkb has joined #openstack-infra | 20:03 | |
| *** cmurphy has joined #openstack-infra | 20:03 | |
| *** ralonsoh has joined #openstack-infra | 20:04 | |
| *** lourot has joined #openstack-infra | 20:05 | |
| *** pcaruana has joined #openstack-infra | 20:05 | |
| *** rosmaita has joined #openstack-infra | 20:06 | |
| *** gyee has quit IRC | 20:09 | |
| *** michael-beaver has quit IRC | 20:10 | |
| *** diablo_rojo has quit IRC | 20:14 | |
| *** ralonsoh has quit IRC | 20:20 | |
| donnyd | ok its working here again | 20:22 |
| fungi | checking | 20:22 |
| donnyd | so weird... did anyone restart anything on the mirror node??? | 20:23 |
| donnyd | There was a power surge that blew up an ups on saturday but didn't effect any workloads until sunday... and things have been bonkers since | 20:23 |
| openstackgerrit | James E. Blair proposed zuul/zuul-jobs master: WIP: openshift speculative containers https://review.opendev.org/696939 | 20:24 |
| *** auristor has quit IRC | 20:25 | |
| fungi | afs is working there again | 20:25 |
| fungi | corvus: your ssh is back to normal? | 20:25 |
| fungi | donnyd: looks like the server was rebooted 2h36m ago at 17:49z | 20:26 |
| donnyd | That was me turning it back on | 20:26 |
| fungi | i'm the only one who has logged in since and i have done no restarts | 20:26 |
| fungi | i'll chec older lastlog | 20:27 |
| fungi | donnyd: no logins at all recorded in /var/log/wtmp.1 | 20:27 |
| donnyd | ok well maybe it just took a while to get going... | 20:29 |
| corvus | fungi: yep, i can ssh in now | 20:29 |
| *** lucasagomes has joined #openstack-infra | 20:29 | |
| fungi | so unless log rotation or unattended upgrades restarted something, i can't imagine what would have happened within the guest to alter things | 20:29 |
| donnyd | I don't have any other answers because nothing was actually changed with the exception of turning the instance back on. | 20:29 |
| *** lucasagomes has quit IRC | 20:30 | |
| donnyd | I know why it went down initially, it was from the power surge... but it took a very very long time to come back up | 20:30 |
| *** lucas_ has joined #openstack-infra | 20:30 | |
| fungi | something something dhcp something? | 20:30 |
| donnyd | LOL. so do we want to turn FN back on? | 20:31 |
| *** lucas_ is now known as lucasagomes | 20:31 | |
| fungi | i'm happy to push up a revert, and if you can +1 it i'll feel better about +2ing it myself | 20:32 |
| *** hashar has quit IRC | 20:33 | |
| openstackgerrit | Jeremy Stanley proposed openstack/project-config master: Revert "Disable fortnebula" https://review.opendev.org/696959 | 20:34 |
| fungi | donnyd: corvus: ^ | 20:34 |
| openstackgerrit | Colleen Murphy proposed opendev/glean master: Add support for SLES https://review.opendev.org/696960 | 20:34 |
| * cmurphy wonders if people still review glean ^ | 20:35 | |
| *** gyee has joined #openstack-infra | 20:36 | |
| *** auristor has joined #openstack-infra | 20:36 | |
| donnyd | fungi: just make sure you read my comments | 20:37 |
| donnyd | done | 20:37 |
| fungi | cmurphy: i think they do, when they find out there's work being done on it | 20:38 |
| fungi | thanks donnyd! | 20:38 |
| fungi | best review comment evar | 20:38 |
| donnyd | cmurphy: there were plenty when we were working out the ipv6 stuff | 20:39 |
| donnyd | i figured someone would get a kick out of it | 20:39 |
| fungi | cmurphy: do you think bindep needs a similar patch, or is it already working? | 20:40 |
| cmurphy | fungi: not sure, i'm not using bindep atm | 20:41 |
| *** lucasagomes has quit IRC | 20:41 | |
| fungi | no worries, just didn't know if you had looked | 20:42 |
| fungi | thanks for the glean fix! | 20:42 |
| cmurphy | thanks for the review! | 20:42 |
| *** lucasagomes has joined #openstack-infra | 20:44 | |
| *** lucasagomes has quit IRC | 20:50 | |
| ianw | corvus: so i can duplicate the nodepool functional test in system-config to use containers built in system-config. but that doesn't automatically test changes from nodepool, openstacksdk & diskimage-builder | 20:52 |
| openstackgerrit | Merged openstack/project-config master: Revert "Disable fortnebula" https://review.opendev.org/696959 | 20:52 |
| ianw | it doesn't seem zuul-ish to say "thanks for the change, can you just up a [dnm] change in system-config and depends-on this" so we can confirm it doesn't break container operation | 20:52 |
| ianw | which is why it seems to make more sense for a master job in zuul/nodepool to run the test; which requires using the system-config image build jobs in zuul/nodepool | 20:54 |
| pabelanger | ianw: couldn't system-config be setup as 3pci job, then run builds and report back to gerrit change on zuul/zuul? | 20:55 |
| corvus | the thing i'd like to avoid happening is repeating the situation where all testing of dib was in nodepool | 20:56 |
| corvus | dib and openstacksdk deserve to have robust standalone testing | 20:56 |
| ianw | i wish to use the same nodepool functional+containers job in all of them | 20:57 |
| corvus | i hope we're not worried about whether sdk works in a container or not, so just looking at dib, why not set up jobs on dib that run builds in containers? that can be done without nodepool. | 20:57 |
| ianw | well the thing i've found so useful is that we test what we run, not simulations of. that's why i'd like to run the opendev containers as the test | 20:58 |
| corvus | i think it would be great for system-config to test what it runs, so it should totally run a real nodepool container image test. | 20:59 |
| corvus | but i don't think that we need to entangle all of these projects to do it. each project can test what it needs | 21:00 |
| ianw | tbh i feel like it's almost a regression not testing the master changes of nodepool, dib and openstacksdk against an opendev functional+containers job ... we have that basically now with the test pip installing, which is the same thing that happens in production | 21:04 |
| corvus | dib and sdk are installed from releases | 21:05 |
| ianw | but each change has passed the nodepool functional tests before merge | 21:06 |
| *** eharney has quit IRC | 21:06 | |
| corvus | i feel like we're getting off the track here | 21:08 |
| corvus | i think it's important to maintain appropriate separation between the projects. | 21:10 |
| corvus | honestly, it's pretty hard to think about this since we're still waffling on whether nodepool should even be building images with dib | 21:11 |
| *** mattw4 has joined #openstack-infra | 21:13 | |
| pabelanger | sorry, I think I missed some discussions. I am processing that as, there has been some talk to stop using dib altogether in nodepool-builder? | 21:14 |
| corvus | nope | 21:14 |
| *** auristor has quit IRC | 21:14 | |
| pabelanger | okay, cool | 21:14 |
| corvus | pabelanger: some of the words in that statement can be read 2 ways, let me clarify | 21:14 |
| corvus | pabelanger: i meant to say: whether the container images that the zuul project creates for nodepool and publishes on dockerhub as zuul/nodepool-builder should contain dib | 21:15 |
| *** jtomasek has quit IRC | 21:15 | |
| pabelanger | ack | 21:16 |
| corvus | (having said that, the project has always been open to other implementations of image builders; it's in the original spec. but no one has suggested that we stop having a dib-based builder) | 21:16 |
| Shrews | Well, on a personal level, I too prefer to keep dependencies unentangled (assuming that's a word) | 21:17 |
| ianw | i feel they should *not* include dib. it's not the idea of containers to bundle the kitchen sink, we should compose an opendev image | 21:17 |
| ianw | tristanC wrote a virt-customize wrapper that obeys dib command-lines, for example | 21:17 |
| pabelanger | yes, that is how I understood it too (other builders) | 21:17 |
| corvus | ianw: as there is currently no other way to use 'nodepool-builder' i think it should include dib. | 21:17 |
| *** diablo_rojo has joined #openstack-infra | 21:17 | |
| *** auristor has joined #openstack-infra | 21:18 | |
| corvus | if/when there are other options, we should make more images | 21:18 |
| pabelanger | ianw: I'm starting to POC a packer.io based builder for nodepool for reasons (not to distract on current topic) | 21:19 |
| corvus | pabelanger: feel free to upstream that -- i'd love to have a second option to make this less theoretical | 21:20 |
| ianw | https://softwarefactory-project.io/r/#/c/16896/ is what i'm thinking of | 21:21 |
| tristanC | corvus: nodepool-builder can be used without dib, in fact we could document the dib-cmd (nodepool attribute) cli interface and let user use custom tool | 21:22 |
| corvus | tristanC: i'm not sure that the dib cli is meant to be a stable api? perhaps it would be better to make new upstream nodepool builder drivers to support other tools first class? | 21:23 |
| tristanC | corvus: it hasn't changed in recent years, but yes, it would be even better to have stable cli arguments | 21:24 |
| ianw | dib-cmd was mostly about making the fake-dib used during testing less of an inbuilt hack. but yes once you give people a hook, they're going to hook into it :) | 21:24 |
| Shrews | hmm, making that a proper interface might actually be fun | 21:25 |
| tristanC | Shrews: corvus: i would vote for a image-builder-playbook attribute that nodepool-builder would execute with -e "image_output_path=Path image_type=raw|qcow|vhd" | 21:25 |
| corvus | yeah, all the way back in https://docs.opendev.org/opendev/infra-specs/latest/specs/nodepool-workers.html we said that we would support multiple builder processes (see "Optional: create nodepool-builder that implements nodepool on-image builds"). but that hasn't happened yet. we still could do that. also, we could abstract a driver interface so that the bulk of the existing builder process can | 21:27 |
| corvus | be reused and extended easily. that would probably be best for most of the things under discussion here, which are basically just "different external processes that emit an image file" | 21:27 |
| pabelanger | tristanC: would be interested to see if you have looked at packer.io for ansible integration, that is currently something we are exploring over ansible nodepool-builder (but maybe that is better route) | 21:29 |
| pabelanger | more info in #zuul | 21:30 |
| tristanC | pabelanger: no we haven't looked at packer, we are pretty satisfied by virt-customize | 21:31 |
| corvus | anyway, it seems reasonable to me that zuul/nodepool-builder should include dib. and as folks upstream other options (packer, virt-customize), we can build other images. | 21:31 |
| corvus | the main thing is that i think it would be best if the images produced by nodepool were useful. | 21:32 |
| pabelanger | tristanC: is there ansible integraiton in virt-customize? I didn't see that, but may have missed | 21:32 |
| tristanC | corvus: i find this reasonable too | 21:32 |
| tristanC | pabelanger: i haven't look, we use ansible to fetch and maintain the cloud image cache, then we call virt-customize with the command module | 21:33 |
| tristanC | pabelanger: you can see the nodepool-builder log when using virt-customize here: https://softwarefactory-project.io/nodepool-log/cloud-centos-7-0000000803.log | 21:34 |
| openstackgerrit | Slawek Kaplonski proposed openstack/project-config master: Add 'Review-Priority' for Neutron stadium repos https://review.opendev.org/696628 | 21:34 |
| pabelanger | tristanC: ah, okay cool. Yah, was hoping to run playbooks against image, over CLI commands. But could be a first step | 21:35 |
| *** dpawlik has joined #openstack-infra | 21:43 | |
| *** auristor has quit IRC | 21:46 | |
| *** dpawlik has quit IRC | 21:48 | |
| *** slaweq has quit IRC | 21:48 | |
| *** StevenK_ is now known as StevenK | 21:49 | |
| *** auristor has joined #openstack-infra | 21:53 | |
| *** fnordahl has joined #openstack-infra | 21:56 | |
| *** rcernin has joined #openstack-infra | 21:57 | |
| *** auristor has quit IRC | 21:57 | |
| *** mugsie_ is now known as mugsie | 21:59 | |
| *** rcernin has quit IRC | 21:59 | |
| *** auristor has joined #openstack-infra | 22:00 | |
| ianw | corvus: so you would like to see nodepool importing container build jobs from openstacksdk and then dib to build it's final nodepool-builder image? | 22:00 |
| ianw | corvus: this is where we started. the issue is that importing openstacksdk jobs into the zuul tenant absolutely explodes the dependency list, creating a ton of config errors | 22:01 |
| corvus | i would rather not share jobs | 22:01 |
| corvus | the tenants are separate for a reason | 22:02 |
| corvus | and i'm sorry, i don't mean to be obtuse, i just don't see the big picture on this yet. i don't know what we should do, but i think i know some things i don't think we should do. | 22:03 |
| corvus | ianw: the last thing i remember on the subject was the suggestion that we just add dib to the nodepool-builder image | 22:05 |
| corvus | could we start with that, then run that image in system-config and see what happens? | 22:06 |
| ianw | corvus: so the method to do this, the way things are written, is to build zuul/nodepool-builder ontop of a openstack/diskimage-builder container | 22:07 |
| corvus | i thought i saw a change to just install dib in the nodepool-builder image | 22:07 |
| corvus | like, a change to nodepool's dockerfile | 22:07 |
| *** factor has joined #openstack-infra | 22:08 | |
| corvus | oh here's the one for vhd util https://review.opendev.org/693306 | 22:09 |
| corvus | maybe thet's what i'm thinking of | 22:09 |
| ianw | corvus: what i would like, however, is for the test to be able to speculatively test dib changes | 22:10 |
| *** pcaruana has quit IRC | 22:11 | |
| ianw | to restrict this to just nodepool to simplify, the current functional test obeys Depends-On: for openstacksdk and dib | 22:12 |
| *** eharney has joined #openstack-infra | 22:13 | |
| corvus | ianw: got it, that's a problem i can hold in my head. the first practical issue is that speculative container dependencies won't work across tenants because zuul won't perform the lookup of previous builds across tenants. so the only way for a job to test a container image with a change to both projects is to build that image from scratch within that job. | 22:14 |
| *** dchen has quit IRC | 22:14 | |
| ianw | corvus: that is what i have implemented in https://review.opendev.org/#/c/693464/29/.zuul.yaml | 22:17 |
| ianw | it is clearer in -> https://review.opendev.org/#/c/694177/18/.zuul.yaml | 22:18 |
| corvus | the second practical issue, is that, to my knowledge, there are no container build jobs that build more than just the project under test (ie, i don't think our python-builder image obeys siblings) | 22:19 |
| ianw | in 694177 the buildset registry starts, builds an openstacksdk image (openstacksdk-build-opendev-image) then a dib image that depends on that (dib-build-opendev-image) and then nodepool-builder images that depend on that | 22:20 |
| corvus | so why not put all of that in the dib repo? | 22:22 |
| *** dpawlik has joined #openstack-infra | 22:22 | |
| ianw | corvus: python-builder does not, and really can-not; because docker runs from a context. so you can not do "pip install /home/zuul/<project>/" to install a speculative checkout; docker has no access to /home | 22:22 |
| ianw | corvus: when you say all of that, what do you mean? | 22:23 |
| corvus | ianw: yep. we could, if we wanted, standardize on a way of supporting that with siblings. probably by doing something similar to what you've done. | 22:23 |
| corvus | ianw: those jobs | 22:23 |
| corvus | ianw: apply https://review.opendev.org/693464 to the dib repo instead | 22:23 |
| corvus | or, something similar (without the reliance on system-config) | 22:24 |
| ianw | i'm not sure that would help? even for just our simplified "we want a container test job for nodepool that obeys Depends-On: for diskimage-builder and openstacksdk" unless that job builds the containers, it won't obey speculative builds? | 22:26 |
| *** dpawlik has quit IRC | 22:26 | |
| ianw | this is what led me to 696000. in there, we keep the dockerfiles for each project, and copy it into the zuul checkout (e.g. https://review.opendev.org/#/c/696000/18/playbooks/zuul/docker/diskimage-builder.yaml) | 22:28 |
| *** auristor has quit IRC | 22:28 | |
| ianw | we then use zuul_work_dir to make sure we're in the directory for that project | 22:28 |
| corvus | let's assume that nodepool does not depend on a diskimage-builder container image. in that case, you can make a job that runs in the dib project that builds a nodepool container image. after solving the sibling problem, you should be set. | 22:29 |
| corvus | if you solve the sibling problem by implementing it as a role in zuul-jobs, you can use the same role in jobs for any of the involved projects. | 22:30 |
| corvus | then we don't really need to share jobs, because the jobs aren't special. the only special thing is the role that builds a container image with siblings, and that's available to any job. | 22:32 |
| ianw | the sibling problem does not so much seem to be a role problem as a python-builder problem | 22:33 |
| corvus | maybe both? | 22:33 |
| ianw | you're essentially saying python-builder should be installing multiple projects at a time | 22:33 |
| corvus | i'm handwaving. i'm saying that no matter what approach we use, the sibling problem needs to be solved. | 22:34 |
| ianw | my understanding was that the best way to solve that problem was to build projects into their own containers and then build them ontop of each other | 22:35 |
| corvus | i don't think that image dependencies are a solution to this -- they are one-dimensional. | 22:35 |
| corvus | so that only works for the first thing you want to install like this. not the second. | 22:35 |
| ianw | i do agree, it's basically multiple inheritance that Dockerfiles don't do | 22:36 |
| corvus | but if we say that the nodepool dockerfile is going to install dib (and whatever else we want), then both nodepool and dib projects can build that image. | 22:37 |
| corvus | also, you save 30 minutes in build time :) | 22:37 |
| corvus | these chains of images can get tediously slow | 22:38 |
| corvus | ianw: i prefer the docker-image-with-siblings approach, mostly because, aside from everything else, i don't think that the nodepool container image should depend on a dib container image. but, for exploration purposes, i had a thought about your stacked image approach i'd like to share: | 22:42 |
| corvus | ianw: if you were to do something like 693464, you might consider rather than chaining 3 jobs together, you could write one job which ran the standard image build roles 3 times. that would speed things up, and possibly reduce the need/desire to share jobs between tenants. | 22:44 |
| corvus | (they would still use the buildset registry as appropriate) | 22:44 |
| *** rh-jelabarre has quit IRC | 22:44 | |
| ianw | corvus: on the siblings approach, basically you are saying install nodepool+dib+openstacksdk together from the zuul checkouts in the one container | 22:47 |
| corvus | ianw: yep | 22:47 |
| ianw | one issue see with that is that the Dockerfile is going to depend on external state of the other repositories being available | 22:48 |
| ianw | it will basically only run under zuul | 22:48 |
| corvus | i think it would somehow need to be optional | 22:48 |
| corvus | and yeah, it's not the first dockerfile-related disappointment we've encountered | 22:49 |
| ianw | in that case, i'd imagine you would keep it as a separate dockerfile (Dockerfile.zuul, say) | 22:49 |
| corvus | and just to level-set: i'm not sure this is a good idea, but it is an idea. :) | 22:49 |
| corvus | i'm thinking maybe there's something the assemble script in python-base could do to handle this | 22:50 |
| corvus | so you could still have one dockerfile | 22:50 |
| mordred | yeah - what if we made a Dockerfile.zuul like ianw suggests - but also have a normal Dockerfile that installs depends not from source | 22:51 |
| corvus | er python-builder | 22:51 |
| corvus | why couldn't they be the same? | 22:51 |
| ianw | corvus: right, that's what i'm thinking on too. the problem is the build context; the Dockerfile (and, ergo, assemble script) has to have access to all the source repositories | 22:52 |
| mordred | well - dockerfiles dont' really do conditionals particularly well - especially since one of the steps is going to nede to be a COPY | 22:52 |
| mordred | yeah | 22:52 |
| corvus | that's where the role comes in | 22:52 |
| ianw | but we are back to having a Dockerfile that, essentially, only runes under zuul (via role magic) | 22:53 |
| mordred | but if the role does that - then the dockefile only works if you're using it via the zuul role, becuase you need extra external context - and in that case it would likely be nicer to just to scrap the dockerfile and write a script that used buildah commands or something | 22:53 |
| corvus | if you run docker build in a checkout, you get an image built in the normal way. if you use a zuul role to build a siblings-image, it moves repos around so the context includes all the repos, and the assemble script knows what to do with it. | 22:53 |
| mordred | ah - I think I get what you're getting at here | 22:53 |
| mordred | basically - put the siblings into a subdir of the normal context | 22:54 |
| mordred | so when "copy ." happens - it includes all of the siblines | 22:54 |
| mordred | and then assemble detects it got a complex context not a simple one | 22:54 |
| mordred | yeah? | 22:54 |
| corvus | yep. that's the vague hand-wavey idea. | 22:54 |
| mordred | that makes enough sense to me that I *think* I could write a wip python-builder patch | 22:55 |
| corvus | ianw: you did a similar experiment, right? | 22:55 |
| ianw | so maybe assemble sees a ".zuul-siblings" directory, that contains hard-links to the siblings projects | 22:56 |
| corvus | ianw: sounds plausible | 22:56 |
| ianw | (i do not think docker will follow symlinks outside its context) | 22:56 |
| mordred | yeah - it might have to be a copy even | 22:56 |
| mordred | but yeah- .zuul-siblings dir sounds right no | 22:56 |
| mordred | on | 22:56 |
| ianw | and then loops over directories in there, building wheels and installing | 22:56 |
| mordred | yup. and getting the sum of the bindeps | 22:57 |
| mordred | so that install-from-bindep works | 22:57 |
| *** rcernin has joined #openstack-infra | 22:57 | |
| ianw | hrm, does cat-ing bindep files together work? | 22:57 |
| mordred | dunno - but catting the output of bindep -b should be fine to do | 22:57 |
| tristanC | fwiw, you can also "podman build -v ~/src:/src" to have all the zuul projects available for RUN command without using copy | 22:58 |
| mordred | (because the builder image runs bindep and saves the output) | 22:58 |
| mordred | tristanC: oh - podman build supports volumes? that's so nice | 22:58 |
| mordred | plain docker doesn't - but I'd personally be fine saying "these dockerfiles only build under podman because it's better" - but we can discuss that as an optimization - for now copy is working ok | 22:59 |
| tristanC | mordred: oh yes, and you can do -v /var/cache/dnf:/var/cache/dnf:O that will create an overlay and discard any modification at the end of the build | 23:00 |
| mordred | corvus: ^^ that seems like functionality we should expose in build-podman-image | 23:00 |
| corvus | ++ | 23:00 |
| mordred | separate from whether we decide to use it for our images | 23:01 |
| corvus | i am in favor of build-podman-image getting better than build-docker-image. lack-of-feature compatibility is not a goal :) | 23:01 |
| mordred | ++ | 23:01 |
| mordred | it sounds like suggesting people just use podman build regardless of whether they want to use podman for running is so much yes | 23:02 |
| ianw | mordred: it seems like we should explicitly tell the building role what sibling projects to map in, rather than just assuming everything in required-projects: is also to be installed as a sibling. there might be ones you don't want installed in the container | 23:03 |
| corvus | (this is getting into #zuul territory, but anyway) i'm personally okay with making that a requirement (eventually). i think we learned from the pbrx experiment that having a non-standard way to build images is not helpful (which is why i'd like our dockerfile to continue to work in a non-zuul context). but i think growing into buildah extra features is reasonable. | 23:03 |
| mordred | ++ | 23:04 |
| mordred | ianw: yeah - I think in this context that makes good sense | 23:04 |
| ianw | so, as step 0) make docker_images: have a "siblings:" entry, that lists projects that will be copied into (say) .zuul-siblings? let me pastebin out a sample config maybe | 23:07 |
| corvus | yeah, and the podman role can maybe do the same without the copy | 23:08 |
| *** tkajinam has joined #openstack-infra | 23:09 | |
| mordred | ianw: ++ | 23:10 |
| mordred | ianw: then the next step is to make assemble look in .zuul-siblings for things to iterate over | 23:11 |
| *** Adri2000 has quit IRC | 23:12 | |
| ianw | mordred: for dib, say, the trick to building the image for functional testing will be to use the nodepool dockerfile i guess ... http://paste.openstack.org/show/787009/ | 23:15 |
| ianw | well, in fact it will want to use the zuu/nodepool image build job. that should be ok | 23:16 |
| mordred | ianw: yes, I think that's right. I think we'll want to make a job where nodeopool is zuul_work_dir and the others are siblings | 23:16 |
| mordred | yeah | 23:16 |
| ianw | so dib (and openstacksdk) can 1) start buildset repo 2) build containers using zuul/nodepool jobs 3) run zuul/nodepool functional+containers test | 23:18 |
| *** gfidente|afk has quit IRC | 23:24 | |
| *** pkopec has quit IRC | 23:27 | |
| *** dchen has joined #openstack-infra | 23:28 | |
| *** diablo_rojo has quit IRC | 23:40 | |
| *** tosky has quit IRC | 23:41 | |
| *** diablo_rojo has joined #openstack-infra | 23:42 | |
| *** diablo_rojo has quit IRC | 23:47 | |
| fungi | ianw: mordred: yes, bindep is designed with the idea that you should be able to just merge the contents of a bunch of package lists and it should properly dedupe and reconcile the results. if it doesn't, that would be considered a bug | 23:47 |
| fungi | basically you shouldn't have to run bindep more than once | 23:48 |
| mordred | cool | 23:48 |
| fungi | now, if you mix in version constraints for packages (i still think they're a terrible idea but they were a half-implemented bit of the original design) then all bets are off | 23:49 |
| fungi | not sure what will happen if you declare that you need conflicting versions of something | 23:49 |
| ianw | mordred/corvus: i still think we need https://review.opendev.org/#/c/696859/1/zuul/main.yaml to make sure the openstacksdk/dib sources are available to zuul/nodepool | 23:50 |
| openstackgerrit | Ian Wienand proposed zuul/zuul-jobs master: [wip] copy listed siblings to docker build https://review.opendev.org/696987 | 23:53 |
| *** mriedem has quit IRC | 23:54 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!