| *** tetsuro has joined #openstack-release | 00:30 | |
| *** tetsuro_ has joined #openstack-release | 00:40 | |
| *** tetsuro has quit IRC | 00:43 | |
| *** redrobot has quit IRC | 01:34 | |
| *** tetsuro has joined #openstack-release | 01:44 | |
| *** tetsuro_ has quit IRC | 01:47 | |
| *** tetsuro_ has joined #openstack-release | 02:12 | |
| *** tetsuro has quit IRC | 02:15 | |
| *** tetsuro has joined #openstack-release | 03:06 | |
| *** tetsuro_ has quit IRC | 03:09 | |
| *** jovial[m1 has quit IRC | 03:15 | |
| *** dustinc has quit IRC | 03:34 | |
| *** tetsuro_ has joined #openstack-release | 04:06 | |
| *** tetsuro has quit IRC | 04:09 | |
| *** ykarel|away is now known as ykarel | 04:26 | |
| *** evrardjp has quit IRC | 04:36 | |
| *** evrardjp has joined #openstack-release | 04:36 | |
| *** diablo_rojo has quit IRC | 04:44 | |
| *** dmellado has quit IRC | 04:45 | |
| *** udesale has joined #openstack-release | 05:36 | |
| *** jbadiapa has joined #openstack-release | 05:36 | |
| *** tetsuro has joined #openstack-release | 05:43 | |
| *** tetsuro_ has quit IRC | 05:47 | |
| *** ysandeep|away is now known as ysandeep | 05:50 | |
| *** tetsuro_ has joined #openstack-release | 06:23 | |
| *** tetsuro has quit IRC | 06:26 | |
| *** tetsuro has joined #openstack-release | 06:29 | |
| *** tetsuro_ has quit IRC | 06:32 | |
| *** slaweq has joined #openstack-release | 06:51 | |
| *** witek_ has joined #openstack-release | 06:59 | |
| *** vishalmanchanda has joined #openstack-release | 07:03 | |
| *** tetsuro_ has joined #openstack-release | 07:24 | |
| *** tetsuro has quit IRC | 07:27 | |
| *** rpittau|afk is now known as rpittau | 07:29 | |
| *** amoralej|off is now known as amoralej | 07:34 | |
| *** tosky has joined #openstack-release | 07:35 | |
| *** dmellado has joined #openstack-release | 07:37 | |
| *** e0ne has joined #openstack-release | 08:05 | |
| *** AJaeger has joined #openstack-release | 08:07 | |
| AJaeger | release team, FYI, I'll start updating docs.o.o to point to Ussuri release in a bit. See https://review.opendev.org/#/c/725199/ | 08:12 |
|---|---|---|
| * AJaeger likes to have it up early in case of problems or bugs... | 08:12 | |
| *** ykarel is now known as ykarel|lunch | 08:30 | |
| hberaud | AJaeger: thanks for the heads up | 08:39 |
| *** tetsuro_ has quit IRC | 08:47 | |
| openstackgerrit | Thierry Carrez proposed openstack/releases master: Test release with brand-new mirror-on-tags jobs https://review.opendev.org/727666 | 08:53 |
| ttx | smcginnis, hberaud: if you could +2a ^ | 09:05 |
| hberaud | ttx: ack | 09:05 |
| AJaeger | ttx, is 727665 released? | 09:06 |
| AJaeger | merged I mean | 09:06 |
| ttx | I added a depend... | 09:06 |
| ttx | not enough? | 09:06 |
| AJaeger | ah, depend is enough ;) | 09:07 |
| ttx | hberaud: maybe hold for 2 min | 09:07 |
| hberaud | ttx: ack | 09:07 |
| *** tetsuro has joined #openstack-release | 09:07 | |
| ttx | or maybe just +2 it and I'll +a it when the other merges | 09:07 |
| smcginnis | The dependency is close. | 09:08 |
| smcginnis | And its merged. | 09:08 |
| ttx | ok it's in, +2a away | 09:08 |
| AJaeger | ttx, just saw a github mirror failure: https://553343c6cc0b3b0c7584-5df0725eab79e913beea805c3c302096.ssl.cf5.rackcdn.com/0c714e51507007a0245f67ec1c1c04aecb46e7b9/post/openstack-upload-github-mirror/b09a37f/job-output.txt | 09:09 |
| ttx | AJaeger: I see the issue | 09:10 |
| smcginnis | Should we block that release-test patch? | 09:10 |
| ttx | no | 09:10 |
| ttx | Let me explain | 09:10 |
| ttx | Basically the old Gerrit-wide replication did sync all references, including all refs/changes refs to Gerrit patchsets | 09:11 |
| ttx | The per-repo job does a straight mirror, so that does not replicate refs/changes | 09:11 |
| ttx | so on the mirror, it does remove all refs/changes on Github | 09:12 |
| ttx | Which will fail miserably on GitHub if there are too many | 09:12 |
| ttx | (that's the error you got) | 09:12 |
| ttx | To avoid that... | 09:12 |
| ttx | I did clean up asynchropnously all refs/changes on github | 09:12 |
| ttx | BUT I probably missed some repos | 09:12 |
| ttx | like openstack-manuals | 09:13 |
| *** dtantsur|afk is now known as dtantsur | 09:13 | |
| ttx | Let me clean it up async now | 09:13 |
| AJaeger | https://zuul.opendev.org/t/openstack/builds?job_name=openstack-upload-github-mirror&result=FAILURE shows a few... | 09:13 |
| AJaeger | just one page going back 2 weeks... | 09:14 |
| AJaeger | thanks, ttx! | 09:14 |
| ttx | I only cleaned up those in projects.yaml... so those in sigs or tc did not get picked up, which explains the majority of those | 09:14 |
| ttx | I should have a look at the others :) | 09:15 |
| ttx | the cleanup just takes forever as GitHub does not like when you try to delete more than 100 refs at at time | 09:17 |
| ttx | the others look like transient fails | 09:19 |
| ttx | https://zuul.opendev.org/t/openstack/builds?job_name=openstack-upload-github-mirror&project=openstack%2Fpuppet-ironic | 09:19 |
| AJaeger | hope that with all the tags we don't run into API limits on github... But sigs or tc repos won't get tagged ;) | 09:20 |
| *** tetsuro has quit IRC | 09:23 | |
| *** ykarel|lunch is now known as ykarel | 09:24 | |
| ttx | OK confirmed all the others were transient errors. Only governance and openstack-manuals were stuck | 09:24 |
| ttx | Cleaning them now | 09:24 |
| AJaeger | thanks | 09:24 |
| openstackgerrit | Merged openstack/releases master: Test release with brand-new mirror-on-tags jobs https://review.opendev.org/727666 | 09:29 |
| smcginnis | Tag complete, mirror job now running. | 09:33 |
| ttx | yep, looks good so far | 09:33 |
| ttx | new tags up | 09:34 |
| smcginnis | Since there aren't any good ussuri songs: https://youtu.be/3Awey7AbXT0 | 09:34 |
| smcginnis | logstash error, but doesn't appear to be important. | 09:35 |
| ttx | I guess one question is whether a failure in mirroring would prevent the rest of the release jobs from running | 09:35 |
| ttx | But I think those are independent | 09:35 |
| smcginnis | I think so. | 09:36 |
| smcginnis | Job: https://opendev.org/openstack/project-config/src/branch/master/zuul.d/jobs.yaml#L777 | 09:36 |
| smcginnis | Tempplate: https://opendev.org/openstack/openstack-zuul-jobs/src/branch/master/zuul.d/project-templates.yaml#L1938 | 09:37 |
| ttx | OK ,let's wait a couple more minutes and send the torpedo away | 09:38 |
| smcginnis | Probably good to let the rest of the jobs finish, just to be safe. | 09:38 |
| ttx | ack | 09:39 |
| AJaeger | So, https://docs.openstack.org now points to Ussuri as new release. If anybody wants to double check... | 09:46 |
| smcginnis | Looking... | 09:46 |
| smcginnis | Looks right to me. | 09:46 |
| AJaeger | thanks - looks good to me as well. Still missing deployment projects but they are not released yet... | 09:47 |
| smcginnis | Yeah, might be awhile yet before the cycle-trailing things are ready. | 09:48 |
| smcginnis | ttx: OK, I think the important things are done. We can probably go ahead now. | 09:49 |
| ttx | ok, anyone else wants to pile up their +2? | 09:49 |
| AJaeger | would be good to remind them when they are ready to send patches for openstack-manuals to get added to the index | 09:50 |
| ttx | ok we are ON | 10:00 |
| smcginnis | Bombs away! | 10:00 |
| AJaeger | \o/ | 10:04 |
| fungi | i'm on hand as well, just in case | 10:08 |
| *** ysandeep is now known as ysandeep|lunch | 10:09 | |
| smcginnis | So far so good. Still running all jobs in gate. | 10:09 |
| AJaeger | morning, fungi | 10:11 |
| fungi | thanks, and a guten morgen to you | 10:12 |
| smcginnis | Validate should be ~20-30 minutes. Better than 3 hours. :) | 10:12 |
| AJaeger | fungi: lunch time now ;) Time to start cooking... | 10:13 |
| fungi | indeed, it must be nearing noon there | 10:15 |
| fungi | i bet the sun isn't even up for smcginnis yet | 10:15 |
| fungi | it's only just starting to rise here | 10:16 |
| smcginnis | The sky is just starting to lighten up. | 10:23 |
| smcginnis | validation job wrapping up. | 10:30 |
| openstackgerrit | Merged openstack/releases master: Ussuri final releases for cycle-with-rc projects https://review.opendev.org/726484 | 10:30 |
| fungi | good so far | 10:31 |
| hberaud | smcginnis: then I think I can start to check missing-releases | 10:32 |
| *** udesale_ has joined #openstack-release | 10:36 | |
| smcginnis | hberaud: Thanks. Just need to wait for the tagging jobs to finish so the expected packages get uploaded. Then we should be all set. | 10:37 |
| hberaud | smcginnis: ack | 10:38 |
| *** udesale has quit IRC | 10:38 | |
| hberaud | smcginnis: can you paste the url to follow this job? | 10:39 |
| fungi | hberaud: the release pipeline at https://zuul.opendev.org/t/openstack/status | 10:40 |
| hberaud | fungi: thx | 10:41 |
| fungi | those are being generated by the one in the release-post pipeline | 10:41 |
| hberaud | yep I seen them | 10:42 |
| fungi | the tag-releases job running for the openstack/releases commit there | 10:42 |
| smcginnis | On masakari-dashboard, so making good progress. | 10:42 |
| fungi | the console stream for that tag-releases build is https://zuul.opendev.org/t/openstack/stream/92d4e5d4ccf941a8a444e9ca7086e7f0?logfile=console.log | 10:43 |
| fungi | looks like cache_remote is trying to use the old /opt/git path still, and spits out a warning | 10:50 |
| fungi | https://zuul.opendev.org/t/openstack/stream/92d4e5d4ccf941a8a444e9ca7086e7f0?logfile=console.log | 10:51 |
| fungi | WARNING: Cache directory /opt/git/openstack/nova does not exist, ignoring. | 10:51 |
| smcginnis | fungi: Is that where it tries to use a local existing clone to speed up the current clone? | 10:51 |
| fungi | i believe so | 10:51 |
| smcginnis | I remember seeing that before. I wasn't aware there were changes. | 10:51 |
| smcginnis | What is the new path? | 10:52 |
| smcginnis | fungi: Looks like our script defaults to that path if ZUUL_CACHE_DIR is not set. | 10:53 |
| fungi | or maybe that's not obsolete, but it certainly doesn't seem to be finding it | 10:53 |
| *** ysandeep|lunch is now known as ysandeep | 10:55 | |
| fungi | looks like it should be using /opt/git/opendev.org/openstack/nova | 10:56 |
| fungi | instead of /opt/git/openstack/nova | 10:56 |
| smcginnis | Ah, additional namespacing for more tenants probably? | 10:57 |
| fungi | well, for more potential source connection names | 10:57 |
| openstackgerrit | Sean McGinnis proposed openstack/releases master: Use updated git cache dir default path https://review.opendev.org/727710 | 10:59 |
| smcginnis | Tagging almost complete. | 11:00 |
| hberaud | done | 11:00 |
| *** rpittau is now known as rpittau|bbl | 11:04 | |
| fungi | now it's a matter of waiting for the release pipeline to empty out and checking for any job failures | 11:05 |
| ttx | looking good so far | 11:10 |
| openstackgerrit | Sean McGinnis proposed openstack/releases master: Use updated git cache dir default path https://review.opendev.org/727710 | 11:13 |
| smcginnis | Looks like only email announcements and update constraints jobs left. | 11:17 |
| openstackgerrit | Dmitriy Rabotyagov (noonedeadpunk) proposed openstack/releases master: Release OpenStack-Ansible Stein https://review.opendev.org/727715 | 11:19 |
| fungi | nothing new at http://lists.openstack.org/pipermail/release-job-failures/2020-May/date.html since friday | 11:20 |
| openstackgerrit | Dmitriy Rabotyagov (noonedeadpunk) proposed openstack/releases master: Release OpenStack-Ansible Train https://review.opendev.org/727716 | 11:20 |
| smcginnis | tacker-horizon announce-release failure. | 11:23 |
| hberaud | `missing-release` return a huge list (922 errors) or I missed something => http://paste.openstack.org/show/793509/ | 11:23 |
| fungi | http://lists.openstack.org/pipermail/release-job-failures/2020-May/001381.html | 11:24 |
| fungi | yep. just showed up | 11:24 |
| smcginnis | Logs make it look like it worked though. | 11:24 |
| fungi | yep, just log collection failed | 11:26 |
| fungi | rsync hit an ssh socket timeout | 11:26 |
| smcginnis | Safe to ignore then. | 11:26 |
| smcginnis | hberaud: I see a lot of RC version in that list. Did you pull the latest from master before running? | 11:26 |
| smcginnis | Running locally now too. I am also getting a lot of missing tarballs and signatures. | 11:27 |
| smcginnis | fungi: Is there a delay in syncing tarballs.o.o? | 11:27 |
| hberaud | smcginnis: normally yes, I recheck | 11:27 |
| fungi | smcginnis: volume release occurs every 5 minutes for that site, but may take longer depending on the amount of data... i'll check on its progress | 11:28 |
| smcginnis | Oh, I think we can drop the check for python 2 and python 2/3 wheels. | 11:28 |
| hberaud | smcginnis: ah wait...I point on 27c15e1028c3a0c097baac273245284d07c5d871 | 11:28 |
| hberaud | smcginnis: I thought to be up-to-date... | 11:29 |
| smcginnis | I've done that many times. ;) | 11:31 |
| fungi | smcginnis: looks like the tarballs site volume should be in sync, what's missing? | 11:31 |
| fungi | i see a nova-21.0.0-py3-none-any.whl and nova-21.0.0.tar.gz and associated .asc files at https://tarballs.opendev.org/openstack/nova/ | 11:35 |
| ttx | looking | 11:37 |
| ttx | example: https://tarballs.openstack.org/blazar-dashboard/blazar-dashboard-3.0.0.tar.gz missing | 11:39 |
| ttx | but it's not... hmm | 11:39 |
| fungi | remember that's a redirect now | 11:40 |
| ttx | yeah the script seems a bit out of date | 11:40 |
| fungi | mostly making sure the script successfully follows permanent redirects | 11:40 |
| hberaud | good point | 11:42 |
| ttx | missing = ((response.status_code // 100 != 2)) | 11:42 |
| openstackgerrit | Tobias Urdin proposed openstack/releases master: Release puppet-ceph 3.1.1 https://review.opendev.org/727094 | 11:42 |
| smcginnis | Sorry, I've been updating the script. I think I might have it. | 11:43 |
| smcginnis | We also need to drop checks for py2 wheels. | 11:43 |
| hberaud | yep | 11:43 |
| ttx | I'll run the W3C link checker in parallel | 11:44 |
| openstackgerrit | Sean McGinnis proposed openstack/releases master: Update missing cmd to follow redirects https://review.opendev.org/727733 | 11:46 |
| ttx | smcginnis: can you paste the report you get with this version? | 11:50 |
| hberaud | looks better now :) | 11:52 |
| smcginnis | Yep, almost done. | 11:52 |
| smcginnis | Looking a LOT better than it had been. :) | 11:52 |
| hberaud | I agree :) | 11:53 |
| smcginnis | http://paste.openstack.org/show/793511/ | 11:54 |
| smcginnis | Barbican one is indeed missing a whl. | 11:54 |
| smcginnis | https://tarballs.opendev.org/openstack/barbican/ | 11:54 |
| smcginnis | 10.0.0 only has the tar.gz file and signature. | 11:55 |
| smcginnis | ceilometermiddleware appears to not have completed the drop-py2 goal, so there is a py2.py3 wheel. | 11:55 |
| smcginnis | I can update the script to still check for that. That may eliminate a few of these. | 11:56 |
| hberaud | good idea | 11:57 |
| *** amoralej is now known as amoralej|lunch | 11:58 | |
| AJaeger | ceilometermiddleware merged https://review.opendev.org/#/c/692065/ in November | 11:58 |
| smcginnis | fungi: Do you know - are py2.py3 wheels built if the universal flag is set? Or some other trigger? | 12:01 |
| fungi | that's what typically causes it to happen, pretty sure. i'll double-check that project | 12:02 |
| AJaeger | I don't see universal flag set recently | 12:02 |
| AJaeger | I'm checking the repo right now and do a few cleanups | 12:04 |
| AJaeger | didn't see anything obvious ;( | 12:04 |
| smcginnis | I had expected to see universal=true in their setup.cfg, but it's not there. | 12:04 |
| smcginnis | So not sure why 2/3 wheel would be created instead of just 3. | 12:05 |
| fungi | their 2.0.0 release was in december | 12:06 |
| smcginnis | Hmm, script returned happy, but I would have expected an error at least for barbican. | 12:08 |
| fungi | i don't see evidence that ceilometermiddleware ever set universal in their setup.cfg, in fact, looking through git history | 12:11 |
| smcginnis | Oh, sorting threw me off for barbican. They do have a 2/3 wheel. | 12:13 |
| smcginnis | Running the script one more time after some updates. | 12:13 |
| fungi | yeah, ceilometermiddleware never even had a [wheel] section in their setup.cfg at all | 12:13 |
| smcginnis | But I think we are in good shape. | 12:13 |
| AJaeger | barbican took ages to get drop-2.7 changes in, that repo was in a sad state | 12:14 |
| smcginnis | I do see it for barbican: https://opendev.org/openstack/barbican/src/branch/stable/ussuri/setup.cfg#L123 | 12:14 |
| smcginnis | I wonder what could be triggering that for ceilometermiddleware | 12:14 |
| AJaeger | did we change our tooling after that release? | 12:16 |
| fungi | unfortunately, with their release happening in early december, we no longer have the job log from it | 12:16 |
| fungi | it's been over 6 months | 12:16 |
| AJaeger | I41de5a1f57eed0a165bcd93ab1585003c3d478fb merged in January | 12:16 |
| AJaeger | that's it, I guess | 12:16 |
| AJaeger | https://review.opendev.org/#/c/703464/ | 12:16 |
| AJaeger | smcginnis: "Stop forcing universal wheel releases" | 12:16 |
| *** ysandeep is now known as ysandeep|brb | 12:17 | |
| fungi | oh, yep, that was it | 12:17 |
| fungi | good memory, AJaegermystery solved | 12:17 |
| AJaeger | smcginnis: so we would need to re-release ceilometermiddleware or timetravel ;) | 12:17 |
| smcginnis | OK, that makes sense. Thanks AJaeger. | 12:18 |
| smcginnis | That one is a cycle-with-intermediary deliverable, and they had just the one release in December. Before that job change. | 12:18 |
| hberaud | make sense | 12:19 |
| smcginnis | And check with current script is happy again. | 12:19 |
| openstackgerrit | Sean McGinnis proposed openstack/releases master: Update missing cmd to follow redirects https://review.opendev.org/727733 | 12:19 |
| fungi | okay, so any outstanding unknowns from the missing report? | 12:19 |
| *** rpittau|bbl is now known as rpittau | 12:19 | |
| smcginnis | None at all. It appears to be happy with everything. | 12:20 |
| AJaeger | everything? Wow! | 12:20 |
| smcginnis | I will send out the announcements. Looks like we are all set. | 12:20 |
| fungi | thanks! smooth as glass | 12:20 |
| ttx | hmm just a sec | 12:21 |
| fungi | ttx: any unexpected issues raised by the linkcheck? | 12:21 |
| smcginnis | Feel free to check my work with https://review.opendev.org/#/c/727733/ | 12:21 |
| ttx | the linkscheck does not like us now that our page is too big. | 12:21 |
| fungi | lovely | 12:21 |
| ttx | smcginnis: announcement should be sent closer to 15utc | 12:22 |
| ttx | we need to set release as released first | 12:22 |
| ttx | or did I miss that step | 12:23 |
| smcginnis | Nope, I'll get that now. | 12:23 |
| ttx | ok :) Was worried about "sending announcements" :) | 12:24 |
| smcginnis | Yeah, getting a little ahead of myself. | 12:24 |
| *** tkajinam has quit IRC | 12:24 | |
| ttx | I'm crossing lines on https://etherpad.opendev.org/p/ussuri-relmgt-tracking | 12:24 |
| openstackgerrit | Sean McGinnis proposed openstack/releases master: Mark Ussuri as released https://review.opendev.org/727746 | 12:27 |
| ttx | all the tags were successfully pushed to GitHub, fwiw | 12:28 |
| smcginnis | Nice! | 12:28 |
| hberaud | \o/ | 12:28 |
| fungi | awesome | 12:28 |
| ttx | fungi: we noted this morning that tags were actually not synced | 12:28 |
| ttx | since the mirroring job only ran in post | 12:29 |
| fungi | right, i warned about that back when the job was implemented | 12:29 |
| fungi | folks didn't seem concerned at the time | 12:29 |
| ttx | I should write down warnings | 12:29 |
| fungi | "tags will show up the next time a change is merged" was the answer, so i figured it wasn't a big deal ;) | 12:30 |
| AJaeger | fungi, we merged a change to openstack-zuul-jobs to sync after release | 12:30 |
| fungi | AJaeger: i saw, good thinking | 12:30 |
| smcginnis | It would be eventually consistent. | 12:30 |
| smcginnis | Just the value of "eventually" could be quite large for some repos. | 12:31 |
| fungi | i still think that job could just push the triggering branch or tag, rather than doing a full mirror on every run, but more optimization | 12:31 |
| smcginnis | ttx: Can you think of any reason we have steps 4 and 5 called out as two separate steps in https://releases.openstack.org/reference/process.html#r-0-week-final-release | 12:32 |
| ttx | it comes with an ominous warning note | 12:33 |
| ttx | "Only workflow when previous step *fully* ready" | 12:33 |
| ttx | so I think we should honor whoever wrote that warning | 12:34 |
| smcginnis | That's what I am wondering about. I don't see any reason why that should be the case. | 12:34 |
| ttx | ok, let me dive into it | 12:34 |
| ttx | there is a lot riding on that default series value | 12:35 |
| openstackgerrit | Sean McGinnis proposed openstack/releases master: Update release week process https://review.opendev.org/727748 | 12:37 |
| smcginnis | Well, assuming we don't have a real reason for the separation, that will be ready. ^ | 12:37 |
| ttx | so... | 12:38 |
| hberaud | sound like a warning at the entrance of a pyramid that warns of calamities incurred by profaners. | 12:38 |
| ttx | That note was added (by you) after the Stein release, referencing "changes in process found during the stein release process at Denver PTG" | 12:38 |
| fungi | abandon all hope, ye who enter here | 12:39 |
| smcginnis | ttx: Just saw that. | 12:39 |
| ttx | smcginnis: at that time you also collapsed the patch into one patch | 12:39 |
| ttx | I'm pretttty sure we added that note for a good reason then | 12:39 |
| smcginnis | I wonder if there was another step we were thinking about after the discussion with changing auto releases and things like that. | 12:39 |
| ttx | If it had worked correctly you probably would not have added that | 12:39 |
| smcginnis | I combined them for train too. | 12:40 |
| fungi | maybe it was because you realized a tag hadn't been pushed for something and needed to be added at the last moment? | 12:41 |
| fungi | could dive into irc logs from that time | 12:41 |
| smcginnis | I fixed up bulleting in that patch. Since it was a separate paragraph, I probably just threw it into it's own section. I really can't recall, but I can't think of any reason now why we would need to break it out. | 12:41 |
| smcginnis | fungi: Was just thinking of finding the timestamp of when that patch was orginally submitted. ;) | 12:42 |
| smcginnis | No comments in IRC around the time. | 12:44 |
| ttx | yeah, we were in person | 12:44 |
| smcginnis | Looking through https://etherpad.opendev.org/p/relmgmt-train-ptg for clues. | 12:45 |
| smcginnis | Was done interactive collab style: https://etherpad.opendev.org/p/DEN-release-process | 12:46 |
| ttx | yellow, what were you thinking | 12:46 |
| smcginnis | Assuming "ready" is approved and merging, and since we say this one can be approved and be staged on top of the previous one, the only separation here would be if we need to do another check of anything before approval. There's nothing else to do, especially since the previous step is just updating the docs. | 12:48 |
| ttx | yeah, let's live dangerously | 12:49 |
| smcginnis | I better shower quick before meetings start up. The "mark released" patch has zuul +1. I will be back soon. | 12:49 |
| ttx | This was far too calm | 12:49 |
| smcginnis | No kidding. I think this was the calmest I've been a part of so far. | 12:49 |
| ttx | we should change it again | 12:49 |
| ttx | we are getting soft | 12:49 |
| smcginnis | Hah | 12:50 |
| smcginnis | When is zuul v4 coming out? | 12:50 |
| smcginnis | brb | 12:50 |
| *** ianychoi has quit IRC | 12:55 | |
| *** amoralej|lunch is now known as amoralej | 13:00 | |
| fungi | smcginnis: not too long actually | 13:08 |
| fungi | v4 and v5 are both planned | 13:08 |
| smcginnis | Let's hope, for our piece at least, that migration goes much smoother than the transition to v3 went. | 13:08 |
| fungi | but the major version bumps aren't for significant user-facing changes, they're for admin-facing changes | 13:08 |
| smcginnis | That's good. | 13:08 |
| fungi | service architecture improvements for statefulness and high availability | 13:09 |
| *** ykarel is now known as ykarel|afk | 13:09 | |
| openstackgerrit | Sean McGinnis proposed openstack/releases master: Mark Ussuri as released https://review.opendev.org/727746 | 13:30 |
| ttx | I'll W+! this ^ | 13:49 |
| *** ykarel|afk is now known as ykarel | 13:59 | |
| ttx | community meeting on Ussuri release starting now | 14:00 |
| smcginnis | ttx: Good to send the announcement now? | 14:00 |
| ttx | https://etherpad.opendev.org/p/CommunityMeeting_Ussuri | 14:01 |
| ttx | smcginnis: yeah, it's fair game now | 14:01 |
| ttx | just wait until https://review.opendev.org/727746 is processed | 14:01 |
| ttx | so that it shows as released on https://releases.openstack.org/ | 14:02 |
| ttx | smcginnis: ^ | 14:02 |
| openstackgerrit | Merged openstack/releases master: Mark Ussuri as released https://review.opendev.org/727746 | 14:04 |
| diablo_rojo_phon | Whoop whoop! | 14:23 |
| ttx | releases.o.o page still not updated | 14:33 |
| smcginnis | Job is done, but I'm not seeing the updates. | 14:35 |
| smcginnis | Maybe there is another sync step now. I think it was stated that happens every 5 minutes, so will keep checking. | 14:35 |
| ttx | Probably waiting for the refresh | 14:35 |
| ttx | cc: fungi | 14:35 |
| fungi | looking | 14:36 |
| smcginnis | It's updated now. | 14:37 |
| fungi | standing down | 14:37 |
| smcginnis | I noticed releases.o.o doesn't have a favicon. Anyone know if that is an openstackdocstheme setting or something? | 14:37 |
| smcginnis | ttx: Sending announcements now. | 14:38 |
| ttx | ++ | 14:38 |
| *** ysandeep|brb is now known as ysandeep | 14:38 | |
| smcginnis | Need moderator approval for openstack-announce. | 14:38 |
| fungi | and yeah, it'll generally get a vos release batch run every minute evenly divisible by 5, but the volumes released together are processed serially so, for example, a large amount of documentation churn could cause the docs.o.o release to take a couple of minutes before it gets to the releases.o.o volume | 14:39 |
| fungi | i can get the announce moderation queue | 14:39 |
| fungi | done | 14:42 |
| *** redrobot has joined #openstack-release | 14:43 | |
| ttx | now back at paying attention to the community meeting | 14:43 |
| mnaser | ttx, smcginnis: can i technically say "ussuri is out" at the end of the call? | 14:45 |
| ttx | mnaser: yes, it's hit the ML | 14:45 |
| mnaser | awesome. | 14:45 |
| ttx | It's out now if you want to mention it between speakers | 14:45 |
| smcginnis | mnaser: Yep! http://lists.openstack.org/pipermail/openstack-announce/2020-May/002035.html | 14:46 |
| mnaser | ill wait till the end :P | 14:46 |
| smcginnis | I feel like someone behind the scenes preparing for the big keynote reveal announcement. :) | 14:47 |
| mnaser | ahaha :p | 14:47 |
| ttx | smcginnis: welcome to marketing | 14:47 |
| smcginnis | 😱 | 14:48 |
| ttx | smcginnis: congrats on yet another on-time release, fearless leader! | 14:57 |
| smcginnis | You too. | 14:57 |
| smcginnis | Great job team! | 14:57 |
| smcginnis | Of course I couldn't make it without at least one typo in an email. | 14:58 |
| mnaser | \o/ | 15:01 |
| *** priteau has joined #openstack-release | 15:02 | |
| clarkb | smcginnis: that is how you let people know its handcrafted open source software. Not factory produced. Just don't let Zuul in on it :) | 15:16 |
| smcginnis | That's it - it was on purpose to be authenticate. Yep, totally. :D | 15:17 |
| *** zxiiro has joined #openstack-release | 15:32 | |
| *** ykarel is now known as ykarel|away | 15:35 | |
| *** dtantsur is now known as dtantsur|afk | 15:36 | |
| openstackgerrit | Sean McGinnis proposed openstack/releases master: Add release note links for newly added pages https://review.opendev.org/727802 | 15:37 |
| smcginnis | ttx: Still around? I think we want to push that through right away. ^ | 15:45 |
| smcginnis | Or hberaud | 15:45 |
| hberaud | omw | 15:45 |
| hberaud | smcginnis: this patch kill my browser :) | 15:46 |
| smcginnis | Hah | 15:50 |
| hberaud | smcginnis: this is something that we want => https://docs.openstack.org/releasenotes/puppet-openstack_extras/ussuri.html ? | 15:57 |
| hberaud | smcginnis: I mean the empty reno | 15:57 |
| hberaud | I guess yes to prepare the floor for ussuri... | 15:58 |
| ttx | ohai | 15:58 |
| smcginnis | hberaud: Yeah. If the landing page is there, then we want to add it. Hopefully eventually they will actually have content in there. | 15:59 |
| ttx | approved | 15:59 |
| hberaud | ok | 15:59 |
| *** AJaeger has left #openstack-release | 16:00 | |
| *** hberaud has quit IRC | 16:13 | |
| *** hberaud has joined #openstack-release | 16:13 | |
| *** witek_ has quit IRC | 16:15 | |
| openstackgerrit | Merged openstack/releases master: Add release note links for newly added pages https://review.opendev.org/727802 | 16:18 |
| *** rpittau is now known as rpittau|afk | 16:21 | |
| *** camelCaser has quit IRC | 16:22 | |
| *** ccamel has joined #openstack-release | 16:22 | |
| openstackgerrit | Andreas Jaeger proposed openstack/releases master: Release openstackdocstheme 2.1.0 https://review.opendev.org/727825 | 16:22 |
| *** AJaeger has joined #openstack-release | 16:24 | |
| AJaeger | smcginnis: hope that docstheme release is not too quick ;) | 16:24 |
| smcginnis | AJaeger: You had that queued up and waiting, didn't you. :D | 16:25 |
| AJaeger | smcginnis: I was waiting for the merges first... | 16:25 |
| smcginnis | Good to get that out. Some good stuff has merged. | 16:25 |
| AJaeger | exactly - and melwitt wants parallel releasenotes building (I saw nova building for 1 h in Zuul) | 16:26 |
| *** witek_ has joined #openstack-release | 16:26 | |
| smcginnis | Hopefully that helps. I think we need to get all repos to raise their openstackdocstheme lower constraint to make sure we can apply that community-wide in project-config though. Or else drop the -W flag. | 16:27 |
| smcginnis | The other issue there was it rebuilds the docs for each translation. Not sure if we can speed anything up with how that is done. Probably not since it really does need a full build of the docs to be done per language. | 16:28 |
| AJaeger | yeah, indeed, we need to update everywhere - but we're at the beginning of a cycle ;) | 16:29 |
| melwitt | yeah. I wanted to ask the zuul ppl if there's a way we could parallelize that task, so that each language build process gets kicked off at the same time and might benefit from being able to run in parallel? | 16:29 |
| smcginnis | Yes, hopefully enough time to get it all in. | 16:29 |
| AJaeger | we really need the translated docs rebuild everytime, otherwise content is missing | 16:29 |
| smcginnis | That's a good idea melwitt. | 16:30 |
| AJaeger | melwitt: the job is in zuul-jobs if you wnat to have a look. | 16:30 |
| AJaeger | melwitt: https://opendev.org/zuul/zuul-jobs/src/branch/master/roles/build-releasenotes/tasks/main.yaml#L10 | 16:31 |
| melwitt | I'm not sure how much '-j auto' will help us but I'm trying it out to see what's the speedup. note that we'd also need a new reno release because it too added parallel_read_safe recently. I dunno if a release is already proposed yet or not | 16:31 |
| AJaeger | melwitt: yes, we need both reno and docstheme | 16:31 |
| smcginnis | It's been queued up, but we didn't want to release a new reno until ussuri was out just to be safe. | 16:31 |
| smcginnis | Should be able to get that out now too. | 16:31 |
| melwitt | AJaeger: thanks! I am aware of that but I wasn't sure if/how a section/label can be prescribed to run in parallel. because "something" is running that label serially somewhere. I will try to find where and also ask if I can't find | 16:32 |
| melwitt | ok cool | 16:32 |
| smcginnis | In fact, hberaud, would you mind taking a look at https://review.opendev.org/#/c/724697/ | 16:32 |
| *** udesale_ has quit IRC | 16:33 | |
| melwitt | *serially, one run per language | 16:33 |
| AJaeger | melwitt: adding -j auto in https://opendev.org/zuul/zuul-jobs/src/branch/master/roles/build-releasenotes/tasks/main.yaml#L95 is the best way IMHO. Otherwise: You need to rewrite that completely | 16:33 |
| hberaud | smcginnis: done | 16:34 |
| smcginnis | Thanks! | 16:34 |
| melwitt | AJaeger: ok, good to know. I will continue to focus on trying that out and see the gain, then go from there. thanks | 16:34 |
| AJaeger | melwitt you can run that script locally to test | 16:34 |
| smcginnis | That is still serially per language. It would be nice to make that part parallel. | 16:35 |
| smcginnis | But not sure of a good way to do that when running a bash script. | 16:35 |
| smcginnis | At least the builds should take less time, so sooner to get to the next language. | 16:35 |
| * melwitt nods | 16:35 | |
| *** evrardjp has quit IRC | 16:36 | |
| *** evrardjp has joined #openstack-release | 16:36 | |
| AJaeger | if you have another script, could be python, to do the building, we can use that... | 16:37 |
| smcginnis | It may not be worth it. We only have a limited number of cores in the VM that runs the job, so we risk adding too much contention with all of this parallelism. | 16:38 |
| AJaeger | so, let's try the sphinx -j auto | 16:39 |
| smcginnis | Thinking it through, that's probably the best option. | 16:39 |
| AJaeger | but first release reno and openstackdocstheme - and update a 'few' repos | 16:41 |
| *** ysandeep is now known as ysandeep|away | 17:12 | |
| openstackgerrit | Merged openstack/releases master: Release openstackdocstheme 2.1.0 https://review.opendev.org/727825 | 17:14 |
| *** amoralej is now known as amoralej|off | 17:18 | |
| *** priteau has quit IRC | 17:38 | |
| *** e0ne has quit IRC | 17:55 | |
| *** jbadiapa has quit IRC | 19:37 | |
| melwitt | I kicked off a build after the openstackdocstheme release merged but it's still saying "WARNING: the openstackdocstheme extension does not declare if it is safe for parallel reading, assuming it isn't - please ask the extension author to check and make it explicit" is there something else that needs to be done to get the latest version? | 19:50 |
| AJaeger | melwitt: local build? | 19:52 |
| AJaeger | are you really using openstackdocstheme 2.1.0? | 19:52 |
| melwitt | no CI build. I thought after the release the new version would get pulled in? sorry I don't know anything | 19:57 |
| AJaeger | melwitt: if you build locally with constraints, the constraints might limit to an older version | 19:58 |
| AJaeger | melwitt: install the theme into your venv/tox and check with pip freeze that it's really there | 19:59 |
| melwitt | I'm just building in upstream zuul, not locally. maybe I need to add the version to test-requirements.txt? I'm just using DNM patches | 19:59 |
| melwitt | upstream zuul job does a completely different thing than our local tox.ini run | 20:00 |
| AJaeger | melwitt: have a link for me? | 20:00 |
| AJaeger | upstream and local both use constraints the same way | 20:01 |
| melwitt | AJaeger: I don't really want to show you :) but it's this. ignore the actual change bc it's a dummy, it's the Depends-On where things are really happening https://review.opendev.org/727429 | 20:01 |
| melwitt | the releasenotes CI job does _not_ run our tox -e releasenotes target at all | 20:02 |
| AJaeger | melwitt: correct, CI does not use tox -e releasenotes since we build the translated docs centrally | 20:03 |
| AJaeger | yeah confusing ;( | 20:03 |
| melwitt | so now I'm thinking if I add the version to test-requirements.txt and lower-constraints.txt in my DNM nova patch maybe that will do it? | 20:03 |
| melwitt | I'm just testing things out to see the job run time effect the -j auto will have | 20:04 |
| melwitt | we're currently taking around an hour to run the job | 20:04 |
| AJaeger | good idea | 20:05 |
| AJaeger | I see the job running but can't find the version of openstackdocstheme yet ;( | 20:05 |
| melwitt | ok cool thanks :) this is my first time working with any docs related jobs, lots of new to me things | 20:06 |
| melwitt | yeah it doesn't even say it in the job output, which surprised me | 20:06 |
| melwitt | (the openstackdocstheme version) | 20:06 |
| melwitt | looking at an older run, it doesn't seem to mention the version anywhere https://zuul.opendev.org/t/openstack/build/019ca51162354e3285e6383402440857 | 20:08 |
| AJaeger | add a "pip freeze " to roles/build-releasenotes/tasks/main.yaml ;) | 20:10 |
| AJaeger | melwitt, https://zuul.opendev.org/t/openstack/build/019ca51162354e3285e6383402440857/console#1/0/25/ubuntu-bionic | 20:11 |
| AJaeger | the pre-playbook installs sphinx and shows the version on that one | 20:11 |
| AJaeger | you'll get that output at the end of the run | 20:12 |
| AJaeger | melwitt: that output is only in the JSON, not in the console | 20:13 |
| melwitt | ohhhh | 20:13 |
| AJaeger | so, please wait a bit longer... | 20:13 |
| melwitt | dang I see now on the old run | 20:13 |
| melwitt | I had been looking at previous runs and didn't know the json could have more info in it | 20:14 |
| AJaeger | I'll have to leave now, will read tomorrow what you figured out and answer any more questions. | 20:14 |
| smcginnis | melwitt: You'll need yet another depends-on I'm afraid: https://review.opendev.org/#/c/727850/ | 20:14 |
| AJaeger | smcginnis: yes, that's most likely the problem | 20:14 |
| AJaeger | bye for today! | 20:14 |
| smcginnis | Good night AJaeger | 20:15 |
| melwitt | smcginnis: ah thanks. np, I was going to try test-requirements anyway but would've been blocked by that :P | 20:15 |
| melwitt | so was planning another rev anyhow | 20:15 |
| melwitt | thanks AJaeger! | 20:15 |
| melwitt | kewls depends-on that works to pull in the new openstackdocstheme version. api-ref job already finished so I could peek | 20:30 |
| smcginnis | Nice! | 20:30 |
| *** vishalmanchanda has quit IRC | 20:31 | |
| melwitt | woohoo | 20:31 |
| smcginnis | Looks like the release notes job is just getting going. | 20:31 |
| melwitt | yeah, I'm glad it got queued up so quickly this time. nice | 20:32 |
| smcginnis | That was quick. I saw a few patches that took nearly two hours to kick off. | 20:32 |
| melwitt | yeah, previous to this PS it was taking hours before jobs started. this time it was instant. I wonder if depends-on openstack/requirements gave it a higher priority | 20:34 |
| smcginnis | These scanning steps of reno seem to really take a long time for some of the older branches. | 20:35 |
| smcginnis | But it starts at ocata, which is still Extended Maintenance, so not sure that could be made static. | 20:36 |
| smcginnis | pike took 1:15 to scan release notes. | 20:36 |
| smcginnis | So if all the extended maintenance branch release note pages were made static, maybe with a note in there to instruct people what to do manually if new releases notes needed to be added, that looks like it could cut off maybe ~5-7 minutes per build. Time the number of translated builds it needs to do, that could add up. | 20:39 |
| smcginnis | Not sure if it would add up enough to stop timeout problems, but at least buy a little more time. | 20:39 |
| * melwitt nods | 20:40 | |
| melwitt | uh oh, looks like our docs job fails with the new openstackdocstheme. guh | 20:42 |
| smcginnis | melwitt: Looks like that failed on the pdf portion. | 20:54 |
| melwitt | yeah, I'm looking at it. not yet understanding it tho | 20:55 |
| smcginnis | I wonder if it could be a side effect of https://review.opendev.org/#/c/682683/ | 20:55 |
| melwitt | yeah maybe.. gonna try running locally and see if I can get more info | 21:02 |
| melwitt | bah turns out my depends-on the reno change didn't work "WARNING: the reno.sphinxext extension is not safe for parallel reading" on the latest DNM patch run | 21:33 |
| smcginnis | Hmm. Well, hopefully we will have that out and upper-constraints raised soon. | 21:42 |
| *** slaweq has quit IRC | 21:57 | |
| *** slaweq has joined #openstack-release | 22:08 | |
| *** slaweq has quit IRC | 22:13 | |
| *** slaweq has joined #openstack-release | 22:23 | |
| *** slaweq has quit IRC | 22:28 | |
| *** tkajinam has joined #openstack-release | 22:46 | |
| *** hongbin has joined #openstack-release | 23:05 | |
| *** tosky has quit IRC | 23:06 | |
Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!