| ianw | mordred: if you're around want to just double check https://review.opendev.org/#/c/713757/ to remove the builder specific config | 00:10 |
|---|---|---|
| *** mlavalle has quit IRC | 00:40 | |
| *** tosky has quit IRC | 00:50 | |
| openstackgerrit | Merged opendev/system-config master: Make Advisory Board a proper noun https://review.opendev.org/713523 | 01:02 |
| openstackgerrit | Merged opendev/system-config master: Fix URLs after OpenDev rename https://review.opendev.org/713695 | 01:19 |
| openstackgerrit | Merged opendev/system-config master: Update git.zuul-ci.org redirects https://review.opendev.org/713702 | 01:21 |
| openstackgerrit | Merged opendev/system-config master: Update git.starlingx/git.airship redirects https://review.opendev.org/713705 | 01:21 |
| openstackgerrit | Merged opendev/system-config master: nodepool config: remove builder template, fix base template https://review.opendev.org/713757 | 01:29 |
| openstackgerrit | Merged opendev/system-config master: nodepool-builder container: give nodepool group acess to config https://review.opendev.org/713759 | 01:31 |
| openstackgerrit | Merged opendev/system-config master: zookeeper: open firewall port to nb04 https://review.opendev.org/713761 | 01:31 |
| openstackgerrit | Merged opendev/system-config master: Remove Fedora 29 mirroring https://review.opendev.org/713177 | 02:00 |
| mordred | ianw: oh hah! | 02:14 |
| mordred | ianw: nice | 02:14 |
| ianw | sorry i wanted to watch this as it's currently offline | 02:16 |
| ianw | penstack.exceptions.ConflictException: ConflictException: 409: Client Error for url: https://image-sjc1.vexxhost.us/v2/images/c10176f9-56a3-4749-a5dc-44ab56ec3771, Image c10176f9-56a3-4749-a5dc-44ab56ec3771 could not be deleted because it is in use: | 02:26 |
| ianw | it's still trying to delete images :/ | 02:26 |
| clarkb | ianw thats "ok" | 02:28 |
| clarkb | as long as they arevexxhostimages and due to the image is in use error | 02:28 |
| clarkb | any builder can try to d elete those oncethe original builder has tried andfailed | 02:29 |
| ianw | oh ok. i'm running a fedora build on the new host now, see if we get the pause issue | 02:29 |
| ianw | no sudo/host errors; mounts/permissions of necessary volumes all looks good though | 02:32 |
| *** vblando has joined #opendev | 03:07 | |
| ianw | sigh, it's stuck on some sort of lvm op again | 03:10 |
| ianw | seems to have paused for a minute or two; but not for hours as it did before. so it's either "that's what it does" and we never noticed ... or maybe that pause gets longer and longer each build if something isn't right | 03:30 |
| ianw | have to keep an eye on it | 03:30 |
| openstackgerrit | Ian Wienand proposed openstack/diskimage-builder master: [wip] Add Fedora 31 support and test jobs https://review.opendev.org/708416 | 05:19 |
| openstackgerrit | Ian Wienand proposed openstack/diskimage-builder master: [wip] Add Fedora 31 support and test jobs https://review.opendev.org/708416 | 05:24 |
| AJaeger | config-core, please review https://review.opendev.org/713609 https://review.opendev.org/713742 https://review.opendev.org/713446 | 05:35 |
| openstackgerrit | Ian Wienand proposed openstack/diskimage-builder master: [wip] Add Fedora 31 support and test jobs https://review.opendev.org/708416 | 05:51 |
| openstackgerrit | Merged openstack/project-config master: Add publish-to-pypi template for ovn-octavia-provider https://review.opendev.org/713446 | 06:06 |
| openstackgerrit | Merged zuul/zuul-jobs master: Update ansible-lint exception https://review.opendev.org/713742 | 06:08 |
| *** DSpider has joined #opendev | 06:42 | |
| *** dpawlik has joined #opendev | 06:52 | |
| *** dpawlik has quit IRC | 06:53 | |
| *** dpawlik has joined #opendev | 06:53 | |
| *** etp has joined #opendev | 07:55 | |
| *** vblando has quit IRC | 08:07 | |
| *** vblando has joined #opendev | 08:08 | |
| *** mnaser has quit IRC | 08:10 | |
| *** mnaser has joined #opendev | 08:11 | |
| *** rpittau|afk is now known as rpittau | 08:11 | |
| *** lpetrut has joined #opendev | 08:18 | |
| *** tosky has joined #opendev | 08:21 | |
| *** hashar has joined #opendev | 08:48 | |
| *** ralonsoh has joined #opendev | 08:53 | |
| *** yoctozepto has quit IRC | 09:07 | |
| *** yoctozepto0 has joined #opendev | 09:16 | |
| *** yoctozepto9 has joined #opendev | 09:27 | |
| *** yoctozepto0 has quit IRC | 09:27 | |
| openstackgerrit | Chandan Kumar (raukadah) proposed openstack/project-config master: Added new project openstack-tempest-skiplist https://review.opendev.org/713809 | 09:40 |
| *** bcafarel has joined #opendev | 10:04 | |
| AJaeger | infra-root, we lost a few specs directories, for example https://specs.openstack.org/openstack/openstack-specs and https://specs.openstack.org/openstack/api-sig . For api-sig, we're merging a change and republish. We also have 404s for https://specs.openstack.org/openstack/blazar-specs . Do we have openstack-specs and blazar-specs still around to copy over or do we need to republish? | 10:09 |
| openstackgerrit | Andreas Jaeger proposed openstack/project-config master: Fix cloudkitty-specs on specs.o.o https://review.opendev.org/713815 | 10:11 |
| AJaeger | and one easy fix for a 404 ^ | 10:11 |
| *** jrosser has joined #opendev | 10:12 | |
| *** yoctozepto9 is now known as yoctozepto | 10:37 | |
| openstackgerrit | Merged openstack/project-config master: Fix cloudkitty-specs on specs.o.o https://review.opendev.org/713815 | 11:11 |
| openstackgerrit | Chandan Kumar (raukadah) proposed openstack/project-config master: Added new project openstack-tempest-skiplist https://review.opendev.org/713809 | 11:20 |
| *** rpittau is now known as rpittau|bbl | 11:30 | |
| *** hashar is now known as hasharLunch | 12:14 | |
| *** rpittau|bbl is now known as rpittau | 12:59 | |
| *** hasharLunch is now known as hashar | 13:01 | |
| AJaeger | So, openstack-specs is fixed with new change merged... | 13:26 |
| openstackgerrit | Merged openstack/project-config master: Use /detail in check-release-approval queries https://review.opendev.org/713609 | 13:26 |
| openstackgerrit | Monty Taylor proposed openstack/project-config master: Run openstacksdk functional jobs on ansible 2.8 and 2.9 https://review.opendev.org/711474 | 13:58 |
| openstackgerrit | Monty Taylor proposed openstack/project-config master: Run openstacksdk functional jobs on ansible 2.8 and 2.9 https://review.opendev.org/711474 | 14:00 |
| AJaeger | mordred: let's merge https://review.opendev.org/#/c/711471/ first ;) | 14:04 |
| mordred | AJaeger: yes! | 14:04 |
| AJaeger | infra-root, there was a discussion on #openstack-infra about fedora-30 images not having /opt/cache/files - if any of you have time, please see backscroll there | 14:14 |
| clarkb | that is because we removed that element on nb01.opemdev.org as it didnt work on fedora | 14:17 |
| AJaeger | bcafarel: ^ | 14:17 |
| clarkb | the update to go to a fixed list should fix that and I think the new builder is using that element again | 14:17 |
| clarkb | and that shouldve uploaded an image semi recently | 14:17 |
| AJaeger | clarkb: AFAIU it's still failing. bcafarel, do you have failures from *today*? | 14:18 |
| AJaeger | thanks, clarkb | 14:18 |
| rm_work | anyone have any suggestions about the MarkupSafe issue | 14:18 |
| rm_work | maybe you already know the one i'm talking about? 1.0 is broken, need 1.1, for rocky gate testing <_< but u-c caps us at 1.0 | 14:18 |
| rm_work | is there a solution besides updating it to 1.1? | 14:19 |
| clarkb | probably a question for the requirements team? K'm not even aaare of the issue | 14:20 |
| rm_work | hmm, well it had to do with an updated setuptools | 14:21 |
| rm_work | which is part of like... infra? maybe? i don't know how that's classified exactly, it comes from ... virtualenv I think? | 14:22 |
| rm_work | basically the newest setuptools release REMOVED some code that had been deprecated, and older libs were using | 14:22 |
| clarkb | ah that issue, ya either fix the deps or usenew ones (or stop hsing that functionality entirely | 14:22 |
| rm_work | those libs had already updated to fix this, but we are capped below those versions in some cases | 14:22 |
| clarkb | not really anything we can help with | 14:23 |
| rm_work | right but we can't "use new ones" in rocky | 14:23 |
| rm_work | and we can't fix the deps in rocky | 14:23 |
| clarkb | thats a question for the requirements team | 14:23 |
| rm_work | so I was hoping maybe there was an infra fix? like, downgrading/pinning setuptools somehow | 14:23 |
| clarkb | thereis no sane way to do that generally | 14:24 |
| rm_work | bleh ok | 14:24 |
| clarkb | if you switched to pyproject.toml you could set those pre install deps | 14:24 |
| clarkb | but in tge generaly case setuptools is typivally already there and yku dont get to choose the version | 14:24 |
| rm_work | bleh | 14:24 |
| rm_work | this is going to be a tough one, i already proposed a fix to requirements but was told "not going to happen" | 14:25 |
| rm_work | we can't bump U-C in such an old release | 14:25 |
| rm_work | since it will affect all kinds of things | 14:25 |
| clarkb | kill the branch and move on? | 14:25 |
| rm_work | which leaves me with hacking around the setuptools version maybe | 14:25 |
| rm_work | lol are we allowed to just... kill rocky? | 14:26 |
| rm_work | I thought it was mandated as a LTS or something | 14:26 |
| clarkb | if it doesnt work and no one is willing to fix it | 14:26 |
| clarkb | thats my suggestion | 14:26 |
| rm_work | heh k, noted | 14:26 |
| rm_work | well I am totally there with you... | 14:26 |
| rm_work | I just don't know if that's... allowed :D will ask around | 14:26 |
| openstackgerrit | Merged zuul/zuul-jobs master: Add support for installing python with pyenv https://review.opendev.org/704266 | 14:52 |
| fungi | rm_work: the extended maintenance description in the stable branch chapter of openstack's project teams guide says you can switch the branch to unmaintained whenever you like, especially if you can no longer effectively run ci jobs | 15:06 |
| rm_work | cool :) | 15:07 |
| openstackgerrit | Merged opendev/irc-meetings master: Consolidate meeting channels https://review.opendev.org/712656 | 15:07 |
| fungi | rm_work: but also that dropping a bunch of ci jobs during extended maintenance is acceptable if you prefer | 15:07 |
| rm_work | may just ... turn off queens/rocky due to CI being essentially unfixable | 15:07 |
| fungi | that was in fact predicted when the extended maintenance model was designed | 15:08 |
| openstackgerrit | Merged opendev/irc-meetings master: Update chairs list https://review.opendev.org/709626 | 15:08 |
| *** lpetrut has quit IRC | 15:23 | |
| rm_work | so what was the thing about pyproject.toml? | 15:27 |
| rm_work | clarkb: that would somehow let us pin setuptools? | 15:27 |
| rm_work | IE, pre-install downgrade it? | 15:27 |
| rm_work | do you have any examples of that handy, or even know some project I can glance at? | 15:28 |
| mordred | no - it's a whole thing that we haven't even begun to investigate | 15:28 |
| mordred | so far there is zero work in figuring out how it fits into the openstack ecosystem | 15:28 |
| bcafarel | AJaeger: clarkb: nice to hear, no periodic run today yet, I just fired a devstack DNM to confirm :) | 15:28 |
| mordred | and I expect it to be non-trivial | 15:28 |
| * bcafarel likes bugs already fixed at report time | 15:29 | |
| mordred | bcafarel: the best type of bug | 15:29 |
| openstackgerrit | Thierry Carrez proposed opendev/irc-meetings master: Update instructions to match 2020 https://review.opendev.org/713903 | 15:29 |
| rm_work | oh. k | 15:32 |
| rm_work | I thought that was advice for a specific thing | 15:32 |
| *** roman_g has joined #opendev | 15:43 | |
| clarkb | bcafarel: let us know when you get your results back | 16:00 |
| clarkb | infra-root (mordred corvus fungi) has anyone started writing down the tasks for tomorrows downtime? We probably want a change in opendev/project-config to record the renames as well as an etherpad with the walkthrough? | 16:01 |
| corvus | clarkb: i'm unaware of that, but i agree that would be good. well, i mean, a change to p-c for the renames is required, right? :) | 16:05 |
| corvus | here's the current steps: https://docs.openstack.org/infra/system-config/gerrit.html#renaming-a-project | 16:06 |
| corvus | i assume we would do that first, then do the gerrit containerization upgrade | 16:06 |
| clarkb | that sounds like a good way to order it | 16:06 |
| corvus | the rename playbook starts gerrit and does a bunch of stuff | 16:07 |
| corvus | renames groups and re-indexes.... | 16:07 |
| corvus | hrm | 16:08 |
| corvus | maybe we should do this in the other order? | 16:08 |
| clarkb | we'd have to update the playbook to restart gerrit in the other new way but that is doable | 16:08 |
| clarkb | (or possibly just cut the playbook down to the spiderweb bits and do the start stops manually?) | 16:09 |
| mordred | actually - should we even have our gerrit playbooks run docker-compose up ? | 16:09 |
| mordred | we don't normally let puppet try to start gerrit for us do we? | 16:09 |
| corvus | mordred: this is for the project rename playbook | 16:09 |
| corvus | oh i see, unrelated question | 16:10 |
| corvus | just prompted by this discussion | 16:10 |
| mordred | yeah - more just thought about it from a "if we do the container patch then the rename patch" | 16:10 |
| corvus | maybe we need to cut loose on that, and just plan on using emergency file more in those cases | 16:11 |
| corvus | because if we keep trying to do the analog of that with containers, we'll end up having our automation do less and less | 16:11 |
| corvus | the rename playbook also does database changes at the start | 16:12 |
| corvus | mordred: is the current plan to continue using the same trove db? | 16:13 |
| mordred | corvus: yes | 16:13 |
| mordred | it should be a mostly no-op change just to how the war is started - config files and dbs and whatnot should all be the same | 16:14 |
| corvus | mordred: have the change link handy? | 16:14 |
| mordred | corvus: making it right now | 16:15 |
| AJaeger | clarkb: will we publish the infra-manual afterwards still to docs.openstack.org? My rename changes moves it over to docs.opendev.org | 16:16 |
| corvus | mordred: so there's not really anything we need to do on the gerrit host for the container move; just stop the service, then start the container? | 16:16 |
| mordred | corvus: yeah | 16:16 |
| corvus | mordred: we'll keep the same ssl certs? | 16:16 |
| openstackgerrit | Monty Taylor proposed opendev/system-config master: Switch to running gerrit via ansible+containers https://review.opendev.org/713920 | 16:17 |
| clarkb | AJaeger: we should move it and add a redirect | 16:17 |
| mordred | corvus: no, it'll shift to the LE certs | 16:17 |
| AJaeger | clarkb: Ok, then my change is good - and let me add a redirect... | 16:17 |
| mordred | oh wait - we're already using LE | 16:17 |
| mordred | yeah - it'll be the same certs | 16:17 |
| corvus | mordred: like, the same file locations? | 16:18 |
| mordred | yeah | 16:18 |
| corvus | ok | 16:18 |
| corvus | mordred, clarkb: https://etherpad.openstack.org/p/gerrit-2020-03-20 | 16:18 |
| mordred | clarkb already switched review.o.o to use LE | 16:18 |
| clarkb | ya weuse LE now becayseI didnt want to pay for new certs with switch so close :) | 16:19 |
| clarkb | AJaeger: tha ks! | 16:19 |
| clarkb | corvus: I'll look after breakfast | 16:19 |
| corvus | mordred: getting the docker stuff installed and ready for docker-compose up without actually running it is going to be tricky | 16:20 |
| mordred | corvus: hrm. yeah. I think I have an idea | 16:21 |
| corvus | (combining these two outages: rename + containerization seems to be the thing making this difficult; what's the rename again? i didn't see anything about it in clarkb's announcement) | 16:23 |
| corvus | i guess it's these 2: https://wiki.openstack.org/wiki/Meetings/InfraTeamMeeting#Upcoming_Project_Renames | 16:23 |
| openstackgerrit | Monty Taylor proposed opendev/system-config master: Switch to running gerrit via ansible+containers https://review.opendev.org/713920 | 16:23 |
| openstackgerrit | Monty Taylor proposed opendev/system-config master: Add flag to allow skipping docker-compose up https://review.opendev.org/713924 | 16:23 |
| mordred | corvus: what if we do that ^^ | 16:23 |
| clarkb | corvus: yesthe two on the wiki | 16:24 |
| mordred | then we can land both - nothing should change with the running gerrit - but we could go doublecheck files on disk and whatnot | 16:24 |
| corvus | clarkb: the second one, 712149 isn't a valid zuul config currently; does that need to be fixed? | 16:24 |
| corvus | mordred: yeah, that should work | 16:25 |
| mordred | and I think it's more in keeping with how we like to do gerrit things anyway atm | 16:25 |
| mordred | we don't _really_ want an auto up with a new image if we land a point release image patch :) | 16:26 |
| clarkb | corvus: AJaeger that needs a zuul tenant config change to have Zuul notice | 16:27 |
| openstackgerrit | Andreas Jaeger proposed openstack/project-config master: Update infra-manual links https://review.opendev.org/713925 | 16:28 |
| clarkb | AJaeger: can you double check 712149? it needs to be renamed in zuul/main.yaml too? | 16:31 |
| corvus | okay, i think that's the broad outline in the etherpad; if that looks good, we can expand it a bit to include more detail for the rename playbook step | 16:32 |
| AJaeger | clarkb: indeed, will fix | 16:33 |
| *** roman_g has quit IRC | 16:33 | |
| *** roman_g has joined #opendev | 16:33 | |
| openstackgerrit | Dmitriy Rabotyagov (noonedeadpunk) proposed openstack/diskimage-builder master: Add ensure-python element https://review.opendev.org/713692 | 16:36 |
| openstackgerrit | Andreas Jaeger proposed openstack/infra-manual master: Follow infra-manual move to opendev https://review.opendev.org/713929 | 16:37 |
| openstackgerrit | Andreas Jaeger proposed openstack/project-config master: Move openstack/infra-manuals to opendev https://review.opendev.org/712149 | 16:38 |
| openstackgerrit | Andreas Jaeger proposed openstack/project-config master: Update infra-manual links https://review.opendev.org/713925 | 16:38 |
| openstackgerrit | Andreas Jaeger proposed openstack/project-config master: Update infra-manual links https://review.opendev.org/713925 | 16:39 |
| AJaeger | clarkb: updated and pushed additional changes to complete it ^ | 16:39 |
| clarkb | thanks. corvus ^ fyi | 16:39 |
| openstackgerrit | Monty Taylor proposed opendev/system-config master: Add flag to allow skipping docker-compose up https://review.opendev.org/713924 | 16:40 |
| openstackgerrit | Monty Taylor proposed opendev/system-config master: Switch to running gerrit via ansible+containers https://review.opendev.org/713920 | 16:40 |
| mordred | corvus, clarkb: ^^ | 16:40 |
| corvus | mordred: nice | 16:42 |
| openstackgerrit | Merged opendev/irc-meetings master: Update instructions to match 2020 https://review.opendev.org/713903 | 16:47 |
| openstackgerrit | James E. Blair proposed openstack/diskimage-builder master: Add containerfile element https://review.opendev.org/700083 | 16:57 |
| clarkb | mordred: a question by ttx elsewhere made me wonder, our images still build our fork of gerrit right? | 17:04 |
| clarkb | because we need that for the mysql fixes and the user cache lokkup bugfix | 17:04 |
| corvus | clarkb: && wget https://tarballs.openstack.org/gerrit/gerrit-v2.13.12.11.1707fec.war -O /var/gerrit/bin/gerrit.war | 17:06 |
| corvus | is from the dockerfile | 17:06 |
| corvus | clarkb: modules/openstack_project/manifests/review.pp: 'https://tarballs.openstack.org/gerrit/gerrit-v2.13.12.11.1707fec.war', | 17:06 |
| corvus | clarkb: and that's from the puppet | 17:07 |
| corvus | so i think the answer is that the gerrit container uses the exact same war that puppet is currently using | 17:07 |
| clarkb | perfect | 17:07 |
| mordred | yes, that's right | 17:11 |
| mordred | once we update to 2.15 we will no longer be using a fork | 17:11 |
| clarkb | mordred: see comment on https://review.opendev.org/#/c/713924/2 I think that is what the linter is angry about (but double check that I guess) | 17:17 |
| *** hashar is now known as hasharAway | 17:17 | |
| clarkb | I've also left a couple questions on the etherpad that mordred and corvus may want to look at | 17:19 |
| clarkb | otherwise I think this is looking good | 17:19 |
| openstackgerrit | Monty Taylor proposed opendev/system-config master: Add flag to allow skipping docker-compose up https://review.opendev.org/713924 | 17:22 |
| mordred | clarkb: replied - and pushed that ^^ how's that look now? | 17:22 |
| *** mlavalle has joined #opendev | 17:23 | |
| clarkb | mordred: https://review.opendev.org/#/c/713920/3 needs a rebase now, but I think 924 should work | 17:25 |
| clarkb | we only need to run init when we upgrade major versions right? | 17:25 |
| mordred | that's right | 17:25 |
| clarkb | so this could be optimized further but its fine to run it as is | 17:25 |
| mordred | which is why it's really an opt-in action | 17:25 |
| mordred | yeah | 17:25 |
| mordred | the biggest issue is that we need that volume list - so it's messy - but yeah | 17:26 |
| clarkb | mordred: is there any way to encode the tag filter in https://review.opendev.org/#/c/713924/3/playbooks/start-review.yaml without needing to supply it on the command line? | 17:26 |
| mordred | clarkb: not that I know of? | 17:28 |
| openstackgerrit | Monty Taylor proposed opendev/system-config master: Switch to running gerrit via ansible+containers https://review.opendev.org/713920 | 17:30 |
| openstackgerrit | Clark Boylan proposed opendev/project-config master: March 20, 2020 project renames https://review.opendev.org/713943 | 17:31 |
| clarkb | there is the rename script input file (and historical recording) | 17:32 |
| clarkb | I've put it on the etherpad | 17:32 |
| openstackgerrit | Monty Taylor proposed opendev/system-config master: Add flag to allow skipping docker-compose up https://review.opendev.org/713924 | 17:37 |
| openstackgerrit | Donald Talton proposed opendev/meetbot master: Update meetbot to work under python3 https://review.opendev.org/713948 | 17:38 |
| mordred | infra-root: TIL about "tasks_from" argument to include_role. see usage in 713924 | 17:38 |
| mnaser | where is the opendev python base and opendev python build images defined? | 17:38 |
| mordred | mnaser: system-config | 17:38 |
| mnaser | im wondering if there's a way to install an extra python package without adding it to requirements | 17:38 |
| mnaser | im thinking maybe add it to extras might be worth while | 17:39 |
| mordred | mnaser: so far we've just been adding them to the Dockefile in RUN steps | 17:39 |
| mnaser | (which btw, i found a bug in, you cant have comments on the same line as a dependency in extras) | 17:39 |
| mordred | mnaser: but - I bet something could be done? you got a concrete example? | 17:39 |
| mnaser | mordred: adding uwsgi to the image without using system packages but without updating the dependencies of the entire project | 17:40 |
| mnaser | (in this case) | 17:40 |
| mordred | mnaser: ah. interesting | 17:40 |
| mnaser | mordred: also, this breaks http://paste.openstack.org/show/790906/ | 17:42 |
| mnaser | with "# is not a valid dependency" | 17:42 |
| mordred | mnaser: two thoughts ... a) we could add an env parameter something that defaults to empty that gets passed to assemble b) or maybe define an extra that we always look for - but then youv'e got to add that extra everywhere ... c) might be worth making openstack downstream versions that just plop uwsgi into the mix | 17:42 |
| mordred | awesome | 17:42 |
| clarkb | comments in ini files are after ; right? | 17:45 |
| clarkb | mnaser: ^ double check that # is valid in the file format | 17:45 |
| mnaser | clarkb: that's in setup.cfg | 17:46 |
| clarkb | right thats a python ini file | 17:46 |
| mnaser | im not familiar with the format, but that might be a thing | 17:46 |
| clarkb | https://docs.python.org/3/library/configparser.html#supported-ini-file-structure # is valid too but maybe only on a line of its own? | 17:47 |
| clarkb | mnaser: I would try that, move the comment onto their own line and see if it is still mad | 17:48 |
| mordred | clarkb, corvus, mnaser: I'm gonna go ahead and land https://review.opendev.org/#/c/711073/ so it's there before mnaser gets too far down his current path | 17:55 |
| clarkb | mordred: ok | 17:55 |
| openstackgerrit | Monty Taylor proposed opendev/system-config master: Switch to running gerrit via ansible+containers https://review.opendev.org/713920 | 17:56 |
| clarkb | mordred: cloudnull says you need to add the excention on the include tasks magic | 17:57 |
| openstackgerrit | Monty Taylor proposed opendev/system-config master: Add a uwsgi-base container image https://review.opendev.org/713953 | 18:03 |
| mordred | clarkb: yeah - I'm waiting to see test results - the docs say we don't | 18:05 |
| clarkb | ah | 18:06 |
| mordred | but since we do run that in tests, we'll find out :) | 18:06 |
| *** rpittau is now known as rpittau|afk | 18:06 | |
| openstackgerrit | Monty Taylor proposed opendev/system-config master: Add a uwsgi-base container image https://review.opendev.org/713953 | 18:07 |
| mnaser | Unable to freeze job graph: Job system-config-build-image-uwsgi-base depends on system-config-build-image-python-base which was not run. | 18:12 |
| mnaser | funny bug | 18:12 |
| mnaser | well, probably not funny | 18:12 |
| mnaser | mordred: i think you need to add the uwsgi image to files: | 18:13 |
| mnaser | for the base | 18:13 |
| mordred | mnaser: oh. right. that does need to be soft | 18:15 |
| mordred | one sec | 18:15 |
| openstackgerrit | Monty Taylor proposed opendev/system-config master: Add a uwsgi-base container image https://review.opendev.org/713953 | 18:16 |
| mordred | mnaser: that should be better | 18:16 |
| mnaser | mordred: cool, i wonder if there's a way i can pull this locally and try it here | 18:20 |
| mnaser | i guess i can build locally and then tag it locally and keep iterating | 18:21 |
| clarkb | mordred: corvus fungi based on the etherpad I think things are coming together. Other than reviews of linked changes is there anything else we should be doing? | 18:22 |
| fungi | this morning has been a meeting scramble for me. i saw it in scrollback and have pulled it up but not gone over it carefully yet | 18:23 |
| fungi | but i trust your assessment of it | 18:24 |
| clarkb | thanks. I think it is coming together, just want to make sure I'm not neglecting things I can help wtih | 18:24 |
| clarkb | I'm about to enter my lunch and bike ride no/less computer time | 18:24 |
| mordred | mnaser: yeah - that's the shortest path for local iteration | 18:27 |
| mordred | clarkb: and no, I don't think so | 18:27 |
| *** roman_g has quit IRC | 18:28 | |
| mordred | clarkb: this afternoon when you're back from bike ride - maybe let's land the gerrit ansible changes while watching them so we can verify that the files it's laying down on disk are all correct | 18:28 |
| clarkb | k | 18:28 |
| mnaser | mordred: just thought about it too i think having CMD ["/usr/bin/uwsgi"] might be a good idea too | 18:35 |
| openstackgerrit | Dmitriy Rabotyagov (noonedeadpunk) proposed openstack/diskimage-builder master: Add python-stow-versions element https://review.opendev.org/713692 | 18:35 |
| openstackgerrit | Dmitriy Rabotyagov (noonedeadpunk) proposed openstack/diskimage-builder master: Add python-stow-versions element https://review.opendev.org/713692 | 18:37 |
| mordred | mnaser: that's a good point | 18:38 |
| openstackgerrit | Monty Taylor proposed opendev/system-config master: Add a uwsgi-base container image https://review.opendev.org/713953 | 18:38 |
| mordred | then the image just works with the appropriate env var and all is peachy | 18:39 |
| mnaser | yep! | 18:40 |
| mnaser | mordred: using your image, im cleaning it up more, but using port 80 means that when i drop privs so i dont run as root, i cant listen on port 80 with uwsgi | 19:04 |
| bcafarel | clarkb: AJaeger good news fedora job passed properly on https://review.opendev.org/#/c/713902/ indeed! I will check the periodic jobs tomorrow and close the LP then (for double checking) | 19:06 |
| mordred | mnaser: ah - good point | 19:07 |
| mordred | mnaser: maybe we should just pick a non-priv internal port - since it's going to be an external port mapping anyway | 19:07 |
| mordred | mnaser: or - honestly - feel free to push up changes as you hack on it :) | 19:07 |
| mnaser | mordred: im kinda tempted to just not include the port and have services define the port in there | 19:08 |
| mordred | mnaser: wfm | 19:08 |
| mordred | mnaser: that would actually be nicer - because it would allow the normal port spread to happen and let people run with network: host | 19:09 |
| mnaser | mordred: yep, i think that would be nice too if someone wants to do that | 19:09 |
| mordred | or not - really whatever the deployer wants to do should work in that context | 19:09 |
| *** hasharAway is now known as hashar | 19:10 | |
| mordred | mnaser: want me to update the patch? | 19:10 |
| mordred | or you wanna push up things? | 19:10 |
| openstackgerrit | Mohammed Naser proposed opendev/system-config master: Add a uwsgi-base container image https://review.opendev.org/713953 | 19:10 |
| mnaser | mordred: ^ i had a few more things in there too | 19:10 |
| AJaeger | bcafarel: great! | 19:12 |
| mordred | mnaser: is that preferable to adding a USER line? | 19:13 |
| mnaser | mordred: interesting, i didnt think about doing that, i think the benefit that this has is that we let uwsgi drop privs instead of doing it in the container | 19:13 |
| mnaser | i guess i will try that alternative | 19:14 |
| mordred | mnaser: but if it's opening ports after the drop - is that any better? mostly just talking out loud here | 19:14 |
| mnaser | mordred: not really in this case honestly | 19:14 |
| mnaser | ok let me revise locally | 19:14 |
| mnaser | ran into another issue with the python3 plugin not being loaded for uwsgi | 19:15 |
| mordred | yay | 19:15 |
| mordred | mnaser: ok - this is a bit more complex | 19:15 |
| mnaser | does that mean this becomes uwsgi-python-base | 19:16 |
| mordred | and we might want to go back to installing this a little different | 19:16 |
| mnaser | or we just assume people are using uwsgi for python only because they are sane | 19:16 |
| mordred | since the python in these images is not distro python | 19:16 |
| mnaser | yeah but that's fine | 19:16 |
| mnaser | we just need to install uwsgi-plugin-python3 | 19:16 |
| mordred | right - but that doesn't do what we want | 19:16 |
| mordred | because it wants to install a bunch of distro python because of deps | 19:17 |
| * mordred is poking | 19:17 | |
| mnaser | oh i see | 19:17 |
| openstackgerrit | Dmitriy Rabotyagov (noonedeadpunk) proposed openstack/diskimage-builder master: Add python-stow-versions element https://review.opendev.org/713692 | 19:18 |
| mordred | mnaser: update coming | 19:19 |
| mnaser | mordred: maybe add "USER nobody:nogroup" in there too | 19:20 |
| openstackgerrit | Dmitriy Rabotyagov (noonedeadpunk) proposed openstack/diskimage-builder master: Add python-stow-versions element https://review.opendev.org/713692 | 19:21 |
| openstackgerrit | Monty Taylor proposed opendev/system-config master: Add a uwsgi-base container image https://review.opendev.org/713953 | 19:30 |
| mordred | mnaser: enjoy | 19:30 |
| mnaser | openstackgerrit: oh interesting | 19:31 |
| mnaser | mordred: poop | 19:33 |
| mnaser | python: can't open file 'setup.py': [Errno 2] No such file or directory | 19:33 |
| mnaser | upon running + python setup.py sdist bdist_wheel -d /output/wheels | 19:33 |
| mordred | sorry - missed a git add | 19:33 |
| openstackgerrit | Monty Taylor proposed opendev/system-config master: Add a uwsgi-base container image https://review.opendev.org/713953 | 19:34 |
| mnaser | i love how that file forever remains (c) 2013 hp | 19:34 |
| mnaser | good times | 19:34 |
| mnaser | mordred: poop x2 | 19:38 |
| mnaser | E: List directory /var/lib/apt/lists/partial is missing. - Acquire (13: Permission denied) | 19:38 |
| mnaser | uwsgi-base drops perms | 19:38 |
| mnaser | so we cant run apt install | 19:38 |
| mordred | uhm | 19:39 |
| mnaser | so either defer to letting consumers do USER nobody:nogroup *or* go back to using uwsgi_uid uwsgi_gid | 19:39 |
| mordred | mnaser: I think that's a good reason why to do the other thing | 19:39 |
| mordred | yeah | 19:39 |
| mnaser | ok ill do that | 19:39 |
| mordred | kk | 19:39 |
| openstackgerrit | Mohammed Naser proposed opendev/system-config master: Add a uwsgi-base container image https://review.opendev.org/713953 | 19:42 |
| openstackgerrit | Mohammed Naser proposed opendev/system-config master: Add a uwsgi-base container image https://review.opendev.org/713953 | 19:45 |
| mnaser | great, i can use that to build keystone successfully | 19:46 |
| mnaser | granted i have "/etc/keystone/fernet-keys/ does not exist" but hey | 19:46 |
| *** mlavalle has quit IRC | 19:49 | |
| mordred | mnaser: yeah- but whatever - that's something youd' want to mount in anyway | 19:52 |
| mnaser | mordred: yep, im trying a small docker-compose now to test it out | 19:53 |
| *** mlavalle has joined #opendev | 19:58 | |
| corvus | mordred, clarkb, ianw: the test for the containerfile element in dib has passed: https://zuul.opendev.org/t/openstack/build/9a24e4812f5c46839b048495c56d5aee/log/logs/containerfile_bionic-build-succeeds.PASS.log#134 | 20:04 |
| corvus | there are 2 things i think we need to address there: 1) how we want to test that since it uses podman which requires installing a ppa on ubuntu | 20:05 |
| corvus | 2) how to test that it actually produces a bootable image | 20:05 |
| corvus | but we have at least achieved the milestone of having theoretically built a vm image from an arbitrary dockerfile | 20:06 |
| mordred | corvus: woot! | 20:08 |
| mordred | corvus: so we need podman ppa installed on the test node that dib is running on right/ | 20:08 |
| mordred | ? | 20:08 |
| mordred | corvus: for 2 - we do have nodepool tests that run a nodepool, build an image and boot it - maybe we build on that? | 20:09 |
| corvus | mordred: yeah; i just stuck the 'install-podman' role in the job for that change in order to demo it; i need more feedback from folks to know the right way to test that. | 20:09 |
| corvus | mordred: yeah re 2 that sounds like it might be the way to go | 20:10 |
| corvus | (re 1, i think i would advocate extending the install-podman role to work on as many dib test platforms as possible, and using it there, and then skipping the test on non-conforming platforms) | 20:11 |
| corvus | (though right now, i think install-podman errors if you run it on an unsupported platform, so we'd have to soften that i think?) | 20:11 |
| corvus | (or otherwise make the inclusion of install-podman conditional) | 20:11 |
| mordred | yes to all of that | 20:17 |
| mordred | I think making it work on as many test platforms as possible seems like a great step - but then it's probably better to conditionally install it? or just not test building container-image dib images on non-podman supporting platforms? | 20:18 |
| *** ralonsoh has quit IRC | 20:19 | |
| corvus | yeah, i think we'll have to skip the test on platforms that don't have podman available; and conditionally running the role is best, but just seems weird having to duplicate the platform logic. but it's probably most correct. | 20:20 |
| openstackgerrit | Monty Taylor proposed opendev/system-config master: Add constraints support to python-builder https://review.opendev.org/713972 | 20:24 |
| openstackgerrit | Monty Taylor proposed opendev/system-config master: Add constraints support to python-builder https://review.opendev.org/713972 | 20:24 |
| mordred | mnaser: ^^ that should allow a pre-playbook to drop upper-constraints in place | 20:25 |
| mnaser | mordred: oh neat. that really just makes it natively happen | 20:25 |
| openstackgerrit | Brian Rosmaita proposed openstack/project-config master: Revise gerrit ACLs for devstack-plugin-open-cas https://review.opendev.org/713974 | 20:27 |
| openstackgerrit | Monty Taylor proposed opendev/system-config master: Add flag to allow skipping docker-compose up https://review.opendev.org/713924 | 20:51 |
| openstackgerrit | Monty Taylor proposed opendev/system-config master: Switch to running gerrit via ansible+containers https://review.opendev.org/713920 | 20:51 |
| mordred | clarkb: ansible-lint actually caught a real error | 20:52 |
| clarkb | mordred: ya it does seem to catch real issues which is why I worry about dropping it. It just also complains about things that don't matter much :) | 20:53 |
| mordred | yup :) | 20:57 |
| mordred | corvus: I got the JWT error again: https://zuul.opendev.org/t/openstack/build/149e503f426444c1add1e0dc721a2f9a | 21:07 |
| corvus | mordred: 2 out of 2? | 21:07 |
| mordred | yup | 21:07 |
| mordred | happily uploaded in gate and failed in promote | 21:07 |
| corvus | mordred: hrm. that's really hard to debug | 21:07 |
| mordred | yeah. I feel like maybe I just did something dumb somewhere? | 21:07 |
| corvus | mordred: i'm in a rabbit hole right now, but i can try to help out later | 21:08 |
| mordred | corvus: k. no rush | 21:08 |
| corvus | mordred: (at this point, my debug strategy would probably be to try to run the role locally with the actual secret data and full debug) | 21:08 |
| mordred | yeah. I'm going to try cargo-culting something first - then that's next on my list | 21:10 |
| openstackgerrit | Ian Wienand proposed zuul/zuul-jobs master: upload-logs-swift: Create a download script https://review.opendev.org/592341 | 21:11 |
| openstackgerrit | Ian Wienand proposed zuul/zuul-jobs master: upload-logs-swift: Add a unicode file https://review.opendev.org/592853 | 21:11 |
| openstackgerrit | Harald Jensås proposed openstack/diskimage-builder master: Use rpm -e instead of dnf for cleaning old kernels https://review.opendev.org/713988 | 21:32 |
| openstackgerrit | Ian Wienand proposed zuul/zuul-jobs master: upload-logs-swift: Create a download script https://review.opendev.org/592341 | 21:35 |
| openstackgerrit | Ian Wienand proposed zuul/zuul-jobs master: upload-logs-swift: Add a unicode file https://review.opendev.org/592853 | 21:35 |
| openstackgerrit | Harald Jensås proposed openstack/diskimage-builder master: Use rpm -e instead of dnf for cleaning old kernels https://review.opendev.org/713988 | 21:36 |
| mordred | infra-root: could I get some reviews on https://review.opendev.org/#/c/713924/ and https://review.opendev.org/#/c/713920/6 ? | 21:40 |
| clarkb | mordred: minor -1 on the second change | 21:44 |
| clarkb | I think its actually ok to land as is but the suggestion should reduce any potential for confusion | 21:45 |
| *** DSpider has quit IRC | 21:45 | |
| fungi | i have a minor question on the first one, but still +2 | 21:45 |
| fungi | ahh, and my question is basically answered by the second change | 21:46 |
| mordred | clarkb: good call | 21:49 |
| openstackgerrit | Monty Taylor proposed opendev/system-config master: Switch to running gerrit via ansible+containers https://review.opendev.org/713920 | 21:49 |
| mordred | fungi: yeah - normally we don't have cfg-mgmt start or stop gerrit - but we need it to start/stop automatically in the gate | 21:49 |
| mordred | thus the flag and stuff | 21:49 |
| mordred | clarkb, fungi: actually - let me add one more thing for clarity | 21:51 |
| openstackgerrit | Monty Taylor proposed opendev/system-config master: Switch to running gerrit via ansible+containers https://review.opendev.org/713920 | 21:53 |
| mordred | clarkb, fungi: I think that's nicer now | 21:53 |
| clarkb | oh and you added the second false flag | 21:53 |
| openstackgerrit | Monty Taylor proposed opendev/system-config master: Rename remote_puppet_git to service-git https://review.opendev.org/713995 | 21:55 |
| mordred | yeah. just to be explicit | 21:55 |
| mordred | it defaults to false | 21:55 |
| fungi | it's like he's running an entire false flag campaign | 21:55 |
| mordred | but we really dont' want it there | 21:55 |
| mordred | hahaha | 21:55 |
| * fungi sees what's really going on | 21:55 | |
| fungi | per note in the etherpad, i approved those so if we *don't* want them merged today, lmk quickly or -2 the first one | 22:02 |
| fungi | or wip or whatever | 22:03 |
| mordred | no - I think that's good | 22:03 |
| mordred | although terrifying still :) | 22:03 |
| fungi | i trust zuul implicitly | 22:04 |
| fungi | (but really i trust our ability to fix things if they're terribly wrong) | 22:04 |
| fungi | and i'm around to scramble if they are | 22:05 |
| mordred | yeah. same here | 22:05 |
| mordred | I made a copy of review_site/etc to review_site/etc.bak - largely so we can easily check the results of the ansible run | 22:06 |
| fungi | oh, that's smart | 22:06 |
| clarkb | I'm semi around, but not fully | 22:07 |
| mordred | I think that's most likely the place where we'll go "oh, crap - look we forgot to ..." | 22:07 |
| fungi | somebody clearly ate their wheaties[tm] this morning | 22:07 |
| openstackgerrit | Ian Wienand proposed zuul/zuul-jobs master: upload-logs-swift: Create a download script https://review.opendev.org/592341 | 22:11 |
| openstackgerrit | Ian Wienand proposed zuul/zuul-jobs master: upload-logs-swift: Add a unicode file https://review.opendev.org/592853 | 22:11 |
| openstackgerrit | Ian Wienand proposed zuul/zuul-jobs master: test-upload-logs-swift: add download script as an artifact https://review.opendev.org/713998 | 22:11 |
| openstackgerrit | Merged opendev/system-config master: Add flag to allow skipping docker-compose up https://review.opendev.org/713924 | 22:11 |
| ianw | infra-root: ^^ could i get some reviews on 713998 there for the *test* of the log download script. if there isn't enough interest to merge, i'll revert the whole thing, but it would be good for reviews to see an actual result | 22:13 |
| fungi | and if anyone needs a humor break, this was a fun blast from the past: https://www.gnu.org/fun/jokes/ed-msg.txt | 22:16 |
| fungi | `man ed` on my systems does still cite "Ed is the standard text editor" | 22:16 |
| ianw | fungi: s/When I log into my Xenix system with my 110 baud teletype, both vi*and* Emacs are just too damn slow./container/ and it's 2020 ready? | 22:18 |
| clarkb | fungi: "Ed is the 'standard' text editor in the sense that it is the original editor for Unix" is what mine says | 22:18 |
| mordred | ianw: ++ | 22:18 |
| fungi | clarkb: yup, i omitted the rest ;) | 22:18 |
| mordred | I have fixed a down system when ed was the only tool available before | 22:18 |
| mordred | jesus | 22:19 |
| mordred | WHAT NOW LINTERS | 22:20 |
| fungi | those are the "uphill both ways in the snow barefoot" stories we tell millennial sysadmins | 22:20 |
| mordred | oh. hahahahaha | 22:20 |
| JayF | fungi: I always thought that was https://www.ibiblio.org/harris/500milemail.html | 22:21 |
| openstackgerrit | Ian Wienand proposed zuul/zuul-jobs master: test-upload-logs-swift: add download script as an artifact https://review.opendev.org/713998 | 22:21 |
| openstackgerrit | Ian Wienand proposed zuul/zuul-jobs master: Revert "upload-logs-swift: test bulk download script change" https://review.opendev.org/714000 | 22:21 |
| fungi | or as i saw someone else say once, (paraphrasing) "ed's primary use is taunting sysadmins with too much hue in their beards" | 22:21 |
| openstackgerrit | Monty Taylor proposed opendev/system-config master: Switch to running gerrit via ansible+containers https://review.opendev.org/713920 | 22:21 |
| mordred | fungi, clarkb: ^^ | 22:22 |
| mordred | forgot to, you know, remove gerrit also from the test fixture for puppet | 22:22 |
| clarkb | did it break again? | 22:22 |
| mordred | clarkb: didn't update a test | 22:22 |
| mordred | clarkb: I thnik you'll enjoy the thing | 22:22 |
| clarkb | group tests | 22:22 |
| mordred | yup | 22:22 |
| fungi | ianw: 592341 seems to depend on an old patchset of 713998 | 22:26 |
| mordred | fungi: mind giving https://review.opendev.org/#/c/713920/ another +A? | 22:26 |
| corvus | mordred: how are we looking for tomorrow? | 22:26 |
| corvus | i lost my etherpad link | 22:27 |
| corvus | ah https://etherpad.openstack.org/p/gerrit-2020-03-20 | 22:27 |
| mordred | corvus: https://etherpad.openstack.org/p/gerrit-2020-03-20 | 22:27 |
| mordred | corvus: and pretty good - wanna look at https://review.opendev.org/#/c/713920/ ? | 22:27 |
| mordred | (we had to iterate on it a couple of times) | 22:28 |
| corvus | mordred: lgtm! | 22:28 |
| mordred | woot | 22:29 |
| mordred | corvus: I copied review_site/etc to review_site/etc.bak - so we can double-check what ansible is splatting out | 22:29 |
| corvus | i'm about to head out for a shopping trip (?!?!?!) so i'm not going to approve right now, but i support you approving it if you're ready | 22:29 |
| mordred | corvus: enjoy your shopping trip! | 22:29 |
| ianw | fungi: umm there is only one revision of 713998 so there couldn't be an old one? | 22:30 |
| openstackgerrit | Monty Taylor proposed opendev/system-config master: Rename remote_puppet_git to service-git https://review.opendev.org/713995 | 22:30 |
| openstackgerrit | Monty Taylor proposed opendev/system-config master: Remove unused gerrit puppet things https://review.opendev.org/714001 | 22:30 |
| corvus | mordred: i will perform only the following essential activity: obtaining essential supplies from essential buisenesses! | 22:30 |
| fungi | ianw: hum, i see two? | 22:30 |
| ianw | fungi: ohh, it rebased | 22:31 |
| fungi | yep | 22:31 |
| fungi | second is a rebase | 22:31 |
| fungi | but gerrit's going to claim that 592341 can't merge | 22:31 |
| fungi | due to the git parent no longer having any chance of appearing in the branch history | 22:32 |
| clarkb | corvus: good luck with the hsopping | 22:32 |
| clarkb | my brothre tried to find tp yesterday | 22:32 |
| clarkb | was not successful so we've had to take local inventory and may start rationing it out | 22:33 |
| mordred | have you considered ordering from amazon? | 22:33 |
| clarkb | you can't right now | 22:33 |
| clarkb | they are completely out of stock | 22:33 |
| clarkb | (we tried) | 22:33 |
| mordred | nice | 22:34 |
| clarkb | thankfully for us amazon did eventually deliver our normal monthly supply via subscribe and save | 22:34 |
| clarkb | so we are set here and can manage with rationing to siblings | 22:34 |
| mordred | I continue to be sad that we haven't ordered japanese toilet seats yet | 22:35 |
| openstackgerrit | Ian Wienand proposed zuul/zuul-jobs master: upload-logs-swift: Create a download script https://review.opendev.org/592341 | 22:37 |
| openstackgerrit | Ian Wienand proposed zuul/zuul-jobs master: upload-logs-swift: Add a unicode file https://review.opendev.org/592853 | 22:37 |
| *** dpawlik has quit IRC | 22:38 | |
| *** hashar has quit IRC | 22:39 | |
| ianw | fungi: thanks, that should have sorted it. short story is add 713998 to test it, vote on 592341 and either merge that or merge 714000 to get rid of it all | 22:39 |
| ianw | (i think the numbers give away that i let that sit for too long) | 22:40 |
| fungi | ianw: yup, makes sense. i'm good with 713998, still looking at the rest | 22:41 |
| ianw | fungi: i know you're on record as thinking curl | bash is an anti-pattern. i'm not sure if the generation coming from a protected job sways you that the convenience for casual users makes it useful. | 22:43 |
| fungi | it's really not so much a case-by-case basis for me, more that any time you tell users it's okay they're likely to be desensitized to the risks of doing it for unrelated cases | 22:46 |
| openstackgerrit | Ian Wienand proposed zuul/zuul-jobs master: Revert "upload-logs-swift: test bulk download script change" https://review.opendev.org/714000 | 22:47 |
| fungi | but hey, at least it's not curl | sudo bash | 22:48 |
| *** diablo_rojo has joined #opendev | 23:11 | |
| fungi | apropos of https://review.opendev.org/625450 the debian community is using peertube now: https://wiki.debian.org/Teams/DebianSocial | 23:17 |
| openstackgerrit | Merged opendev/system-config master: Switch to running gerrit via ansible+containers https://review.opendev.org/713920 | 23:24 |
| corvus | clarkb: at the local supermarket: still no tp, clorox, flour, sugar. and they're rationing eggs now. | 23:34 |
| clarkb | corvus: rice and beans at least? | 23:34 |
| clarkb | I did my part and picked up sardines because I actually like them :) | 23:35 |
| corvus | clarkb: very little rice and beans; but they did actually have a few bags of generic "long grain" rice | 23:35 |
| corvus | and a few boutique grains | 23:36 |
| corvus | (our local produce market had no bulk grains/beans except split peas. fully stocked on split peas) | 23:36 |
| *** shubjero has joined #opendev | 23:38 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!