Wednesday, 2020-12-02

*** sboyron has quit IRC00:07
clarkbfungi: looks like your change has entered the gate now00:30
fungithanks, and yeah i've been stalking it00:31
clarkbwe should double check there isn't any unexpected changes to the other files that are being modified too before restarting though I reviewed the change multiple times and I expect htat is fine00:32
fungii mean, i'll keep an eye on it during the restart too. in the unlikely event that it balks, i can manually unroll stuff00:36
*** weshay|ruck is now known as weshay|pto00:41
*** mlavalle has quit IRC00:56
ianwi manged to a "Zuul Summary" tab @ just as a proof of concept.  filling stuff into the tab, etc. is obviously to-do, but i think it's more the way plugins are supposed to work00:58
ianwit builds with bazel, which, i don't think is a blocker, but again is just more "gerrity" way to do things01:00
clarkbianw: we're building gerrit with bazel. Also maybe this is somethign we can host upstream and then the upstream zuul can bazel build it for us or something01:00
ianwyeah, sorry i meant that *not* using bazel probably isn't a blocker; but it does make it look more just like everything else in the gerrit tree01:01
ianwit's all cribbed from the code editor and checks plugins.  i'll clean it up and add a link to that mail, see if anyone wants to collaborate01:04
fungimmm, the jgit config change hot a timeout on system-config-run-review01:41
fungii wonder if it's causing gerrit to not start01:41
fungiof so, yay testing!01:41
chengsheng1I am a new contributor. My code has been submitted for a long time, but there is no review. I ask for help if there is a problem with which process.
clarkbchengsheng1: your best bet will be to ask in #openstack-nova01:54
clarkbchengsheng1: the people there are responsible for reviewing changes to that repo01:54
clarkbfungi: looks like it timed out more "normally". It appears the deployment happened but then there wasn't enough time to do the testinfra testing01:55
chengsheng1ok, thank you.01:56
clarkbfungi: task install ansible seems to not have been fast. Also there is a large gap after 01:24 that is odd01:56
clarkbchengsheng1: one thing to keep in mind is that last week was a big holiday week for those of us in the USA01:57
clarkbnot sure if that would impact nova reviews but it seems possible01:57
chengsheng1maybe this is the reason02:01
fungiclarkb: ahh, yeah, digging into the logs it looks like it was just unusually slow/stalling at that point. i'll recheck it02:02
*** ysandeep|away is now known as ysandeep02:18
*** iurygregory|pto has quit IRC03:01
ianwsigh, it hit a limestone mirror error?03:08
*** whoami-rajat__ has quit IRC03:29
*** whoami-rajat__ has joined #opendev04:06
fungiyeah, looks like it04:21
fungiif it actually merges this time, i should still be around to do a restart after it's deployed04:59
openstackgerritMerged opendev/system-config master: Move jgit autogc disablement into correct config
*** ykarel has joined #opendev05:23
fungithe deploy claims to have been successful too, checking configs now05:25
fungilooks right05:28
fungidoing a quick gerrit restart nw05:28
fungion its way back up now05:29
fungi#status log restarted the gerrit service on review.o.o for the config change from
openstackstatusfungi: finished logging05:29
fungilooks like it's up and working fine05:30
fungiianw: ^05:32
fungii'll stick around for a bit longer in case anyone reports problems05:32
fungiokay, suspending my higher order cognitive functions and entering low-power standby for a while. i'll wake on interrupt in 8 hours or so05:55
*** marios has joined #opendev06:12
*** zbr has quit IRC06:13
*** zbr has joined #opendev06:14
*** hamalq has quit IRC06:17
*** hamalq has joined #opendev06:18
*** sboyron has joined #opendev06:42
*** brinzhang0 has joined #opendev06:46
*** brinzhang_ has quit IRC06:49
*** sboyron has quit IRC06:55
*** eolivare has joined #opendev07:27
*** brinzhang_ has joined #opendev07:32
*** brinzhang0 has quit IRC07:35
*** lpetrut has joined #opendev07:38
*** fressi has joined #opendev07:49
*** slaweq has joined #opendev07:56
*** hashar has joined #opendev08:00
*** hashar has quit IRC08:04
*** hashar has joined #opendev08:04
*** sboyron has joined #opendev08:05
*** rpittau|afk is now known as rpittau08:28
*** andrewbonney has joined #opendev08:34
*** DSpider has joined #opendev08:36
*** zbr has quit IRC08:44
*** zbr has joined #opendev08:45
*** zbr has quit IRC08:47
*** zbr has joined #opendev08:47
*** zbr has quit IRC08:49
*** tkajinam has quit IRC08:49
*** zbr has joined #opendev08:49
*** zbr has quit IRC08:52
*** zbr has joined #opendev08:52
*** tosky has joined #opendev09:05
*** marios has quit IRC09:06
*** sshnaidm is now known as sshnaidm|ruck09:34
*** sshnaidm|ruck has quit IRC10:01
*** sshnaidm has joined #opendev10:01
*** mklyus has joined #opendev10:08
mklyusHi, I need some one from core reviewers could take a look on this change and help to get +2 and merge it10:09
openstackgerritMerged openstack/project-config master: Remove openstack-python and openstack-docs job for x/vmtp project.
*** ykarel_ has joined #opendev10:27
*** ykarel_ is now known as ykarel|lunch10:28
*** ykarel has quit IRC10:29
*** hashar is now known as hasharLunch10:32
*** icey_ has joined #opendev10:43
*** icey has quit IRC10:44
mklyusThanks a lot Ian10:52
openstackgerritStephen Finucane proposed opendev/elastic-recheck master: Add query for bug 1906428
openstackgerritStephen Finucane proposed opendev/elastic-recheck master: gitignore: Ignore .eggs directory
openstackbug 1906428 in OpenStack Compute (nova) "test_cold_migrate_unshelved_instance failing with cat: can't open '/mnt/timestamp': No such file or directory" [Critical,Confirmed]
*** sshnaidm is now known as sshnaidm|afk10:55
*** ysandeep is now known as ysandeep|afk11:04
*** marios has joined #opendev11:07
*** mklyus has quit IRC11:11
*** dtantsur|afk is now known as dtantsur11:19
*** ykarel|lunch is now known as ykarel11:30
*** ykarel_ has joined #opendev11:35
*** ykarel has quit IRC11:38
*** sshnaidm|afk is now known as sshnaidm|ruck11:39
*** icey_ has quit IRC11:44
openstackgerritSorin Sbârnea proposed zuul/zuul-jobs master: Enable ensure ansible to install local collection
*** brinzhang_ has quit IRC11:52
*** brinzhang_ has joined #opendev11:52
*** icey has joined #opendev11:58
*** iurygregory has joined #opendev12:03
openstackgerritSorin Sbârnea proposed zuul/zuul-jobs master: Enable ensure ansible to install local collection
openstackgerritSorin Sbârnea proposed zuul/zuul-jobs master: Enable ensure ansible to install local collection
*** rishabhhpe has joined #opendev12:14
openstackgerritSorin Sbârnea proposed opendev/gerritbot master: Skip notifications about WIP changes
*** dtantsur is now known as dtantsur|brb12:31
*** toma4 has quit IRC12:55
*** rishabhhpe has quit IRC12:55
*** hamalq has quit IRC13:01
*** hamalq has joined #opendev13:05
*** ykarel_ is now known as ykarel13:30
*** ysandeep|afk is now known as ysandeep13:30
*** hasharLunch is now known as hashar13:43
*** dtantsur|brb is now known as dtantsur13:46
*** eharney has joined #opendev13:56
*** auristor has quit IRC14:10
openstackgerritMerged opendev/gerritlib master: Avoid crash due to POLLIN event check
*** hashar is now known as hasharAway15:05
openstackgerritSorin Sbârnea proposed opendev/gerritbot master: Skip notifications about WIP changes
*** auristor has joined #opendev15:09
*** ysandeep is now known as ysandeep|away15:23
*** mlavalle has joined #opendev15:28
openstackgerritSorin Sbârnea proposed opendev/gerritbot master: Skip notifications about WIP changes
*** lpetrut has quit IRC15:41
openstackgerritSorin Sbârnea proposed opendev/gerritbot master: Use yaml.safe_load instead of load.
*** slaweq has quit IRC15:54
*** slaweq has joined #opendev15:56
*** redrobot has quit IRC16:11
*** brinzhang0 has joined #opendev16:14
*** brinzhang_ has quit IRC16:16
fungiwe're around 2/3 full on the changeid_project cache after the gerrit restart from just before i went to sleep. 39% cache hit ratio at the moment16:38
*** marios is now known as marios|out16:38
*** marios|out has quit IRC16:44
openstackgerritMerged zuul/zuul-jobs master: upload-logs-s3: fix test-jobs files: attribute
openstackgerritSorin Sbârnea proposed opendev/gerritbot master: Use yaml.safe_load instead of load.
openstackgerritAlbin Vass proposed zuul/zuul-jobs master: GZIPCompressedStream default read length to -1
openstackgerritAlbin Vass proposed zuul/zuul-jobs master: GZIPCompressedStream default read size to -1
openstackgerritMerged opendev/elastic-recheck master: Add query for bug 1906428
openstackbug 1906428 in OpenStack Compute (nova) "test_cold_migrate_unshelved_instance failing with cat: can't open '/mnt/timestamp': No such file or directory" [Critical,In progress]
clarkbmelwitt: ^ reminded me to check the health of the processing pipeline and we still have all 80 workers reporting to gearman says gearman status. I think your change made a big improvement. THank you!17:03
*** ykarel is now known as ykarel|away17:04
*** rpittau is now known as rpittau|afk17:04
melwittclarkb: that's great to hear :)17:06
openstackgerritMerged opendev/elastic-recheck master: gitignore: Ignore .eggs directory
*** ykarel|away has quit IRC17:09
*** ykarel|away has joined #opendev17:17
openstackgerritPaul Belanger proposed zuul/zuul-jobs master: ensure-podman: Use official podman repos for ubuntu
*** ykarel|away has quit IRC17:22
openstackgerritPaul Belanger proposed zuul/zuul-jobs master: ensure-podman: Use official podman repos for ubuntu
*** eolivare has quit IRC17:44
*** sshnaidm|ruck is now known as sshnaidm|afk17:47
*** toma4 has joined #opendev17:48
openstackgerritPaul Belanger proposed zuul/zuul-jobs master: ensure-podman: Use official podman repos for ubuntu
openstackgerritPaul Belanger proposed zuul/zuul-jobs master: ensure-podman: Use official podman repos for ubuntu
*** lpetrut has joined #opendev18:09
*** hasharAway is now known as hashar18:15
*** andrewbonney has quit IRC18:19
openstackgerritMerged opendev/system-config master: gerrit: set ownership on ~gerrit2/.ssh directory
openstackgerritMerged opendev/system-config master: gerrit: fix db/ mount for gate testing
openstackgerritJeremy Stanley proposed opendev/system-config master: Update gerrit docker image to java 11
openstackgerritJeremy Stanley proposed opendev/system-config master: Build Gerrit 3.3 images
openstackgerritPaul Belanger proposed zuul/zuul-jobs master: ensure-podman: Use official podman repos for ubuntu
*** dtantsur is now known as dtantsur|afk18:47
openstackgerritMerged zuul/zuul-jobs master: GZIPCompressedStream default read size to -1
*** toma4 has quit IRC19:00
fungii'm heading out to run some errands, should be back in less than an hour. hopefully by then the openjdk 11 update should have made it into our gerrit images19:01
melwittwith the new gerrit, I read that there's no integration with updating launchpad for blueprints, but what about bugs? I noticed my bug didn't get a comment and state update on it19:28
*** iurygregory has quit IRC19:29
fungimelwitt: yes, see currently around line #6019:48
fungiwe need to rewrite some chunks of jeepyb which used to perform database queries. there's no more rdbms containing that data so some bits need to switch to using gerrit rest api calls19:48
fungiit's a priority, but the gerrit performance problems have been taking more priority for now19:49
melwittthanks fungi19:51
*** hashar has quit IRC20:23
*** lpetrut has quit IRC20:29
*** fressi has quit IRC20:47
fungilooks like we've got a sizeable node request backlog20:48
fungione of the jobs for the openjdk 11 change took nearly an hour to get a node assignment in the gate, as the only change in that queue20:49
fungilooks like tripleo may be suffering some gate thrash20:49
clarkboh ya change near the top had a failure20:50
*** brinzhang_ has joined #opendev20:58
*** brinzhang0 has quit IRC21:01
openstackgerritMerged opendev/system-config master: Update gerrit docker image to java 11
fungiyay! deploy running now21:11
fungioh, and promote21:11
fungii wonder if those race one another21:11
fungiclarkb: ^ do you think?21:11
ianwi think they're all ordered?21:12
fungiin this particular case i guess it didn't21:12
fungiianw: well, we tag the image in dockerhub in promote and also pull in deploy21:12
fungiit's conceivable that we may pull before the image is tagged21:13
fungiahh, though we also run the promote job in the deploy pipeline21:13
fungiso maybe that's how we get around it21:13
ianwalthough some jobs in there have dependencies on their image promote, and others don't21:15
fungiso the image promotion in the promote pipeline isn't guaranteed to run before the deploy job pulls it, but the copy of the (presumably idempotent?) promote job which also runs in the deploy pipeline is strictly ordered before the deploy job which would pull it21:15
ianwlike all these have dependencies explicitly listed :
ianwbut not review
ianwi'm responsible for a bunch of them in there, i've just been copy-pasting21:18
fungicopy-pasting is the best kind of pasting21:19
mordredmmm pasta21:20
clarkbfor some reason I thought we didn't run them in the promote pipeline at all21:20
clarkbbut I also think that job is indempotent? it scans tags and does the right thing21:21
fungimaybe not, i saw it the change show up in the promote pipeline for a little while but didn't check what jobs were run for it21:21
ianwgerrit we have templates for
fungiit was done by the time i realized it was also already in deploy and started to question my assumptions21:22
ianwit might be nice to split all the services up into templates in separate files like that21:22
fungianyway, infra-prod-service-review is finally running in deploy for it, so i'll check the docker image ls shortly21:22
fungiand assuming that looks good we can start thinking about when will be a good time for yet still another service restart21:23
clarkbfungi: we should double check ianw's chagnes landed happily too21:23
fungioh, yup21:23
fungii also need to start cooking dinner soon21:24
fungibut i'm confident i can have this restarted on openjdk 11 before i head to bed21:24
ianwdo we know why infra-prod-manage-projects failed?21:25
clarkbI do not21:25
clarkbfwiw I did manually test project creation on review-test with jeepyb but maybe something isn't working around that?21:26
clarkbianw: do you have a log link?21:26
ianwi have to run out for about 30 mins now, but will then be around to monitor for explosions.  i guess worse case we roll back a tag21:26
clarkbI too am juggling some other stuff21:26
clarkbbut am somewhat around21:26
ianw2020-12-02 18:40:16,860: manage_projects - INFO - Writing cache file /opt/lib/jeepyb/project.cache21:27
ianw2020-12-02 18:40:16,911: manage_projects - ERROR - 1 problems has been caught during run:21:27
ianw ['Problems creating openstack/tripleo-ci, moving on.']21:27
*** iurygregory has joined #opendev21:28
ianwException: Gerrit error executing gerrit create-group --visible-to-all "tripleo-ci-core"21:28
fungithat's decidedly odd21:28
clarkbso I think maybe that file stopped getting bind mounted in maybe when we cleaned up the local replication21:29
fungioh! yes manage-projects may have wanted to init repos for the local replica?21:30
ianwi have to get kids to school, bib21:30
clarkbhrm we bind mount in that path in the manage-projects script21:30
fungiand this is about the create-group api call too21:31
clarkbwe only dropped the local replica not /opt/lib/jeepyb21:31
clarkbfungi: we should set the jeepyb setting to not do a local replica21:31
fungitripleo-ci-core also definitely already exists in the existing gerrit groups21:33
fungii too need to step away momentarily to start cooking dinner21:34
clarkbya I think the bit about the project.cache confused me I thought it was related to the error but the error is in creating that group for some reason21:34
*** sboyron has quit IRC21:43
clarkbthe tripleo-ci.config update landed recently. jeepyb sees that as an udpated config (properly so) so thats all good. I think the issue is in doing group listings to determine if we need to create the group or not21:56
clarkbwhen I tested I tested the new project creation and not udpate old project situation21:57
clarkbI think I figured it out. We use -q and that returns -q is not a valid option21:58
fungiooh, we used to rely on gerrit db checks for the group existing to avoid race conditions, maybe that's still in there?21:58
clarkbno I "fixed" that by querying the api21:59
fungiahh, that's far more straightforward21:59
clarkbbut the documented -q option doesn't seem to be working21:59
clarkb(nad i just checked current gerrit docs show -q as valid, maybe it is an order problem I'll check that next)21:59
clarkbnerp seems that -q is just not valid not to rtfs and see if it was renamed21:59
clarkbrtfs'ing seems to show only -v is a valid option. So I think we need to update gerritlib to do a full listing then filter from there22:01
clarkbwhich is less than ideal22:01
clarkbthough I'm still poking around in the source22:02
fungii'm poking chicken and mushrooms around in a skillet, sorry22:02
fungican help take a look after we eat though22:02
clarkbI don't think it is urgent22:04
clarkbthis seems to be failing in a safe way22:04
clarkbmore just trying to sort out what gerrit expects you to do now22:04
ianwclarkb: do you have a quick link to what you're looking at, for those of us who maybe get a little lost in these layers? :)22:05
clarkbianw: ya one sec22:08
clarkbianw: this condition is true beacuse the acl was updated ( change merged last week). We then call process_acls which calls create_groups_file which calls get_group_uuid which calls _get_group_uuid.22:10
clarkbwe fail to get the group uuid because the -q used in is no longer a valid argument to that command in gerrit22:11
clarkbwe then bubble back up to _get_group_uuid which tries to create teh group when the group does not exist22:11
clarkbI think new project creations may also not work right because we won't get their group uuids properly22:11
clarkbthe flag is now -g22:13
clarkbI don't know when that changed :/22:13
clarkbbut I guess I'll try to figure that out next so we can check the gerrit version and then switch on that?22:14
ianw does show -q right?22:18
clarkbyup the docs are wrong. They depreacted -q in 2.14 and 2.14 supported -q and -g22:20
clarkbI think the docs just never updated22:20
clarkbI can make a docs fix for gerrit 3.3 in a bit too22:20
clarkbya confirmed not depreactedin 2.1322:21
clarkbis depreacted in 2.14 and the docs are just wrong22:21
ianwyay!  already got a -3 on and it's not even 9:30am! :)22:22
clarkbjust working on improving our testing around this then I'll push22:27
clarkbas a side note we appaer to still be managing projects on review-dev despite it being in emergency.yaml22:32
clarkbI don't think we want that behavior because one reason we put things in emergency.yaml is so that eg manage projects doesn't run? maybe someone else can look at that22:32
ianwthe match looks right ... - hosts: "review:review-dev:!!disabled"22:34 is being managed, emergency has openstack.org22:36
ianw- # mordred 2019/09/19 working on dockerification22:38
ianwi have the file open, so if you want me to update i can22:38
openstackgerritClark Boylan proposed opendev/gerritlib master: Handle newer Gerrit ls-groups command syntax
clarkbianw: no we want it disabled22:39
clarkbbeacuse its on the odl code and is effectively a dead server22:39
clarkbI didn't expect that manage projects would run on it22:40
clarkbis review-dev group not matching ?22:40
clarkbianw: fungi: I believe should hopefully fix this problem and I added a bit more robust testing to the tests we run there22:40
clarkbBut I still think we should only land that once we're happy there aren't other gremlins in manage-projects and we should avoid adding more projects or acl updates until we land this fix22:41
ianwclarkb: i mean in emergency it's listed as; so that's why isn't being skipped22:41
clarkbianw: yes please fix it :)22:42
ianwok, done, noted that it was the old server in the comment22:43
fungiokay, dinner is done, i'm catching up22:45
clarkbI'm going to work on a upstream docs fix now then I promised I would help with some foundation stuff then I need to help kids with lights on a tree22:45
ianwclarkb: do you think it's worth a manual run with that fix to confirm?22:46
clarkbianw: ya that would probably be a good idea, we can limit it to just that project too22:47
clarkbianw: I think to do that we need to put review in emergency, then manually pull the new image once it lands, then run manage-projects?22:47
clarkbotherwise our daily run will hit it ? (I think we have ad aily run)22:47
fungiand now i've discovered rusty's fir the first time ever, my day is complete22:47
corvusfungi: i need way more context for that last22:49
corvusmy mental pattern matching is coming up with lots of possibilities but most are very weird with very little discrimination between the levels of weirdness22:50
ianwclarkb: ok, i can probably figure out babysitting a manual run if we're happy with the change.  it lgtm, though i claim no expertise22:51
clarkb is the upstream fix22:51
clarkbianw: ya I dunno. manage-projects always scares me. That makes me both want to be around and not be around when we run things :)22:52
clarkbianw: the testing is getting better though so maybe we want to focus on that a bit more if we think we're missing any other cases we shoudl check22:53
clarkbI think the reason that testing didn't have problems with this before is we only used Registered Users in the test which is a built in group and handled differently iirc22:55
clarkbthat is why I added the two new groups22:55
clarkbthose should get created "normally" and be more representative22:55
fungicorvus: sorry, s/fir/for/ and responding to ianw's link to
fungiit's apparently like a bingo card for api design22:57
ianwoh, i think in looking, i've answered my question on creating the initial admin gerrit22:59
ianwyou just do a GET on http://localhost:8080/a/accounts/admin/sshkeys and that creates the initial user?22:59
ianwthat is way easier than i thought23:00
*** tkajinam has joined #opendev23:00
corvusianw: we do more than that in the zuul quickstart23:00
fungijeepyb fix lgtm, approved23:00
clarkbfungi: should we put review in emergency so that we can manually run it before the daily run?23:01
clarkbor are we happy with the testing and goign to go for it?23:01
* clarkb is running out of day so will defer to others on that23:01
fungithe test adjustments you added satisfied me23:01
*** slaweq has quit IRC23:02
ianwclarkb: yeah, i feel like we assume the best and monitor it23:04
clarkbthough suddenly I wonder if we need a new gerritlib release too?23:04
clarkband/or need to update jeepyb or gerrit to get the image updated23:04
clarkbits possible just landing that chagne won't get the code where we need it23:05
ianwhrm, i made gerritbot use gerritlib master, but not sure about other paths23:07
fungialso i +2'd a change toady which would make gerritlib master uninstallable on ubuntu xenial, just a heads up that if we want to continuously deploy gerritlib we probably should stick to doing it in containers (so for jeepyb with our manage-projects run it's probably fine)23:10
clarkbfungi: whcih change is that?23:10
clarkbif it is the python3.6 chagne i meant to -2 it23:11
clarkbI don't recall if I did or not23:11
openstackgerritMerged opendev/gerritlib master: Handle newer Gerrit ls-groups command syntax
fungiplease -2 if you have concerns23:11
fungii'm trying not to stand in the way of progress like arthur dent and the bulldozer23:12
clarkbin this particular care I think the order of operations should be "update all the things running gerritlib on xenial" first then23:13
* fungi checks his bathrobe pocket for an anaklgesic23:13
clarkbfrom where I'm sitting we don't gain much in dropping python3.5 support on that library23:14
clarkbwe can add new python support easily enough23:14
fungiseems like the oprimary concern raised by that change is that we weren't testing it on 3.523:14
clarkbwe are testing on 3523:15
clarkbor we were when that was first pushed up23:15
clarkbmaybe the bug is taht it got removed elsewhere and side effected gerritlib23:15
ianwcorvus: thanks for pointing that out.  i'm not sure we need to deploy for the gate testing scenario but the bones there are very helpful23:17
clarkblooking at jeepyb I think we do need a release of gerritlib and then a jeepyb update (or gerrit update) to update the code in thei mage23:18
corvusianw: yeah i would expect this is what we'd need to do:
clarkbI think there is a small bug in my testing for gerritlib too shows that test-repo1 isn't getting updated acls. but if we look below in the log testrepo2 is updated. Does lineinfile only modify the last occurence? I expected it to do all of them23:25
clarkbthe testing seems to show things are still working as expected, but we're not getting the behavior out of the test itself that I expected23:25
clarkbyup ansible docs confirm it "only the last line found will be replaced23:26
fungithat's... unfortunate23:27
clarkbI'm getting a brute force hack to amke it work better :)23:28
openstackgerritClark Boylan proposed opendev/gerritlib master: Test manage-projects repos harder
clarkbfungi: ianw corvus ^ that should replace both lines and then do the more aggressive testing that I expected23:29
clarkbhowever, I think the checks against just test-repo-2 show that the change should work as is23:30

Generated by 2.17.2 by Marius Gedminas - find it at!