Wednesday, 2020-10-21

fungi#status ok The Gerrit service at is back up and running; for outage details see analysis here:
openstackstatusfungi: sending ok00:34
*** ChanServ changes topic to "OpenDev is a space for collaborative Open Source software development | | channel logs"00:35
-openstackstatus- NOTICE: The Gerrit service at is back up and running; for outage details see analysis here:
openstackstatusfungi: finished sending ok00:40
*** whoami-rajat__ has joined #opendev00:44
*** chandankumar is now known as raukadah00:47
SotKbig thanks to you all for resolving what sounds like a very painful day, your work is very appreciated!00:48
fungiSotK: that day was a week long :/00:50
SotKeven worse :(00:50
portdirectfungi: thanks so much00:50
portdirectpeople like you, and all the infra team, are the backbone of openstack00:51
fungibut you're welcome! i'm just glad to have the opportunity to collaborate with all of you on these systems00:51
*** owalsh has quit IRC01:08
*** owalsh has joined #opendev01:09
ianwkevinz: looks like some issues with the linaro API HTTPSConn01:54
ianwectionPool(host='', port=5000)01:54
jbryceportdirect: totally agree. the level of effort has been really impressive02:15
*** DSpider has quit IRC02:34
melwitt+1, thank you for all the work you do. we <3 infra03:23
*** hamalq has quit IRC03:46
*** lajoskatona has joined #opendev04:02
*** lajoskatona has left #opendev04:02
*** fressi has joined #opendev04:17
*** fressi has quit IRC04:41
*** marios has joined #opendev05:04
*** pto has joined #opendev05:28
*** sboyron has joined #opendev06:36
*** rpittau|afk is now known as rpittau06:42
*** ralonsoh has joined #opendev06:45
*** eolivare has joined #opendev06:49
*** slaweq has joined #opendev06:53
*** andrewbonney has joined #opendev07:06
noonedeadpunkseems gerritbot has been affected and down at the moment07:32
noonedeadpunk(probably token needs to be updated?)07:32
ianwahh, probably07:33
*** slaweq has quit IRC07:34
*** slaweq has joined #opendev07:35
*** openstackgerrit has quit IRC07:38
ianwok, should be back07:39
*** slaweq has quit IRC07:40
*** tosky has joined #opendev07:42
*** sshnaidm is now known as sshnaidm|afk07:43
*** slaweq has joined #opendev07:44
*** ttx has quit IRC07:48
noonedeadpunkyep, it is, thanks!07:48
*** ttx has joined #opendev07:51
*** priteau has joined #opendev07:54
*** pto has quit IRC07:55
slaweqianw: hi07:59
slaweqianw: since today my ssh key added in gerrit don't work anymore, is that due to this yesterday outage and should I create new key in gerrit?07:59
slaweqI read Your email but I understood from it that ssh keys added before 1.10 should be ok07:59
ianwslaweq: that's right, you didn't update it since then?08:00
ianware you sure you're not using HTTP?  all HTTP API keys have been cleared08:00
*** nautik has joined #opendev08:05
slaweqianw: yes, here is how I have it configured
*** slaweq has quit IRC08:08
*** roman_g has joined #opendev08:10
*** slaweq has joined #opendev08:11
slaweqianw: and I didn't change it since looong time for sure :)08:12
*** gnuoy has quit IRC08:19
*** gnuoy has joined #opendev08:20
*** slaweq has quit IRC08:35
*** slaweq has joined #opendev08:36
*** slaweq has quit IRC08:37
*** slaweq has joined #opendev08:37
*** sshnaidm|afk is now known as sshnaidm08:39
*** fressi has joined #opendev08:43
fricklerthat was the known F33 ssh config issue, not related to our incident08:43
slaweqfrickler: ianw: thank You a lot for Your help, it was issue with Fedora 33 ssh config as frickler pointed me. It's now fixed08:43
*** priteau has quit IRC10:39
*** DSpider has joined #opendev11:00
cgoncalveskevinz, hey! is Linaro cloud still under maintenance? Zuul reported NODE_FAILURE twice in
*** priteau has joined #opendev11:15
*** priteau has quit IRC11:26
*** mkalcok has quit IRC11:38
*** mkalcok has joined #opendev11:39
*** priteau has joined #opendev12:00
*** mattd01 has joined #opendev12:04
fricklerinfra-root: sean-k-mooney noticed a problem with a job on limestone, because the node had no IPv4 address at all and thus devstack failed to find the interface with the default route
frickleris that intended (no v4 at all)? guess we would have to tune devstack for that scenario12:50
fungifrickler: no, the expectation is that it has some rfc 1918 address which then uses pat/overflow nat to reach v4 addresses on the internet12:50
fungiif it didn't get an ipv4 address, maybe there was a problem with dhcpd12:51
fricklerfungi: o.k., let's wait and see whether that was a one-off or we see this more often12:53
sean-k-mooneyill let ye know if i see this again12:53
sean-k-mooneyfor now im goign to just fix the ordering in the world dump script12:54
*** slaweq has quit IRC13:08
*** slaweq has joined #opendev13:12
*** openstackgerrit has joined #opendev13:12
openstackgerritzbr proposed zuul/zuul-jobs master: WIP: Add ensure-vagrant role
openstackgerritzbr proposed zuul/zuul-jobs master: Improve errors from updat-test-platforms
*** ysandeep is now known as ysandeep|ruck13:18
*** slittle1 has joined #opendev13:46
corvusi'm replying to mnaser's email14:03
*** ysandeep|ruck is now known as ysandeep|ruck|af14:04
fungithanks, i did as well earlier14:16
fungii'm trying to reply on the ironic and tripleo threads on openstack-discuss currently14:16
fungias well as questions which came up in the cinder weekly meeting underway just now14:16
corvusoh, i'll check my reply against yours14:17
fungii can see this is going to be most of my day (probably most of my month, and maybe the rest of my year)14:17
corvuswelp i didn't say anything that fungi didn't say, so i will discard my reply :)14:19
fungisorry about that!14:21
corvusnp i was only about half done :)14:21
*** ysandeep|ruck|af is now known as ysandeep|ruck14:21
*** sshnaidm is now known as sshnaidm|rover14:22
*** leuben has joined #opendev14:47
leubenHello I have intermittent network issues with FIP access using neutron (DVR). Can someone guide me to the proper channel to discuss it further ?14:50
AJaegerleuben: #openstack in general or #openstack-neutron.14:52
*** ysandeep|ruck is now known as ysandeep|away14:57
*** mkalcok has quit IRC14:59
mnaserand i was waiting all morning for corvus reply :)15:10
*** mlavalle has joined #opendev15:11
*** raukadah is now known as chandankumar15:12
corvusmnaser: thanks for being 'that guy'.  i'm glad you care.  :)15:19
mnasercorvus: thanks for you and fungi for hearing me out that this honestly comes from a place of care, and not frustration, cause i know what being on the other end of this this like, it's not fun15:20
fungimnaser: you're not "that guy" at all, you're basically echoing what most of us are thinking, we just need some time to recover before we get into deep debate on our options i think15:21
mnaserfungi: agreed, it's not like monday was any easier, so hopefully things do ease up in the next few days -- and the ptg might be in a very good time15:22
fungiall the points you raise are great15:22
*** dmsimard1 has joined #opendev15:30
*** dmsimard has quit IRC15:32
*** dmsimard1 is now known as dmsimard15:32
*** noonedeadpunk has quit IRC15:51
*** marios has quit IRC15:56
*** lyarwood has quit IRC16:01
*** fressi has quit IRC16:03
*** rpittau is now known as rpittau|afk16:04
*** noonedeadpunk_ has joined #opendev16:13
*** tosky has quit IRC16:16
*** portdirect has quit IRC16:19
*** portdirect has joined #opendev16:20
*** hashar has joined #opendev16:20
*** portdirect has quit IRC16:21
*** portdirect has joined #opendev16:21
*** hamalq has joined #opendev16:27
*** fressi has joined #opendev16:28
*** hamalq has quit IRC16:29
*** hamalq has joined #opendev16:30
*** eolivare has quit IRC16:30
*** noonedeadpunk_ has quit IRC16:33
*** noonedeadpunk has joined #opendev16:39
roman_gOpenDev Infra team, you rock! Did a great job on resolving Gerrit/security issue. Thank you!16:40
fungiroman_g: it's arguably the most important service we run, so we took it extremely seriously16:45
fungieven though there's been no sign of successful tampering, we're trying to be as cautious and transparent as possible16:45
roman_gThank you for that.16:46
fungihaving the openid of one of our administrative users come under the control of an untrusted party is essentially one of our worst nightmares16:46
fungiso we're also busy planning for how to make that less risky going forward16:47
roman_gHas the case been reported to the police or such?16:47
fungii expect it to be heavily discussed during our ptg sessions next week16:47
fungiroman_g: well, for starters, jurisdiction is hard to establish in cases where this occurs across oceans, but also we can't be certain that the identity we've found isn't just a smokescreen or false flag trying to smear someone else and implicate them16:48
fungiultimately the unauthorized access occurred in the openid provider, which we also don't control, so any legal action may need to be initiated on their behalf or on behalf of the user of that platform16:49
roman_gAnd that's second question. If you have cooperated with provider and collected evidence.16:50
fungiyep, we've been in touch with the launchpad admins since the first few hours into the incident, and they were instrumental in helping us identify other avenues of intrusion16:51
roman_gGerrit version which is run by you supports GPG signing, I would suggest to make code sign a requirement for infra repositories. This is easy to implement and would add additional security layer.16:52
roman_gOther than that I have nothing to add.16:52
*** fressi has quit IRC16:52
clarkbya were were talking about that yesterday. I don't think you can enforce such a requirement on a per repo basis (I think newer gerrit allows setting it for all repos?)16:52
roman_gYou did a great job and deserve at least congrats and warm hugs.16:52
fungiyep, we've started discussing that as well. thankfully it's also configurable on a per-project basis16:52
clarkbits definitely something to look at16:52
clarkbfungi: oh cool I didn't realize that16:52
fungiclarkb: at least last i looked it was a setting similar to cla or cco enforcement16:53
fungibut we do need to double-check that16:53
roman_gEven if it's not possible to enforce per-repo via API/GUI, either Zuul merger and/or Gerrit Prolog rules could allow to make it enforced per-repo.16:55
fungiyep, absolutely worth investigating. also if it's not yet enforceable per-repo, it might be a feature we can work with the gerrit upstream maintainers to add16:58
openstackgerritwes hayutin proposed zuul/zuul-jobs master: wip, allow rdo repos to be turned off for openvswitch install
*** andrewbonney has quit IRC17:10
fungitristanC: did we miss adding someone from sf to or were we waiting to find out who to add?17:11
fungiwith our upcoming upgrade to newer gerrit, we're eager to try out your in progress lib for that17:11
fungii got a taste yesterday of seeing how painful it was to find and roll back changes to authorized_keys in the user refs in All-Users.git17:12
fungiimagining all of gerrit's db content being similarly harder to query/update17:13
sshnaidm|roverI see many retry_limits in jobs, is something going on? For example:,217:20
*** sshnaidm|rover is now known as sshnaidm|afk17:24
fungithere were some reports earlier of limestone nodes with no ipv4 addresses at all, looking to see if this is more of that, maybe we need to turn that provider down for a bit17:28
*** ralonsoh has quit IRC17:28
clarkb that job failed to ssh near th eend of the job17:30
clarkbanother one that didn't log logfiles shows a similar failure to ssh17:30
clarkb(but it only reports unreachable via ansible exit code 4 I think)17:30
clarkbusually this is a case of arp conflicts?17:30
fungior network connectivity problems to some provider, or to rackspace's dfw region where the executors are hosted17:31
fungithe more characteristic arp overwrite failures manifest as host key mismatches17:32
clarkbthe one in the link shows permission denied so maybe in that case the job changed things under zuul17:33
*** priteau has quit IRC18:01
fungiclarkb: this was the option i was thinking of, so not enforcing validation of signed commits (though gerrit allows users to push them), but rather validating signed pushes on a per-project basis...
*** mattd01 has quit IRC18:07
fungicoupled with keys stored in user preferences18:08
fungiagain, a possibly useful feature, though not one which would have helped with this case18:08
fungisince the attacker could also have simply added their key and signed the push with that18:09
*** fressi has joined #opendev18:17
*** iurygregory has quit IRC18:18
corvusfungi: right, but that's more easily auditable... essentially, the audit suspected diffs process becomes verifying signatures are valid and the keys belong to their authors.  point well taken that you wouldn't be able to trust gerrit for that last part, but asking ppl on a ml to verify their fingerprints in a keyring is probably not too hard.18:19
fungii also agree that cryptographically signed commits could help in an audit. if a commit in question has a signature which verifies against the committer's key and you have some confirmation that key was not altered, then you can (mostly, modulo attacks on sha-1) rule out tampering18:19
*** iurygregory has joined #opendev18:20
fungicorvus: well, don't confuse signed pushes with signed commits. they're different things. one signs the action, the other the material18:20
corvusoh i am doing that18:20
corvusare we not talking about signed commits?18:20
fungiwe started talking about signed commits. then i realized the enforcement configuration i was remembering in gerrit is for signed pushes18:21
corvusoh.  i see the value for signed commits; i'm unaware of how signed pushes would help.  that's new to me.18:21
fungistill, even voluntarily signing commits and keeping the relevant keys somewhere (like in the user preferences gerrit happens to use for confirming signed pushes) would give us the ability to narrow the scope of possible tampering18:22
corvusthe effort for verification of signed commits is also on the order of the number of authors, whereas inspecting content is on the order of number of commits.  so as your exposure period increases, signed commits become increasingly advantageous over the status quo.18:23
fungicorvus: some background:
*** iurygregory has quit IRC18:24
*** iurygregory has joined #opendev18:25
corvusfungi: i wonder to what extent a signed push can imply validation of commit contents.  ie, if we had signed pushes, since those push certs all reference git shas, can we then infer that if all the git shas which appeared in the repo have valid corresponding push certs indicating they were pushed to gerrit, that all the contents are unmodified since the push and truly originated with the push signer?18:27
fungipossibly. i too need to learn more about push certificates, whether they are retained, and how they could be accessed later in an audit18:28
fungior whether they are discarded after validating the push action18:28
*** lyarwood has joined #opendev18:29
fungiat least with commit signatures, those are served as part of the commits themselves18:29
corvusat any rate, i suspect that the best case scenario is that if they can, then still only after a layer of indirection, so it's more difficult to verify.  seems like maybe a good enhancement after signed commits are the norm.  :)18:29
fungiyes, at a minimum they look like complimentary features18:29
fungiespecially since they'll probably usually (in the case of our projects anyway) leverage the same keys for both constructs18:30
*** mattd01 has joined #opendev18:30
*** fressi has quit IRC18:30
fungifor workflows where pushing commits from other contributors is the norm, the keys are likely to differ18:30
corvusgood point18:31
corvusfungi: thanks for making me aware of it so i don't confuse them as we talk about what options we might pursue; i suspect we may need to watch out for that and inform/remind people that both things exist.18:31
fungithe reminder was partly to myself, because i had clearly misremembered gerrit having a configuration option for requiring commit signatures, when in actuality it's push certificates it can enforce18:33
*** fressi has joined #opendev18:37
*** fressi has quit IRC18:56
*** tosky has joined #opendev19:12
*** leuben has quit IRC19:39
*** priteau has joined #opendev20:02
*** whoami-rajat__ has quit IRC20:06
*** mattd01 has left #opendev20:20
*** hashar has quit IRC20:55
*** sboyron has quit IRC20:55
*** slaweq has quit IRC21:24
tristanCfungi: could you please add (for pynotedb)21:47
fungitristanC: yep, gimme a sec21:48
fungitristanC: i have invited that user to become an owner21:51
fungionce you accept, feel free to remove openstackci21:51
tristanCfungi: alright, thanks a lot21:51
fungi#status log handed off ownership of to softwarefactory account21:52
openstackstatusfungi: finished logging21:52
tristanCfungi: and well, the existing implementation is working well for us, we use it to automate gerrit upgrade to v3.x, you can find function to list user references and access their account config, for example:
fungitristanC: awesome, we're going to need something like that following our coming upgrade. glad to hear you have working code already!21:54
*** roman_g has quit IRC22:00
tristanCfungi: upload-pypi job ran successfully, thanks again!22:04
*** tosky has quit IRC22:10
*** tosky has joined #opendev22:11
fungitristanC: awesome, we also got the ownership change notifications, so all yours now22:14
*** qchris has quit IRC22:42
*** tosky has quit IRC22:51
*** qchris has joined #opendev22:54
*** hamalq has quit IRC22:57
*** mlavalle has quit IRC22:58
*** hamalq has joined #opendev23:01

Generated by 2.17.2 by Marius Gedminas - find it at!