Wednesday, 2020-10-21

*** gyee has quit IRC00:11
*** gyee has joined #openstack-infra00:25
*** ChanServ changes topic to "Discussion of OpenStack's Developer tooling and CI jobs | Discussion around infrastructure services now in #opendev"00:37
-openstackstatus- NOTICE: The Gerrit service at review.opendev.org is back up and running; for outage details see analysis here: http://lists.opendev.org/pipermail/service-announce/2020-October/000011.html00:37
*** Goneri has quit IRC00:46
*** chandankumar is now known as raukadah00:47
*** sreejithp has joined #openstack-infra00:56
*** sreejithp has quit IRC01:03
*** owalsh has quit IRC01:08
*** owalsh has joined #openstack-infra01:09
*** dhill has quit IRC01:11
*** ramishra has quit IRC01:11
*** gyee has quit IRC01:12
*** sreejithp has joined #openstack-infra01:20
*** sreejithp has quit IRC01:28
*** jamesmcarthur has joined #openstack-infra01:31
*** jamesmcarthur has quit IRC01:48
*** jamesmcarthur has joined #openstack-infra01:48
*** jamesmcarthur has quit IRC01:50
*** jamesmcarthur has joined #openstack-infra01:50
*** jamesmcarthur has quit IRC01:53
*** jamesmcarthur has joined #openstack-infra01:56
*** jamesmcarthur has quit IRC02:02
*** jamesmcarthur has joined #openstack-infra02:04
*** piotrowskim has quit IRC02:12
*** stevebaker has quit IRC02:15
*** jamesmcarthur has quit IRC02:31
*** jamesmcarthur has joined #openstack-infra02:32
*** jamesmcarthur has quit IRC02:36
*** sai438 has joined #openstack-infra03:14
*** jamesmcarthur has joined #openstack-infra03:16
*** pmannidi has quit IRC03:18
*** jamesmcarthur has quit IRC03:25
*** pmannidi has joined #openstack-infra03:26
*** sai438 has quit IRC03:29
*** psachin has joined #openstack-infra03:30
*** psachin has quit IRC03:30
*** psachin has joined #openstack-infra03:31
tkajinamhi. will we recover gerrit bot as gerrit has been recovered ?03:42
*** rfolco has joined #openstack-infra03:43
*** hamalq has quit IRC03:46
*** rfolco has quit IRC03:47
*** sai438 has joined #openstack-infra03:49
*** pmannidi has quit IRC03:52
*** pmannidi has joined #openstack-infra03:53
*** pmannidi_ has joined #openstack-infra03:55
*** sai438 has quit IRC03:57
*** pmannidi has quit IRC03:58
*** jamesmcarthur has joined #openstack-infra03:59
*** lajoskatona has joined #openstack-infra04:02
*** lajoskatona has left #openstack-infra04:02
*** jamesmcarthur has quit IRC04:04
*** evrardjp has quit IRC04:33
*** evrardjp has joined #openstack-infra04:33
*** matt_kosut has joined #openstack-infra04:53
*** soniya29|ruck is now known as soniya2904:58
*** zxiiro has quit IRC04:59
*** jamesmcarthur has joined #openstack-infra05:05
*** jamesmcarthur has quit IRC05:10
*** ldenny has quit IRC05:21
*** sreejithp has joined #openstack-infra05:24
*** bdodd_ has joined #openstack-infra05:28
*** sreejithp has quit IRC05:28
*** bdodd has quit IRC05:29
*** ldenny has joined #openstack-infra05:36
*** jamesmcarthur has joined #openstack-infra05:41
*** jamesmcarthur has quit IRC05:46
*** jamesmcarthur has joined #openstack-infra05:56
*** jamesmcarthur has quit IRC05:57
*** jamesmcarthur has joined #openstack-infra05:57
*** jamesmcarthur has quit IRC06:02
*** jamesmcarthur has joined #openstack-infra06:34
*** dciabrin has joined #openstack-infra06:35
*** sboyron has joined #openstack-infra06:36
*** rpittau|afk is now known as rpittau06:42
*** jcapitao has joined #openstack-infra06:43
*** jtomasek has joined #openstack-infra06:43
*** ralonsoh has joined #openstack-infra06:45
*** vishalmanchanda has joined #openstack-infra06:47
*** dklyle has quit IRC06:47
*** eolivare has joined #openstack-infra06:49
*** jamesmcarthur has quit IRC06:52
*** slaweq has joined #openstack-infra06:54
*** andrewbonney has joined #openstack-infra07:06
*** sreejithp has joined #openstack-infra07:25
*** sreejithp has quit IRC07:31
*** hberaud has quit IRC07:34
*** slaweq has quit IRC07:34
*** slaweq has joined #openstack-infra07:35
*** hberaud has joined #openstack-infra07:36
*** openstackgerrit has quit IRC07:38
*** rcernin has quit IRC07:39
*** slaweq has quit IRC07:40
*** tosky has joined #openstack-infra07:42
*** sshnaidm is now known as sshnaidm|afk07:43
*** slaweq has joined #openstack-infra07:45
*** jtomasek has quit IRC07:47
*** ttx has quit IRC07:48
*** ttx has joined #openstack-infra07:51
*** priteau has joined #openstack-infra07:54
*** jpena|off is now known as jpena07:56
*** pmannidi_ has quit IRC08:02
*** lucasagomes has joined #openstack-infra08:07
*** slaweq has quit IRC08:08
*** rcernin has joined #openstack-infra08:08
*** dtantsur|afk is now known as dtantsur08:10
*** slaweq has joined #openstack-infra08:11
*** rcernin has quit IRC08:14
*** gnuoy has quit IRC08:19
*** pmannidi has joined #openstack-infra08:19
*** gnuoy has joined #openstack-infra08:20
*** rcernin has joined #openstack-infra08:28
*** nightmare_unreal has joined #openstack-infra08:34
*** derekh has joined #openstack-infra08:34
*** slaweq has quit IRC08:35
*** slaweq has joined #openstack-infra08:37
*** slaweq has joined #openstack-infra08:37
*** sshnaidm|afk is now known as sshnaidm08:39
*** jamesmcarthur has joined #openstack-infra08:50
*** ociuhandu has joined #openstack-infra08:51
*** rcernin has quit IRC08:56
*** jamesmcarthur has quit IRC08:59
*** matbu has quit IRC09:18
*** gfidente has joined #openstack-infra09:19
*** matbu has joined #openstack-infra09:25
*** sreejithp has joined #openstack-infra09:27
*** pmannidi has quit IRC09:30
*** pmannidi has joined #openstack-infra09:30
*** sreejithp has quit IRC09:32
*** pmannidi has quit IRC09:35
tkajinamis there any good way to download gerrit-diffs as an archive ?09:37
tkajinamwe have bunch of puppet-foo projects so it would be helpful if I can download all files for puppet-* projects so that I can check them in my local machine09:38
*** pmannidi has joined #openstack-infra09:39
*** lmiccini has quit IRC09:46
*** pmannidi has quit IRC09:47
*** pmannidi has joined #openstack-infra09:47
*** lmiccini has joined #openstack-infra09:51
*** sai438 has joined #openstack-infra09:51
fricklertkajinam: wget has some recursive download option, not sure if can also filter the way you need. we could likely also create a tarball for the whole tree, I can look into that in a bit09:54
*** pmannidi has quit IRC09:55
*** pmannidi has joined #openstack-infra09:59
*** sai438 has quit IRC10:02
*** sai438 has joined #openstack-infra10:02
*** ramishra has joined #openstack-infra10:03
*** ramishra has quit IRC10:04
*** ramishra has joined #openstack-infra10:04
*** pmannidi has quit IRC10:05
*** pmannidi_ has joined #openstack-infra10:06
*** sai438 has quit IRC10:07
tkajinamhmm. we need to somehow filter the uri otherwise it searches all files :-P10:07
*** bradm has quit IRC10:23
*** ldenny has quit IRC10:24
*** ldenny has joined #openstack-infra10:24
*** jcapitao is now known as jcapitao_lunch10:24
*** aluria has quit IRC10:28
*** aluria has joined #openstack-infra10:30
*** aluria has quit IRC10:38
*** priteau has quit IRC10:39
*** aluria has joined #openstack-infra10:39
*** dhill has joined #openstack-infra11:02
*** jtomasek has joined #openstack-infra11:09
*** priteau has joined #openstack-infra11:15
*** priteau has quit IRC11:26
*** jpena is now known as jpena|lunch11:32
*** jcapitao_lunch is now known as jcapitao11:33
*** jamesmcarthur has joined #openstack-infra11:36
*** jamesmcarthur has quit IRC11:41
*** ramishra has quit IRC11:44
*** rlandy has joined #openstack-infra11:53
*** rlandy is now known as rlandy|rover11:53
*** jtomasek has quit IRC12:00
*** priteau has joined #openstack-infra12:01
*** rfolco has joined #openstack-infra12:04
*** jamesmcarthur has joined #openstack-infra12:07
*** dchen has quit IRC12:14
*** jpena|lunch is now known as jpena12:30
*** jamesmcarthur has quit IRC12:33
*** Goneri has joined #openstack-infra12:46
*** zxiiro has joined #openstack-infra12:50
*** ramishra has joined #openstack-infra13:00
*** ramishra has quit IRC13:07
*** slaweq has quit IRC13:08
*** jamesmcarthur has joined #openstack-infra13:09
*** slaweq has joined #openstack-infra13:12
*** tdasilva has quit IRC13:15
*** tdasilva has joined #openstack-infra13:15
*** rlandy|rover is now known as rlandy13:17
*** ysandeep is now known as ysandeep|ruck13:18
*** psachin has quit IRC13:19
*** nhicher has joined #openstack-infra13:21
*** nhicher has quit IRC13:22
*** nhicher has joined #openstack-infra13:24
*** ramishra has joined #openstack-infra13:29
*** jamesmcarthur has quit IRC13:29
*** pmannidi has joined #openstack-infra13:47
*** pmannidi_ has quit IRC13:49
*** jamesmcarthur has joined #openstack-infra13:50
*** ysandeep|ruck is now known as ysandeep|ruck|af14:04
*** jamesmcarthur has quit IRC14:12
*** ysandeep|ruck|af is now known as ysandeep|ruck14:21
*** sshnaidm is now known as sshnaidm|rover14:22
*** dklyle has joined #openstack-infra14:39
*** jtomasek has joined #openstack-infra14:41
*** jamesmcarthur has joined #openstack-infra14:54
*** ysandeep|ruck is now known as ysandeep|away14:57
*** raukadah is now known as chandankumar15:12
*** tdasilva_ has joined #openstack-infra15:27
*** tdasilva has quit IRC15:30
*** dmsimard1 has joined #openstack-infra15:30
*** dmsimard has quit IRC15:32
*** dmsimard1 is now known as dmsimard15:32
*** thedaveking has quit IRC15:42
*** ramishra has quit IRC15:47
smcginnistkajinam: Could clone locally and grep there.15:50
smcginnisfor repo in $(curl -s https://review.opendev.org/projects/ | grep "openstack.puppet-.*" | cut -d '"' -f 2); do    git clone "https://opendev.org/$repo"; done15:50
*** thedaveking has joined #openstack-infra15:51
*** noonedeadpunk has quit IRC15:51
*** dtantsur is now known as dtantsur|afk15:55
fungiyeah, the main reason we generated a specific set of refs is that we can't necessarily trust the commit times in git repositories to indicate whether a change truly originated at the times claimed, since anyone can backdate a commit15:56
fungiso instead we took two copies of all the repositories, one from a snapshot prior to the compromise and the other current, and identified every commit which was in the current set but not in the snapshot15:57
fungithat way we avoided overlooking any such "backdated" commits15:57
*** kaisers2 has quit IRC15:58
*** mihalis68_ has quit IRC15:58
*** pmannidi has quit IRC15:58
*** pmannidi has joined #openstack-infra16:01
*** lyarwood has quit IRC16:01
auristordoes openstack install the gerrit reviewnotes plugin?  If so, I think you can trust the Submitted-at timestamp16:03
fungiauristor: yes, folks if they configure fetching notes can check that metadata16:04
fungi(i rely on it extensively)16:04
*** rpittau is now known as rpittau|afk16:04
fungihowever, in this case, the attacker had write access to the database (via gerrit's gsql "feature") and those timestamps are writeable fields, so...16:05
fungii wouldn't entirely trust the submitted-at timestamps in this particular case16:05
auristorin that case you can't trust anything.16:06
fungiright. we basically diffed things against a known good snapshot from prior to the compromise16:06
fungiidentified every new commit which was not in the snapshot, rolled back every ssh key which had been added since the snapshot, blew away all api keys in the db, and so on16:07
auristorgot it.16:09
fungialso diffed the account_external_ids tables between current and the snapshot to check for extra openid urls which may have been added to accounts, or any which might have been surreptitiously replaced16:10
fungibasically anything we could think of which could provide a trampoline for the attacker to regain access to existing accounts16:10
*** Topner has joined #openstack-infra16:11
*** noonedeadpunk_ has joined #openstack-infra16:13
*** tosky has quit IRC16:16
auristorI might have restored the known good snapshot and then resubmitted the other changes to the restored gerrit for review.16:16
fungithe main challenges there is that we've got published release artifacts resulting from those changes, so would also have needed to revoke them all and make all new releases of many projects (hundreds of openstack deliverables were released in that three-week period)16:18
fungiso a post-hoc commit audit was ultimately seen as less work. most of the teams already seem to have knocked theirs out since the announcement hours ago16:19
*** portdirect has quit IRC16:19
*** portdirect has joined #openstack-infra16:20
*** hashar has joined #openstack-infra16:20
*** portdirect has quit IRC16:21
*** portdirect has joined #openstack-infra16:21
*** jcapitao has quit IRC16:22
*** hamalq has joined #openstack-infra16:27
*** hamalq has quit IRC16:29
*** hamalq has joined #openstack-infra16:30
*** eolivare has quit IRC16:30
*** noonedeadpunk_ has quit IRC16:33
*** lucasagomes has quit IRC16:34
*** openstackgerrit has joined #openstack-infra16:34
openstackgerritMoisés Guimarães proposed openstack/pbr master: Adding pre-commit  https://review.opendev.org/74216016:34
*** dklyle has quit IRC16:39
*** Topner has quit IRC16:39
*** dklyle has joined #openstack-infra16:39
*** ociuhandu_ has joined #openstack-infra16:39
*** noonedeadpunk has joined #openstack-infra16:39
*** jtomasek has quit IRC16:42
*** ociuhandu has quit IRC16:42
*** ociuhandu_ has quit IRC16:44
openstackgerritMoisés Guimarães proposed openstack/pbr master: Adding pre-commit  https://review.opendev.org/74216016:48
*** jtomasek has joined #openstack-infra16:48
*** gfidente is now known as gfidente|afk16:51
*** jtomasek has quit IRC16:58
*** jpena is now known as jpena|off17:07
*** andrewbonney has quit IRC17:10
*** derekh has quit IRC17:18
*** sshnaidm|rover is now known as sshnaidm|afk17:24
*** artom has quit IRC17:24
*** artom has joined #openstack-infra17:25
*** artom has quit IRC17:25
*** artom has joined #openstack-infra17:26
*** Topner has joined #openstack-infra17:27
*** dhill has quit IRC17:28
*** ralonsoh has quit IRC17:28
*** Topner has quit IRC17:34
*** jamesmcarthur has quit IRC17:37
*** xek has joined #openstack-infra17:40
*** xek has quit IRC17:40
*** priteau has quit IRC18:01
*** dhill has joined #openstack-infra18:05
*** gyee has joined #openstack-infra18:09
*** jamesmcarthur has joined #openstack-infra18:12
*** thedaveking has quit IRC18:13
*** iurygregory has quit IRC18:18
*** iurygregory has joined #openstack-infra18:20
*** iurygregory has quit IRC18:24
*** iurygregory has joined #openstack-infra18:25
*** lyarwood has joined #openstack-infra18:29
*** jamesmcarthur_ has joined #openstack-infra18:33
*** stevebaker has joined #openstack-infra18:33
*** jamesmcarthur has quit IRC18:36
*** d34dh0r53 has quit IRC18:41
*** d34dh0r53 has joined #openstack-infra18:44
*** Topner has joined #openstack-infra19:03
*** tosky has joined #openstack-infra19:12
*** jamesmcarthur_ has quit IRC19:13
*** jamesmcarthur has joined #openstack-infra19:14
*** Topner has quit IRC19:30
*** nightmare_unreal has quit IRC19:38
*** dciabrin_ has joined #openstack-infra19:51
*** dciabrin has quit IRC19:54
*** vishalmanchanda has quit IRC19:57
clarkbthe dependency resolver will be changing in about a week when pip 20.3 is released19:57
clarkbprometheanfire: I believe you had tested with the new resolving and openstack requirements handled it fine, but figured I'd mention it in case things pop up19:57
clarkbgmann: smcginnis ^ you will probably be interested in that as well19:58
clarkbI think that openstack's use of constraints makes it less of an issue though it may affect constraint generation19:58
*** priteau has joined #openstack-infra20:02
*** ddurst has quit IRC20:15
*** ddurst has joined #openstack-infra20:17
prometheanfireclarkb: yep I did, good to know that they are finally releasing though20:37
*** lbragstad_ is now known as lbragstad20:49
*** hashar has quit IRC20:55
*** sboyron has quit IRC20:55
*** kwazar has joined #openstack-infra21:04
*** tdasilva_ is now known as tdasilva21:11
*** rfolco has quit IRC21:16
*** jamesmcarthur_ has joined #openstack-infra21:23
*** slaweq has quit IRC21:24
*** jamesmcarthur has quit IRC21:26
*** matt_kosut has quit IRC21:36
*** matt_kosut has joined #openstack-infra21:37
*** matt_kosut has quit IRC21:37
*** bradm has joined #openstack-infra21:45
*** jamesmcarthur_ has quit IRC21:51
*** jamesmcarthur has joined #openstack-infra22:03
*** tosky has quit IRC22:10
*** tosky has joined #openstack-infra22:11
*** jamesmcarthur has quit IRC22:17
*** gfidente|afk has quit IRC22:20
*** rlandy is now known as rlandy|bbl22:25
*** jamesmcarthur has joined #openstack-infra22:28
*** ociuhandu has joined #openstack-infra22:34
*** jamesmcarthur has quit IRC22:35
*** jamesmcarthur has joined #openstack-infra22:35
*** ociuhandu has quit IRC22:39
*** jamesmcarthur has quit IRC22:41
*** jamesmcarthur has joined #openstack-infra22:42
*** rfolco has joined #openstack-infra22:44
*** jamesmcarthur_ has joined #openstack-infra22:45
*** jamesmcarthur has quit IRC22:49
*** rcernin has joined #openstack-infra22:51
*** tosky has quit IRC22:51
*** hamalq has quit IRC22:57
*** jamesmcarthur_ has quit IRC22:59
*** jamesmcarthur has joined #openstack-infra23:00
*** hamalq has joined #openstack-infra23:01
*** jamesmcarthur has quit IRC23:02
*** rfolco has quit IRC23:17
*** sai438 has joined #openstack-infra23:31
tkajinamI'm afraid that gerrit is veeery slow now23:32
prometheanfireright23:32
*** dchen has joined #openstack-infra23:33
*** pmannidi has quit IRC23:34
clarkbit looks like someone in google cloud may be crawling all commits23:37
clarkbs/commits/changes23:37
prometheanfireouch23:40
tkajinamhmm... maybe that is for auditing ? I guess23:44
clarkbcross checking gerrit thread activity the lucene threads which talk to the indexes seem quite busy which I think correleates to ueries like that23:50
clarkbbecause gerrit has to load the change data and it talks to lucene to do that quickly23:51
clarkblooks like whatever is doing it may be buggy because it is requesting the same series of change sover and over23:52

Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!