*** gyee has quit IRC | 00:11 | |
*** gyee has joined #openstack-infra | 00:25 | |
*** ChanServ changes topic to "Discussion of OpenStack's Developer tooling and CI jobs | Discussion around infrastructure services now in #opendev" | 00:37 | |
-openstackstatus- NOTICE: The Gerrit service at review.opendev.org is back up and running; for outage details see analysis here: http://lists.opendev.org/pipermail/service-announce/2020-October/000011.html | 00:37 | |
*** Goneri has quit IRC | 00:46 | |
*** chandankumar is now known as raukadah | 00:47 | |
*** sreejithp has joined #openstack-infra | 00:56 | |
*** sreejithp has quit IRC | 01:03 | |
*** owalsh has quit IRC | 01:08 | |
*** owalsh has joined #openstack-infra | 01:09 | |
*** dhill has quit IRC | 01:11 | |
*** ramishra has quit IRC | 01:11 | |
*** gyee has quit IRC | 01:12 | |
*** sreejithp has joined #openstack-infra | 01:20 | |
*** sreejithp has quit IRC | 01:28 | |
*** jamesmcarthur has joined #openstack-infra | 01:31 | |
*** jamesmcarthur has quit IRC | 01:48 | |
*** jamesmcarthur has joined #openstack-infra | 01:48 | |
*** jamesmcarthur has quit IRC | 01:50 | |
*** jamesmcarthur has joined #openstack-infra | 01:50 | |
*** jamesmcarthur has quit IRC | 01:53 | |
*** jamesmcarthur has joined #openstack-infra | 01:56 | |
*** jamesmcarthur has quit IRC | 02:02 | |
*** jamesmcarthur has joined #openstack-infra | 02:04 | |
*** piotrowskim has quit IRC | 02:12 | |
*** stevebaker has quit IRC | 02:15 | |
*** jamesmcarthur has quit IRC | 02:31 | |
*** jamesmcarthur has joined #openstack-infra | 02:32 | |
*** jamesmcarthur has quit IRC | 02:36 | |
*** sai438 has joined #openstack-infra | 03:14 | |
*** jamesmcarthur has joined #openstack-infra | 03:16 | |
*** pmannidi has quit IRC | 03:18 | |
*** jamesmcarthur has quit IRC | 03:25 | |
*** pmannidi has joined #openstack-infra | 03:26 | |
*** sai438 has quit IRC | 03:29 | |
*** psachin has joined #openstack-infra | 03:30 | |
*** psachin has quit IRC | 03:30 | |
*** psachin has joined #openstack-infra | 03:31 | |
tkajinam | hi. will we recover gerrit bot as gerrit has been recovered ? | 03:42 |
---|---|---|
*** rfolco has joined #openstack-infra | 03:43 | |
*** hamalq has quit IRC | 03:46 | |
*** rfolco has quit IRC | 03:47 | |
*** sai438 has joined #openstack-infra | 03:49 | |
*** pmannidi has quit IRC | 03:52 | |
*** pmannidi has joined #openstack-infra | 03:53 | |
*** pmannidi_ has joined #openstack-infra | 03:55 | |
*** sai438 has quit IRC | 03:57 | |
*** pmannidi has quit IRC | 03:58 | |
*** jamesmcarthur has joined #openstack-infra | 03:59 | |
*** lajoskatona has joined #openstack-infra | 04:02 | |
*** lajoskatona has left #openstack-infra | 04:02 | |
*** jamesmcarthur has quit IRC | 04:04 | |
*** evrardjp has quit IRC | 04:33 | |
*** evrardjp has joined #openstack-infra | 04:33 | |
*** matt_kosut has joined #openstack-infra | 04:53 | |
*** soniya29|ruck is now known as soniya29 | 04:58 | |
*** zxiiro has quit IRC | 04:59 | |
*** jamesmcarthur has joined #openstack-infra | 05:05 | |
*** jamesmcarthur has quit IRC | 05:10 | |
*** ldenny has quit IRC | 05:21 | |
*** sreejithp has joined #openstack-infra | 05:24 | |
*** bdodd_ has joined #openstack-infra | 05:28 | |
*** sreejithp has quit IRC | 05:28 | |
*** bdodd has quit IRC | 05:29 | |
*** ldenny has joined #openstack-infra | 05:36 | |
*** jamesmcarthur has joined #openstack-infra | 05:41 | |
*** jamesmcarthur has quit IRC | 05:46 | |
*** jamesmcarthur has joined #openstack-infra | 05:56 | |
*** jamesmcarthur has quit IRC | 05:57 | |
*** jamesmcarthur has joined #openstack-infra | 05:57 | |
*** jamesmcarthur has quit IRC | 06:02 | |
*** jamesmcarthur has joined #openstack-infra | 06:34 | |
*** dciabrin has joined #openstack-infra | 06:35 | |
*** sboyron has joined #openstack-infra | 06:36 | |
*** rpittau|afk is now known as rpittau | 06:42 | |
*** jcapitao has joined #openstack-infra | 06:43 | |
*** jtomasek has joined #openstack-infra | 06:43 | |
*** ralonsoh has joined #openstack-infra | 06:45 | |
*** vishalmanchanda has joined #openstack-infra | 06:47 | |
*** dklyle has quit IRC | 06:47 | |
*** eolivare has joined #openstack-infra | 06:49 | |
*** jamesmcarthur has quit IRC | 06:52 | |
*** slaweq has joined #openstack-infra | 06:54 | |
*** andrewbonney has joined #openstack-infra | 07:06 | |
*** sreejithp has joined #openstack-infra | 07:25 | |
*** sreejithp has quit IRC | 07:31 | |
*** hberaud has quit IRC | 07:34 | |
*** slaweq has quit IRC | 07:34 | |
*** slaweq has joined #openstack-infra | 07:35 | |
*** hberaud has joined #openstack-infra | 07:36 | |
*** openstackgerrit has quit IRC | 07:38 | |
*** rcernin has quit IRC | 07:39 | |
*** slaweq has quit IRC | 07:40 | |
*** tosky has joined #openstack-infra | 07:42 | |
*** sshnaidm is now known as sshnaidm|afk | 07:43 | |
*** slaweq has joined #openstack-infra | 07:45 | |
*** jtomasek has quit IRC | 07:47 | |
*** ttx has quit IRC | 07:48 | |
*** ttx has joined #openstack-infra | 07:51 | |
*** priteau has joined #openstack-infra | 07:54 | |
*** jpena|off is now known as jpena | 07:56 | |
*** pmannidi_ has quit IRC | 08:02 | |
*** lucasagomes has joined #openstack-infra | 08:07 | |
*** slaweq has quit IRC | 08:08 | |
*** rcernin has joined #openstack-infra | 08:08 | |
*** dtantsur|afk is now known as dtantsur | 08:10 | |
*** slaweq has joined #openstack-infra | 08:11 | |
*** rcernin has quit IRC | 08:14 | |
*** gnuoy has quit IRC | 08:19 | |
*** pmannidi has joined #openstack-infra | 08:19 | |
*** gnuoy has joined #openstack-infra | 08:20 | |
*** rcernin has joined #openstack-infra | 08:28 | |
*** nightmare_unreal has joined #openstack-infra | 08:34 | |
*** derekh has joined #openstack-infra | 08:34 | |
*** slaweq has quit IRC | 08:35 | |
*** slaweq has joined #openstack-infra | 08:37 | |
*** slaweq has joined #openstack-infra | 08:37 | |
*** sshnaidm|afk is now known as sshnaidm | 08:39 | |
*** jamesmcarthur has joined #openstack-infra | 08:50 | |
*** ociuhandu has joined #openstack-infra | 08:51 | |
*** rcernin has quit IRC | 08:56 | |
*** jamesmcarthur has quit IRC | 08:59 | |
*** matbu has quit IRC | 09:18 | |
*** gfidente has joined #openstack-infra | 09:19 | |
*** matbu has joined #openstack-infra | 09:25 | |
*** sreejithp has joined #openstack-infra | 09:27 | |
*** pmannidi has quit IRC | 09:30 | |
*** pmannidi has joined #openstack-infra | 09:30 | |
*** sreejithp has quit IRC | 09:32 | |
*** pmannidi has quit IRC | 09:35 | |
tkajinam | is there any good way to download gerrit-diffs as an archive ? | 09:37 |
tkajinam | we have bunch of puppet-foo projects so it would be helpful if I can download all files for puppet-* projects so that I can check them in my local machine | 09:38 |
*** pmannidi has joined #openstack-infra | 09:39 | |
*** lmiccini has quit IRC | 09:46 | |
*** pmannidi has quit IRC | 09:47 | |
*** pmannidi has joined #openstack-infra | 09:47 | |
*** lmiccini has joined #openstack-infra | 09:51 | |
*** sai438 has joined #openstack-infra | 09:51 | |
frickler | tkajinam: wget has some recursive download option, not sure if can also filter the way you need. we could likely also create a tarball for the whole tree, I can look into that in a bit | 09:54 |
*** pmannidi has quit IRC | 09:55 | |
*** pmannidi has joined #openstack-infra | 09:59 | |
*** sai438 has quit IRC | 10:02 | |
*** sai438 has joined #openstack-infra | 10:02 | |
*** ramishra has joined #openstack-infra | 10:03 | |
*** ramishra has quit IRC | 10:04 | |
*** ramishra has joined #openstack-infra | 10:04 | |
*** pmannidi has quit IRC | 10:05 | |
*** pmannidi_ has joined #openstack-infra | 10:06 | |
*** sai438 has quit IRC | 10:07 | |
tkajinam | hmm. we need to somehow filter the uri otherwise it searches all files :-P | 10:07 |
*** bradm has quit IRC | 10:23 | |
*** ldenny has quit IRC | 10:24 | |
*** ldenny has joined #openstack-infra | 10:24 | |
*** jcapitao is now known as jcapitao_lunch | 10:24 | |
*** aluria has quit IRC | 10:28 | |
*** aluria has joined #openstack-infra | 10:30 | |
*** aluria has quit IRC | 10:38 | |
*** priteau has quit IRC | 10:39 | |
*** aluria has joined #openstack-infra | 10:39 | |
*** dhill has joined #openstack-infra | 11:02 | |
*** jtomasek has joined #openstack-infra | 11:09 | |
*** priteau has joined #openstack-infra | 11:15 | |
*** priteau has quit IRC | 11:26 | |
*** jpena is now known as jpena|lunch | 11:32 | |
*** jcapitao_lunch is now known as jcapitao | 11:33 | |
*** jamesmcarthur has joined #openstack-infra | 11:36 | |
*** jamesmcarthur has quit IRC | 11:41 | |
*** ramishra has quit IRC | 11:44 | |
*** rlandy has joined #openstack-infra | 11:53 | |
*** rlandy is now known as rlandy|rover | 11:53 | |
*** jtomasek has quit IRC | 12:00 | |
*** priteau has joined #openstack-infra | 12:01 | |
*** rfolco has joined #openstack-infra | 12:04 | |
*** jamesmcarthur has joined #openstack-infra | 12:07 | |
*** dchen has quit IRC | 12:14 | |
*** jpena|lunch is now known as jpena | 12:30 | |
*** jamesmcarthur has quit IRC | 12:33 | |
*** Goneri has joined #openstack-infra | 12:46 | |
*** zxiiro has joined #openstack-infra | 12:50 | |
*** ramishra has joined #openstack-infra | 13:00 | |
*** ramishra has quit IRC | 13:07 | |
*** slaweq has quit IRC | 13:08 | |
*** jamesmcarthur has joined #openstack-infra | 13:09 | |
*** slaweq has joined #openstack-infra | 13:12 | |
*** tdasilva has quit IRC | 13:15 | |
*** tdasilva has joined #openstack-infra | 13:15 | |
*** rlandy|rover is now known as rlandy | 13:17 | |
*** ysandeep is now known as ysandeep|ruck | 13:18 | |
*** psachin has quit IRC | 13:19 | |
*** nhicher has joined #openstack-infra | 13:21 | |
*** nhicher has quit IRC | 13:22 | |
*** nhicher has joined #openstack-infra | 13:24 | |
*** ramishra has joined #openstack-infra | 13:29 | |
*** jamesmcarthur has quit IRC | 13:29 | |
*** pmannidi has joined #openstack-infra | 13:47 | |
*** pmannidi_ has quit IRC | 13:49 | |
*** jamesmcarthur has joined #openstack-infra | 13:50 | |
*** ysandeep|ruck is now known as ysandeep|ruck|af | 14:04 | |
*** jamesmcarthur has quit IRC | 14:12 | |
*** ysandeep|ruck|af is now known as ysandeep|ruck | 14:21 | |
*** sshnaidm is now known as sshnaidm|rover | 14:22 | |
*** dklyle has joined #openstack-infra | 14:39 | |
*** jtomasek has joined #openstack-infra | 14:41 | |
*** jamesmcarthur has joined #openstack-infra | 14:54 | |
*** ysandeep|ruck is now known as ysandeep|away | 14:57 | |
*** raukadah is now known as chandankumar | 15:12 | |
*** tdasilva_ has joined #openstack-infra | 15:27 | |
*** tdasilva has quit IRC | 15:30 | |
*** dmsimard1 has joined #openstack-infra | 15:30 | |
*** dmsimard has quit IRC | 15:32 | |
*** dmsimard1 is now known as dmsimard | 15:32 | |
*** thedaveking has quit IRC | 15:42 | |
*** ramishra has quit IRC | 15:47 | |
smcginnis | tkajinam: Could clone locally and grep there. | 15:50 |
smcginnis | for repo in $(curl -s https://review.opendev.org/projects/ | grep "openstack.puppet-.*" | cut -d '"' -f 2); do git clone "https://opendev.org/$repo"; done | 15:50 |
*** thedaveking has joined #openstack-infra | 15:51 | |
*** noonedeadpunk has quit IRC | 15:51 | |
*** dtantsur is now known as dtantsur|afk | 15:55 | |
fungi | yeah, the main reason we generated a specific set of refs is that we can't necessarily trust the commit times in git repositories to indicate whether a change truly originated at the times claimed, since anyone can backdate a commit | 15:56 |
fungi | so instead we took two copies of all the repositories, one from a snapshot prior to the compromise and the other current, and identified every commit which was in the current set but not in the snapshot | 15:57 |
fungi | that way we avoided overlooking any such "backdated" commits | 15:57 |
*** kaisers2 has quit IRC | 15:58 | |
*** mihalis68_ has quit IRC | 15:58 | |
*** pmannidi has quit IRC | 15:58 | |
*** pmannidi has joined #openstack-infra | 16:01 | |
*** lyarwood has quit IRC | 16:01 | |
auristor | does openstack install the gerrit reviewnotes plugin? If so, I think you can trust the Submitted-at timestamp | 16:03 |
fungi | auristor: yes, folks if they configure fetching notes can check that metadata | 16:04 |
fungi | (i rely on it extensively) | 16:04 |
*** rpittau is now known as rpittau|afk | 16:04 | |
fungi | however, in this case, the attacker had write access to the database (via gerrit's gsql "feature") and those timestamps are writeable fields, so... | 16:05 |
fungi | i wouldn't entirely trust the submitted-at timestamps in this particular case | 16:05 |
auristor | in that case you can't trust anything. | 16:06 |
fungi | right. we basically diffed things against a known good snapshot from prior to the compromise | 16:06 |
fungi | identified every new commit which was not in the snapshot, rolled back every ssh key which had been added since the snapshot, blew away all api keys in the db, and so on | 16:07 |
auristor | got it. | 16:09 |
fungi | also diffed the account_external_ids tables between current and the snapshot to check for extra openid urls which may have been added to accounts, or any which might have been surreptitiously replaced | 16:10 |
fungi | basically anything we could think of which could provide a trampoline for the attacker to regain access to existing accounts | 16:10 |
*** Topner has joined #openstack-infra | 16:11 | |
*** noonedeadpunk_ has joined #openstack-infra | 16:13 | |
*** tosky has quit IRC | 16:16 | |
auristor | I might have restored the known good snapshot and then resubmitted the other changes to the restored gerrit for review. | 16:16 |
fungi | the main challenges there is that we've got published release artifacts resulting from those changes, so would also have needed to revoke them all and make all new releases of many projects (hundreds of openstack deliverables were released in that three-week period) | 16:18 |
fungi | so a post-hoc commit audit was ultimately seen as less work. most of the teams already seem to have knocked theirs out since the announcement hours ago | 16:19 |
*** portdirect has quit IRC | 16:19 | |
*** portdirect has joined #openstack-infra | 16:20 | |
*** hashar has joined #openstack-infra | 16:20 | |
*** portdirect has quit IRC | 16:21 | |
*** portdirect has joined #openstack-infra | 16:21 | |
*** jcapitao has quit IRC | 16:22 | |
*** hamalq has joined #openstack-infra | 16:27 | |
*** hamalq has quit IRC | 16:29 | |
*** hamalq has joined #openstack-infra | 16:30 | |
*** eolivare has quit IRC | 16:30 | |
*** noonedeadpunk_ has quit IRC | 16:33 | |
*** lucasagomes has quit IRC | 16:34 | |
*** openstackgerrit has joined #openstack-infra | 16:34 | |
openstackgerrit | Moisés Guimarães proposed openstack/pbr master: Adding pre-commit https://review.opendev.org/742160 | 16:34 |
*** dklyle has quit IRC | 16:39 | |
*** Topner has quit IRC | 16:39 | |
*** dklyle has joined #openstack-infra | 16:39 | |
*** ociuhandu_ has joined #openstack-infra | 16:39 | |
*** noonedeadpunk has joined #openstack-infra | 16:39 | |
*** jtomasek has quit IRC | 16:42 | |
*** ociuhandu has quit IRC | 16:42 | |
*** ociuhandu_ has quit IRC | 16:44 | |
openstackgerrit | Moisés Guimarães proposed openstack/pbr master: Adding pre-commit https://review.opendev.org/742160 | 16:48 |
*** jtomasek has joined #openstack-infra | 16:48 | |
*** gfidente is now known as gfidente|afk | 16:51 | |
*** jtomasek has quit IRC | 16:58 | |
*** jpena is now known as jpena|off | 17:07 | |
*** andrewbonney has quit IRC | 17:10 | |
*** derekh has quit IRC | 17:18 | |
*** sshnaidm|rover is now known as sshnaidm|afk | 17:24 | |
*** artom has quit IRC | 17:24 | |
*** artom has joined #openstack-infra | 17:25 | |
*** artom has quit IRC | 17:25 | |
*** artom has joined #openstack-infra | 17:26 | |
*** Topner has joined #openstack-infra | 17:27 | |
*** dhill has quit IRC | 17:28 | |
*** ralonsoh has quit IRC | 17:28 | |
*** Topner has quit IRC | 17:34 | |
*** jamesmcarthur has quit IRC | 17:37 | |
*** xek has joined #openstack-infra | 17:40 | |
*** xek has quit IRC | 17:40 | |
*** priteau has quit IRC | 18:01 | |
*** dhill has joined #openstack-infra | 18:05 | |
*** gyee has joined #openstack-infra | 18:09 | |
*** jamesmcarthur has joined #openstack-infra | 18:12 | |
*** thedaveking has quit IRC | 18:13 | |
*** iurygregory has quit IRC | 18:18 | |
*** iurygregory has joined #openstack-infra | 18:20 | |
*** iurygregory has quit IRC | 18:24 | |
*** iurygregory has joined #openstack-infra | 18:25 | |
*** lyarwood has joined #openstack-infra | 18:29 | |
*** jamesmcarthur_ has joined #openstack-infra | 18:33 | |
*** stevebaker has joined #openstack-infra | 18:33 | |
*** jamesmcarthur has quit IRC | 18:36 | |
*** d34dh0r53 has quit IRC | 18:41 | |
*** d34dh0r53 has joined #openstack-infra | 18:44 | |
*** Topner has joined #openstack-infra | 19:03 | |
*** tosky has joined #openstack-infra | 19:12 | |
*** jamesmcarthur_ has quit IRC | 19:13 | |
*** jamesmcarthur has joined #openstack-infra | 19:14 | |
*** Topner has quit IRC | 19:30 | |
*** nightmare_unreal has quit IRC | 19:38 | |
*** dciabrin_ has joined #openstack-infra | 19:51 | |
*** dciabrin has quit IRC | 19:54 | |
*** vishalmanchanda has quit IRC | 19:57 | |
clarkb | the dependency resolver will be changing in about a week when pip 20.3 is released | 19:57 |
clarkb | prometheanfire: I believe you had tested with the new resolving and openstack requirements handled it fine, but figured I'd mention it in case things pop up | 19:57 |
clarkb | gmann: smcginnis ^ you will probably be interested in that as well | 19:58 |
clarkb | I think that openstack's use of constraints makes it less of an issue though it may affect constraint generation | 19:58 |
*** priteau has joined #openstack-infra | 20:02 | |
*** ddurst has quit IRC | 20:15 | |
*** ddurst has joined #openstack-infra | 20:17 | |
prometheanfire | clarkb: yep I did, good to know that they are finally releasing though | 20:37 |
*** lbragstad_ is now known as lbragstad | 20:49 | |
*** hashar has quit IRC | 20:55 | |
*** sboyron has quit IRC | 20:55 | |
*** kwazar has joined #openstack-infra | 21:04 | |
*** tdasilva_ is now known as tdasilva | 21:11 | |
*** rfolco has quit IRC | 21:16 | |
*** jamesmcarthur_ has joined #openstack-infra | 21:23 | |
*** slaweq has quit IRC | 21:24 | |
*** jamesmcarthur has quit IRC | 21:26 | |
*** matt_kosut has quit IRC | 21:36 | |
*** matt_kosut has joined #openstack-infra | 21:37 | |
*** matt_kosut has quit IRC | 21:37 | |
*** bradm has joined #openstack-infra | 21:45 | |
*** jamesmcarthur_ has quit IRC | 21:51 | |
*** jamesmcarthur has joined #openstack-infra | 22:03 | |
*** tosky has quit IRC | 22:10 | |
*** tosky has joined #openstack-infra | 22:11 | |
*** jamesmcarthur has quit IRC | 22:17 | |
*** gfidente|afk has quit IRC | 22:20 | |
*** rlandy is now known as rlandy|bbl | 22:25 | |
*** jamesmcarthur has joined #openstack-infra | 22:28 | |
*** ociuhandu has joined #openstack-infra | 22:34 | |
*** jamesmcarthur has quit IRC | 22:35 | |
*** jamesmcarthur has joined #openstack-infra | 22:35 | |
*** ociuhandu has quit IRC | 22:39 | |
*** jamesmcarthur has quit IRC | 22:41 | |
*** jamesmcarthur has joined #openstack-infra | 22:42 | |
*** rfolco has joined #openstack-infra | 22:44 | |
*** jamesmcarthur_ has joined #openstack-infra | 22:45 | |
*** jamesmcarthur has quit IRC | 22:49 | |
*** rcernin has joined #openstack-infra | 22:51 | |
*** tosky has quit IRC | 22:51 | |
*** hamalq has quit IRC | 22:57 | |
*** jamesmcarthur_ has quit IRC | 22:59 | |
*** jamesmcarthur has joined #openstack-infra | 23:00 | |
*** hamalq has joined #openstack-infra | 23:01 | |
*** jamesmcarthur has quit IRC | 23:02 | |
*** rfolco has quit IRC | 23:17 | |
*** sai438 has joined #openstack-infra | 23:31 | |
tkajinam | I'm afraid that gerrit is veeery slow now | 23:32 |
prometheanfire | right | 23:32 |
*** dchen has joined #openstack-infra | 23:33 | |
*** pmannidi has quit IRC | 23:34 | |
clarkb | it looks like someone in google cloud may be crawling all commits | 23:37 |
clarkb | s/commits/changes | 23:37 |
prometheanfire | ouch | 23:40 |
tkajinam | hmm... maybe that is for auditing ? I guess | 23:44 |
clarkb | cross checking gerrit thread activity the lucene threads which talk to the indexes seem quite busy which I think correleates to ueries like that | 23:50 |
clarkb | because gerrit has to load the change data and it talks to lucene to do that quickly | 23:51 |
clarkb | looks like whatever is doing it may be buggy because it is requesting the same series of change sover and over | 23:52 |
Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!