16479 Commits

Author SHA1 Message Date
Jeremy Stanley
abc66ed38c Add service-incident@opendev mailing list
Create a mailing list for private coordination of security incidents
for the OpenDev Collaboratory. The intent is that this can be used
to share sensitive information between sysadmins and council members
in the event of any suspected breach. For the sake of transparency,
all information discussed on this list which can safely be made
public should also be communicated to the service-announce or
service-discuss mailing lists at the earliest opportunity.

Change-Id: I32bef68eb7019261471c167d19eee733457078a2
2020-10-22 16:16:04 +00:00
Zuul
b5ee5e6eb8 Merge "Don't recompress db backups" 2020-10-19 22:06:36 +00:00
Clark Boylan
fa0753668d Add two more openstackid servers so that we can load balnace them
Virtual summit fun! We need more capacity for openstackid. smarcet says
there isn't really any shared state so this hsould be safe then we can
sort out a load balancer.

Change-Id: I2a7d62bc980a797f7c45d119a5398925f5607df0
2020-10-19 09:56:34 -07:00
Clark Boylan
6e368cd61e Don't recompress db backups
We're using logrotate to keep a small number of db backups locally. We
write these backups to disk compressed. We don't want logrotate to
recompress them. This is unnecessary extra work.

Change-Id: Iafe1628ff421f47cf3e5cbee14998eeceb60be4c
2020-10-19 07:06:47 -07:00
Zuul
be8e13c139 Merge "Update static Apache configs to 2.4 ACL primitives" 2020-10-16 21:06:28 +00:00
Zuul
35ca675476 Merge "Remove docker v1 registry proxy from our mirrors" 2020-10-16 20:42:09 +00:00
Zuul
01585bcd98 Merge "Use the apache-ua-filter role on Gitea servers" 2020-10-16 20:42:07 +00:00
Zuul
e3728575b9 Merge "Block restricted user agents for the tarballs site" 2020-10-16 20:06:51 +00:00
Clark Boylan
9b6398394d Remove docker v1 registry proxy from our mirrors
Docker has long planned to turn this off and it appears that they have
done so. Planning details can be found at:
https://www.docker.com/blog/registry-v1-api-deprecation/

Removing this simplifies our configs as well as testing. Do this as part
of good hygiene.

Change-Id: I11281167a87ba30b4ebaa88792032aec1af046c1
2020-10-16 12:35:37 -07:00
Clark Boylan
fa362b813c More old apache acl cleanups
We can rely on Require instead of Order, Allow, Deny, Satisfy since we
are all on apache 2.4 now. This simplifies reasoning about acl rules.

Change-Id: Idedba1558ccaa1c753d1175e356bf26a8d4b1084
2020-10-16 11:16:26 -07:00
Jeremy Stanley
9cabc58a1a Use the apache-ua-filter role on Gitea servers
The user agent filter has been turned into a reusable Ansible role
containing a macro definition. Add that role and replace the
hard-coded copy of the user agent filter here with that
UserAgentFilter macro.

Change-Id: Ic24a38c93f0f68fab9ef1168de91ffad477fe13c
2020-10-16 17:45:19 +00:00
Jeremy Stanley
deaae4f665 Block restricted user agents for the tarballs site
A copy of the filter used for our Gitea farm, this same activity has
been showing up on our tarballs.opendev.org site as well which is
consuming available connection slots for all vhosts on the static
server.

This is implemented as a macro so that it can be included into
additional vhosts, and put into a separate role so that it can be
added to all playbooks which need it. A subsequent change will add
it to the Gitea servers, eliminating the redundant copy there.

Change-Id: Ic2020b753076209f7708f76744fdf746bf933bd9
2020-10-16 17:45:12 +00:00
Jeremy Stanley
8924835baf Update static Apache configs to 2.4 ACL primitives
We don't need to keep using the old Apache 2.2 Satisfy ACL primitive
because we are now running Apache 2.4 everywhere. Stick to Require
as it simplifies understanding of ACLs by being consistent.

Change-Id: Ib2f7ea1909b9798279efc77a42b632e7129bd1d0
2020-10-16 16:15:00 +00:00
Zuul
ef94bf7645 Merge "OpenstackId v3.0.16" 2020-10-15 15:22:23 +00:00
Zuul
e76ace6dcc Merge "Add four more gitea ddos UA strings" 2020-10-15 05:34:47 +00:00
smarcet
807ea2608e OpenstackId v3.0.16
added new search criteria for endpoint
GET /api/v1/users

primary_email (==,@=)

Change-Id: Ib643a8c1ba4e79444463777197fc86a64a1912be
Signed-off-by: smarcet <smarcet@gmail.com>
2020-10-15 00:23:08 -03:00
Ian Wienand
a719ac4b9e tarballs: remove incorrect redirects
We found a couple of projects that were initially moved under "x/" but
then moved back under "openstack/" later.  The original scripts didn't
take this into account (I5bf2ddf09b3df71a3428a8a0c535b131ecbc0bca has
been updated to note this).

The affected projects have been moved back manually on AFS, and this
corrects the website redirects.

Change-Id: I59ba05923ec5aa1ca8fed337b6384064b3038836
2020-10-15 09:11:49 +11:00
Clark Boylan
ebc2fa976c Add four more gitea ddos UA strings
The existing filtered UAs seem to catch the bulk of the traffic but
there are a few common ones that are still sneaking through. Add four
new rules for these cases.

All three are MSIE variants from version 6 to 9. All old enough that we
should be able to do this safely without impacting real users.

Change-Id: I8ae59f38de8b30bd06e1643ddbccf81ea32858aa
2020-10-14 10:30:27 -07:00
Clark Boylan
ccc91ba9c9 Switch to filtered apache backends
We seem to be under a similar attack to last time. The new apache filter
in front of gitea was implemented to be used if this happened again.
Switch to it.

Change-Id: Ib9ed3029dad7fc26cca209fece547a2a94d8da4a
2020-10-14 08:55:21 -07:00
Jeremy Stanley
16d661affe Switch openstack/compute-hyperv->x tarball redir
When moving unofficial repositories out of the openstack git
namespace, compute-hyperv was included because it was not under the
governance of any official team or SIG. Later the OpenStack
Winstackers team adopted it and we moved it back into the openstack
git namespace.

More recently, when moving and redirecting tarballs site
content/URLs based on the original git namespace moves, we failed to
take into account that this project had moved back into the
openstack git namespace. Undo the redirect, and then the old
tarballs will be moved manually to match.

Change-Id: I208d3196ac38ccfbad6269a75848339c95e08c2b
2020-10-14 11:48:38 +00:00
Zuul
21ffde3bec Merge "borg-backups: add some extra excludes" 2020-10-14 04:15:37 +00:00
Ian Wienand
faa296d37d borg-backups: add some extra excludes
A few extra things to not bother with in our default backup
directories

Change-Id: I693e80020d852f4d09978ddcd7ecf94acc2d17c3
2020-10-14 10:01:07 +11:00
Zuul
2c21f08b86 Merge "Remove Ubuntu Xenial ARM64 base testing" 2020-10-13 21:48:57 +00:00
Zuul
3796649b66 Merge "borg backup : add ethercalc02" 2020-10-12 22:22:53 +00:00
Zuul
65a4a4efe7 Merge "Stop replicating to local git mirror on gerrit" 2020-10-12 21:57:19 +00:00
Zuul
4edf521794 Merge "Add gerrit static files that were lost in ansiblification" 2020-10-12 21:57:17 +00:00
smarcet
e394198d03 OpenstackID v3.0.15
* updated registration emails
* updated registration form
* updated password policies
* bug fixing

Change-Id: Ibd644e9daa9bd345cf883db3dfa75b58b4ad7a18
Signed-off-by: smarcet <smarcet@gmail.com>
2020-10-12 17:20:54 -03:00
Ian Wienand
27f67136f7 borg backup : add ethercalc02
Add ethercalc as the first borg backup client.  We will monitor this
as it roles into production.

Change-Id: I1ac71e92a8e5c779cd98af16ee4958877c6162ce
2020-10-12 16:30:46 +11:00
Ian Wienand
a86ba4590b install-borg: bump to latest version
Since we haven't used this anywhere yet, let's start with the latest
version.

Fix role matching for job too.

Change-Id: I22620fc7ade8fbdb664100ef6b6ab98c93d6104f
2020-10-12 15:07:38 +11:00
Ian Wienand
dd50f6b732 borg : match install-borg role to run deploy job
This was forgotten in the original addition.

Change-Id: I0725b99e938b993e68106f6ba1b2704e9413e902
2020-10-12 13:06:10 +11:00
Zuul
482a874862 Merge "install-borg: also install python3-venv" 2020-10-11 23:06:35 +00:00
Clark Boylan
cae58690dc Stop replicating to local git mirror on gerrit
We've disabled access to the local gerrit git mirrors at the /p/ prefix
previously as newer gerrit uses that path for something else. The next
step is to stop replicating to that location entirely.

Another reason for this is when we switch to notedb this local
replication will replicate everything then if we expose it we'd
potentially expose content we don't want to via git (rather than the
gerrit APIs).

Change-Id: I795466af3e1608eefe506ca56828327491f73c27
2020-10-09 10:39:30 -07:00
Clark Boylan
388ec27173 Add gerrit static files that were lost in ansiblification
The title.svg logo for opendev and two jquery js files are no longer
managed by ansible nor do they appear to be in our docker image. They
appear to have been lost when we converted from puppet to ansible +
docker. Add them back in. We are also missing the icla html content (but
not the other clas) add this one back in.

We vendor the js contents even though in the past we copied them from a
git repo clone and a distro package installation. This way we don't have
unexpected surprises, record that the files are used, and can always
update them later.

Change-Id: I981b4b0f233ece45d03a80dc1724a4e496f66eb8
2020-10-09 10:38:45 -07:00
Ian Wienand
bc7ce1b9a7 install-borg: also install python3-venv
This is failing in production as python3-venv isn't installed on the
server.  Add it.

Change-Id: I748ea612fc20fbebbddff122f907eb59c6f39f94
2020-10-09 16:50:21 +11:00
Ian Wienand
2a63ce7d30 Remove Ubuntu Xenial ARM64 base testing
We no longer have any ARM64 Xenial nodes using this, we can remove it.

Change-Id: I20f22a4538bbaa9cb3a3ca11e156c3d02ca5406f
2020-10-08 14:31:31 +11:00
Ian Wienand
0d0f8ffe14 Add initial borg backup server
To catch up -- because this work is moving slowly ... the two backup
servers are currently the vexxhost and RAX ORD hosts.  The vexxhost
node is deployed with Ansible on Bionic, but the old ORD host still
needs to be upgraded and moved out of puppet.  Instead of dealing with
the unmaintained bup and getting it to work on the current LTS Focal,
we are doing an initial borg deployment with plans to switch to it
globally.

This adds the backup02.ca-ymq-1.vexxhost.opendev.org to the inventory
and borg-backup-server group, so it will be deployed as a borg backup
server (note, no hosts are backing up to it, yet).

To avoid the original bup roles matching, we restrict the
backup-server group to backup01.ca-ymq-1.vexxhost.opendev.org
explicitly.

Change-Id: Id30a2ffad75236fc23ed51b2c67d0028da988de5
2020-10-08 11:54:27 +11:00
Zuul
4ed23af8fe Merge "Build images for gerrit 3.1 and 3.2" 2020-10-07 21:57:43 +00:00
Zuul
eb263527ea Merge "Mirror the only Fedora Atomic image used by Magnum" 2020-10-07 20:21:04 +00:00
Clark Boylan
1008a1fc90 Build images for gerrit 3.1 and 3.2
This will allow us to test further gerrit upgrades while we sort out how
far into the gerrit releases we will be upgrading to on our next
upgrade.

Change-Id: Ic9d07b76e41ad4262cc0e2e1ff8a5d554f88239e
2020-10-06 15:37:06 -07:00
Zuul
04cfc9fac3 Merge "Cleanup graphite01" 2020-10-06 22:03:23 +00:00
Zuul
f3bbc8171f Merge "gitea: fix 3081 proxy" 2020-10-06 20:55:00 +00:00
Zuul
30cd357408 Merge "Make gitea description update failures nonfatal" 2020-10-05 17:58:13 +00:00
Zuul
172bef3967 Merge "Exclude private dirs when mirroring fedora" 2020-10-05 17:24:23 +00:00
Zuul
81bebbf966 Merge "Record stacktraces when logging errors in gitea" 2020-10-02 21:48:35 +00:00
Zuul
870dd0ecd2 Merge "Use the new stable-2.16 branch of its-storyboard" 2020-10-02 15:39:17 +00:00
Ian Wienand
03727e4941 tarballs.opendev.org: better redirects
This matches the file, which got lost in my original script because I
didn't quote a $.  Also add some quotes for better grouping.

Change-Id: I335e89616f093bdd2f0599b1ea1125ec642515ba
2020-10-02 12:22:28 +10:00
Zuul
93dd7b59d5 Merge "tarballs: Add rewrite rules for tenant moves" 2020-10-02 00:43:54 +00:00
Zuul
083e8b43ea Merge "Add borg-backup roles" 2020-10-01 07:36:47 +00:00
Clark Boylan
512349a28d Make gitea description update failures nonfatal
There appears to be a gitea bug that causes PATCH updates to projects to
fail when the cache is in a bad state for that project. We use PATCH
updates to a project to set the project descriptions. Since project
descriptions are not critical to gitea functionality (we weren't
updating them until last week) we can treat this as best effort and
ignore these failures.

We'll log these cases to aid in further debugging but continue on. The
next pass can retry.

Change-Id: I625bdc0856caaccb6b55931b0cdc6cf11a0bf3e1
2020-09-30 10:15:04 -07:00
Clark Boylan
941f746767 Record stacktraces when logging errors in gitea
Gitea has added a STACKTRACE_LEVEL config option to set which log level
will also generate stack traces when logging. We want them for at least
Error and above so set this to Error for now. In particular there seems
to be a commit cache issue which results in errors that having stack
traces for would be helpful to debug.

Change-Id: I0491373ef143dfa753c011d02e3c670c699d2a52
2020-09-30 10:04:49 -07:00