Long story short, my master list of contacts are in iCloud, not a selfhosted CardDAV provider. This is how I access them on other devices.
App-specific passwords
iCloud requires app-specific passwords for most things instead of your primary password. To create them go to sign in to your Apple ID ( I had to do this on my iOS device because it refeused to log me in via any other browser )
Scroll to Security, click "Generate Password…", enter a specific label such as "$DEVICE_NAME - $ APP_NAME" and click create. Take note of the password but I don't add them to my password manager, if I ever need to reauthenticate I revoke the current password and create a new one.
Connecting Android to iCloud contacts
To use CardDAV an Android device you'll need DavDroid F-Droid Google Play.
Once running the DavDroid click add account, and "Login with URL and user name"
the "Base URL" is https://contacts.icloud.com the "User name"
is your email/Apple ID and "Password" is the app-specific password.
You'll also need to enable the adress book if it hasn't automatically.
Now your contacts will show up in any contacts app on Android.
Connecting GNOME to iCloud contacts
GNOME Online Accounts can connect to many providers but iCloud or CardDAV isn't one. Instead we will have to use Evolution to add out account.
Open Evolution and go to the contacts section, click the drop down next to "New"
then "Adress Book". Change the type to CardDAV, put the "URL" to
https://contacts.icloud.com and "User" to your email/Apple ID and
click "Find Address Books". You will now need your app-specific password.
Select your adress book and click "OK", you can now set a "Name" and click "OK"
again.
Now GNOME Contacts, and any app that uses Evolution Data Server should show your contacts.