API Headaches

[mbrown]

Hi all,

I'm just getting DT set up for our organization and it's going pretty well. I have a multisite set up underway.

One thing I'm struggling with is that we did some prototyping on a different hosting group (before I got involved) and we want to move all the contacts over from that site. We are really struggling to figure out how to get it done without a lot of manual work.

I've dug into the documentation and for me I would love to simply write a python or whatever program to move things exactly the way we want them, but I can't seem to get the API working. I've set up a Site Link and am trying to use the token. I know I'm using the correct token, as I get the "invalid token" message when I mess it up, but now I'm getting the message "No permissions to read contacts" when just trying to read a contact - e.g. https://staff.equipleaders.org/dt/wp-json/dt-posts/v2/contacts/10

The thing is that the doc is a bit vague on this. It says that to do this, I need permission "access_{post_type}" but there is no such permission in the connection type dropdown and every option I have there doesn't seem to work, I just get this same permission error.

Any pointers, examples, etc would be greatly appreciated!

[Corsac]

Hey @mbrown,

Thanks for messaging! We love the mission and vision of https://equipleaders.org!
A huge congrats on exploring the API docs and getting authentication working.

I'll start off with a couple thoughts about migrating without using the API. In case that would prove easier.

• A basic import export could be used using the CSV export on the list and then importing the csv using the Import plugin. This solution does have limitations as it won't copy over comments or connection fields.

• We've used this plugin to migrate successfully between Wordpress multisite installs: https://deliciousbrains.com/wp-migrate-db-pro/

Now for using the API.

• The site link system was initially designed to enable creating contacts from outside D.T, like from the webform plugin. You are right that none of the default options allow for reading contacts. I've just played with adding a "all permissions" option. We'll need some time to think through the security implications, but you could go ahead and download that branch and use the new option. Github Pull Request: https://github.com/DiscipleTools/disciple-tools-theme/pull/2554.
  You'll need to go to the site links and select the "All permissions" connection type.
  Please let us know if this works as expected if you try it out.

• Another option would be to use a JTW token authentication with your user instead of the site link. This will let you do any API call your user has permissions to do.
  JWT docs.

Hope this lets you move forward. Please post again if you run into any issues.
Also we'd love any help or suggestions on improving the dev docs to make it simpler for the next person.

[mbrown]

Ok, that's very helpful.

I went ahead and pulled the dev repo and took a look at the changes, but I'm a bit leery of pushing the dev repo into a production use case at the moment.

I think I'm going to tackle this on two fronts. First, I'll investigate the JWT method. I had read up on that a bit but it threw me with the authenticaiton use of the v1 REST API vs v2, it was unclear if the v2 API calls would accept the v1 JWT tokens; from what you are saying I gather it will.

If that doesn't work, I'll grab a dev copy and put it in as a separate template. That said, I have a suggestion for your "develop" branch repo. I suggest changing the style.css:

Theme Name: Disciple Tools (DEV)

or something similar, then have a mechanism that changes it back when you push it to dev. That way, if I clone the dev repo into the wp-content/themes directory, I can do a:

git clone <url> disciple-tools-dev

That will allow users to easily see and flip between the dev version and the production version without having the mess of continuously copying updates and re-customizing style.css.

Another suggestion I would have is that an "all permissions" is perhaps not needed. What I find odd is that I was unable to use a get call on a specific contact instance even with the unidirectional transfer permission. That's what made me confused. It seems that such a permission implies read? Or is it write only and the other side gathers the information internally and sends it?

Ultimately I think a read contact permission would be enormously helpful. I would guess that an all permissions would be useful for development purposes but in real life use I would basic read or read/write would be common.

[mbrown]

Ok, well. It works!!!

I got it all to work on the command line with curl commands.

But then I tried to convert it to python and was stymied for a couple of hours. I couldn't figure out why but every request bombed with python.

I don't know exactly why, but the API seems to be very sensitive to the User-Agent coming across. It does NOT like the default user-agent injected by the python requests module. So, since curl worked, I forced it to use the curl UA.

Bash Example:

#!/usr/bin/env bash

function get_token() {
    url="${1}/wp-json/jwt-auth/v1/token"
    user="$2"
    passwd="$3"
    printf -v auth '{"username": "%s", "password": "%s"}' "$user" "$passwd"
    curl -s -X post -d "$auth" -H 'Content-Type: application/json' "$url" || exit 1
}

function list_contacts() {
    url="${1}/wp-json/dt-posts/v2/contacts"
    token="$2"
    curl -s -X get -H 'Content-Type: application/json' -H 'Accept: application/json' -H "Authorization: Bearer $token" "$url"
}
i='https://staff.equipleaders.org/dt'
u='eliadmin'
p='your-password-here'
t=$(get_token "$i" "$u" "$p"|jq -r .token)
r=$(list_contacts "$i" "$t")
echo $r|jq

Python Example:

#!/usr/bin/env python3
import requests
import json
import logging

def get_token(url, user, passwd):
    url = f"{url}/wp-json/jwt-auth/v1/token"
    headers = {
        'Content-Type': 'application/json',
        'Accept': 'application/json',
        'User-Agent': 'curl/7.68.0'
    }
    auth = {"username": user, "password": passwd}

    response = requests.post(url, json=auth, headers=headers)

    response.raise_for_status()  # Handle HTTP errors
    return response.json().get('token')

def list_contacts(url, token):
    url = f"{url}/wp-json/dt-posts/v2/contacts"
    headers = {
        'Authorization': f'Bearer {token}',
        'Content-Type': 'application/json',
        'Accept': 'application/json',
        'User-Agent': 'curl/7.68.0'
    }
    response = requests.get(url, headers=headers)
    response.raise_for_status()  # Handle HTTP errors
    return response.json()

i = 'https://staff.equipleaders.org/dt'
u = 'eliadmin'
p = 'your-password-goes-here'
t = get_token(i, u, p)
r = list_contacts(i, t)
print(json.dumps(r, indent=4))

[Corsac]

So glad you got it working @mbrown!

Most of our dev has been in php and js so I'm not sure why the python script didn't work.

Thanks for your suggestions on the dev branch.
Have a look at https://wppusher.com/, it allows installing themes and plugins from github with easy switching between branches. We use it a lot when testing new features.

For the sites links. None of the default connection types allow for reading.
So adding a read contacts option is a great idea.
All permissions makes sense if you need to work with multiple record types: contacts and groups (or other record types).
What would probably be best here is changing the system to use a checkbox list of permission so the admin can granularly set which permissions they want enabled. [ read_contacts, create_contacts, update_contacts, read_groups, etc ]

[mbrown]

@Corsac

Ultimately, the JWT method works great. Feel free to publish the above examples if you want. I've done a chunk more development though... as always, the code is self documenting: code tells the computer what to do, so it should tell us what it's doing too (in other words, I've not added clear documentation yet but to a python programmer this should be self-evident). I don't really see a need to go back to the other method other than curiosity if you do eventually add a read scope. IMO that would be safer for things like reporting tools, as JWT inherits the capabilities of the user.

Here's my more fully fleshed out starter code, converted to a class structure.

#!/usr/bin/env python3
import argparse
import getpass
import os,sys
import requests
import json
import logging

class DiscipleTools:
    def __init__(self):
        interactive=False
        self.token = None
        self.instance = None
        self.username = None
        self.password = None

        self.set_logging()

        self.session=requests.session()
        self.session.headers.update({
            'Content-Type': 'application/json',
            'Accept': 'application/json',
            'User-Agent': 'curl/*'
        })

        parser = argparse.ArgumentParser()
        parser.add_argument("--token", "-t", help="JWT Token")
        parser.add_argument("--username", "-u", help="WordPress Username")
        parser.add_argument("--password", "-p", help="WordPress Password")
        parser.add_argument("--interactive", "-I", action="store_true", help="Interactive mode, prompt for credentials if needed")
        parser.add_argument("instance", help="WordPress instance URL")

        args = parser.parse_args()
        self.token = args.token
        self.username = args.username
        self.password = args.password
        self.instance = args.instance
        if args.interactive:
            interactive=True

        # Check environment variables if properties are still None
        self.token = self.token or os.getenv('WP_JWT_TOKEN')
        self.username = self.username or os.getenv('WP_USER')
        self.password = self.password or os.getenv('WP_PASS')
        self.instance = self.instance or os.getenv('WP_INSTANCE')

        # Interactive prompt if token is still None and interactive is True
        if interactive and self.token is None:
            self.username = self.username or input("Enter username: ")
            self.password = self.password or getpass.getpass("Enter password: ")

        # Get token from API if token is still None and username and password are available
        if self.token is None and self.username and self.password:
            self.get_token()
        if self.token is not None:
            self.session.headers.update({ 'Authorization': f'Bearer {self.token}' })

    def set_logging(self,level=logging.WARNING):
        logging.basicConfig(
            level=level,
            format='%(asctime)s - %(levelname)s - %(message)s',
            stream=sys.stderr
        )
        self.loglevel=level

    def get_token(self):
        '''
        If we don't have a token passed via ENV or arguments, get one using the specified username/password via args, ENV, or, if interactive, prompted
        '''
        url = f"{self.instance}/wp-json/jwt-auth/v1/token"
        auth = {"username": self.username, "password": self.password}
        response = self.session.post(url, json=auth)
        response.raise_for_status()  # Handle HTTP errors
        # we have a token, lets store it and adjust headers for other methods
        self.token=response.json().get('token')
        return True

    def list_contacts(self, contact = None):
        '''
        lists contacts from the Disciples Tools repository.
        If nothing is passed, it returns a json struct in self.data containing an array of all contacts.
        If a single contact number is passed, it returns the json struct for that contact.
        If an array of contact numbers is passed, it returns an array of the contacts requested.

        '''
        base_url = f"{self.instance}/wp-json/dt-posts/v2/contacts"
        if contact is None:   # return all contacts
            response=self.session.get(base_url)
            response.raise_for_status() # dump on HTTP errors
            self.data=response.json().get('posts')
        elif isinstance(contact,int):
            url = f'{base_url}/{contact}'
            response = self.session.get(url)
            if response.status_code == 200:
                self.data=response.json()
            elif response.status_code == 403:
                logging.warning(f"Contact id: {contact} not found.")
                self.data=None
                return False
            else:
                response.raise_for_status()  # Handle HTTP errors
        elif isinstance(contact,list):
            self.data=[]
            for contact_id in contact:
                if isinstance(contact_id,int):
                    url = f'{base_url}/{contact_id}'
                    response=self.session.get(url)
                    if response.status_code == 200:
                        self.data.append(response.json())
                    elif response.status_code == 403:
                        logging.warning(f"Contact id: {contact_id} not found.")
                        # this happens when an id code doesn't exist
                        continue
                    else:
                        response.raise_for_status()  # Handle HTTP errors
                else:
                    logging.warning(f"Invalid Contact id: {contact_id} is type {type(contact_id)}")
            return True
        else:
            logging.error(f"Invalid Contact id: {contact} is type {type(contact)}")
            return False
        return True

'''
Example usage so far
'''

app=DiscipleTools()

if app.token:
    if app.list_contacts():
        print(json.dumps(app.data,indent=4))
    print(app.token)

[Corsac]

Thanks @mbrown!