Three Steps in Doing Research with Generative AI Tools Such as GPT-4o and CoPilot
Case Study: Was There A Petrodollar Agreement Signed 50 Years Ago or Not?
Generative AI in aid of the search for the petrodollar agreement
Generative AI (GenAI) is all the rage. Dominating headlines ever since ChatGPT was launched on November 30, 2022, it has swept up the world in a whirlwind of excitement. Although doubts linger about its accuracy given all the hallucination issues users have experienced, many have gotten used to using it in a variety of support functions. In this article, I will share an example of how I use it as a research assistant.
A timely topic that has risen in Google searches this month is whether the petrodollar agreement the U.S. and Saudi Arabia signed 50 years ago expired. According to a MarketWatch article on June 15, “Stories about the collapse of a longstanding ‘petrodollar’ agreement between the U.S. and Saudi Arabia spread like wildfire on social media. But the agreement never existed.”
As an exercise of using AI in research, I’ll demonstrate how to investigate if the agreement actually existed. I will show you how you can find the answer from credible sources that are not covered by your usual go-to data and news vendors. And you can do so far quicker than someone not using AI tools. Or in popular lingo today, how you can improve productivity with ease. Spoiler alert: we found evidence that the agreement did exist.
(A word of caution: this is just that, an exercise to investigate a historical fact. This is NOT an overall assessment of the petrodollar situation today, particularly the impact of its demise or not on the financial markets. For that, I have a book that I can recommend.)
Step 0. Take a minute to think through the process
Always start with a purpose and define a specific objective. Break down the issues if they are complex and feed the questions into your AI tool one by one where necessary. This is not only because AI tools are built on human data so that they can be lazy and push back like humans, each interaction with the tool also only gets so much resources allocated to it by the provider given the substantial cost involved in operating these models.
Our objective in this case, as outlined earlier, is simple enough so I wanted to ask the AI assistant to do it in three steps: scan for hopefully credible sources that support the existence of a petrodollar agreement signed 50 years ago and those that reject the idea, have it read through the sources and find evidence that is as direct as possible in support or reject the fact, have it compare the final batch of evidence collected and check for consistency.
It is certainly conceivable that a fourth step can be added: compile the research report. As AI tools mature and our skills for using them improve, I am sure that will become more common practice. For this exercise we are not doing so though for the sake of simplicity (and a variety of other considerations).
Step 1. Scan online sources with CoPilot to establish a baseline
There are also two points worth noting in this stage:
Why CoPilot? Isn’t it powered by OpenAI’s ChatGPT series of models including the latest GPT-4o?
Yes, indeed. That said, CoPilot provides sources of its answers systematically while ChatGPT only does so if you insist, and grudgingly at best. The advantage of having sources is two folds: First, it is helpful for research documentation. This is what your research assistant would do and it is simply best practice in doing research. More importantly, this helps you check its sources so that you can be better informed in terms of the answers’ credibility and reduce the issue with hallucination.
What conversation style should I use on CoPilot?
The Large Language Models (LLM) such as ChatGPT has a built-in switch called “temperature”, which controls how much liberty the model takes on the data in coming up with an answer. CoPilot has limited the choices of temperature to three levels. Since we want to be 100% accurate with our facts and cannot tolerate the model to daydream about any conspiracy theories, choose Precise, which should be the default for all research functions except for those rare moments when you may want to show some flair in your language when compiling a research report.
Once those are decided, I typed in the simple question “was there a petrodollar agreement signed 50 years ago that required Saudi Arabia to price oil only in US dollars?”
And immediately, CoPilot responded with yes and listed 5 sources. Four of them I had never heard of, and the one source I did know was simply part of their news aggregation from another source that I did not know. So indeed it looked like what MarketWatch reported: there is no mainstream coverage. And you see the importance of understanding the sources of your AI tool’s answers.
Since this is an event that took place 50 years ago, I asked CoPilot to focus on history: “Look through documents and reports from the 1970s, was there a petrodollar agreement signed then that required Saudi Arabia to price oil only in US dollars?”
This time it came back with some of the same sources but also offered up a couple of new ones including a peer-reviewed energy history journal.
Step 2. Ask CoPilot or ChatGPT to delve deeper into the sources you identified as more relevant and credible
The paper is 30 some paragraphs long and offers 78 notes and a tantalizing list of references. To expedite the process, I simply added the URL into my question and ask the AI tool to answer the question based on this article.
Reading is what these tools are designed for and I have not detected if either one is better at it than the other. I used GPT-4o in this case as an illustration. Below is its answer.
So, just like that, you have found the answer that has evaded so many of your peers.
Before declaring victory, you should of course read the article carefully, especially the paragraphs directly linked to the statement and references the author cited to support that fact, such as the following (see below). Those are of course what you are really after and the AI tools have simply provided you with a treasure map.
Step 3. Check additional sources for consistency
For high stakes research, you always want to have multiple sources that can back each other up. This is, of course, time-consuming. The good news is you can now often do that with your AI tools in minutes if not seconds, provided these are public sources, or private sources that you have licensed to use.
In this case I was fortunate enough to learn that there was a book written on the subject and the stories all checked out. In reality, this was probably one of many books written, but a reference suggested by a trusted source adds a different layer of credibility as it is more likely to be completely independent of the AI tools and their training dataset.
There you have it.
GenAI tools can come in extremely handy and give you a leg up in the competitive world of research.