{
"results": [
{
"id": "https://exa.ai",
"url": "https://exa.ai/",
"title": "Exa API",
"author": "exa",
"text": "AIs need powerful access to knowledge. But search engines haven't improved since 1998. API features built for AI apps FLEXIBLE SEARCH Handle any type of query For queries that need semantic understanding, search with our SOTA web embeddings model\nover our custom index. For all other queries, we offer keyword-based search. PAGE CONTENT Instantly retrieve the content from any page Stop learning how to web scrape or parse HTML. Get the clean, full text of any page in our\nindex, or intelligent embeddings-ranked\nhighlights related to a query. CUSTOMIZABILITY Apply filters to search over your own subset of the web Select any date range, include or exclude any domain, select a custom data vertical, or get up to 10 million results. SIMILARITY SEARCH Search the web using examples of what you’re looking for Your query can even be a URL or multiple paragraphs on a webpage. Explore the documentation",
"image": "https://exa.imgix.net/og-image.png",
"subpages": [
{
"id": "https://docs.exa.ai/reference/getting-started",
"url": "https://docs.exa.ai/reference/getting-started",
"title": "Getting Started",
"author": "",
"text": "Exa provides search for AI. \n Exa is a knowledge API for LLMs..\n Search\nUsing the /search endpoint, your LLM can search using natural language and return a list of relevant webpages from our neural database. Exa's neural search allows your LLM to query in natural language. And if a query doesn't benefit from neural search, Exa also supports traditional Google-style keyword search.\n Contents\nRight alongside the search results, you can obtain clean, up-to-date HTML content, no web crawling or scraping required. We also support returning highlights from pages - highly intelligent extracts calculated using RAG models.\nSee \"Examples\" to see Exa in action. Or just ask one of our GPTs how to build what you want!\n 🔑 Getting a Exa API Key Exa is free to use up to 1000 requests per month, for individual developers. Get an API key here. \nThere's no need to learn how to use our API all alone. Below are ChatGPT GPTs that you can ask about how to implement anything.\nFor Python SDK assistance, go here.\nFor TypeScript SDK assistance, go here.\nFor any other language, go here.\n # pip install exa_py\nfrom exa_py import Exa\nexa = Exa(\"EXA_API_KEY\")\nresults = exa.search('hottest AI agent startups', use_autoprompt=True)\n \n Recent News Summarizer \n Exa Researcher \n Basic search \n Basic link similarity \n Basic content retrieval \n Table of Contents \n What is Exa? \n Getting Access \n GPT-assisted implementation \n A simple example \n More examples"
},
{
"id": "https://docs.exa.ai/reference/recent-news-summarizer",
"url": "https://docs.exa.ai/reference/recent-news-summarizer",
"title": "Recent News Summarizer",
"author": null,
"publishedDate": "2024-03-02T11:36:31.000Z",
"text": "https://docs.exa.ai/reference/recent-news-summarizer\nRecent News Summarizer\n2024-03-02T11:36:31Z\n \nIn this example, we will build a LLM-based news summarizer app with the Exa API to keep us up-to-date with the latest news on a given topic.\nThis Jupyter notebook is available on Colab for easy experimentation. You can also check it out on Github, including a plain Python version if you want to skip to a complete product.\nTo play with this code, first we need a Exa API key and an OpenAI API key. Get 1000 Exa searches per month free just for signing up!\n # install Exa and OpenAI SDKs\n!pip install exa_py\n!pip install openai\n \n from google.colab import userdata # comment this out if you're not using Colab\nEXA_API_KEY = userdata.get('EXA_API_KEY') # replace with your api key, or add to Colab Secrets\nOPENAI_API_KEY = userdata.get('OPENAI_API_KEY') # replace with your api key, or add to Colab Secrets\n \nFirst, let's try building the app just by using the OpenAI API. We will use GPT 3.5-turbo as our LLM. Let's ask it for the recent news, like we might ask ChatGPT.\n import openai\nopenai.api_key = OPENAI_API_KEY\nUSER_QUESTION = "What's the recent news in physics this week?"\ncompletion = openai.chat.completions.create(\nmodel="gpt-3.5-turbo",\nmessages=[\n{"role": "system", "content": "You are a helpful assistant."},\n{"role": "user", "content": USER_QUESTION},\n],\n)\nresponse = completion.choices[0].message.content\nprint(response)\n \n One recent news in physics is that researchers at the University of Illinois have discovered a new topological state of matter. They created a material composed of interacting particles called quadrupoles, which can exhibit unique behavior in their electrical properties. This finding has the potential to pave the way for the development of new types of electronic devices and quantum computers.\nAnother interesting development is in the field of cosmology. The European Space Agency's Planck satellite has provided new insights into the early universe. By analyzing the cosmic microwave background radiation, scientists have obtained more accurate measurements of the rate at which the universe is expanding, which could challenge current theories of physics.\nAdditionally, scientists at CERN's Large Hadron Collider (LHC) have observed a rare phenomenon called charm mixing. They found that particles containing both charm and strange quarks can spontaneously transition between their matter and antimatter states. This discovery could contribute to our understanding of the puzzle of why the universe is primarily made of matter and why there is very little antimatter.\n \nOh no! Since the LLM is unable to use recent data, it doesn't know the latest news. It might tell us some information, but that info isn't recent, and we can't be sure it's trustworthy either since it has no source. Luckily, Exa API allows us to solve these problems by connecting our LLM app to the internet. Here's how:\nLet's use the Exa neural search engine to search the web for relevant links to the user's question.\nFirst, we ask the LLM to generate a search engine query based on the question.\n import openai\nfrom exa_py import Exa\nopenai.api_key = OPENAI_API_KEY\nexa = Exa(EXA_API_KEY)\nSYSTEM_MESSAGE = "You are a helpful assistant that generates search queries based on user questions. Only generate one search query."\nUSER_QUESTION = "What's the recent news in physics this week?"\ncompletion = openai.chat.completions.create(\nmodel="gpt-3.5-turbo",\nmessages=[\n{"role": "system", "content": SYSTEM_MESSAGE},\n{"role": "user", "content": USER_QUESTION},\n],\n)\nsearch_query = completion.choices[0].message.content\nprint("Search query:")\nprint(search_query)\n \n Search query:\nRecent news in physics this week\n \nLooks good! Now let's put the search query into Exa. Let's also use start_published_date to filter the results to pages published in the last week:\n from datetime import datetime, timedelta\none_week_ago = (datetime.now() - timedelta(days=7))\ndate_cutoff = one_week_ago.strftime("%Y-%m-%d")\nsearch_response = exa.search_and_contents(\nsearch_query, use_autoprompt=True, start_published_date=date_cutoff\n)\nurls = [result.url for result in search_response.results]\nprint("URLs:")\nfor url in urls:\nprint(url)\n \n URLs:\nhttps://phys.org/news/2024-01-carrots-reveals-mechanics-root-vegetable.html?utm_source=twitter.com&utm_medium=social&utm_campaign=v2\nhttps://phys.org/news/2024-01-astrophysicists-theoretical-proof-traversable-wormholes.html\nhttps://gizmodo.com/proton-physics-strong-force-quarks-measurement-1851192840\nhttps://www.nytimes.com/2024/01/24/science/space/black-holes-photography-m87.html\nhttps://phys.org/news/2024-01-liquid-lithium-walls-fusion-device.html?utm_source=twitter.com&utm_medium=social&utm_campaign=v2\nhttps://physics.aps.org/articles/v17/s13\nhttps://phys.org/news/2024-01-validating-hypothesis-complex.html?utm_source=twitter.com&utm_medium=social&utm_campaign=v2\nhttps://phys.org/news/2024-01-scientists-previously-unknown-colonies-emperor.html?utm_source=twitter.com&utm_medium=social&utm_campaign=v2\nhttps://phys.org/news/2024-01-reveals-quantum-topological-potential-material.html?utm_source=twitter.com&utm_medium=social&utm_campaign=v2\nhttps://phys.org/news/2024-01-shallow-soda-lakes-cradles-life.html?utm_source=twitter.com&utm_medium=social&utm_campaign=v2\n \nNow we're getting somewhere! Exa gave our app a list of relevant, useful URLs based on the original question.\nBy the way, we might be wondering what makes Exa special. Why can't we just search with Google? Well, let's take a look for ourselves at the Google search results. It gives us the front page of lots of news aggregators, but not the news articles themselves. And since we used Exa's search_and_contents, our search came with the webpage contents, so can use Exa to skip writing a web crawler and access the knowledge directly!\n results = search_response.results\nresult_item = results[0]\nprint(f"{len(results)} items total, printing the first one:")\nprint(result_item.text)\n \n 10 items total, printing the first one:\nCredit: CC0 Public Domain\nChopped carrot pieces are among the most universally enjoyed foods and a snacking staple—a mainstay of school lunchboxes, picnics and party platters year-round.\nNow researchers from the University of Bath have uncovered the secret science of prepping the popular root vegetable and quantified the processes that make them curl up if left uneaten for too long.\nMechanical Engineering student Nguyen Vo-Bui carried out the research as part of his final-year studies, in the limited circumstances of COVID-19 lockdowns of 2021. The research paper, "Modelling of longitudinally cut carrot curling induced by the vascular cylinder-cortex interference pressure", is published in Royal Society Open Science.\nWithout access to labs, Nguyen aimed to identify the geometrical and environmental factors that have the most influence on carrots' longevity. Working in his kitchen, he characterized, analytically modeled and verified the aging of over 100 Lancashire Nantes carrot halves, cut lengthways, using finite-element (FE) models normally used in structural engineering.\nThe research team concluded that residual stresses and dehydration were the two key factors behind the curling behavior. The starchy outer layer of the carrot (the cortex) is stiffer than the soft central vein (also known as the vascular cylinder). When cut lengthwise, the two carrot halves curl because the difference in stress becomes unbalanced. Dehydration leads to further loss of stiffness, further driving the curling effect.\nTheir recommendations to manufacturers include handling carrots in cold, moist, airtight and humidity-controlled environments to protect their natural properties and increase their edible life span.\nThey say the study provides a methodology to predict the deformation of cut root vegetables, adding that the procedure is likely to apply to other plant structures. The study gives food producers a new mathematical tool that could be applied to the design of packaging and food handling processes, potentially reducing food waste.\nOne of the world's top crops by market value, carrots are known for their high production efficiency—but despite this, wastage is high. Around 25–30% of this occurs prior to processing and packaging—due to deformities, mechanical damage or infected sections. Fresh cut and minimally processed carrots are a convenient ready-to-use ingredient that make possible the use of carrots that might otherwise be discarded, reducing food waste.\nDr. Elise Pegg, a senior lecturer in Bath's Department of Mechanical Engineering, is one of the research paper authors and oversaw the study. She said, "We have mathematically represented the curl of a cut carrot over time, and showed the factors that contribute to curling.\n"Our motivation was to look for ways to improve the sustainability of carrot processing and make them as long-lasting as possible. We have produced a methodology that a food producer could use to change their processes, reducing food waste and making packaging and transportation more efficient. Understanding the bending behavior in such systems can help us to design and manufacture products with higher durability.\n"A question like this would normally be investigated from a biological perspective, but we have done this work using purely mechanical principles. I'm so pleased for Nguyen—it's a measure of his resourcefulness and dedication to produce such interesting research in a challenging situation."\nOver the course of a week, the curl of the carrot halves increased—with the average radius of each carrot's curvature falling from 1.61m to 1.1m. A 1.32-times reduction in stiffness was also seen, correlating with the carrots drying out; on average, their weight fell by 22%.\nNguyen added, "This was interesting research—to apply mechanical principles to vegetables was surprising and fun.\n"One of the big challenges was to devise an experiment that could be done in a lockdown setting, without access to normal labs and equipment. To now be in a position to have this work published in an academic journal and potentially be used by the food industry is really rewarding.\n"This project has inspired me to continue my studies at the University of Bath and I now study residual stresses in porous ferroelectric ceramics for my Ph.D."\nAs well as having to use a suitcase to collect the 30kg of carrots the experiment demanded from a farmers' market, a further challenge was finding ways to use them afterward. Carrot cake, the Indian carrot dessert Gajar Ka Halwa, carrot pesto and many other dishes kept Nguyen and his flatmates fed for several days.\nMore information:\nModelling of longitudinally cut carrot curling induced by the vascular cylinder-cortex interference pressure, Royal Society Open Science (2024). DOI: 10.1098/rsos.230420. royalsocietypublishing.org/doi/10.1098/rsos.230420\nCitation:\nWhy do carrots curl? Research reveals the mechanics behind root vegetable aging (2024, January 23)\nretrieved 24 January 2024\nfrom https://phys.org/news/2024-01-carrots-reveals-mechanics-root-vegetable.html\nThis document is subject to copyright. Apart from any fair dealing for the purpose of private study or research, no\npart may be reproduced without the written permission. The content is provided for information purposes only.\n \nAwesome! That's really interesting, or it would be if we had bothered to read it all. But there's no way we're doing that, so let's ask the LLM to summarize it for us:\n import textwrap\nSYSTEM_MESSAGE = "You are a helpful assistant that briefly summarizes the content of a webpage. Summarize the users input."\ncompletion = openai.chat.completions.create(\nmodel="gpt-3.5-turbo",\nmessages=[\n{"role": "system", "content": SYSTEM_MESSAGE},\n{"role": "user", "content": result_item.text},\n],\n)\nsummary = completion.choices[0].message.content\nprint(f"Summary for {urls[0]}:")\nprint(result_item.title)\nprint(textwrap.fill(summary, 80))\n \n Summary for https://phys.org/news/2024-01-carrots-reveals-mechanics-root-vegetable.html?utm_source=twitter.com&utm_medium=social&utm_campaign=v2:\nWhy do carrots curl? Research reveals the mechanics behind root vegetable aging\nResearchers from the University of Bath have conducted a study on the curling\nbehavior of chopped carrot pieces. The study found that residual stresses and\ndehydration were the main factors behind the curling effect. The starchy outer\nlayer of the carrot is stiffer than the soft central vein, and when cut\nlengthwise, the difference in stress causes the carrot to curl. Dehydration\nfurther contributes to the curling effect. The research provides recommendations\nto manufacturers on how to handle carrots to increase their edible lifespan. The\nstudy also offers a methodology that can be used to predict the deformation of\ncut root vegetables and potentially reduce food waste. The findings have\nimplications for the design of packaging and food handling processes. Carrots\nare a highly produced crop, but wastage is still high, with a significant amount\noccurring before processing and packaging. The study was carried out by\nMechanical Engineering student Nguyen Vo-Bui during the COVID-19 lockdowns of\n2021.\n \nAnd we're done! We built an app that translates a question into a search query, uses Exa to search for useful links, uses Exa to grab clean content from those links, and summarizes the content to effortlessly answer your question about the latest news, or whatever we want.\nWe can be sure that the information is fresh, we have the source in front of us, and we did all this with a Exa queries and LLM calls, no web scraping or crawling needed!\n With Exa, we have empowered our LLM application with the Internet. The possibilities are endless."
},
{
"id": "https://docs.exa.ai/reference/company-analyst",
"url": "https://docs.exa.ai/reference/company-analyst",
"title": "Company Analyst",
"author": null,
"publishedDate": "2024-03-02T11:36:42.000Z",
"text": "https://docs.exa.ai/reference/company-analyst\nCompany Analyst\n2024-03-02T11:36:42Z\n In this example, we'll build a company analyst tool that researches companies relevant to what you're interested in. If you just want to see the code, check out the Colab notebook.\nThe code requires an Exa API key and an OpenAI API key. Get 1000 Exa searches per month free just for signing up!\nSay we want to find companies similar to Thrifthouse, a platform for selling secondhand goods on college campuses. Unfortunately, googling “companies similar to Thrifthouse” doesn't do a very good job. Traditional search engines rely heavily on keyword matching. In this case we get results about physical thrift stores. Hm, that's not really what I want.\nLet’s try again, this time searching based on a description of the company, like by googling “community based resale apps.” But, this isn’t very helpful either and just returns premade listicles...\n What we really need is neural search.\nExa is a fully neural search engine built using a foundational embeddings model trained for webpage retrieval. It’s capable of understanding entity types (company, blog post, Github repo), descriptors (funny, scholastic, authoritative), and any other semantic qualities inside of a query. Neural search can be far more useful for these types of complex queries.\nLet's try Exa, using the Python SDK! We can use thefind_similar_and_contents function to find similar links and get contents from each link. The input is simply a URL, https://thrift.house and we set num_results=10(this is customizable up to thousands of results in Exa).\nBy specifying highlights={"num_sentences":2} for each search result, Exa will also identify and return a 2 sentence excerpt from the content that's relevant to our query. This will allow us to quickly understand each website that we find.\n !pip install exa_py\nfrom exa_py import Exa\nimport os\nEXA_API_KEY= os.environ.get("EXA_API_KEY")\nexa = Exa(api_key=EXA_API_KEY)\ninput_url = "https://thrift.house"\nsearch_response = exa.find_similar_and_contents(\ninput_url,\nhighlights={"num_sentences":2},\nnum_results=10)\ncompanies = search_response.results\nprint(companies[0])\n \nThis is an example of the full first result:\n [Result(url='https://www.mystorestash.com/',\nid='lMTt0MBzc8ztb6Az3OGKPA',\ntitle='The Airbnb of Storage',\nscore=0.758899450302124,\npublished_date='2023-01-01',\nauthor=None,\ntext=None,\nhighlights=["I got my suitcase picked up right from my dorm and didn't have to worry for the whole summer.Angela Scaria /Still have questions?Where are my items stored?"],\nhighlight_scores=[0.23423566609247845])]\n \nAnd here are the 10 titles and urls I got:\n # to just see the 10 titles and urls\nurls = {}\nfor c in companies:\nprint(c.title + ':' + c.url)\n \n rumie - College Marketplace:https://www.rumieapp.com/\nThe Airbnb of Storage:https://www.mystorestash.com/\nBunction.net:https://bunction.net/\nHome - Community Gearbox:https://communitygearbox.com/\nNOVA SHOPPING:https://www.novashoppingapp.com/\nRe-Fridge: Buy, sell, or store your college fridge - Re-Fridge:https://www.refridge.com/\nJamble: Social Fashion Resale:https://www.jambleapp.com/\nBranded Resale | Treet:https://www.treet.co/\nSwapskis:https://www.swapskis.co/\nEarn Money for Used Clothing:https://www.thredup.com/cleanout?redirectPath=%2Fcleanout%2Fsell\n \nLooks pretty darn good! As a bonus specifically for companies data, specifying category="company" in the SDK will search across a curated, larger companies dataset - if you're interested in this, let us know at [email protected] !\nNow that we have 10 companies we want to dig into further, let’s do some research on each of these companies.\nNow let's get more information by finding additional webpages about each company. To do this, we're going to do a keyword search of each company's URL. We're using keyword because we want to find webpages that exactly match the company we're inputting. We can do this with the search_and_contents function, and specify type="keyword" and num_results=5. This will give me 5 websites about each company.\n # doing an example with the first companies\nc = companies[0]\nall_contents = ""\nsearch_response = exa.search_and_contents(\nc.url, # input the company's URL\ntype="keyword",\nnum_results=5\n)\nresearch_response = search_response.results\nfor r in research_response:\nall_contents += r.text\n \nHere's an example of the first result for the first company, Rumie App. You can see the first result is the actual link contents itself.\n <div><div><div><div><p><a href="https://www.rumieapp.com/"></a></p></div><div><p>The <strong>key</strong> to <strong>your</strong> college experience. </p><p><br/>Access the largest college exclusive marketplace to buy, sell, and rent with other students.</p></div></div><div><h2>320,000+</h2><p>Users in Our Network</p></div><div><div><p><h2>Selling is just a away.</h2></p><p>Snap a pic, post a listing, and message buyers all from one intuitive app.</p><div><p></p><p>Quick setup and .edu verification</p></div><div><p></p><p>Sell locally or ship to other campuses</p></div><div><p></p><p>Trade with other students like you</p></div></div><div><p><h2>. From local businesses around your campus</h2></p><h4>Get access to student exclusive discounts</h4><p>rumie students get access to student exclusive discounts from local and national businesses around their campus.</p></div></div><div><p><h2>Rent dresses from </h2></p><p>Wear a new dress every weekend! Just rent it directly from a student on your campus.</p><div><p></p><p>Make money off of the dresses you've already worn</p></div><div><p></p><p>rumie rental guarantee ensures your dress won't be damaged</p></div><div><p></p><p>Find a new dress every weekend and save money</p></div></div><div><p><h2>. The only place to buy student tickets at student prices</h2></p><h4>Buy or Sell students Football and Basketball tickets with your campus</h4><p>rumie students get access to the first-ever student ticket marketplace. No more getting scammed trying to buy tickets from strangers on the internet.</p></div><div><div><div><p></p><h4>Secure</h4><p>.edu authentication and buyer protection on purchases.</p></div><div><p></p><h4>Lightning-fast</h4><p>Post your first listing in under a minute.</p></div><div><p></p><h4>Verified Students</h4><p>Trade with other students, not strangers.</p></div><div><p></p><h4>Intuitive</h4><p>List an item in a few simple steps. Message sellers with ease.</p></div></div><p><a href="https://apps.apple.com/us/app/rumie-college-marketplace/id1602465206">Download the app now</a></p></div><div><p><h2>Trusted by students.</h2></p><div><div><p></p><p>Saves me money</p><p>Facebook Marketplace and Amazon are great but often times you have to drive a long way to meet up or pay for shipping. rumie let’s me know what is available at my school… literally at walking distance. </p></div><div><p></p><p>5 stars!</p><p>Having this app as a freshman is great! It makes buying and selling things so safe and easy! Much more efficient than other buy/sell platforms!</p></div><div><p></p><p>Amazing!</p><p>5 stars for being simple, organized, safe, and a great way to buy and sell in your college community.. much more effective than posting on Facebook or Instagram!</p></div><div><p></p><p>The BEST marketplace for college students!!!</p><p>Once rumie got to my campus, I was excited to see what is has to offer! Not only is it safe for students like me, but the app just has a great feel and is really easy to use. The ONLY place I’ll be buying and selling while I’m a student.</p></div></div></div><div><p><h2>Easier to than GroupMe or Instagram.</h2></p><p>Forget clothing instas, selling groupme's, and stress when buying and selling. Do it all from the rumie app.</p></div></div></div>\n \nFinally, let's create a summarized report that lists our 10 companies and gives us an easily digestible summary of each company. We can input all of this web content into an LLM and have it generate a nice report!\n import textwrap\nimport openai\nimport os\nSYSTEM_MESSAGE = "You are a helpful assistant writing a research report about a company. Summarize the users input into multiple paragraphs. Be extremely concise, professional, and factual as possible. The first paragraph should be an introduction and summary of the company. The second paragraph should include pros and cons of the company. Things like what are they doing well, things they are doing poorly or struggling with. And ideally, suggestions to make the company better."\nopenai.api_key = os.environ.get("OPENAI_API_KEY")\ncompletion = openai.chat.completions.create(\nmodel="gpt-4",\nmessages=[\n{"role": "system", "content": SYSTEM_MESSAGE},\n{"role": "user", "content": all_contents},\n],\n)\nsummary = completion.choices[0].message.content\nprint(f"Summary for {c.url}:")\nprint(textwrap.fill(summary, 80))\n \n Summary for https://www.rumieapp.com/:\nRumie is a college-exclusive marketplace app that allows students to buy, sell,\nand rent items with other students. It has over 320,000 users in its network and\noffers features such as quick setup, .edu verification, local and campus-wide\nselling options, and exclusive discounts from local businesses. Students can\nalso rent dresses from other students, buy or sell student tickets at student\nprices, and enjoy secure and intuitive transactions. The app has received\npositive feedback from users for its convenience, safety, and effectiveness in\nbuying and selling within the college community.\nPros of Rumie include its focus on college students' needs, such as providing a\nsafe platform and exclusive deals for students. The app offers an intuitive and\nfast setup process, making it easy for students to start buying and selling.\nThe option to trade with other students is also appreciated. Users find it convenient\nthat they can sell locally or ship items to other campuses. The app's rental\nguarantee for dresses provides assurance to users that their dresses won't be\ndamaged. Overall, Rumie is highly regarded as a simple, organized, and safe\nplatform for college students to buy and sell within their community.\nSuggestions to improve Rumie include expanding its reach to more colleges and\nuniversities across the nation and eventually internationally. Enhancing\nmarketing efforts and fundraising can aid in raising awareness among college\nstudents. Additionally, incorporating features such as improved search filters\nand a rating/review system for buyers and sellers could enhance the user\nexperience. Continual updates and improvements to the app's interface and\nfunctionality can also ensure that it remains user-friendly and efficient.\n \nAnd we’re done! We’ve built an app that takes in a company webpage and uses Exa to \nDiscover similar startups\nFind information about each of those startups\nGather useful content and summarize it with OpenAI\nHopefully you found this tutorial helpful and are ready to start building your very own company analyst! Whether you want to generate sales leads or research competitors to your own company, Exa's got you covered."
},
{
"id": "https://docs.exa.ai/reference/exa-researcher",
"url": "https://docs.exa.ai/reference/exa-researcher",
"title": "Exa Researcher",
"author": null,
"publishedDate": "2024-03-02T11:36:30.000Z",
"text": "https://docs.exa.ai/reference/exa-researcher\nExa Researcher\n2024-03-02T11:36:30Z\n In this example, we will build Exa Researcher, a Javascript app that given a research topic, automatically searches for different sources about the topic with Exa and synthesizes the searched contents into a research report.\nThis interactive notebook was made with the Deno Javascript kernel for Jupyter. Check out the plain JS version if you prefer a regular Javascript file you can run with NodeJS, or want to skip to the final result. If you'd like to run this notebook locally, Installing Deno and connecting Deno to Jupyter is fast and easy.\nTo play with this code, first we need a Exa API key and an OpenAI API key. Get 1000 Exa searches per month free just for signing up! \nLet's import the Exa and OpenAI SDKs and put in our API keys to create a client object for each.\nMake sure to pick the right imports for your runtime and paste or load your API keys.\n // Deno imports\nimport Exa from 'npm:exa-js';\nimport OpenAI from 'npm:openai';\n// NodeJS imports\n//import Exa from 'exa-js';\n//import OpenAI from 'openai';\nconst EXA_API_KEY = "" // insert or load your API key here\nconst OPENAI_API_KEY = ""// insert or load your API key here\nconst exa = new Exa(EXA_API_KEY);\nconst openai = new OpenAI({ apiKey: OPENAI_API_KEY });\n \nSince we'll be making several calls to the OpenAI API to get a completion from GPT 3.5-turbo, let's make a simple utility wrapper function so we can pass in the system and user messages directly, and get the LLM's response back as a string.\n async function getLLMResponse({system = 'You are a helpful assistant.', user = '', temperature = 1, model = 'gpt-3.5-turbo'}){\nconst completion = await openai.chat.completions.create({\nmodel,\ntemperature,\nmessages: [\n{'role': 'system', 'content': system},\n{'role': 'user', 'content': user},\n]\n});\nreturn completion.choices[0].message.content;\n}\n \nOkay, great! Now let's starting building Exa Researcher. The app should be able to automatically generate research reports for all kinds of different topics. Here's two to start:\n const SAMA_TOPIC = 'Sam Altman';\nconst ART_TOPIC = 'renaissance art';\n \nThe first thing our app has to do is decide what kind of search to do for the given topic. \nExa offers two kinds of search: neural and keyword search. Here's how we decide:\nNeural search is preferred when the query is broad and complex because it lets us retrieve high quality, semantically relevant data. Neural search is especially suitable when a topic is well-known and popularly discussed on the Internet, allowing the machine learning model to retrieve contents which are more likely recommended by real humans. \nKeyword search is useful when the topic is specific, local or obscure. If the query is a specific person's name, and identifier, or acronym, such that relevant results will contain the query itself, keyword search may do well. And if the machine learning model doesn't know about the topic, but relevant documents can be found by directly matching the search query, keyword search may be necessary.\nSo, Exa Researcher is going to get a query, and it needs to automatically decide whether to use keyword or neural search to research the query based on the criteria. Sounds like a job for the LLM! But we need to write a prompt that tells it about the difference between keyword and neural search-- oh wait, we have a perfectly good explanation right there.\n // Let's generalize the prompt and call the search types (1) and (2) in case the LLM is sensitive to the names. We can replace them with different names programmatically to see what works best.\nconst SEARCH_TYPE_EXPLANATION = `- (1) search is usually preferred when the query is a broad topic or semantically complex because it lets us retrieve high quality, semantically relevant data. (1) search is especially suitable when a topic is well-known and popularly discussed on the Internet, allowing the machine learning model to retrieve contents which are more likely recommended by real humans.\n- (2) search is useful when the topic is specific, local or obscure. If the query is a specific person's name, and identifier, or acronym, such that relevant results will contain the query itself, (2) search may do well. And if the machine learning model doesn't know about the topic, but relevant documents can be found by directly matching the search query, (2) search may be necessary.\n`;\n \nHere's a function that instructs the LLM to choose between the search types and give its answer in a single word. Based on its choice, we return keyword or neural.\n async function decideSearchType(topic, choiceNames = ['neural', 'keyword']){\nlet userMessage = 'Decide whether to use (1) or (2) search for the provided research topic. Output your choice in a single word: either "(1)" or "(2)". Here is a guide that will help you choose:\\n';\nuserMessage += SEARCH_TYPE_EXPLANATION;\nuserMessage += `Topic: ${topic}\\n`;\nuserMessage += `Search type: `;\nuserMessage = userMessage.replaceAll('(1)', choiceNames[0]).replaceAll('(2)', choiceNames[1]);\nconst response = await getLLMResponse({\nsystem: 'You will be asked to make a choice between two options. Answer with your choice in a single word.',\nuser: userMessage,\ntemperature: 0\n});\nconst useKeyword = response.trim().toLowerCase().startsWith(choiceNames[1].toLowerCase());\nreturn useKeyword ? 'keyword' : 'neural';\n}\n \nLet's test it out:\n console.log(SAMA_TOPIC, 'expected: keyword, got:', await decideSearchType(SAMA_TOPIC));\nconsole.log(ART_TOPIC, 'expected: neural, got:', await decideSearchType(ART_TOPIC));\n \n Sam Altman expected: keyword, got: keyword\nrenaissance art expected: neural, got: neural\n \nGreat! Now we have to craft some search queries for the topic and the search type. There are two cases here: keyword search and neural search. Let's do the easy one first. LLMs already know what Google-like keyword searches look like. So let's just ask the LLM for what we want:\n function createKeywordQueryGenerationPrompt(topic, n){\nreturn `I'm writing a research report on ${topic} and need help coming up with Google keyword search queries.\nGoogle keyword searches should just be a few words long. It should not be a complete sentence.\nPlease generate a diverse list of ${n} Google keyword search queries that would be useful for writing a research report on ${topic}. Do not add any formatting or numbering to the queries.`\n}\nconsole.log(await getLLMResponse({\nsystem: 'The user will ask you to help generate some search queries. Respond with only the suggested queries in plain text with no extra formatting, each on it\\'s own line.',\nuser: createKeywordQueryGenerationPrompt(SAMA_TOPIC, 3),\n}));\n \n Sam Altman biography\nY Combinator founder\nInvestments made by Sam Altman\n \nThose are some good ideas!\nNow we have to handle the neural Exa search. This is tougher: you can read all about crafting good Exa searches here. But this is actually a really good thing: making the perfect Exa search is hard because Exa is so powerful! Exa allows us to express so much more nuance in our searches and gives us unparalleled ability to steer our search queries towards our real objective.\nWe need to our app to understand our goal, what Exa is, and how to use it to achieve the goal. So let's just tell the LLM everything it needs to know.\n function createNeuralQueryGenerationPrompt(topic, n){\nreturn `I'm writing a research report on ${topic} and need help coming up with Exa keyword search queries.\nExa is a fully neural search engine that uses an embeddings based approach to search. Exa was trained on how people refer to content on the internet. The model is trained given the description to predict the link. For example, if someone tweets "This is an amazing, scientific article about Roman architecture: <link>", then our model is trained given the description to predict the link, and it is able to beautifully and super strongly learn associations between descriptions and the nature of the content (style, tone, entity type, etc) after being trained on many many examples. Because Exa was trained on examples of how people talk about links on the Internet, the actual Exa queries must actually be formed as if they are content recommendations that someone would make on the Internet where a highly relevant link would naturally follow the recommendation, such as the example shown above.\nExa neural search queries should be phrased like a person on the Internet indicating a webpage to a friend by describing its contents. It should end in a colon :.\nPlease generate a diverse list of ${n} Exa neural search queries for informative and trustworthy sources useful for writing a research report on ${topic}. Do not add any quotations or numbering to the queries.`\n}\nconsole.log(await getLLMResponse({\nsystem: 'The user will ask you to help generate some search queries. Respond with only the suggested queries in plain text with no extra formatting, each on it\\'s own line.',\nuser: createNeuralQueryGenerationPrompt(ART_TOPIC, 3),\n//model: 'gpt-4'\n}));\n \n Hey, check out this comprehensive guide to Renaissance art:\nCan you recommend any scholarly articles on Renaissance art?\nI found an excellent website that explores the influence of religion on Renaissance art:\n \nNow let's put them together into a function that generates queries for the right search mode.\n async function generateSearchQueries(topic, n, searchType){\nif(searchType !== 'keyword' && searchType !== 'neural'){\nthrow 'invalid searchType';\n}\nconst userPrompt = searchType === 'neural' ? createNeuralQueryGenerationPrompt(topic, n) : createKeywordQueryGenerationPrompt(topic, n);\nconst completion = await getLLMResponse({\nsystem: 'The user will ask you to help generate some search queries. Respond with only the suggested queries in plain text with no extra formatting, each on it\\'s own line.',\nuser: userPrompt,\ntemperature: 1\n});\nconst queries = completion.split('\\n').filter(s => s.trim().length > 0).slice(0, n);\nreturn queries;\n}\n \nLet's make sure it works, and check out some more queries:\n const samaQueries = await generateSearchQueries(SAMA_TOPIC, 3, 'keyword');\nconst artQueries = await generateSearchQueries(ART_TOPIC, 3, 'neural');\n \n console.log(samaQueries);\nconsole.log(artQueries);\n \n [\n"Sam Altman biography",\n"Y Combinator founder",\n"Sam Altman startup advice"\n]\n[\n"Check out this comprehensive guide to Renaissance art:",\n"Discover the key characteristics of Renaissance art and its influential artists.",\n"Explore the development of perspective and human anatomy in Renaissance paintings."\n]\n \nNow it's time to use Exa to do the search, either neural or keyword. Using searchAndContents, we can get clean text contents bundled with each link.\n async function getSearchResults(queries, type, linksPerQuery=2){\nlet results = [];\nfor (const query of queries){\nconst searchResponse = await exa.searchAndContents(query, { type, numResults: linksPerQuery, useAutoprompt: false });\nresults.push(...searchResponse.results);\n}\nreturn results;\n}\n \n const artLinks = await getSearchResults(artQueries, 'neural');\nconsole.log(artLinks[0]); // first result of six\n \n {\ntitle: "How to Look at and Understand Great Art",\nurl: "https://www.wondrium.com/how-to-look-at-and-understand-great-art?lec=29%3Futm_source%3DSocialMedia&p"... 5 more characters,\npublishedDate: "2013-11-19",\nauthor: "Doc",\nid: "dq0L1GOKroUBuryT3ypSsQ",\ntext: "\\n" +\n" \\n" +\n" \\n" +\n" \\n" +\n" \\n" +\n" \\n" +\n" \\n" +\n" Trailer\\n" +\n" \\n" +\n" \\n" +\n"\\n" +\n" \\n" +\n" \\n" +\n" \\n" +\n" \\n" +\n" \\n" +\n" \\n" +\n" 01: The Importance of First Impressions\\n" +\n" \\n" +\n" Examine the conte"... 14543 more characters,\nscore: 0.1785949170589447\n}\n \nIn just a couple lines of code, we've used Exa to go from some search queries to useful Internet content.\nThe final step is to instruct the LLM to synthesize the content into a research report, including citations of the original links. We can do that by pairing the content and the urls and writing them into the prompt.\n async function synthesizeReport(topic, searchContents, contentSlice = 750){\nconst inputData = searchContents.map(item => `--START ITEM--\\nURL: ${item.url}\\nCONTENT: ${item.text.slice(0, contentSlice)}\\n--END ITEM--\\n`).join('');\nreturn await getLLMResponse({\nsystem: 'You are a helpful research assistant. Write a report according to the user\\'s instructions.',\nuser: 'Input Data:\\n' + inputData + `Write a two paragraph research report about ${topic} based on the provided information. Include as many sources as possible. Provide citations in the text using footnote notation ([#]). First provide the report, followed by a single "References" section that lists all the URLs used, in the format [#] <url>.`,\n//model: 'gpt-4' //want a better report? use gpt-4\n});\n}\n \n const artReport = await synthesizeReport(ART_TOPIC, artLinks);\n \n Research Report: Renaissance Art\nRenaissance art is a significant period in the history of art characterized by technical innovation and a richly symbolic visual language. It is known for combining the advancements in technique with the exploration of deeper layers of meaning in artworks. In Renaissance paintings, the identification of the patron or donor often provides insights into the intended message of the artwork. For example, a painting discussed in an article from the Royal Academy[^1] was commissioned by Jacopo Pesaro, the Bishop of Paphos on the island of Cyprus, and was likely painted by Titian during his early twenties. Analyzing the patronage and symbolism in Renaissance paintings allows for a better understanding of the multifaceted meanings conveyed by the artists.\nAnother key aspect of studying Renaissance art is understanding the contexts and environments in which art is encountered and viewed. The influence of the viewer's point of view and focal point plays a critical role in shaping the experience of art. Lectures provided by Wondrium[^3] discuss the importance of first impressions and explore how the artist positions the viewer with respect to the image. Additionally, the genres of Western art and the artist's media, tools, and techniques are explored in these lectures, providing a comprehensive understanding of the various elements that contribute to the creation and perception of Renaissance art.\nOverall, studying Renaissance art encompasses an exploration of not only the technical skills and innovations of the artists but also the cultural and historical contexts in which the artworks were created. By analyzing the patronage, symbolism, and viewing experience, researchers gain a deeper appreciation and interpretation of this significant period in the history of art.\nReferences:\n[1] How to Read a Renaissance Painting. (2016, April 1). Royal Academy. Retrieved from https://www.royalacademy.org.uk/article/how-to-read-a-renaissance-painting\n[3] How to Look at and Understand Great Art (#29). Wondrium. Retrieved from https://www.wondrium.com/how-to-look-at-and-understand-great-art?lec=29\n \nLet's wrap up by putting it all together into one researcher() function that starts from a topic and returns us the finished report. We can also let Exa Researcher generate us a report about our keyword search topic as well.\n async function researcher(topic){\nconst searchType = await decideSearchType(topic);\nconst searchQueries = await generateSearchQueries(topic, 3, searchType);\nconsole.log(searchQueries);\nconst searchResults = await getSearchResults(searchQueries, searchType);\nconsole.log(searchResults[0]);\nconst report = await synthesizeReport(topic, searchResults);\nreturn report;\n}\n \n console.log(await researcher(SAMA_TOPIC));\n \n [\n"Sam Altman biography",\n"Y Combinator founder",\n"Sam Altman startup advice"\n]\n{\ntitle: "Sam Altman - Wikipedia",\nurl: "https://en.wikipedia.org/wiki/Sam_Altman",\nauthor: null,\nid: "8942e4e1-a37d-42fd-bec1-cf1715ef8d35",\ntext: "\\n" +\n"From Wikipedia, the free encyclopedia\\n" +\n"\\n" +\n"Sam AltmanAltman in 2019BornSamuel Harris AltmanApril 22, 19"... 7424 more characters\n}\nResearch Report: Sam Altman\nSam Altman is an American entrepreneur, investor, and former CEO of OpenAI[^1^]. He is widely known for his significant contributions as the president of Y Combinator from 2014 to 2019[^1^]. Altman was born on April 22, 1985, in Chicago, Illinois[^2^]. He grew up in St. Louis, Missouri, where he attended John Burroughs School[^3^]. Altman's interest in computers began at a young age, and he received his first computer, an Apple Macintosh, at the age of eight[^3^]. His childhood idol was Steve Jobs[^4^]. Sam Altman dropped out of Stanford University after one year to pursue his entrepreneurial endeavors[^1^].\nAltman's career has been marked by his involvement in various tech ventures. He notably served as the CEO of OpenAI from 2019 to 2023[^1^]. Additionally, Altman played a pivotal role as the president of Y Combinator, a startup accelerator[^1^]. His influence in the tech industry has drawn comparisons to renowned figures like Steve Jobs and Bill Gates[^2^]. Altman firmly believes in the potential of artificial general intelligence (AGI) and its ability to accomplish tasks comparable to those performed by humans[^2^].\nReferences:\n[^1^] Wikipedia. (n.d.). Sam Altman. Retrieved from https://en.wikipedia.org/wiki/Sam_Altman\n[^2^] Britannica. (n.d.). Sam Altman. Retrieved from https://www.britannica.com/biography/Sam-Altman\n \nFor a link to a complete, cleaned up version of this project that you can execute in your NodeJS environment, check out the alternative JS-only version."
},
{
"id": "https://docs.exa.ai/reference/exa-rag",
"url": "https://docs.exa.ai/reference/exa-rag",
"title": "Exa RAG",
"author": null,
"publishedDate": "2024-03-02T11:36:43.000Z",
"text": "https://docs.exa.ai/reference/exa-rag\nExa RAG\n2024-03-02T11:36:43Z\n \nLLMs are powerful because they compress large amounts of data and patterns into a format that allows convenient access, but this compressions isn't lossless. Exa can bring the most relevant data into context. This lets us to combine the compressed data of the LLM with a select quantity of uncompressed data for the problem at hand for the best generations possible.\nExa's SDKs make incorporating quality data into your LLM pipelines quick and painless. Install the SDK by running this command in your terminal:\n pip install exa-py \n # Now, import the Exa class and pass your API key to it.\nfrom exa_py import Exa\nmy_exa_api_key = "YOUR_API_KEY_HERE"\nexa = Exa(my_exa_api_key)\n \nFor our first example, we'll set up Exa to answer questions with OpenAI's popular GPT 3.5 model. (You can use GPT 4 or another model if you prefer!) We'll use Exa's highlight feature, which directly returns relevant text of customizable length for a query. You'll need to run pip install openai to get access to OpenAI's SDK if you haven't used it before. More information about the OpenAI Python SDK can be found here.\n # Set up OpenAI' SDK\nfrom openai import OpenAI\nopenai_api_key = "YOUR_API_KEY_HERE"\nopenai_client = OpenAI(api_key=openai_api_key)\n \nNow, we just need some questions to answer!\n questions = [\n"How did bats evolve their wings?",\n"How did Rome defend Italy from Hannibal?",\n]\n \nWhile LLMs can answer some questions on their own, they have limitations:\nLLMs don't have knowledge past when their training was stopped, so they can't know about recent events\nIf an LLM doesn't know the answer, it will often 'hallucinate' a correct-sounding response, and it can be difficult and inconvenient to distinguish these from correct answers\nBecause of the opaque manner of generation and the problems mentioned above, it is difficult to trust an LLM's responses when accuracy is important \nRobust retrieval helps solve all of these issues by providing quality sources of ground truth for the LLM (and their human users) to leverage. Let's use Exa to get some information about our questions:\n # Parameters for our Highlights search\nhighlights_options = {\n"num_sentences": 7, # how long our highlights should be\n"highlights_per_url": 1, # just get the best highlight for each URL\n}\n# Let the magic happen!\ninfo_for_llm = []\nfor question in questions:\nsearch_response = exa.search_and_contents(question, highlights=highlights_options, num_results=3, use_autoprompt=True)\ninfo = [sr.highlights[0] for sr in search_response.results]\ninfo_for_llm.append(info)\n \n [['As the only mammals with powered flight, the evolutionary\\xa0history of their wings has been poorly understood. However, research published Monday in Nature and PLoS Genetics has provided the first comprehensive look at the genetic origins of their incredible wings.But to appreciate the genetics of their wing development, it’s important to know how crazy a bat in flight truly\\xa0looks.Try a little experiment: Stick your arms out to the side, palms facing forward, thumbs pointing up toward the ceiling. Now imagine that your fingers are\\xa0long, arching down toward the floor like impossibly unkempt fingernails — but still made of bone, sturdy and spread apart. Picture the sides of your body connecting to your hands, a rubbery membrane attaching your leg and torso to those long fingers, binding you with strong, stretchy skin. Then, finally, imagine using your muscles to flap those enormous hands.Bats, man.As marvelous as bat flight is to behold, the genetic origins of their storied wings has remained murky. However, new findings from an international team of researchers led by Nadav Ahituv, PhD, of the University of California at San Francisco, Nicola Illing, PhD, of the University of Cape Town\\xa0in\\xa0South Africa\\xa0and Katie Pollard, PhD of the UCSF-affiliated Gladstone Institutes has shed new light on how, 50 million years ago, bats took a tetrapod blueprint for arms and legs and went up into the sky.Using a sophisticated set of genetic tools, researchers approached the question of how bats evolved flight by looking not only at which genes were used in the embryonic development of wings, but at what point during development the genes were turned on and off, and — critically — what elements in the genome were regulating the expression of these genes. Genes do not just turn themselves on without input; genetic switches, called enhancers, act to regulate the timing and levels of gene expression in the body.',\n"Since flight evolved millions of years ago in all of the groups that are capable of flight today, we can't observe the changes in behavior and much of the morphology that the evolution of flight involves. We do have the fossil record, though, and it is fairly good for the three main groups that evolved true flight. We'll spare you an in-depth description of how each group evolved flight for now; see the later exhibits for a description of each group and how they developed flight.",\n"It's easy to forget that one in five species of mammal on this planet have wings capable of delivering spectacularly acrobatic flying abilities. Equally incredibly, two-thirds of these 1,200 species of flying mammal can fly in the dark, using exquisite echolocation to avoid obstacles and snatch airborne prey with stunning deftness. These amazing feats have helped make bats the focus not only of folkloric fascination, but also of biological enquiry and mimicry by human engineers from Leonardo da Vinci onwards. Recent research in PLOS journals continues to add surprising new findings to what we know about bats, and how they might inspire us to engineer manmade machines such as drones to emulate their skills. Bats, unlike most birds and flying insects, have relatively heavy wings – something that might appear disadvantageous. But a recent study in PLOS Biology by Kenny Breuer and colleagues shows that bats can exploit the inertia of the wings to make sharp turns that would be near-impossible using aerodynamic forces alone. The authors combined high-speed film of real bats landing upside-down on ceiling roosts with computational modelling to tease apart aerodynamic and inertial effects."],\n["things, gold and silver, could buy a victory. And this Other Italian cities, inspired by Rome's example, overpowered occupying troops, shut their gates again and invited a second siege. Hannibal could not punish them without dividing his he had no competent leadership to do so, what with one member of",\n'A group of Celts known as the Senone was led through Italy by their commander, Brennus. The Senone Gauls were threatening the nearby town of Clusium, when Roman Ambassadors from the Fabii family were sent to negotiate peace for Clusium. The Romans were notoriously aggressive, and so it is only a little surprising that when a scuffle broke out between the Gauls and Clusians, the Fabii joined in and actually killed a Senone chieftain. The Roman people voted to decide the fate of those who broke the sacred conduct of ambassadors, but the Fabii were so popular that they were instead voted to some of the highest positions in Rome. This absolutely infuriated Brennus and his people and they abandoned everything and headed straight for Rome. Rome was woefully unprepared for this sudden attack. The Gauls had marched with purpose, declaring to all the towns they passed that they would not harm them, they were heading straight for Rome.',\n"Hannibal had no intention to sit and recieve the romans in spain.Hannibal clearly considered the nature of roman power-and came to the conclusion that Rome could only be defeated in Italy.The cornerstone of Rome's power was a strategic manpower base that in theory could produce 7,00,000 infantry and 70,000 cavalry.More than half of this manpower base (4,00,000) was provided by rome's Italian allies,who paid no taxes but had to render military service to rome's armies.Not all were content.Carthage on the other hand rarely used its own citizens for war,bulk of its army being mercenaries.In any case its manpower could never even come close to Rome,the fact that had aided roman victory in the 1st Punic war.Hannibal thus understood that Rome could afford to raise and send army after army to spain and take losses. Meanwhile any carthiginian losses in spain would encourage the recently conquered iberian tribes to defect. The only way to defeat Rome,was to fight in italy itself.By winning battle after battle on italian soil and demonstrating to the italian allies rome's inability to protect them and weakness,he could encourage them to break free of Rome eroding Rome's manpower to sizeable proportions. But there was one problem,his fleet was tiny and Rome ruled the seas.By land,the coastal route would be blocked by Roman forces and her ally-the great walled city of massalia.Hannibal thus resolved to think and do the impossible - move thousands of miles by land through the pyranees mountains,uncharted territory inhabited by the fierce gauls ,then through the Alps mountains and invade italy. Even before the siege of Saguntum had concluded,Hannibal had set things in motion.Having sent a number of embassies to the Gallic tribes in the Po valley with the mission of establishing a safe place for Hannibal to debouch from the Alps into the Po valley. He did not desire to cross this rugged mountain chain and to descend into the Po valley with exhausted troops only to have to fight a battle.Additionally the fierce gauls would provide a source of manpower for Hannibal's army.The romans had recently conquered much territory from the gauls in this area,brutally subjagating them ,seizing their land and redistributing it to roman colonists.Thus securing an alliance proved to be easy. After the sack of Saguntum he dismissed his troops to their own localities."]]\n \nNow, let's give the context we got to our LLM so it can answer our questions with solid sources backing them up!\n responses = []\nfor question, info in zip(questions, info_for_llm):\nsystem_prompt = "You are RAG researcher. Read the provided contexts and, if relevant, use them to answer the user's question."\nuser_prompt = f"""Sources: {info}\nQuestion: {question}"""\ncompletion = openai_client.chat.completions.create(\nmodel="gpt-3.5-turbo",\nmessages=[\n{"role": "system", "content": system_prompt},\n{"role": "user", "content": user_prompt},\n]\n)\nresponse = f"""\nQuestion: {question}\nAnswer: {completion.choices[0].message.content}\n"""\nresponses.append(response)\n \n from pprint import pprint # pretty print\npprint(responses)\n \n ['\\n'\n' Question: How did bats evolve their wings?\\n'\n' Answer: Recent research has shed new light on how bats evolved their '\n'wings. An international team of researchers used genetic tools to study the '\n'embryonic development of bat wings and the genes involved in their '\n'formation. They also investigated the regulatory elements in the genome that '\n'control the expression of these genes. By analyzing these factors, the '\n'researchers discovered that bats took a tetrapod blueprint for arms and legs '\n'and adapted it to develop wings, allowing them to fly. This research '\n'provides a comprehensive understanding of the genetic origins of bat wings '\n'and how they evolved over 50 million years ago.\\n'\n' ',\n'\\n'\n' Question: How did Rome defend Italy from Hannibal?\\n'\n' Answer: Rome defended Italy from Hannibal by using various strategies. One '\n'of the main defenses relied on the Roman manpower base, which consisted of a '\n'large army made up of Roman citizens and Italian allies who were obligated '\n"to render military service. Rome's strategic manpower base was a cornerstone "\n'of their power, as it could produce a significant number of infantry and '\n'cavalry. This posed a challenge for Hannibal, as Carthage relied heavily on '\n"mercenaries and could not match Rome's manpower.\\n"\n'\\n'\n'Hannibal realized that in order to defeat Rome, he needed to fight them in '\n'Italy itself. His plan was to win battles on Italian soil and demonstrate '\n"Rome's inability to protect their Italian allies, with the intention of "\n"encouraging them to break free from Rome. This would erode Rome's manpower "\n'base to a sizeable proportion. However, Hannibal faced several obstacles. '\n'Rome ruled the seas, making it difficult for him to transport troops and '\n'supplies by sea. Additionally, the coastal route to Italy would be blocked '\n'by Roman forces and their ally, the walled city of Massalia.\\n'\n'\\n'\n'To overcome these challenges, Hannibal devised a daring plan. He decided to '\n'lead his troops on a treacherous journey through the Pyrenees mountains, '\n'inhabited by fierce Gauls, and then through the Alps mountains to invade '\n'Italy. He sent embassies to Gallic tribes in the Po valley, securing '\n'alliances and establishing a safe place for his army to enter the Po valley '\n'from the Alps.\\n'\n'\\n'\n'Overall, Rome defended Italy from Hannibal by leveraging their manpower '\n'base, their control of the seas, and their strategic alliances with Italian '\n'allies. They also had the advantage of better infrastructure and control '\n'over resources within Italy itself. These factors ultimately played a '\n"significant role in Rome's defense against Hannibal's invasion.\\n"\n' ']\n \nExa can be used for more than simple question answering. One superpower of embeddings-based search is that we can search for the meaning of sentences or even paragraphs:\n paragraph = """\nGeorgism, also known as Geoism, is an economic philosophy and ideology named after the American\npolitical economist Henry George (1839–1897).This doctrine advocates for the societal collective,\nrather than individual property owners, to capture the economic value derived from land and other\nural resources. To this end, Georgism proposes a single tax on the unimproved value of land, known\nas a "land value tax," asserting that this would deter speculative land holding and promote efficient\nuse of valuable resources. Adherents argue that because the supply of land is fundamentally inelastic,\ntaxing it will not deter its availability or use, unlike other forms of taxation. Georgism differs\nfrom Marxism and capitalism, underscoring the distinction between common and private property while\nlargely contending that individuals should own the fruits of their labor."""\nquery = f"The best academic source about {paragraph} is (paper: "\ngeorgism_search_response = exa.search_and_contents(paragraph, highlights=highlights_options, num_results=5, use_autoprompt=False)\n \n for result in georgism_search_response.results:\nprint(result.title)\nprint(result.url)\npprint(result.highlights)\n \n Henry George\nhttps://www.newworldencyclopedia.org/entry/Henry_George\n["George's theory of interest is nowadays dismissed even by some otherwise "\n'Georgist authors, who see it as mistaken and irrelevant to his ideas about '\n'land and free trade. The separation of the value of land into improved and '\n"unimproved is problematic in George's theory. Once construction has taken "\n'place, not only the land on which such improvements were made is affected, '\n'the value of neighboring, as yet unimproved, land is impacted. Thus, while '\n'the construction of a major attraction nearby may increase the value of '\n'land, the construction of factories or nuclear power plants decreases its '\n'value. Indeed, location is the single most important asset in real estate. '\n'George intended to propose a tax that would have the least negative impact '\n'on productive activity. However, even unimproved land turns out to be '\n'affected in value by productive activity in the neighborhood.']\nWikiwand\nhttps://www.wikiwand.com/en/Georgism\n['Georgism is concerned with the distribution of economic rent caused by land '\n'ownership, natural monopolies, pollution rights, and control of the commons, '\n'including title of ownership for natural resources and other contrived '\n'privileges (e.g. intellectual property). Any natural resource which is '\n'inherently limited in supply can generate economic rent, but the classical '\n'and most significant example of land monopoly involves the extraction of '\n'common ground rent from valuable urban locations. Georgists argue that '\n'taxing economic rent is efficient, fair and equitable. The main Georgist '\n'policy recommendation is a tax assessed on land value, arguing that revenues '\n'from a land value tax (LVT) can be used to reduce or eliminate existing '\n'taxes (such as on income, trade, or purchases) that are unfair and '\n'inefficient. Some Georgists also advocate for the return of surplus public '\n"revenue to the people by means of a basic income or citizen's dividend. The "\n'concept of gaining public revenues mainly from land and natural resource '\n'privileges was widely popularized by Henry George through his first book, '\n'Progress and Poverty (1879).']\nHenry George\nhttps://www.conservapedia.com/Henry_George\n['He argued that land, unlike other factors of production, is supplied by '\n'nature and that rent is unearned surplus. The landless deserve their share '\n'of this surplus as a birthright, according to George. Henry George was born '\n'in Philadelphia, Pennsylvania, on the 2nd of September 1839. He settled in '\n'California in 1858; then later removed to New York in 1880; was first a '\n'printer, then an editor, but finally devoted all his life to economic and '\n'social questions. In 1860, George met Annie Corsina Fox. Her family was very '\n'opposed to the relationship, and in 1861 they eloped. In 1871 he published '\n'Our Land Policy, which, as further developed in 1879 under the title of '\n'Progress and Poverty, speedily attracted the widest attention both in '\n'America and in Europe.']\nGeorgism - Wikipedia\nhttps://en.wikipedia.org/wiki/Georgism\n['A key issue to the popular adoption of Georgism is that homes are illiquid '\n'yet governments need cash every year. Some economists have proposed other '\n'ways of extracting value from land such as building government housing and '\n'selling homes to new buyers in areas of fast-rising land value. The '\n'government would theoretically collect revenue from home sales without much '\n'cost to current homeowners while slowing down land value appreciation in '\n'high-demand areas. Henry George, whose writings and advocacy form the basis '\n'for Georgism Georgist ideas heavily influenced the politics of the early '\n'20th century. Political parties that were formed based on Georgist ideas '\n'include the Commonwealth Land Party in the United States, the Henry George '\n'Justice Party in Victoria, the Single Tax League in South Australia, and the '\n"Justice Party in Denmark. In the United Kingdom, George's writings were "\n'praised by emerging socialist groups in 1890s such as the Independent Labour '\n'Party and the Fabian Society, which would each go on to help form the '\n'modern-day Labour Party.']\nGeorgism\nhttps://rationalwiki.org/wiki/Georgism\n['Even with mostly primitive methods, land values are already assessed around '\n'the world wherever property/council taxes exist, and some municipalities '\n'even collect all their revenue from land values. Though these are '\n'market-based measures, they can still prove difficult and require upfront '\n'investment. Georgists believe that the potential value of land is greater '\n'than the current sum of government spending, since the abolition of taxes on '\n'labor and investment would further increase the value of land. Conversely, '\n'the libertarian strain in Georgism is evident in the notion that their land '\n'tax utopia also entails reducing or eliminating the need for many of the '\n'things governments currently supply, such as welfare, infrastructure to '\n'support urban sprawl, and military & foreign aid spending to secure '\n"resources abroad. Therefore, many Georgists propose a citizen's dividend. "\n'This is a similar concept to basic income but its proponents project its '\n'potential to be much larger due to supposedly huge takings from the land '\n'tax, combined with lowered government spending. It has been recognized since '\n'Adam Smith and David Ricardo that a tax on land value itself cannot be '\n'passed on to tenants, but instead would be paid for by the owners of the '\n'land:']\n \nUsing Exa, we can easily find related papers, either for further research or to provide a source for our claims. This is just a brief intro into what Exa can do. For a look at how you can leverage getting full contents, check out this article."
},
{
"id": "https://docs.exa.ai/",
"url": "https://docs.exa.ai/",
"title": "Introduction",
"author": "",
"publishedDate": "2023-03-03T23:47:48.000Z",
"text": "Exa is a search engine made for AIs. \n Exa has three core functionalities. \n Search Find webpages using Exa's embeddings-based or Google-style keyword search.\n Get contents Obtain clean, up-to-date, parsed HTML from Exa search results. Supports full text, LLM summaries, and highlights (snippets of relevant text).\n Find similar pages Based on a link, find and return pages that are similar in meaning\nLearn how to do an Exa search in your project\nPython Quickstart\nJavaScript Quickstart\ncURL Quickstart\nLearn how to do Exa-powered RAG\nGetting Started with RAG in Python\nGetting Started with RAG in TypeScript\nGetting Started with LangChain\nGetting Started with OpenAI\nGetting Started with CrewAI\nGetting Started with LlamaIndex\n Table of Contents \n Exa finds the exact content you're looking for on the web."
},
{
"id": "https://exa.ai/blog/announcing-exa",
"url": "https://exa.ai/blog/announcing-exa",
"title": "Exa API",
"author": "exa",
"text": "Steps toward the mission Today, we're excited to announce that we're renaming Metaphor to \"Exa” to better reflect our\nmission. We're also launching a new type of search experience – highlights. With highlights, you can instantly extract any webpage content from each search result using a\ncustomizable embedding model. All together, we’re taking big steps toward our core mission – organize the world’s knowledge. Why Exa? The internet contains the collective knowledge output of mankind – all the great works of art\nand literature, millions of essays, hundreds of millions of research papers, billions of\nimages and videos, trillions of ideas sprinkled across tweets, forums, and memes. Searching the internet should feel like navigating a grand library of knowledge, where you\ncould weave insights across cultures, industries, and millenia. Of course it doesn’t feel that way. Today, searching the internet feels more like navigating a\nlandfill. Many. have. debated. what's. wrong. with. search. But the core problem is actually simple – knowledge on the internet is buried under an\noverwhelming amount of information. The core solution is also simple – we need a better search algorithm to filter all that\ninformation and organize the knowledge buried inside. Exa is going to organize the world’s knowledge. To illustrate the problem, try googling \"startups working on climate change\". You get 43,800,000 results. That’s a lot of information! But how much actual knowledge? I see\nmany listicle results, but no actual startups working on climate change. That's because Google still uses a keyword based algorithm. Keywords as a filtering\nmechanism may have worked in 1998, but they don't work for an internet with a thousand times more\ncontent and an SEO industry devoted to hacking keywords. Exa, in contrast, is the first web-scale neural search engine. Our algorithm uses\ntransformers end-to-end, the same technology that built ChatGPT. This enables us to filter the\ninternet by meaning, not by keyword. Here's the same search on Exa: startups working on climate change Rewind Climate Change https://www.rewind.earth/ Stored on the bottom of the sea, the carbon will be sequestered for thousands of years. Holocene — Harnessing organic chemistry to remove CO2 https://www.theholocene.co/ Harnessing organic chemistry to remove CO 2 from the atmosphere, forever. A NOVEL PHOTOCHEMICAL PROCESS TO CAPTURE CARBON DIOXIDE https://www.banyucarbon.com/ Combined they have >130 peer-reviewed publications, have managed research projects worth millions of dollars, and have both presented before the National Academies of Science. Living Carbon https://www.livingcarbon.com/ Restoring ecosystems from the ground up More biomass and faster growth means more carbon capture Our trees accumulate up to 53% more biomass than control seedlings We incorporated a photosynthesis enhancement trait to help trees grow faster Weâve developed a metal accumulation trait so trees can absorb more metals in their roots and stem These metals naturally slow wood decay, creating durable wood products and retaining carbon in wood for a longer period of time Our trees are unique in their ability to grow on degraded land with high concentrations of heavy metals As our trees grow, they can clean soil made toxic by industrial activity, store more carbon, and create investment opportunities on otherwise abandoned land Phykos, PBC https://www.phykos.co/ We've built houses and farms and families, and we've travelled widely to see firsthand just how precious this world is. Eion Carbon | Carbon removal that checks all the right boxes https://eioncarbon.com/ For every 10,000 tons of carbon removal, we create about five local jobs. Climate Robotics https://www.climaterobotics.com/ In order to save ourselves from catastrophic climate change, we must automate sequestering carbon as well. Carbon Dioxide Removal | Carba https://www.carba.com/ When biomass biodegrades, up to half of the carbon can be converted into methane, which carries 28 times the greenhouse warming potential. Unemit https://unemit.com/ We firmly believe humanity can and will reach net zero greenhouse gas emissions, and ours will be a significant contribution to stopping and then reversing climate change by driving permanent Carbon Dioxide Removal (CDR) of more than 10% of historical emissions by the end of the century.\t Twelve | The Carbon Transformation Company https://www.twelve.co/ To get there, we are working with brands and industrial partners to implement our technology at scale and eliminate emissions from global manufacturing supply chains. Note that these results don't necessarily mention the words \"climate change\" or \"startup\". Exa\nfiltered out all the noisy webpages – the listicles talking about startups – and returned the\nactual knowledge – the startups themselves. Exa's goal is to understand any query – no matter how complex – and filter the internet to\nexactly the knowledge required for that query. This is the solution to the problem of overwhelming information. Instead of 40 million\nnoisy/redundant links, we’ll filter down to exactly the pieces of knowledge needed. In doing\nso, we’ll convert an internet landfill into a library of knowledge with each and every search. That's why we chose the name Exa. Exa means 10^18th power. Google means 10^100th power. While\nGoogle aims to surface all information, Exa aims to filter all information into organized\nknowledge. And when it comes to organized knowledge, 10^18 is greater than 10^100. Why .ai? It just so happens that a creature recently emerged on this planet with a particularly strong\nneed for organized, high quality knowledge. AIs, powered by LLMs like ChatGPT, are overhauling the way we do knowledge work. But to\nperform knowledge work optimally, AIs must incorporate the internet’s knowledge. That means\nthey need to be paired with a web retrieval system, like Exa. The quality of web retrieval particularly matters for LLMs. If retrieval returns junk\ninformation, the LLM will output junk content. If the retrieval is powerful enough to filter\nthe internet to the right knowledge, the LLM will output the highest quality content possible. Most of our customers have tried search APIs like Bing and Google SERP. They come to us\nbecause those APIs just don’t work for their use cases. Those APIs work for humans, but not\nfor LLMs, which have quite different needs. We’ve been on the frontlines witnessing this transformation of knowledge work and of\nretrieval. We believe AI applications will soon search the internet more than humans. That’s why Exa wasn’t just built with AI, it was also built for AI. Building with the Exa API Exa was designed with all the features necessary for AI applications: Filter out noisy SEO results to only the type of content the AI requests Handle long queries – a sentence, a paragraph, or even a whole webpage can be a query Retrieve the page content of any url for downstream processing Today we're adding highlights. Highlights means Exa can instantly extract any\npiece of content from any result's webpage. Behind the scenes, we’re chunking and embedding\nfull webpages with a paragraph prediction model. Because this happens live, you can customize\nhighlights length, # per page, and specify a secondary query specific to what content you want\nto find. Customers using highlights have seen significant increases in user conversion compared to Bing\nand other search providers. It's exciting to see customers using Exa for so many applications that just weren't possible a\nyear ago, from research paper writing assistants, to lead generation research bots, to\nlearning tools for students. A mission we take seriously As builders at the doorway to the world’s knowledge, we recognize the power we wield. The\nknowledge people consume underlies nearly everything, from our politics to our scientific\nprogress to our daily perceptions of the world. We believe that organizing the world's knowledge, and thereby giving users immense power to\nfilter the internet, is critical societal infrastructure that does not currently exist. Luckily, our incentives are aligned with our values. We have no ad-based need to monetize\nattention. Our API customers want the highest quality search possible. So do we. When the\ninternet feels like a library of all knowledge, we'll know we succeeded. If you want to help us build some of the coolest technology – designing internet-scale web\ncrawling infrastructure, training foundation models for search on our own GPU cluster, serving\ncustom vector databases at scale – you can check out our roles here. And if you're ready to build with the Exa API, you can sign up here. Thanks for reading and we'll keep writing more posts soon so you can join us on the ride! See more",
"image": "https://exa.imgix.net/og-image.png"
},
{
"id": "https://dashboard.exa.ai/",
"url": "https://dashboard.exa.ai/",
"title": "Exa API Dashboard",
"author": "Exa",
"publishedDate": "2012-01-06T00:00:00.000Z",
"text": "Get started with Exa No credit card required If you are supposed to join a team, please contact your team's administrator for an invite link."
}
]
}
]
}