{ "cells": [ { "cell_type": "markdown", "id": "30eb1704-8d76-4bc9-9308-93243aeb69cb", "metadata": {}, "source": [ "## This demo app shows:\n", "* how to use LlamaIndex, an open source library to help you build custom data augmented LLM applications;\n", "* how to ask Llama questions about breaking news via the You.com live search API and LlamaIndex.\n", "The LangChain package is used to facilitate the call to Llama2 hosted on Replicate." ] }, { "cell_type": "code", "execution_count": null, "id": "1d0005d6-e928-4d1a-981b-534a40e19e56", "metadata": {}, "outputs": [], "source": [ "!pip install llama-index langchain" ] }, { "cell_type": "code", "execution_count": 1, "id": "21fe3849", "metadata": {}, "outputs": [], "source": [ "# use ServiceContext to configure the LLM used and the custom embeddings \n", "from llama_index import ServiceContext\n", "\n", "# VectorStoreIndex is used to index custom data \n", "from llama_index import VectorStoreIndex\n", "\n", "from langchain.llms import Replicate" ] }, { "cell_type": "code", "execution_count": 2, "id": "d9d76e33", "metadata": {}, "outputs": [ { "name": "stdin", "output_type": "stream", "text": [ " ········\n" ] } ], "source": [ "from getpass import getpass\n", "import os\n", "\n", "REPLICATE_API_TOKEN = getpass()\n", "os.environ[\"REPLICATE_API_TOKEN\"] = REPLICATE_API_TOKEN" ] }, { "cell_type": "code", "execution_count": 3, "id": "75275628-5235-4b55-8033-601c76107528", "metadata": {}, "outputs": [ { "name": "stdin", "output_type": "stream", "text": [ " ········\n" ] } ], "source": [ "# You.com’s Search API is used to augment LLM responses with the snippets that contains rich info on the query results\n", "# To use the You.com Search API, you can email api@you.com to request an API key. \n", "YOUCOM_API_KEY = getpass()\n", "os.environ[\"YOUCOM_API_KEY\"] = YOUCOM_API_KEY" ] }, { "cell_type": "code", "execution_count": null, "id": "c12fc2cb", "metadata": {}, "outputs": [], "source": [ "# set llm to be using Llama2 hosted on Replicate\n", "llama2_13b_chat = \"meta/llama-2-13b-chat:f4e2de70d66816a838a89eeeb621910adffb0dd0baba3976c96980970978018d\"\n", "\n", "llm = Replicate(\n", " model=llama2_13b_chat,\n", " model_kwargs={\"temperature\": 0.01, \"top_p\": 1, \"max_new_tokens\":500}\n", ")" ] }, { "cell_type": "code", "execution_count": 5, "id": "effc9656-b18d-4d24-a80b-6066564a838b", "metadata": {}, "outputs": [], "source": [ "# chat about breaking news by first making an API request to the You.com search API\n", "import requests\n", "\n", "query = \"breaking news\"\n", "headers = {\"X-API-Key\": os.environ[\"YOUCOM_API_KEY\"]}\n", "data = requests.get(\n", " f\"https://api.ydc-index.io/search?query={query}\",\n", " headers=headers,\n", ").json()" ] }, { "cell_type": "code", "execution_count": 6, "id": "8bed3baf-742e-473c-ada1-4459012a8a2c", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "{\n", " \"hits\": [\n", " {\n", " \"description\": \"View the latest news and breaking news today for U.S., world, weather, entertainment, politics and health at CNN.com.\",\n", " \"snippets\": [\n", " \"Labor negotiations\\n\\u2018I feel bad for the nurses, but I feel worse for the patients\\u2019: Kaiser members struggle during the strike\\nWatch the latest CNN Headlines\\nMore top stories\\nShirtless man with gun who asked to see governor was arrested at Wisconsin Capitol. He returned with assault rifle after posting bail\\nTrump asks judge to dismiss election interference case, citing presidential immunity and Senate acquittal\\nAfghan girls can\\u2019t attend school past the 6th grade. This secret network of schools is defying the Taliban\\nLatest politics\\nSenate Democrats reject offer from GOP donor friend of Clarence Thomas for 5 years of financial records\\nAd Feedback\\nAd Feedback\\nCheck these out\\nAd Feedback\\nCNN Underscored\\nAd Feedback\\nScience and health\\nScientists say they\\u2019ve confirmed evidence that humans arrived in the Americas far earlier than previously thought\\nDeSantis goes off on Trump, says we don\\u2019t need \\u2018any more presidents\\u2019 who\\u2019ve \\u2018lost the zip on their fastball\\u2019\",\n", " \"Texas school district that suspended student over locs asks court to clarify if dress policy violates law\\nVideo\\nPaid Content\\nMore from CNN\\nAd Feedback\",\n", " \"Trump floats serving as House speaker for \\u2018short period of time\\u2019 if needed until GOP settles on leader\\nVideo\\nPaid Content\\nMore from CNN\\nAd Feedback\"\n", " ],\n", " \"title\": \"Breaking News, Latest News and Videos | CNN\",\n", " \"url\": \"https://www.cnn.com/\"\n", " },\n", " {\n", " \"description\": \"Go to NBCNews.com for breaking news, videos, and the latest top stories in world news, business, politics, health and pop culture.\",\n", " \"snippets\": [\n", " \"Evan Vucci / APBiden border backlashBorder debate shifts as Biden is confronted with anger \\u2014 from DemocratsBiden administration waives federal laws to allow border wall construction in TexasBiden administration to resume direct deportation flights for Venezuelan migrants\"\n", " ],\n", " \"title\": \"NBC News - Breaking News & Top Stories - Latest World, US & Local ...\",\n", " \"url\": \"https://www.nbcnews.com\"\n", " },\n", " {\n", " \"description\": \"Breaking News, Latest News and Current News from FOXNews.com. Breaking news and video. Latest Current News: U.S., World, Entertainment, Health, Business, Technology, Politics, Sports\",\n", " \"snippets\": [\n", " \"CRISIS LOOMINGOPINION\\nThis material may not be published, broadcast, rewritten, or redistributed. \\u00a92023 FOX News Network, LLC. All rights reserved. Quotes displayed in real-time or delayed by at least 15 minutes. Market data provided by Factset. Powered and implemented by FactSet Digital Solutions. Legal Statement. Mutual Fund and ETF data provided by Refinitiv Lipper.\"\n", " ],\n", " \"title\": \"Fox News - Breaking News Updates | Latest News Headlines | Photos ...\",\n", " \"url\": \"https://www.foxnews.com/\"\n", " },\n", " {\n", " \"description\": \"To the anger of his supporters, the US president is ramping up deportation and expanding the wall.\",\n", " \"snippets\": [\n", " \"To the anger of his supporters, the US president is ramping up deportation and expanding the wall.\"\n", " ],\n", " \"title\": \"Biden cannot escape Trump's shadow in border crisis - BBC News\",\n", " \"url\": \"https://www.bbc.com/news/67033249\"\n", " },\n", " {\n", " \"description\": \"Your trusted source for breaking news, analysis, exclusive interviews, headlines, and videos at ABCNews.com\",\n", " \"snippets\": [\n", " \"Paula Lobo/ABC\\n24/7 Coverage of Breaking News and Live Events.\\nON NOW \\u2022 LIVE\\n10:30 pm\\nAiring LIVE\\nABCNL Prime with Linsey Davis\\nTrending video\\nOct 05\\nTom Williams/CQ-Roll Call via Getty Images\\nPolitics\\nCornel West switches parties again, and other campaign trail takeawaysOct 05, 2023\\nWhy Biden claims he has no choice but to build more of Trump's border wall4 hours ago\\nTrump seeks dismissal of charges in Stormy Daniels hush money caseOct 05, 2023\\nTrump considers attending speaker forum, in first post-Jan. 6 Capitol visit: SourcesOct 05, 2023\\nGOP megadonors tell DeSantis and Haley: Pitch us on how you can beat TrumpOct 05, 2023\\nAla. congressional map to include 2nd district for Black voters after legal fightOct 05, 2023\\nMcCarthy didn't want Dems to save him. Does GOP need them to save the House? ANALYSISOct 04, 2023\\nBill limiting ballot hand counting in California now law; one county to defy statuteOct 04, 2023\",\n", " \"Sen. Feinstein's memorial no longer open to the public due to security concernsOct 04, 2023\\nGeorgia election case defendant wants charges dropped due to alleged paperwork errorOct 04, 2023\\nMore News\\nMore News\\nOct 05\\nEducation Images/Universal Images Group via Getty Images, FILE\\nOct 05\\nFrederic J. Brown/AFP via Getty Images, FILE\\nExtreme Weather\\nExtreme Weather\\nOct 05\\nCatherine Powell/Getty Images\\nOct 05\\nKevin Winter/Getty Images\\nOct 05\\nUnited States Consumer Product Safety Commission\\nIn Case You Missed It\\nIn Case You Missed It\\nSimone Biles leads Team USA to historic 7th consecutive world championship goldOct 05, 2023\\nElite pilots prepare for \\u2018camping out in the sky\\u2019 in prestigious gas balloon raceOct 05, 2023\\nDad loses 100 pounds in 6 months after photo motivates him to lose weightOct 04, 2023\\nWhy this woman had a quinceanera to celebrate her 60th birthdayOct 04, 2023\\nNo time to return a package? Let Uber do itOct 04, 2023\",\n", " \"Lauren Perez launches Anablue hair care: 'We're here to honor our roots'Oct 04, 2023\\nZooey Deschanel, Jonathan Scott on their love story and 'basking in engaged bliss'Oct 04, 2023\\nTeen has hands, legs amputated after rare complication from the fluOct 03, 2023\\n'Mean Girls' musical movie adaptation to be released Jan. 12Oct 03, 2023\\nSee the larger-than-life Taylor Swift Halloween decoration with a Travis Kelce nodOct 03, 2023\\nOct 05\\nWarner Bros. Pictures via YouTube\\nJan 28\\nErin Scott/The White House\"\n", " ],\n", " \"title\": \"ABC News \\u2013 Breaking News, Latest News and Videos\",\n", " \"url\": \"https://abcnews.go.com/\"\n", " },\n", " {\n", " \"description\": \"A series of powerful earthquakes struck western Afghanistan on Saturday, killing at least 100 people and wounding more than 500 others, according to the UN Office for the Coordination of Humanitarian Affairs (OCHA).\",\n", " \"snippets\": [\n", " \"A series of powerful earthquakes struck western Afghanistan on Saturday, killing at least 100 people and wounding more than 500 others, according to the UN Office for the Coordination of Humanitarian Affairs (OCHA).\"\n", " ],\n", " \"title\": \"Afghanistan earthquake: Powerful earthquakes kill at least 100 ...\",\n", " \"url\": \"https://www.cnn.com/2023/10/07/world/afghanistan-herat-earthquake-devastation-intl-hnk/index.html\"\n", " },\n", " {\n", " \"description\": \"The latest tweets from CNN Breaking News (@cnnbrk)\",\n", " \"snippets\": [\n", " \"JavaScript is not available.\\nWe\\u2019ve detected that JavaScript is disabled in this browser. Please enable JavaScript or switch to a supported browser to continue using twitter.com. You can see a list of supported browsers in our Help Center.\\nHelp Center\"\n", " ],\n", " \"title\": \"CNN Breaking News (@cnnbrk) / X\",\n", " \"url\": \"https://twitter.com/cnnbrk\"\n", " },\n", " {\n", " \"description\": \"Sarah Rumpf-Whitten is a writer on the breaking news team for Fox News Digital. You can reach her on Twitter at @s_rumpfwhitten.\",\n", " \"snippets\": [\n", " \"Sarah Rumpf-Whitten is a writer on the breaking news team for Fox News Digital. You can reach her on Twitter at @s_rumpfwhitten.\",\n", " \"Almost 1,000 migrating songbirds have perished after smashing into windows at an exhibition center in Chicago on Thursday night.\"\n", " ],\n", " \"title\": \"Nearly 1,000 migrating birds die crashing into windows at Chicago ...\",\n", " \"url\": \"https://www.foxnews.com/us/nearly-1000-migrating-birds-die-after-crashing-into-windows-chicago-exhibition-hall-carpet-dead-birds\"\n", " },\n", " {\n", " \"description\": \"Read the latest headlines, breaking news, and videos at APNews.com, the definitive source for independent journalism from every corner of the globe.\",\n", " \"snippets\": [\n", " \"A law abolishing cash bail will take effect in Illinois on Sept. 18. The change makes Illinois the first state to eliminate the practice and a nationally watched testing ground for whether such a change can work. (Sept. 12) (AP Video: Melissa Winder)\\nMediterranean storm Daniel has caused devastating floods, breaking dams and sweeping away neighborhoods in northeastern Libya, leaving many feared dead.\\nPhiladelphia State Police posted early Tuesday on X, formerly known as Twitter, that the department was pursuing escaped prison inmate Danelo Souza Cavalcante in Chester County and that \\u2018he is armed.\\u2019 (Sept. 12)\",\n", " \"- Apple\\u2019s new iPhones get faster chips, better cameras and new charging ports\\n- Thousands are feared dead and thousands more are missing in flood-ravaged eastern Libya\\n- Ex-NFL receiver Mike Williams dies 2 weeks after being injured in construction accident\\n- Larry Nassar survivor says Michigan State\\u2019s latest mess shows it hasn\\u2019t learned from past\\n- Candidate in high-stakes Virginia election performed sex acts with husband in live videos\\n- Houston Rockets\\u2019 Kevin Porter Jr. fractured girlfriend\\u2019s vertebra in NYC assault, prosecutors say\\n- North Korea\\u2019s leader is in Russia to meet Putin, with both locked in standoffs with the West\\nThe Associated Press explains what an impeachment inquiry entails now that House Speaker Kevin McCarthy wants to investigate President Joe Biden and his son Hunter as the election-year clash between Congress and the White House unfolds. (September 12)\",\n", " \"Philadelphia State Police posted early Tuesday on X, formerly known as Twitter, that the department was pursuing escaped prison inmate Danelo Souza Cavalcante in Chester County and that \\u2018he is armed.\\u2019 (Sept. 12)\\nPhiladelphia State Police posted early Tuesday on X, formerly known as Twitter, that the department was pursuing escaped prison inmate Danelo Souza Cavalcante in Chester County and that \\u2018he is armed.\\u2019 (Sept. 12)\\nRehearsals began in August for the return of Ringling Bros. and Barnum & Bailey\\u2019s \\u201cThe Greatest Show on Earth,\\u201d after a six year hiatus. The new edition of the circus show will no longer feature animals. (Sept. 11) (AP Video: Laura Bargfeld)\\nRehearsals began in August for the return of Ringling Bros. and Barnum & Bailey\\u2019s \\u201cThe Greatest Show on Earth,\\u201d after a six year hiatus. The new edition of the circus show will no longer feature animals. (Sept. 11) (AP Video: Laura Bargfeld)\",\n", " \"The Associated Press explains what an impeachment inquiry entails now that House Speaker Kevin McCarthy wants to investigate President Joe Biden and his son Hunter as the election-year clash between Congress and the White House unfolds. (September 12)\\nHeavy rainfall has flooded parts of Massachusetts and Rhode Island as water poured into homes, creating moats around their foundations, and stranded drivers. (Sept. 12) (AP Video: Rodrique Ngowi)\\nGoogle has exploited its dominance of the internet search market to lock out competitors and smother innovation, the Department of Justice charged Tuesday at the opening of the biggest U.S. antitrust trial in a quarter century. (September 12)\\nNorth Korea\\u2019s Kim Jong Un rolled into Russia on an armored train Tuesday to see President Vladimir Putin, a rare meeting between leaders driven together by their need for support in escalating standoffs with the West.\",\n", " \"North Korean state media has published images of the country\\u2019s leader Kim Jong Un getting on a train to go to Russia. (Sep. 12)\\nNorth Korean state media has published images of the country\\u2019s leader Kim Jong Un getting on a train to go to Russia. (Sep. 12)\\nMichael Kors presented his spring 2024 runway show on Monday in Brooklyn with famous faces like Halle Berry, Anitta, Tiffany Haddish, Blake Lively, Jenna Dewan and more in attendance. (Sept. 11)\\nMichael Kors presented his spring 2024 runway show on Monday in Brooklyn with famous faces like Halle Berry, Anitta, Tiffany Haddish, Blake Lively, Jenna Dewan and more in attendance. (Sept. 11)\\nA new battle over abortion in the U.S. is threatening the President\\u2019s Emergency Plan for AIDS Relief, PEPFAR, which provides HIV/AIDS medications to millions around the world including at an orphanage in Nairobi, Kenya. (Sept.11) (AP video: Josphat Kasire & Dan Huff)\",\n", " \"A new battle over abortion in the U.S. is threatening the President\\u2019s Emergency Plan for AIDS Relief, PEPFAR, which provides HIV/AIDS medications to millions around the world including at an orphanage in Nairobi, Kenya. (Sept.11) (AP video: Josphat Kasire & Dan Huff)\",\n", " \"Dictionary.com Editorial Vice President John Kelly says pop culture and social media have a strong influence on the new words and phrases \\u2013 and the changes to existing definitions\\u2014in the site\\u2019s latest update. (Sept. 7)\"\n", " ],\n", " \"title\": \"Associated Press News: Breaking News, Latest Headlines and Videos ...\",\n", " \"url\": \"https://apnews.com\"\n", " },\n", " {\n", " \"description\": \"Chicago breaking news, weather and live video. Covering local politics, health, traffic and sports for Chicago, the suburbs and northwest Indiana.\",\n", " \"snippets\": [\n", " \"Johnson planning border trip; Migrants dropped off at Willis Tower\\nMayor Brandon Johnson said Wednesday that he will be making a trip to the border to see the migrant crisis first-hand.\\nShow More\\nTop Videos\\nABC7 Chicago 24/7 Stream\\nJohnson planning border trip; Migrants dropped off at Willis Tower\\nAttorneys for ex-NU coach Fitzgerald expected to announce lawsuit\\nU.S. & World\\nShow More\\nMore Accuweather\\nMore Shop\\nFeatured\\nShow More\\nShow Fewer\\nMore Promotions\"\n", " ],\n", " \"title\": \"ABC Chicago - Breaking News and Weather\",\n", " \"url\": \"https://abc7chicago.com/\"\n", " }\n", " ],\n", " \"latency\": 0.7186734676361084\n", "}\n" ] } ], "source": [ "# check the query result in JSON\n", "import json\n", "\n", "print(json.dumps(data, indent=2))" ] }, { "cell_type": "code", "execution_count": 7, "id": "7c40e73f-ca13-4f4a-a753-e613df3d389e", "metadata": {}, "outputs": [], "source": [ "# one way to load the JSON result with key info stored as \"snippets\"\n", "from llama_index import download_loader\n", "\n", "JsonDataReader = download_loader(\"JsonDataReader\")\n", "loader = JsonDataReader()\n", "documents = loader.load_data([hit[\"snippets\"] for hit in data[\"hits\"]])\n", "\n", "# another way to add the snippets in the query result to documents - this can be handy if you just need to add a list of text strings to doc:\n", "# from llama_index import Document\n", "# snippets = [snippet for hit in data[\"hits\"] for snippet in hit[\"snippets\"]]\n", "# documents = [Document(text=s) for s in snippets]" ] }, { "cell_type": "code", "execution_count": 9, "id": "a5de3080-2c4b-479c-baba-793b3bee36ed", "metadata": {}, "outputs": [], "source": [ "# use HuggingFace embeddings\n", "from langchain.embeddings.huggingface import HuggingFaceEmbeddings\n", "from llama_index import LangchainEmbedding\n", "\n", "embeddings = LangchainEmbedding(HuggingFaceEmbeddings())\n", "\n", "# create a ServiceContext instance to use Llama2 and custom embeddings\n", "service_context = ServiceContext.from_defaults(llm=llm, chunk_size=800, chunk_overlap=20, embed_model=embeddings)\n", "\n", "# create vector store index from the documents created above\n", "index = VectorStoreIndex.from_documents(documents, service_context=service_context)\n", "\n", "# create query engine from the index\n", "query_engine = index.as_query_engine(streaming=True)" ] }, { "cell_type": "code", "execution_count": 10, "id": "de91a191-d0f2-498e-88dc-b2b43423e0e5", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ " Sure! Based on the provided context information, here is a summary of the text:\n", "\n", "The article discusses several current events and news stories, including:\n", "\n", "1. The latest update from Dictionary.com, which highlights new words and phrases influenced by pop culture and social media.\n", "2. A battle over abortion in the US that threatens the President's Emergency Plan for AIDS Relief (PEPFAR).\n", "3. Johnson planning a border trip and migrants being dropped off at Willis Tower.\n", "4. Labor negotiations and a shirtless man with a gun who" ] } ], "source": [ "# ask Llama2 to query about the search result\n", "response = query_engine.query(\"give me a summary\")\n", "response.print_response_stream()" ] }, { "cell_type": "code", "execution_count": 11, "id": "72814b20-06aa-4da8-b4dd-f0b0d74a2ea0", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ " Based on the provided context information, here are some other breaking news stories:\n", "\n", "* North Korea's leader Kim Jong Un has arrived in Russia to meet with President Vladimir Putin amid escalating standoffs with the West. (September 12)\n", "* A series of powerful earthquakes struck western Afghanistan, killing at least 100 people and wounding over 500 others. (October 3)\n", "* Michael Kors presented his spring 2024 runway show in Brooklyn with famous faces like Halle Berry and T" ] } ], "source": [ "# more queries\n", "query_engine.query(\"what about other breaking news\").print_response_stream()" ] }, { "cell_type": "code", "execution_count": 12, "id": "a65bc037-a689-476d-b529-0059a27bc949", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ " Sure! Based on the provided context information, there are several pieces of news related to North Korea that I can provide more information on:\n", "\n", "1. Kim Jong Un's trip to Russia: According to the Associated Press video, North Korea's leader Kim Jong Un rolled into Russia on an armored train Tuesday to meet with President Vladimir Putin. This is a rare meeting between the two leaders, who are both facing escalating standoffs with the West.\n", "2. Flooding in eastern Libya: The AP video reports that thousands are feared dead and thousands more are missing" ] } ], "source": [ "query_engine.query(\"tell me more about the North Korea news\").print_response_stream()" ] }, { "cell_type": "code", "execution_count": 13, "id": "16a56542", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ " Based on the provided context information, there is no direct mention of Donald Trump in the given text. However, there are some mentions of the White House and Congress, which could potentially be related to Trump's presidency. Here are the relevant sentences:\n", "\n", "* \"The Associated Press explains what an impeachment inquiry entails now that House Speaker Kevin McCarthy wants to investigate President Joe Biden and his son Hunter as the election-year clash between Congress and the White House unfolds.\"\n", "* \"Heavy rainfall has flooded parts of Massachusetts and Rhode Island as water pou" ] } ], "source": [ "query_engine.query(\"any news about Trump?\").print_response_stream()" ] }, { "cell_type": "code", "execution_count": null, "id": "148fe1d6-502d-4f8e-952f-b52594dd7dfa", "metadata": {}, "outputs": [], "source": [] } ], "metadata": { "kernelspec": { "display_name": "Python 3 (ipykernel)", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.8.18" } }, "nbformat": 4, "nbformat_minor": 5 }