{"id":75682,"date":"2023-12-06T08:00:37","date_gmt":"2023-12-06T16:00:37","guid":{"rendered":"https:\/\/github.blog\/?p=75682"},"modified":"2025-03-14T10:27:06","modified_gmt":"2025-03-14T17:27:06","slug":"how-were-experimenting-with-llms-to-evolve-github-copilot","status":"publish","type":"post","link":"https:\/\/github.blog\/ai-and-ml\/llms\/how-were-experimenting-with-llms-to-evolve-github-copilot\/","title":{"rendered":"How we\u2019re experimenting with LLMs to evolve GitHub Copilot"},"content":{"rendered":"<p>Earlier this year, it seemed like every headline or dinner conversation was earmarked by the buzzwords \u201cgenerative AI.\u201d And while 2023 has been a benchmark year for the adoption of generative AI, it\u2019s not entirely a new technology. Arguably, AI has been around since the \u201860s, but the AI as we know it today came to be with the invention of machine learning frameworks known as neural networks (<a href=\"https:\/\/github.blog\/2023-04-07-what-developers-need-to-know-about-generative-ai\/\">you can read more about that here<\/a>).<\/p>\n<p>For the past few years at GitHub, we\u2019ve been experimenting with generative AI models to create new, meaningful tools for developers\u2014which is <a href=\"https:\/\/github.blog\/2023-05-17-inside-github-working-with-the-llms-behind-github-copilot\/\">how GitHub Copilot was born<\/a>. And since GitHub Copilot\u2019s initial preview release in 2021, we\u2019ve been thinking a lot about how generative AI can (and should) empower developers to be more productive at every stage of the software development lifecycle. That led us to <a href=\"https:\/\/github.blog\/2023-11-08-universe-2023-copilot-transforms-github-into-the-ai-powered-developer-platform\/\">our vision for the future of AI-powered software development with GitHub Copilot<\/a>, which we covered in detail this year at GitHub Universe 2023.<\/p>\n<p>In this blog post, we\u2019ll explore some of the experiments we\u2019ve conducted with generative AI models over the past few years, as well as take a behind-the-scenes look at some of our key learnings. We\u2019ll also explore what going from a concept to a product looks like with a radically new technology.<\/p>\n<p><a href=\"https:\/\/resources.github.com\/newsletter\/\"><img data-recalc-dims=\"1\" decoding=\"async\" loading=\"lazy\" src=\"https:\/\/github.blog\/wp-content\/uploads\/2023\/12\/1200x300-Insider_Ad-2.png?w=1024&#038;resize=1024%2C256\" alt=\"\" width=\"1024\" height=\"256\" class=\"aligncenter size-large wp-image-75727 width-fit\" srcset=\"https:\/\/github.blog\/wp-content\/uploads\/2023\/12\/1200x300-Insider_Ad-2.png?w=1200 1200w, https:\/\/github.blog\/wp-content\/uploads\/2023\/12\/1200x300-Insider_Ad-2.png?w=300 300w, https:\/\/github.blog\/wp-content\/uploads\/2023\/12\/1200x300-Insider_Ad-2.png?w=768 768w, https:\/\/github.blog\/wp-content\/uploads\/2023\/12\/1200x300-Insider_Ad-2.png?w=1024 1024w\" sizes=\"auto, (max-width: 1000px) 100vw, 1000px\" \/><\/a><\/p>\n<h2 id=\"key-pillars-of-experimentation-with-ai-at-github\"><a class=\"heading-link\" href=\"#key-pillars-of-experimentation-with-ai-at-github\">Key pillars of experimentation with AI at GitHub<span class=\"heading-hash pl-2 text-italic text-bold\" aria-hidden=\"true\"><\/span><\/a><\/h2>\n<p><a href=\"https:\/\/github.blog\/2023-06-13-survey-reveals-ais-impact-on-the-developer-experience\/\">As developers increasingly use AI tools to improve overall productivity<\/a>, we have four key pillars at GitHub that are guiding our work and how we experiment with AI. We want a developer\u2019s AI experience to be:<\/p>\n<ul>\n<li><strong>Predictable<\/strong>. We want to create tools that guide developers towards their end goals but don\u2019t surprise or overwhelm them.<\/li>\n<li><strong>Tolerable<\/strong>. As we\u2019ve seen, AI models can be wrong. Users should be able to spot incorrect suggestions easily, and address them at a low cost to focus and productivity.<\/li>\n<li><strong>Steerable<\/strong>. When a response isn\u2019t right or isn\u2019t what a user is looking for, they should be able to steer the AI towards a solution. Otherwise, we\u2019re optimistically banking on the models producing perfect answers.<\/li>\n<li><strong>Verifiable<\/strong>. Solutions must be easy to evaluate. The models are not perfect, but they can be very helpful tools if users verify their outputs. <\/li>\n<\/ul>\n<p>Now that we have a baseline understanding of how we prioritize experimenting with AI, let\u2019s take a look at <a href=\"https:\/\/github.blog\/2023-11-08-universe-2023-copilot-transforms-github-into-the-ai-powered-developer-platform\/\">the events that led to the conception of the latest evolution of GitHub Copilot<\/a>.<\/p>\n<h2 id=\"before-github-copilots-evolution-came-gpt-4\"><a class=\"heading-link\" href=\"#before-github-copilots-evolution-came-gpt-4\">Before GitHub Copilot\u2019s evolution came GPT-4<span class=\"heading-hash pl-2 text-italic text-bold\" aria-hidden=\"true\"><\/span><\/a><\/h2>\n<p>Last year, researchers from <a href=\"https:\/\/githubnext.com\/\">GitHub Next<\/a>, our R&amp;D department focused on the future of software development, were given advanced access to OpenAI\u2019s large language model (LLM) that would soon be released as GPT-4.<\/p>\n<p>\u201cAt the time, no one had seen anything like this,\u201d <a href=\"https:\/\/githubnext.com\/team\/idan\/\">Idan Gazit<\/a>, senior director of research for GitHub Next recalls. \u201cIt became a race to discover what the new models are capable of doing and what kinds of applications are possible tomorrow that were impossible yesterday.\u201d<\/p>\n<p>So, the GitHub Next team did what they do best: experiment. Over the course of several months, researchers from GitHub Next used the GPT-4 model to develop potential new tools and features that could be used across the GitHub platform. Once the team identified the projects that showed true value, the sprint to build began.<\/p>\n<p>\u201cIn classic GitHub Next fashion, we sat down and spiked a bunch of ideas and saw what looked promising or exciting to us,\u201d Gazit explains. \u201cAnd then we doubled down on the things that we believed would bear fruit.\u201d<\/p>\n<p>In the time between receiving the model and the slated announcement of the model\u2019s release in March 2023, the team had come up with several concepts and technical previews.<\/p>\n<figure class=\"gh-full-blockquote mx-0 pl-6 mt-6 mt-md-7 mb-7 mb-md-8\"><blockquote><p>At the time, no one had seen anything like this. It became a race to discover what the new models are capable of doing and what kinds of applications are possible tomorrow that were impossible yesterday.<\/p><\/blockquote><figcaption class=\"text-mono color-fg-muted f5-mktg mt-3\"> - Idan Gazit, Senior Director of Research \/\/ GitHub Next<\/figcaption><\/figure>\n<p>As these projects came together, senior leadership at GitHub began to think about what these meant for the future of GitHub Copilot. Mario Rodriguez, VP of product management, says, \u201cWe knew we wanted to make an announcement of our own around the joint Microsoft and OpenAI announcement of GPT-4. At that time, GitHub Next had a set of investments that they were making that they thought were worthwhile for the announcement. Those investments were not production-ready\u2014they were more future-focused.\u201d He explains, \u201cBut that got us thinking, so we put pen to paper and came up with the ambition behind <a href=\"https:\/\/github.com\/features\/copilot\/\">the latest evolution of GitHub Copilot<\/a>.\u201d<\/p>\n<h3 id=\"thinking-ahead-%f0%9f%a4%94\"><a class=\"heading-link\" href=\"#thinking-ahead-%f0%9f%a4%94\">Thinking ahead \ud83e\udd14<span class=\"heading-hash pl-2 text-italic text-bold\" aria-hidden=\"true\"><\/span><\/a><\/h3>\n<p>As teams at GitHub thought about evolving GitHub Copilot beyond a pair programmer in the IDE, they imagined a future where GitHub Copilot was:<\/p>\n<ul>\n<li><strong>Ubiquitous<\/strong> across every tool that developers use and integrated into every task that developers perform.<\/li>\n<li><strong>Conversational<\/strong> by default, so that natural language can be used to achieve anything.<\/li>\n<li><strong>Personalized<\/strong> to the context and knowledge of the individual, project, team, and community.<\/li>\n<\/ul>\n<p>This thought exercise, in conjunction with GitHub Next\u2019s work to conceptualize and create new tools that could revolutionize the developer workflow, crystallized what would make up <a href=\"https:\/\/github.blog\/2023-11-08-universe-2023-copilot-transforms-github-into-the-ai-powered-developer-platform\/\">the latest evolution of GitHub Copilot<\/a>. And on March 22, 2023, the technical preview for what GitHub Copilot would evolve into was released to the world with <a href=\"https:\/\/github.com\/github-copilot\/chat_waitlist_signup\/join\">GitHub Copilot Chat<\/a> and the following technical previews created by GitHub Next:<\/p>\n<ul>\n<li><a href=\"https:\/\/githubnext.com\/projects\/copilot-for-pull-requests\/\">Copilot for Pull Requests<\/a><\/li>\n<li><a href=\"https:\/\/githubnext.com\/projects\/copilot-for-docs\/\">Copilot for Docs<\/a><\/li>\n<li><a href=\"https:\/\/githubnext.com\/projects\/copilot-cli\/\">Copilot for CLI<\/a><\/li>\n<\/ul>\n<p>So, what happened behind the scenes to come up with these previews? Let\u2019s find out.<\/p>\n<p><a href=\"https:\/\/github.com\/features\/copilot?utm_source=blog&#038;utm_medium=blogB&#038;utm_campaign=cta&#038;utm_content=copilot\"><img data-recalc-dims=\"1\" decoding=\"async\" loading=\"lazy\" src=\"https:\/\/github.blog\/wp-content\/uploads\/2023\/09\/1200x300-Blog_Ad-Master-3.png?w=1024&#038;resize=1024%2C257\" alt=\"\" width=\"1024\" height=\"257\" class=\"aligncenter size-large wp-image-74341 width-fit\" srcset=\"https:\/\/github.blog\/wp-content\/uploads\/2023\/09\/1200x300-Blog_Ad-Master-3.png?w=1200 1200w, https:\/\/github.blog\/wp-content\/uploads\/2023\/09\/1200x300-Blog_Ad-Master-3.png?w=300 300w, https:\/\/github.blog\/wp-content\/uploads\/2023\/09\/1200x300-Blog_Ad-Master-3.png?w=768 768w, https:\/\/github.blog\/wp-content\/uploads\/2023\/09\/1200x300-Blog_Ad-Master-3.png?w=1024 1024w\" sizes=\"auto, (max-width: 1000px) 100vw, 1000px\" \/><\/a><\/p>\n<h2 id=\"experimenting-with-ais-place-in-the-developer-experience\"><a class=\"heading-link\" href=\"#experimenting-with-ais-place-in-the-developer-experience\">Experimenting with AI\u2019s place in the developer experience<span class=\"heading-hash pl-2 text-italic text-bold\" aria-hidden=\"true\"><\/span><\/a><\/h2>\n<p>If you asked just about any developer what\u2019s something that is specifically unique to GitHub, it would be pretty shocking if they <em>didn\u2019t<\/em> say \u201cpull requests.\u201d Pull requests play a central role in the GitHub developer experience\u2014they\u2019re not only a point of collaboration, but a gateway for teams to view and approve any changes to code.<\/p>\n<p>So when <a href=\"https:\/\/githubnext.com\/team\/acr31\/\">Andrew Rice<\/a>, <a href=\"https:\/\/githubnext.com\/team\/dsyme\/\">Don Syme<\/a>, <a href=\"https:\/\/githubnext.com\/team\/drifkin\/\">Devon Rifkin<\/a>, <a href=\"https:\/\/github.com\/mattrothenberg\">Matt Rothenberg<\/a>, <a href=\"https:\/\/github.com\/max-schaefer\">Max Schaefer<\/a>, <a href=\"https:\/\/githubnext.com\/team\/wunderalbert\/\">Albert Ziegler<\/a>, and <a href=\"https:\/\/githubnext.com\/team\/aqeelsidd\/\">Aqeel Siddiqui<\/a> were given the GPT-4 model, they were tasked with the challenge of finding ways to incorporate AI into GitHub.com.<\/p>\n<p>\u201cGitHub invented pull requests, so we started thinking, how could we add AI smarts around pull requests?\u201d Rice says. \u201cWe tried a bunch of stuff\u2014we prototyped automatic code suggestions for reviews, we had a sort of summarize mode, and a bunch of other things around test generation.\u201d But as the deadline of March 22 approached, a few of these prototyped features weren\u2019t working as desired, so Rice and team began focusing their attention and efforts solely on the summary feature.<\/p>\n<p>With the early version of <a href=\"https:\/\/githubnext.com\/projects\/copilot-for-pull-requests\/\">Copilot for Pull Requests<\/a>, a developer could submit their pull request and the AI model would generate a description and walkthrough of the code in the first comment to provide important context for the reviewer.<\/p>\n<p>\u201cWe did an internal study of the feature with Hubbers and it didn&#8217;t go well,\u201d Rice laughs. It wasn\u2019t that the developers didn\u2019t like what the feature was trying to achieve, it was the user experience, Rice believes, they were having challenges with. \u201cThe developers were concerned that the AI would be wrong. But there\u2019s two things: you have the content the AI generates and then you have the way that it\u2019s presented to the user and how it interacts with the workflow. At first, we focused a lot on the first bit, the AI-generated content, but it turned out that the second bit was far more crucial in getting this thing to fly,\u201d he explains.<\/p>\n<p>To work around this, Rice and team decided to pivot and use the same AI-generated content but frame it differently. \u201cInstead of a comment, we put it as a suggestion to the developer that let them get a preview of what the description of their pull request <em>could<\/em> look like that they could then edit,\u201d Rice says. \u201cSo, we moved it to a suggestion system, and all of a sudden the feedback changed to \u2018wow, these are helpful suggestions.\u2019 The content was exactly the same as before, it was just presented differently.\u201d<\/p>\n<h2 id=\"nobodys-perfect-not-even-ai\"><a class=\"heading-link\" href=\"#nobodys-perfect-not-even-ai\">Nobody\u2019s perfect\u2014not even AI<span class=\"heading-hash pl-2 text-italic text-bold\" aria-hidden=\"true\"><\/span><\/a><\/h2>\n<p>For Rice, the key takeaway during this process was the importance of how the AI output is presented to the developer, rather than the total accuracy of the suggestion. That doesn\u2019t mean that it\u2019s acceptable for the AI to be completely wrong, but it does mean that a developer\u2019s demand for the quality of the suggestion sits on a spectrum\u2014developers will view something as it fits within their workflow regardless of what is served to them. When the content was served as a suggestion that the developer had the authority to accept and edit, the typical attitude toward the feature changed.<\/p>\n<p><a href=\"https:\/\/githubnext.com\/team\/eaftan\/\">Eddie Aftandilian<\/a>, a principal researcher that headed up the development of another GitHub Copilot feature, shared some similar sentiments and takeaways throughout the process of building <a href=\"https:\/\/githubnext.com\/projects\/copilot-for-docs\/\">Copilot for Docs<\/a>. In late 2022, Aftandilian and Johan Rosenkilde <a href=\"https:\/\/github.blog\/2023-05-17-how-github-copilot-is-getting-better-at-understanding-your-code\/\">were examining embeddings and retrievals<\/a>, and they prototyped a vector database for a different GitHub Copilot experiment. \u201cThis got us thinking, what if we could use this for retrievals of things other than just code,\u201d Aftandilian remembers. \u201cOnce we got access to GPT-4, we realized we could use the retrieval engine to search a large corpus of documentation, and then compose those search results into a prompt that elicits better, more topical answers based on the documentation,\u201d he explains.<\/p>\n<p>\u201cSince GitHub is all about developer tools, we thought, how can we make this into a useful developer tool?\u201d Aftandilian says. Developers spend an enormous amount of time poring over docs to find solutions\u2014and as Aftandilian plainly puts it, \u201cNo one really likes reading documentation!\u201d He continues, \u201cIt also can be hard to get the right answer out of docs, too. So, it seemed like there was an opportunity here for something that could answer a developer\u2019s question more directly and unblock them. It\u2019s also an area of the development process that we felt was underexplored. We spend a lot of time searching around for answers, which can be a real pain point, and we thought we could do better with these new LLMs.\u201d<\/p>\n<p>Aftandilian, along with Devon Rifkin, Jake Donham, and Amelia Wattenberger, also deployed their early version of Copilot for Docs to Hubbers, extending GitHub Copilot\u2019s reach to GitHub\u2019s internal docs in addition to public documentation. But once the preview reached public testing, he got some interesting feedback about the quality of the AI outputs.<\/p>\n<p>\u201cOne challenge we came across during the development process was that the models don\u2019t always give the right answer or the right document,\u201d Aftandilian says. \u201cTo address this, we built in the capability for our answers to provide references or links to other documentation. We found that when we deployed it, the feedback we received was that developers didn&#8217;t mind if the output wasn&#8217;t always perfectly correct if the linked references made it easier to evaluate what the AI produced. They were using Copilot for Docs as a search engine,\u201d he says.<\/p>\n<figure class=\"gh-full-blockquote mx-0 pl-6 mt-6 mt-md-7 mb-7 mb-md-8\"><blockquote><p>The UX needs to be tolerant of AI's mistakes\u2014you can't assume that the AI will always be right.<\/p><\/blockquote><figcaption class=\"text-mono color-fg-muted f5-mktg mt-3\"> - Eddie Aftandilian, Principal Researcher \/\/ GitHub Next<\/figcaption><\/figure>\n<p>Another key learning for Aftandilian was that human feedback is the true gold standard for developing AI-based tools. \u201cOne of our conclusions was that you should ship something sooner rather than later to get real, human feedback to drive improvements,\u201d he says.<\/p>\n<p>And similar to Rice\u2019s earlier point, user experience is also critical to the success of these AI-powered tools. \u201cThe UX needs to be tolerant of AI\u2019s mistakes\u2014you can\u2019t assume that the AI will always be right,\u201d Aftandilian says. \u201cInitially we were focused on getting everything right, but we soon learned that the chat-like modality of Copilot for Docs makes the answers feel less authoritative and folks are more tolerant of the responses when they point the user in the right direction. The AI isn\u2019t always perfect, but it\u2019s a great start.\u201d<\/p>\n<h2 id=\"small-but-mighty\"><a class=\"heading-link\" href=\"#small-but-mighty\">Small but mighty<span class=\"heading-hash pl-2 text-italic text-bold\" aria-hidden=\"true\"><\/span><\/a><\/h2>\n<p>In October 2022, the entire GitHub Next team met up in Oxford, England to get together and discuss all of the projects that they were currently working on, as well as some exciting\u2014and maybe even far-fetched\u2014ideas.<\/p>\n<p>\u201cOne of the things that I pitched at this crazy ideas session was a project that would use LLMs to help you figure out CLI commands,\u201d <a href=\"https:\/\/githubnext.com\/team\/johanrosenkilde\/\">Johan Rosenkilde<\/a>, a principal researcher for GitHub Next, recalls. \u201cI was thinking about something that could use natural language prompts to describe what you wanted to do in the command line, then some sort of GUI or interface pops up that helps you narrow down what you want to do.\u201d<\/p>\n<p>As Rosenkilde talked through his pitch, one of his colleagues, Matt Rothenberg, began writing an application that did almost exactly that. \u201cBy the time my talk ended, he asked if he could show me something, and my mind was just blown,\u201d Rosenkilde laughs. That thirty-minute prototype was the genesis for what would become <a href=\"https:\/\/githubnext.com\/projects\/copilot-cli\/\">Copilot for CLI<\/a>.<\/p>\n<p>\u201cWhat he had created clearly showed that there was something of value here, but it lacked maturity of course,\u201d Rosenkilde says. \u201cAnd so what we did was carve out time to refine this rough demo into something that we could deliver to developers,\u201d he says. By the time March 2023 rolled around, they had a preview that brought the power of GitHub Copilot right to the CLI for developers to quickly ask for and receive their desired shell commands, including a breakdown that explains each part of the command\u2014without ever needing to search the web for answers.<\/p>\n<p>When reflecting on the process of taking this app from that original, scrappy version to a technical preview, Rosenkilde echoes Rice and Aftandilian in his appreciation for the subtlety of UX decisions.<\/p>\n<p>\u201cI\u2019m a backend person: I\u2019m heavy on theory and I like really difficult problems that cause me to think for weeks about a solution,\u201d Rosenkilde says. \u201cMatt was the UX guy, and he iterated extremely quickly through a lot of options. So much of the success of this application hinged on the UX, and that\u2019s a lesson that I\u2019ve taken with me. All that we do in GitHub Next, in the end, is think up tools that will add value to the user experience, so it\u2019s crucial that we get the design right and that it fits in with what the AI model can do. As we know, the AI models aren\u2019t perfect, but when they are imperfect, the cost to the user should be as low as possible,\u201d Rosenkilde says.<\/p>\n<p>That simple fact is what informs the explanation field that can be found in Copilot for CLI. \u201cThis actually wasn\u2019t part of the original UI. As the product matured, we came up with the explanation field, but we had some difficulty with the LLM producing the structured type of explanations we sought. It\u2019s very unnatural for a language model to produce something that looks like this, I had to hit it with a very large hammer,\u201d he jokes. \u201cWe wanted it to be clearly structured, but if you just ask the AI to explain a shell command, it would feed you a long paragraph that is not readily scannable and might not include the details you want.\u201d<\/p>\n<p><img decoding=\"async\" loading=\"lazy\" src=\"https:\/\/github.blog\/wp-content\/uploads\/2023\/12\/copilot-cli-explanation.gif\" alt=\"Example of the explanation field in Copilot for CLI\"><\/p>\n<p>Rosenkilde also felt that it was important to add the explanation field to help developers learn about shell scripts and double check that they have received the correct command. \u201cIt\u2019s also a security feature because you can read in natural language whether the command will change files you didn\u2019t expect to change,\u201d he explains. This multifaceted explanation field is not only useful, it\u2019s a testament to the UX of the application. \u201cWhen you have such a small application, you want every feature to have multiple different uses so that you can package up a lot of complexity in something that visually is very simple.\u201d<\/p>\n<h2 id=\"where-were-headed-%f0%9f%9a%80\"><a class=\"heading-link\" href=\"#where-were-headed-%f0%9f%9a%80\">Where we\u2019re headed \ud83d\ude80<span class=\"heading-hash pl-2 text-italic text-bold\" aria-hidden=\"true\"><\/span><\/a><\/h2>\n<p>We\u2019re focused on something great here: creating delightful AI experiences for everyone who interacts with the GitHub platform. And while we\u2019re working on it, we invite you to be part of the process. You can get involved by <a href=\"https:\/\/github.com\/features\/preview\">joining the waitlists for our current previews<\/a> and giving us your honest feedback on what you think and what you want to see going forward.<\/p>\n<p>And if you\u2019re not already using GitHub Copilot, <a href=\"https:\/\/github.com\/features\/copilot\">give it a try with a free, 30-day trial for individual developers<\/a>.<\/p>\n","protected":false},"excerpt":{"rendered":"<p>Learn how we\u2019re experimenting with generative AI models to extend GitHub Copilot across the developer lifecycle.<\/p>\n","protected":false},"author":2127,"featured_media":75685,"comment_status":"closed","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"_gh_post_show_toc":"no","_gh_post_is_no_robots":"no","_gh_post_is_featured":"no","_gh_post_is_excluded":"no","_gh_post_is_unlisted":"no","_gh_post_related_link_1":"","_gh_post_related_link_2":"","_gh_post_related_link_3":"","_gh_post_sq_img":"","_gh_post_sq_img_id":"","_gh_post_cta_title":"","_gh_post_cta_text":"","_gh_post_cta_link":"","_gh_post_cta_button":"Click Here to Learn More","_gh_post_recirc_hide":"no","_gh_post_recirc_col_1":"78957","_gh_post_recirc_col_2":"78959","_gh_post_recirc_col_3":"78961","_gh_post_recirc_col_4":"65316","_featured_video":"","_gh_post_additional_query_params":"","_jetpack_memberships_contains_paid_content":false,"footnotes":"","jetpack_publicize_message":"","jetpack_publicize_feature_enabled":true,"jetpack_social_post_already_shared":true,"jetpack_social_options":{"image_generator_settings":{"template":"highway","default_image_id":0,"font":"","enabled":false},"version":2},"_wpas_customize_per_network":false,"_links_to":"","_links_to_target":""},"categories":[3293,3296],"tags":[2837,3241,3028,2535,3064],"coauthors":[3065],"class_list":["post-75682","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-ai-and-ml","category-llms","tag-ai","tag-ai-insights","tag-generative-ai","tag-github-copilot","tag-llm"],"yoast_head":"<!-- This site is optimized with the Yoast SEO Premium plugin v27.3 (Yoast SEO v27.3) - https:\/\/yoast.com\/product\/yoast-seo-premium-wordpress\/ -->\n<title>How we\u2019re experimenting with LLMs to evolve GitHub Copilot - The GitHub Blog<\/title>\n<meta name=\"description\" content=\"Learn how we\u2019re experimenting with generative AI models to extend GitHub Copilot across the developer lifecycle.\" \/>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/github.blog\/ai-and-ml\/llms\/how-were-experimenting-with-llms-to-evolve-github-copilot\/\" \/>\n<meta property=\"og:locale\" content=\"en_US\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"How we\u2019re experimenting with LLMs to evolve GitHub Copilot\" \/>\n<meta property=\"og:description\" content=\"Learn how we\u2019re experimenting with generative AI models to extend GitHub Copilot across the developer lifecycle.\" \/>\n<meta property=\"og:url\" content=\"https:\/\/github.blog\/ai-and-ml\/llms\/how-were-experimenting-with-llms-to-evolve-github-copilot\/\" \/>\n<meta property=\"og:site_name\" content=\"The GitHub Blog\" \/>\n<meta property=\"article:published_time\" content=\"2023-12-06T16:00:37+00:00\" \/>\n<meta property=\"article:modified_time\" content=\"2025-03-14T17:27:06+00:00\" \/>\n<meta property=\"og:image\" content=\"https:\/\/github.blog\/wp-content\/uploads\/2023\/12\/AI-DarkMode-3.png\" \/>\n\t<meta property=\"og:image:width\" content=\"1200\" \/>\n\t<meta property=\"og:image:height\" content=\"630\" \/>\n\t<meta property=\"og:image:type\" content=\"image\/png\" \/>\n<meta name=\"author\" content=\"Sara Verdi\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:label1\" content=\"Written by\" \/>\n\t<meta name=\"twitter:data1\" content=\"Sara Verdi\" \/>\n\t<meta name=\"twitter:label2\" content=\"Est. reading time\" \/>\n\t<meta name=\"twitter:data2\" content=\"13 minutes\" \/>\n<script type=\"application\/ld+json\" class=\"yoast-schema-graph\">{\"@context\":\"https:\\\/\\\/schema.org\",\"@graph\":[{\"@type\":\"Article\",\"@id\":\"https:\\\/\\\/github.blog\\\/ai-and-ml\\\/llms\\\/how-were-experimenting-with-llms-to-evolve-github-copilot\\\/#article\",\"isPartOf\":{\"@id\":\"https:\\\/\\\/github.blog\\\/ai-and-ml\\\/llms\\\/how-were-experimenting-with-llms-to-evolve-github-copilot\\\/\"},\"author\":{\"name\":\"Sara Verdi\",\"@id\":\"https:\\\/\\\/github.blog\\\/#\\\/schema\\\/person\\\/af3bd38b386dfb8c531a67884597cfaf\"},\"headline\":\"How we\u2019re experimenting with LLMs to evolve GitHub Copilot\",\"datePublished\":\"2023-12-06T16:00:37+00:00\",\"dateModified\":\"2025-03-14T17:27:06+00:00\",\"mainEntityOfPage\":{\"@id\":\"https:\\\/\\\/github.blog\\\/ai-and-ml\\\/llms\\\/how-were-experimenting-with-llms-to-evolve-github-copilot\\\/\"},\"wordCount\":2811,\"image\":{\"@id\":\"https:\\\/\\\/github.blog\\\/ai-and-ml\\\/llms\\\/how-were-experimenting-with-llms-to-evolve-github-copilot\\\/#primaryimage\"},\"thumbnailUrl\":\"https:\\\/\\\/github.blog\\\/wp-content\\\/uploads\\\/2023\\\/12\\\/AI-DarkMode-3.png?fit=1200%2C630\",\"keywords\":[\"AI\",\"AI Insights\",\"generative AI\",\"GitHub Copilot\",\"LLM\"],\"articleSection\":[\"AI &amp; ML\",\"LLMs\"],\"inLanguage\":\"en-US\"},{\"@type\":\"WebPage\",\"@id\":\"https:\\\/\\\/github.blog\\\/ai-and-ml\\\/llms\\\/how-were-experimenting-with-llms-to-evolve-github-copilot\\\/\",\"url\":\"https:\\\/\\\/github.blog\\\/ai-and-ml\\\/llms\\\/how-were-experimenting-with-llms-to-evolve-github-copilot\\\/\",\"name\":\"How we\u2019re experimenting with LLMs to evolve GitHub Copilot - The GitHub Blog\",\"isPartOf\":{\"@id\":\"https:\\\/\\\/github.blog\\\/#website\"},\"primaryImageOfPage\":{\"@id\":\"https:\\\/\\\/github.blog\\\/ai-and-ml\\\/llms\\\/how-were-experimenting-with-llms-to-evolve-github-copilot\\\/#primaryimage\"},\"image\":{\"@id\":\"https:\\\/\\\/github.blog\\\/ai-and-ml\\\/llms\\\/how-were-experimenting-with-llms-to-evolve-github-copilot\\\/#primaryimage\"},\"thumbnailUrl\":\"https:\\\/\\\/github.blog\\\/wp-content\\\/uploads\\\/2023\\\/12\\\/AI-DarkMode-3.png?fit=1200%2C630\",\"datePublished\":\"2023-12-06T16:00:37+00:00\",\"dateModified\":\"2025-03-14T17:27:06+00:00\",\"author\":{\"@id\":\"https:\\\/\\\/github.blog\\\/#\\\/schema\\\/person\\\/af3bd38b386dfb8c531a67884597cfaf\"},\"description\":\"Learn how we\u2019re experimenting with generative AI models to extend GitHub Copilot across the developer lifecycle.\",\"breadcrumb\":{\"@id\":\"https:\\\/\\\/github.blog\\\/ai-and-ml\\\/llms\\\/how-were-experimenting-with-llms-to-evolve-github-copilot\\\/#breadcrumb\"},\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"ReadAction\",\"target\":[\"https:\\\/\\\/github.blog\\\/ai-and-ml\\\/llms\\\/how-were-experimenting-with-llms-to-evolve-github-copilot\\\/\"]}]},{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\\\/\\\/github.blog\\\/ai-and-ml\\\/llms\\\/how-were-experimenting-with-llms-to-evolve-github-copilot\\\/#primaryimage\",\"url\":\"https:\\\/\\\/github.blog\\\/wp-content\\\/uploads\\\/2023\\\/12\\\/AI-DarkMode-3.png?fit=1200%2C630\",\"contentUrl\":\"https:\\\/\\\/github.blog\\\/wp-content\\\/uploads\\\/2023\\\/12\\\/AI-DarkMode-3.png?fit=1200%2C630\",\"width\":1200,\"height\":630},{\"@type\":\"BreadcrumbList\",\"@id\":\"https:\\\/\\\/github.blog\\\/ai-and-ml\\\/llms\\\/how-were-experimenting-with-llms-to-evolve-github-copilot\\\/#breadcrumb\",\"itemListElement\":[{\"@type\":\"ListItem\",\"position\":1,\"name\":\"Home\",\"item\":\"https:\\\/\\\/github.blog\\\/\"},{\"@type\":\"ListItem\",\"position\":2,\"name\":\"AI &amp; ML\",\"item\":\"https:\\\/\\\/github.blog\\\/ai-and-ml\\\/\"},{\"@type\":\"ListItem\",\"position\":3,\"name\":\"LLMs\",\"item\":\"https:\\\/\\\/github.blog\\\/ai-and-ml\\\/llms\\\/\"},{\"@type\":\"ListItem\",\"position\":4,\"name\":\"How we\u2019re experimenting with LLMs to evolve GitHub Copilot\"}]},{\"@type\":\"WebSite\",\"@id\":\"https:\\\/\\\/github.blog\\\/#website\",\"url\":\"https:\\\/\\\/github.blog\\\/\",\"name\":\"The GitHub Blog\",\"description\":\"Updates, ideas, and inspiration from GitHub to help developers build and design software.\",\"potentialAction\":[{\"@type\":\"SearchAction\",\"target\":{\"@type\":\"EntryPoint\",\"urlTemplate\":\"https:\\\/\\\/github.blog\\\/?s={search_term_string}\"},\"query-input\":{\"@type\":\"PropertyValueSpecification\",\"valueRequired\":true,\"valueName\":\"search_term_string\"}}],\"inLanguage\":\"en-US\"},{\"@type\":\"Person\",\"@id\":\"https:\\\/\\\/github.blog\\\/#\\\/schema\\\/person\\\/af3bd38b386dfb8c531a67884597cfaf\",\"name\":\"Sara Verdi\",\"image\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\\\/\\\/secure.gravatar.com\\\/avatar\\\/5a529ba6b3a94c246cdf02cc58cedfe4aab668ccd467df21f78cb0ea5bd0ab8e?s=96&d=mm&r=gac5ed945152c5dd3c4acb9d35b8b5068\",\"url\":\"https:\\\/\\\/secure.gravatar.com\\\/avatar\\\/5a529ba6b3a94c246cdf02cc58cedfe4aab668ccd467df21f78cb0ea5bd0ab8e?s=96&d=mm&r=g\",\"contentUrl\":\"https:\\\/\\\/secure.gravatar.com\\\/avatar\\\/5a529ba6b3a94c246cdf02cc58cedfe4aab668ccd467df21f78cb0ea5bd0ab8e?s=96&d=mm&r=g\",\"caption\":\"Sara Verdi\"},\"url\":\"https:\\\/\\\/github.blog\\\/author\\\/saraverdi\\\/\"}]}<\/script>\n<!-- \/ Yoast SEO Premium plugin. -->","yoast_head_json":{"title":"How we\u2019re experimenting with LLMs to evolve GitHub Copilot - The GitHub Blog","description":"Learn how we\u2019re experimenting with generative AI models to extend GitHub Copilot across the developer lifecycle.","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/github.blog\/ai-and-ml\/llms\/how-were-experimenting-with-llms-to-evolve-github-copilot\/","og_locale":"en_US","og_type":"article","og_title":"How we\u2019re experimenting with LLMs to evolve GitHub Copilot","og_description":"Learn how we\u2019re experimenting with generative AI models to extend GitHub Copilot across the developer lifecycle.","og_url":"https:\/\/github.blog\/ai-and-ml\/llms\/how-were-experimenting-with-llms-to-evolve-github-copilot\/","og_site_name":"The GitHub Blog","article_published_time":"2023-12-06T16:00:37+00:00","article_modified_time":"2025-03-14T17:27:06+00:00","og_image":[{"width":1200,"height":630,"url":"https:\/\/github.blog\/wp-content\/uploads\/2023\/12\/AI-DarkMode-3.png","type":"image\/png"}],"author":"Sara Verdi","twitter_card":"summary_large_image","twitter_misc":{"Written by":"Sara Verdi","Est. reading time":"13 minutes"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"Article","@id":"https:\/\/github.blog\/ai-and-ml\/llms\/how-were-experimenting-with-llms-to-evolve-github-copilot\/#article","isPartOf":{"@id":"https:\/\/github.blog\/ai-and-ml\/llms\/how-were-experimenting-with-llms-to-evolve-github-copilot\/"},"author":{"name":"Sara Verdi","@id":"https:\/\/github.blog\/#\/schema\/person\/af3bd38b386dfb8c531a67884597cfaf"},"headline":"How we\u2019re experimenting with LLMs to evolve GitHub Copilot","datePublished":"2023-12-06T16:00:37+00:00","dateModified":"2025-03-14T17:27:06+00:00","mainEntityOfPage":{"@id":"https:\/\/github.blog\/ai-and-ml\/llms\/how-were-experimenting-with-llms-to-evolve-github-copilot\/"},"wordCount":2811,"image":{"@id":"https:\/\/github.blog\/ai-and-ml\/llms\/how-were-experimenting-with-llms-to-evolve-github-copilot\/#primaryimage"},"thumbnailUrl":"https:\/\/github.blog\/wp-content\/uploads\/2023\/12\/AI-DarkMode-3.png?fit=1200%2C630","keywords":["AI","AI Insights","generative AI","GitHub Copilot","LLM"],"articleSection":["AI &amp; ML","LLMs"],"inLanguage":"en-US"},{"@type":"WebPage","@id":"https:\/\/github.blog\/ai-and-ml\/llms\/how-were-experimenting-with-llms-to-evolve-github-copilot\/","url":"https:\/\/github.blog\/ai-and-ml\/llms\/how-were-experimenting-with-llms-to-evolve-github-copilot\/","name":"How we\u2019re experimenting with LLMs to evolve GitHub Copilot - The GitHub Blog","isPartOf":{"@id":"https:\/\/github.blog\/#website"},"primaryImageOfPage":{"@id":"https:\/\/github.blog\/ai-and-ml\/llms\/how-were-experimenting-with-llms-to-evolve-github-copilot\/#primaryimage"},"image":{"@id":"https:\/\/github.blog\/ai-and-ml\/llms\/how-were-experimenting-with-llms-to-evolve-github-copilot\/#primaryimage"},"thumbnailUrl":"https:\/\/github.blog\/wp-content\/uploads\/2023\/12\/AI-DarkMode-3.png?fit=1200%2C630","datePublished":"2023-12-06T16:00:37+00:00","dateModified":"2025-03-14T17:27:06+00:00","author":{"@id":"https:\/\/github.blog\/#\/schema\/person\/af3bd38b386dfb8c531a67884597cfaf"},"description":"Learn how we\u2019re experimenting with generative AI models to extend GitHub Copilot across the developer lifecycle.","breadcrumb":{"@id":"https:\/\/github.blog\/ai-and-ml\/llms\/how-were-experimenting-with-llms-to-evolve-github-copilot\/#breadcrumb"},"inLanguage":"en-US","potentialAction":[{"@type":"ReadAction","target":["https:\/\/github.blog\/ai-and-ml\/llms\/how-were-experimenting-with-llms-to-evolve-github-copilot\/"]}]},{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/github.blog\/ai-and-ml\/llms\/how-were-experimenting-with-llms-to-evolve-github-copilot\/#primaryimage","url":"https:\/\/github.blog\/wp-content\/uploads\/2023\/12\/AI-DarkMode-3.png?fit=1200%2C630","contentUrl":"https:\/\/github.blog\/wp-content\/uploads\/2023\/12\/AI-DarkMode-3.png?fit=1200%2C630","width":1200,"height":630},{"@type":"BreadcrumbList","@id":"https:\/\/github.blog\/ai-and-ml\/llms\/how-were-experimenting-with-llms-to-evolve-github-copilot\/#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https:\/\/github.blog\/"},{"@type":"ListItem","position":2,"name":"AI &amp; ML","item":"https:\/\/github.blog\/ai-and-ml\/"},{"@type":"ListItem","position":3,"name":"LLMs","item":"https:\/\/github.blog\/ai-and-ml\/llms\/"},{"@type":"ListItem","position":4,"name":"How we\u2019re experimenting with LLMs to evolve GitHub Copilot"}]},{"@type":"WebSite","@id":"https:\/\/github.blog\/#website","url":"https:\/\/github.blog\/","name":"The GitHub Blog","description":"Updates, ideas, and inspiration from GitHub to help developers build and design software.","potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/github.blog\/?s={search_term_string}"},"query-input":{"@type":"PropertyValueSpecification","valueRequired":true,"valueName":"search_term_string"}}],"inLanguage":"en-US"},{"@type":"Person","@id":"https:\/\/github.blog\/#\/schema\/person\/af3bd38b386dfb8c531a67884597cfaf","name":"Sara Verdi","image":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/secure.gravatar.com\/avatar\/5a529ba6b3a94c246cdf02cc58cedfe4aab668ccd467df21f78cb0ea5bd0ab8e?s=96&d=mm&r=gac5ed945152c5dd3c4acb9d35b8b5068","url":"https:\/\/secure.gravatar.com\/avatar\/5a529ba6b3a94c246cdf02cc58cedfe4aab668ccd467df21f78cb0ea5bd0ab8e?s=96&d=mm&r=g","contentUrl":"https:\/\/secure.gravatar.com\/avatar\/5a529ba6b3a94c246cdf02cc58cedfe4aab668ccd467df21f78cb0ea5bd0ab8e?s=96&d=mm&r=g","caption":"Sara Verdi"},"url":"https:\/\/github.blog\/author\/saraverdi\/"}]}},"jetpack_publicize_connections":[],"jetpack_featured_media_url":"https:\/\/github.blog\/wp-content\/uploads\/2023\/12\/AI-DarkMode-3.png?fit=1200%2C630","jetpack_shortlink":"https:\/\/wp.me\/pamS32-jGG","jetpack_sharing_enabled":true,"_links":{"self":[{"href":"https:\/\/github.blog\/wp-json\/wp\/v2\/posts\/75682","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/github.blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/github.blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/github.blog\/wp-json\/wp\/v2\/users\/2127"}],"replies":[{"embeddable":true,"href":"https:\/\/github.blog\/wp-json\/wp\/v2\/comments?post=75682"}],"version-history":[{"count":10,"href":"https:\/\/github.blog\/wp-json\/wp\/v2\/posts\/75682\/revisions"}],"predecessor-version":[{"id":83256,"href":"https:\/\/github.blog\/wp-json\/wp\/v2\/posts\/75682\/revisions\/83256"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/github.blog\/wp-json\/wp\/v2\/media\/75685"}],"wp:attachment":[{"href":"https:\/\/github.blog\/wp-json\/wp\/v2\/media?parent=75682"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/github.blog\/wp-json\/wp\/v2\/categories?post=75682"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/github.blog\/wp-json\/wp\/v2\/tags?post=75682"},{"taxonomy":"author","embeddable":true,"href":"https:\/\/github.blog\/wp-json\/wp\/v2\/coauthors?post=75682"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}