Part 2 - Enhancing Repository Descriptions with OpenAI for Maximum Impact

In case you missed it, here is Part 1

So, you’ve built a GitHub crawler that fetches JavaScript repositories working with AI. Impressive! But what if we could go one step further and give those descriptions a sensational makeover? Imagine turning a bland description like “A library for machine learning” into something electrifying like “Revolutionize your AI projects with this groundbreaking machine learning library!”

In this tutorial, we’ll integrate OpenAI into our Node.js script to rewrite repository descriptions, making them more captivating and attention-grabbing. Let’s dive in.

Let's not forget to install the dotenv package, if not already

npm install dotenv

Part 1: Enhancing the Crawler

Update your crawler.js to include OpenAI.

Step 1: Import OpenAI and Configure the API

At the top of your file, add the necessary imports and set up OpenAI:

require('dotenv').config(); // Load environment variables
const { Configuration, OpenAIApi } = require('openai');
const axios = require('axios');
const cheerio = require('cheerio');

// Configure OpenAI
const configuration = new Configuration({
    apiKey: process.env.OPENAI_API_KEY,
});
const openai = new OpenAIApi(configuration);

Step 2: Add a Function to Rewrite Descriptions

Create a function that uses OpenAI to rewrite descriptions:

const rewriteDescription = async (description) => {
    try {
        const response = await openai.createCompletion({
            model: 'text-davinci-003',
            prompt: `Rewrite the following repository description to make it more sensational and engaging:\n"${description}"`,
            max_tokens: 100,
            temperature: 0.7,
        });

        return response.data.choices[0].text.trim();
    } catch (error) {
        console.error('Error rewriting description:', error.message);
        return description; // Return the original description on error
    }
};

This function:

1. Sends the original description to OpenAI with a prompt to rewrite it.
2. Returns the enhanced version or the original if an error occurs.

Step 3: Integrate the Rewriter into the Crawler

Update your fetchRepositories function to enhance descriptions:

const fetchRepositories = async () => {
    try {
        const { data } = await axios.get(SEARCH_URL);
        const $ = cheerio.load(data);

        const repositories = [];
        for (const element of $('.repo-list-item').toArray()) {
            const repoName = $(element).find('a').text().trim();
            const repoUrl = `https://github.com${$(element).find('a').attr('href')}`;
            const repoDescription = $(element).find('.mb-1').text().trim();

            const enhancedDescription = await rewriteDescription(repoDescription);

            repositories.push({
                name: repoName,
                url: repoUrl,
                description: enhancedDescription,
            });
        }

        return repositories;
    } catch (error) {
        console.error('Error fetching repositories:', error.message);
        return [];
    }
};

Here’s the key change:

• For each repository, we call rewriteDescription to enhance its description before adding it to the results.

1400+ Free HTML Templates
359+ Free News Articles
69+ Free AI Prompts
323+ Free Code Libraries
52+ Free Code Snippets & Boilerplates for Node, Nuxt, Vue, and more!
25+ Free Open Source Icon Libraries

Part 2: Displaying the Results

Finally, log the enhanced repositories:

(async () => {
    const repositories = await fetchRepositories();
    console.log('Enhanced AI-Powered JavaScript Repositories:', repositories);
})();

Part 3: Running the Script

Run your updated crawler script:

node crawler.js

You’ll see a list of AI-related JavaScript repositories with their descriptions transformed into sensational, engaging text.

Example Output

Before:

{
    "name": "ai-library",
    "url": "https://github.com/user/ai-library",
    "description": "A library for AI models in JavaScript."
}

After:

{
    "name": "ai-library",
    "url": "https://github.com/user/ai-library",
    "description": "Unlock the full potential of JavaScript with this cutting-edge AI library—designed to empower developers with next-gen models!"
}

Part 4: Enhancing and Scaling

1. Batch Processing: If GitHub returns a large number of repositories, implement batching to avoid exceeding OpenAI’s API limits.
2. Customization: Adjust the prompt or OpenAI parameters (temperature, max_tokens) to suit your desired tone and creativity level.

Save Results: Save the enhanced repositories to a JSON file for easy reference:

const fs = require('fs');

const saveToFile = (data) => {
    fs.writeFileSync('enhanced_repositories.json', JSON.stringify(data, null, 2));
    console.log('Enhanced data saved to enhanced_repositories.json');
};

// Save data after fetching
(async () => {
    const repositories = await fetchRepositories();
    saveToFile(repositories);
})();

For more tips on web development, check out DailySandbox and sign up for our free newsletter to stay ahead of the curve!