-Clean and deduplicate URLs
-Clean extracted text
-Semantic selection to keep only text relevant to the query
-Limit number of extracted characters per URL and also limit the total characters extracted to avoid returning too much text
Arguments:
urls:
List of URLs to process.
query:
User question or information wanted
Semantic ranking is performed against this query.
Optional Arguments:
max_chars_per_url:
Maximum characters returned for each URL.
Default set to 6000
max_total_chars:
Maximum characters returned across all URLs.
Default set to 30000
top_k_chunks:
Maximum number of relevant chunks per URL.
Default set to 6
min_relevance_score:
Minimum semantic similarity score.
Default set to 0.25
Returns:
List of relevant web content results.
Below an example of info return for one URL:
========================================
=== SOURCE URL: https://www.marketsandmarkets.com/Market-Reports/3d-scanner-market-119952472.html ===
STATUS: success
CONTENT:
[Semantic Retrieval]
Query: Extract information about companies, products, pricing, key features, target customers, market trends, growth, and competitive information related to 3D scanners.
Chunks considered: 109
Chunks selected: 6
Relevance scores: [0.759, 0.757, 0.749, 0.71, 0.7, 0.696]
Content characters: 3046
Truncated: False
--- RELEVANT CONTENT ---
Chunk 1
Chunk 2
Chunk 3
Chunk 4
Chunk 5
Chunk 6
========================================
vincent