vincent
2b43592151
-Clean and deduplicate URLs -Clean extracted text -Semantic selection to keep only text relevant to the query -Limit number of extracted characters per URL and also limit the total characters extracted to avoid returning too much text Arguments: urls: List of URLs to process. query: User question or information wanted Semantic ranking is performed against this query. Optional Arguments: max_chars_per_url: Maximum characters returned for each URL. Default set to 6000 max_total_chars: Maximum characters returned across all URLs. Default set to 30000 top_k_chunks: Maximum number of relevant chunks per URL. Default set to 6 min_relevance_score: Minimum semantic similarity score. Default set to 0.25 Returns: List of relevant web content results. Below an example of info return for one URL: ======================================== === SOURCE URL: https://www.marketsandmarkets.com/Market-Reports/3d-scanner-market-119952472.html === STATUS: success CONTENT: [Semantic Retrieval] Query: Extract information about companies, products, pricing, key features, target customers, market trends, growth, and competitive information related to 3D scanners. Chunks considered: 109 Chunks selected: 6 Relevance scores: [0.759, 0.757, 0.749, 0.71, 0.7, 0.696] Content characters: 3046 Truncated: False --- RELEVANT CONTENT --- Chunk 1 Chunk 2 Chunk 3 Chunk 4 Chunk 5 Chunk 6 ========================================
7 lines
73 B
Plaintext
7 lines
73 B
Plaintext
requests
|
|
#beautifulsoup4
|
|
trafilatura
|
|
sentence-transformers
|
|
fastmcp
|
|
torch
|