Re: Talksearch.io - Advanced Bitcointalk Search Engine
by on 09/12/2025, 07:20:16 UTC
Already the mass edited/deleted posts updater using the sitemaps has refreshed 100k threads (a pittance when you take note that there's over 5 million topics, but we're getting there!)
Here are some logs from this operation, which runs simultaneously with the main indexer by the way:
It will be added soon!
Here are some logs from this operation, which runs simultaneously with the main indexer by the way:
Code:
🔄 Starting real-time scrape and ingest for topic 101198 (mode: update)
📄 Topic: [WTS] MTGOX USD
📁 Board path: Bitcoin Forum > Economy > Marketplace
📖 Pages: 1
2025-12-09 01:16:02,987 - core.scraper - INFO - Topic 101198 has 1 pages
2025-12-09 01:16:16,389 - core.scraper - INFO - Reached detected total pages (1), stopping pagination
📦 Uploading final batch of 1 posts | 📋 Queue: 1 waiting | 📤 Uploaded: 0 | 📦 Batches: 0
✅ Uploaded 1 posts | 📋 Queue: 0 waiting | 📤 Uploaded: 1 | 📦 Batches: 1
📊 Topic 101198 Summary:
✅ Successfully uploaded: 1 posts
❌ Failed uploads: 0 posts
📈 Success rate: 100.0%
⏱️ Waiting 1.0s before next topic...
🔄 Processing topic 16767/20000: 101199
🔄 Starting real-time scrape and ingest for topic 101199 (mode: update)
📄 Topic: "Mintchip is designed to track you" claims anonymous insider
📁 Board path: Bitcoin Forum > Alternate cryptocurrencies > Altcoin Discussion
📖 Pages: 1
2025-12-09 01:16:17,761 - core.scraper - INFO - Topic 101199 has 1 pages
2025-12-09 01:16:31,183 - core.scraper - INFO - Reached detected total pages (1), stopping pagination
📦 Uploading final batch of 3 posts | 📋 Queue: 3 waiting | 📤 Uploaded: 0 | 📦 Batches: 0
✅ Uploaded 3 posts | 📋 Queue: 0 waiting | 📤 Uploaded: 3 | 📦 Batches: 1
📊 Topic 101199 Summary:
✅ Successfully uploaded: 3 posts
❌ Failed uploads: 0 posts
📈 Success rate: 100.0%
⏱️ Waiting 1.0s before next topic...
🔄 Processing topic 16768/20000: 101200
🔄 Starting real-time scrape and ingest for topic 101200 (mode: update)
📄 Topic: VPN that works in China
📁 Board path: Bitcoin Forum > Economy > Marketplace
📖 Pages: 1
2025-12-09 01:16:32,581 - core.scraper - INFO - Topic 101200 has 1 pages
2025-12-09 01:16:46,006 - core.scraper - INFO - Reached detected total pages (1), stopping pagination
📦 Uploading final batch of 5 posts | 📋 Queue: 5 waiting | 📤 Uploaded: 0 | 📦 Batches: 0
✅ Uploaded 5 posts | 📋 Queue: 0 waiting | 📤 Uploaded: 5 | 📦 Batches: 1
📊 Topic 101200 Summary:
✅ Successfully uploaded: 5 posts
❌ Failed uploads: 0 posts
📈 Success rate: 100.0%
⏱️ Waiting 1.0s before next topic...
🔄 Processing topic 16769/20000: 101201
🔄 Starting real-time scrape and ingest for topic 101201 (mode: update)
📄 Topic: Anyone feel like lending me 120 BTC ?
📁 Board path: Bitcoin Forum > Economy > Marketplace
📖 Pages: 1
2025-12-09 01:16:47,330 - core.scraper - WARNING - Server error (attempt 1), using adaptive delay: 503 Server Error: Service Unavailable for url: https://bitcointalk.org/index.php?topic=101201.0
📄 Topic: [WTS] MTGOX USD
📁 Board path: Bitcoin Forum > Economy > Marketplace
📖 Pages: 1
2025-12-09 01:16:02,987 - core.scraper - INFO - Topic 101198 has 1 pages
2025-12-09 01:16:16,389 - core.scraper - INFO - Reached detected total pages (1), stopping pagination
📦 Uploading final batch of 1 posts | 📋 Queue: 1 waiting | 📤 Uploaded: 0 | 📦 Batches: 0
✅ Uploaded 1 posts | 📋 Queue: 0 waiting | 📤 Uploaded: 1 | 📦 Batches: 1
📊 Topic 101198 Summary:
✅ Successfully uploaded: 1 posts
❌ Failed uploads: 0 posts
📈 Success rate: 100.0%
⏱️ Waiting 1.0s before next topic...
🔄 Processing topic 16767/20000: 101199
🔄 Starting real-time scrape and ingest for topic 101199 (mode: update)
📄 Topic: "Mintchip is designed to track you" claims anonymous insider
📁 Board path: Bitcoin Forum > Alternate cryptocurrencies > Altcoin Discussion
📖 Pages: 1
2025-12-09 01:16:17,761 - core.scraper - INFO - Topic 101199 has 1 pages
2025-12-09 01:16:31,183 - core.scraper - INFO - Reached detected total pages (1), stopping pagination
📦 Uploading final batch of 3 posts | 📋 Queue: 3 waiting | 📤 Uploaded: 0 | 📦 Batches: 0
✅ Uploaded 3 posts | 📋 Queue: 0 waiting | 📤 Uploaded: 3 | 📦 Batches: 1
📊 Topic 101199 Summary:
✅ Successfully uploaded: 3 posts
❌ Failed uploads: 0 posts
📈 Success rate: 100.0%
⏱️ Waiting 1.0s before next topic...
🔄 Processing topic 16768/20000: 101200
🔄 Starting real-time scrape and ingest for topic 101200 (mode: update)
📄 Topic: VPN that works in China
📁 Board path: Bitcoin Forum > Economy > Marketplace
📖 Pages: 1
2025-12-09 01:16:32,581 - core.scraper - INFO - Topic 101200 has 1 pages
2025-12-09 01:16:46,006 - core.scraper - INFO - Reached detected total pages (1), stopping pagination
📦 Uploading final batch of 5 posts | 📋 Queue: 5 waiting | 📤 Uploaded: 0 | 📦 Batches: 0
✅ Uploaded 5 posts | 📋 Queue: 0 waiting | 📤 Uploaded: 5 | 📦 Batches: 1
📊 Topic 101200 Summary:
✅ Successfully uploaded: 5 posts
❌ Failed uploads: 0 posts
📈 Success rate: 100.0%
⏱️ Waiting 1.0s before next topic...
🔄 Processing topic 16769/20000: 101201
🔄 Starting real-time scrape and ingest for topic 101201 (mode: update)
📄 Topic: Anyone feel like lending me 120 BTC ?
📁 Board path: Bitcoin Forum > Economy > Marketplace
📖 Pages: 1
2025-12-09 01:16:47,330 - core.scraper - WARNING - Server error (attempt 1), using adaptive delay: 503 Server Error: Service Unavailable for url: https://bitcointalk.org/index.php?topic=101201.0
I also hope you'll update the OP to include the Arabic translation in the main thread.
It will be added soon!