2026-03-27 14:32:09 [scrapy.utils.log] INFO: Scrapy 2.11.2 started (bot: SourcingV2)
2026-03-27 14:32:09 [scrapy.utils.log] INFO: Versions: lxml 5.2.2.0, libxml2 2.12.6, cssselect 1.2.0, parsel 1.9.1, w3lib 2.1.2, Twisted 24.3.0, Python 3.11.2 (main, Mar 13 2023, 12:18:29) [GCC 12.2.0], pyOpenSSL 24.1.0 (OpenSSL 3.2.1 30 Jan 2024), cryptography 42.0.7, Platform Linux-5.15.0-1098-azure-x86_64-with-glibc2.36
2026-03-27 14:32:09 [auto_selling_autotrader.co.uk] INFO: Starting auto_selling_autotrader.co.uk spider
2026-03-27 14:32:09 [auto_selling_autotrader.co.uk] INFO: Quick mode enabled: batching detail requests in groups of 10
2026-03-27 14:32:09 [auto_selling_autotrader.co.uk] INFO: Limiting photo downloads to 1 per item
2026-03-27 14:32:09 [auto_selling_autotrader.co.uk] INFO: _job: bbc8d04a29e911f18f76c264925fbcfa, SCRAPY_JOB: None
2026-03-27 14:32:09 [scrapy.addons] INFO: Enabled addons:
[]
2026-03-27 14:32:09 [asyncio] DEBUG: Using selector: EpollSelector
2026-03-27 14:32:09 [scrapy.utils.log] DEBUG: Using reactor: twisted.internet.asyncioreactor.AsyncioSelectorReactor
2026-03-27 14:32:09 [scrapy.utils.log] DEBUG: Using asyncio event loop: asyncio.unix_events._UnixSelectorEventLoop
2026-03-27 14:32:09 [scrapy.extensions.telnet] INFO: Telnet Password: b461756c6c1d75f6
2026-03-27 14:32:09 [scrapy.middleware] INFO: Enabled extensions:
['scrapy.extensions.corestats.CoreStats',
 'scrapy.extensions.telnet.TelnetConsole',
 'scrapy.extensions.memusage.MemoryUsage',
 'scrapy.extensions.feedexport.FeedExporter',
 'scrapy.extensions.logstats.LogStats',
 'scrapy.extensions.closespider.CloseSpider']
2026-03-27 14:32:09 [scrapy.crawler] INFO: Overridden settings:
{'BOT_NAME': 'SourcingV2',
 'CLOSESPIDER_TIMEOUT': 3600,
 'DOWNLOAD_MAXSIZE': 52428800,
 'DOWNLOAD_WARNSIZE': 10485760,
 'FEED_EXPORT_ENCODING': 'utf-8',
 'LOG_FILE': '/var/log/scrapyd/logs/sourcing_v2/auto_selling_autotrader.co.uk/bbc8d04a29e911f18f76c264925fbcfa.log',
 'LOG_FORMATTER': 'crawlers.log_formatter.SourcingLogFormatter',
 'MEMUSAGE_LIMIT_MB': 2048,
 'MEMUSAGE_WARNING_MB': 1536,
 'NEWSPIDER_MODULE': 'spiders',
 'REQUEST_FINGERPRINTER_CLASS': 'scrapy_zyte_api.ScrapyZyteAPIRequestFingerprinter',
 'REQUEST_FINGERPRINTER_IMPLEMENTATION': '2.7',
 'SPIDER_MODULES': ['spiders', 'auth_check'],
 'TWISTED_REACTOR': 'twisted.internet.asyncioreactor.AsyncioSelectorReactor',
 'USER_AGENT': ''}
2026-03-27 14:32:09 [scrapy_zyte_api.handler] INFO: Using a Zyte API key starting with '5857011'
2026-03-27 14:32:09 [scrapy_zyte_api.handler] INFO: Using a Zyte API key starting with '5857011'
2026-03-27 14:32:09 [scrapy.middleware] INFO: Enabled downloader middlewares:
['scrapy.downloadermiddlewares.offsite.OffsiteMiddleware',
 'scrapy.downloadermiddlewares.httpauth.HttpAuthMiddleware',
 'scrapy.downloadermiddlewares.downloadtimeout.DownloadTimeoutMiddleware',
 'scrapy.downloadermiddlewares.defaultheaders.DefaultHeadersMiddleware',
 'scrapy.downloadermiddlewares.useragent.UserAgentMiddleware',
 'scrapy_zyte_api.ScrapyZyteAPIDownloaderMiddleware',
 'scrapy.downloadermiddlewares.retry.RetryMiddleware',
 'scrapy.downloadermiddlewares.redirect.MetaRefreshMiddleware',
 'scrapy.downloadermiddlewares.httpcompression.HttpCompressionMiddleware',
 'scrapy.downloadermiddlewares.redirect.RedirectMiddleware',
 'scrapy.downloadermiddlewares.cookies.CookiesMiddleware',
 'scrapy.downloadermiddlewares.httpproxy.HttpProxyMiddleware',
 'scrapy.downloadermiddlewares.stats.DownloaderStats']
2026-03-27 14:32:09 [crawlers.auto_selling.middlewares.id_gen_middleware] INFO: Setting up IdGenerationMiddleware for auto_selling
2026-03-27 14:32:09 [scrapy.middleware] INFO: Enabled spider middlewares:
['scrapy.spidermiddlewares.httperror.HttpErrorMiddleware',
 'scrapy_zyte_api.ScrapyZyteAPISpiderMiddleware',
 'scrapy.spidermiddlewares.referer.RefererMiddleware',
 'scrapy.spidermiddlewares.urllength.UrlLengthMiddleware',
 'scrapy.spidermiddlewares.depth.DepthMiddleware',
 'crawlers.middlewares.photo_download_middleware.PhotoDownloadMiddleware',
 'crawlers.auto_selling.middlewares.id_gen_middleware.IdGenMiddleware']
2026-03-27 14:32:09 [scrapy.middleware] INFO: Enabled item pipelines:
['crawlers.pipelines.post_to_azurebus_auto_selling.AzureBusPipelineAutoSelling',
 'crawlers.pipelines.save_to_cdlq_db_auto_selling.SaveToCdlqDbAutoSelling']
2026-03-27 14:32:09 [scrapy.core.engine] INFO: Spider opened
2026-03-27 14:32:09 [azure.identity._credentials.environment] INFO: Incomplete environment configuration for EnvironmentCredential. These variables are set: AZURE_TENANT_ID, AZURE_CLIENT_ID
2026-03-27 14:32:09 [common.service_bus.service_bus_client_factory] INFO: ServiceBusClient initialized via DefaultAzureCredential for alx-production-bus.servicebus.windows.net
2026-03-27 14:32:09 [crawlers.pipelines.base_azure_bus_pipeline] INFO: [AzureBusPipeline] Connected to topic auto.sourcing-v2-quickmode
2026-03-27 14:32:09 [scrapy.extensions.logstats] INFO: Crawled 0 pages (at 0 pages/min), scraped 0 items (at 0 items/min)
2026-03-27 14:32:09 [scrapy.extensions.telnet] INFO: Telnet console listening on 127.0.0.1:6023
2026-03-27 14:32:09 [scrapy.extensions.memusage] INFO: Peak memory usage is 142MiB
2026-03-27 14:32:09 [scrapy-playwright] INFO: Starting download handler
2026-03-27 14:32:09 [scrapy-playwright] INFO: Starting download handler
2026-03-27 14:32:14 [auto_selling_autotrader.co.uk] INFO: Scraper started with retailer ID: 10038525 and multipost_profile_id: 0d84cb11-4c5f-4102-9735-aa9b85d0f865 and job_id: bbc8d04a29e911f18f76c264925fbcfa and url: https://www.autotrader.co.uk/dealers/essex/south-ockendon/arbat-motors-10038525
2026-03-27 14:32:14 [auto_selling_autotrader.co.uk] INFO: Getting list for retailer ID: 10038525, page: 1
2026-03-27 14:32:14 [zyte_api._retry] DEBUG: Starting call to 'zyte_api._async.AsyncZyteAPI.get.<locals>.request', this is the 1st time calling it.
2026-03-27 14:32:16 [scrapy.core.engine] DEBUG: Crawled (200) <POST https://www.autotrader.co.uk/at-gateway?opname=SearchResultsListingsGridQuery&opname=SearchResultsFacetsWithGroupsQuery> (referer: None) ['zyte-api']
2026-03-27 14:32:16 [auto_selling_autotrader.co.uk] DEBUG: response: <200 https://www.autotrader.co.uk/at-gateway?opname=SearchResultsListingsGridQuery&opname=SearchResultsFacetsWithGroupsQuery>
2026-03-27 14:32:16 [auto_selling_autotrader.co.uk] INFO: Page: 1/1, total results: 12
2026-03-27 14:32:16 [azure.identity._credentials.environment] INFO: Incomplete environment configuration for EnvironmentCredential. These variables are set: AZURE_TENANT_ID, AZURE_CLIENT_ID
2026-03-27 14:32:16 [urllib3.connectionpool] DEBUG: Starting new HTTPS connection (1): login.microsoftonline.com:443
2026-03-27 14:32:16 [urllib3.connectionpool] DEBUG: https://login.microsoftonline.com:443 "POST /8ea908c1-4e85-4692-bc3f-3646b9b40891/oauth2/v2.0/token HTTP/1.1" 200 2123
2026-03-27 14:32:16 [azure.identity._credentials.chained] INFO: DefaultAzureCredential acquired a token from WorkloadIdentityCredential
2026-03-27 14:32:16 [urllib3.connectionpool] DEBUG: Starting new HTTPS connection (1): alxsourcingstorageprod.table.core.windows.net:443
2026-03-27 14:32:16 [urllib3.connectionpool] DEBUG: https://alxsourcingstorageprod.table.core.windows.net:443 "POST /Tables HTTP/1.1" 409 None
2026-03-27 14:32:16 [urllib3.connectionpool] DEBUG: https://alxsourcingstorageprod.table.core.windows.net:443 "GET /ScrapedListings()?$filter=PartitionKey%20eq%20%27auto_selling_autotrader.co.uk%27%20and%20RowKey%20eq%20%27202603261038863%27 HTTP/1.1" 200 None
2026-03-27 14:32:16 [auto_selling_autotrader.co.uk] INFO: Scrape type for 202603261038863: 1
2026-03-27 14:32:16 [urllib3.connectionpool] DEBUG: https://alxsourcingstorageprod.table.core.windows.net:443 "POST /Tables HTTP/1.1" 409 None
2026-03-27 14:32:16 [urllib3.connectionpool] DEBUG: https://alxsourcingstorageprod.table.core.windows.net:443 "GET /ScrapedListings()?$filter=PartitionKey%20eq%20%27auto_selling_autotrader.co.uk%27%20and%20RowKey%20eq%20%27202603160742123%27 HTTP/1.1" 200 None
2026-03-27 14:32:16 [auto_selling_autotrader.co.uk] INFO: Scrape type for 202603160742123: 1
2026-03-27 14:32:16 [urllib3.connectionpool] DEBUG: https://alxsourcingstorageprod.table.core.windows.net:443 "POST /Tables HTTP/1.1" 409 None
2026-03-27 14:32:16 [urllib3.connectionpool] DEBUG: https://alxsourcingstorageprod.table.core.windows.net:443 "GET /ScrapedListings()?$filter=PartitionKey%20eq%20%27auto_selling_autotrader.co.uk%27%20and%20RowKey%20eq%20%27202603140701425%27 HTTP/1.1" 200 None
2026-03-27 14:32:16 [auto_selling_autotrader.co.uk] INFO: Scrape type for 202603140701425: 1
2026-03-27 14:32:16 [urllib3.connectionpool] DEBUG: https://alxsourcingstorageprod.table.core.windows.net:443 "POST /Tables HTTP/1.1" 409 None
2026-03-27 14:32:16 [urllib3.connectionpool] DEBUG: https://alxsourcingstorageprod.table.core.windows.net:443 "GET /ScrapedListings()?$filter=PartitionKey%20eq%20%27auto_selling_autotrader.co.uk%27%20and%20RowKey%20eq%20%27202603060470389%27 HTTP/1.1" 200 None
2026-03-27 14:32:16 [auto_selling_autotrader.co.uk] INFO: Scrape type for 202603060470389: 1
2026-03-27 14:32:16 [urllib3.connectionpool] DEBUG: https://alxsourcingstorageprod.table.core.windows.net:443 "POST /Tables HTTP/1.1" 409 None
2026-03-27 14:32:16 [urllib3.connectionpool] DEBUG: https://alxsourcingstorageprod.table.core.windows.net:443 "GET /ScrapedListings()?$filter=PartitionKey%20eq%20%27auto_selling_autotrader.co.uk%27%20and%20RowKey%20eq%20%27202603060462594%27 HTTP/1.1" 200 None
2026-03-27 14:32:16 [auto_selling_autotrader.co.uk] INFO: Scrape type for 202603060462594: 1
2026-03-27 14:32:16 [urllib3.connectionpool] DEBUG: https://alxsourcingstorageprod.table.core.windows.net:443 "POST /Tables HTTP/1.1" 409 None
2026-03-27 14:32:16 [urllib3.connectionpool] DEBUG: https://alxsourcingstorageprod.table.core.windows.net:443 "GET /ScrapedListings()?$filter=PartitionKey%20eq%20%27auto_selling_autotrader.co.uk%27%20and%20RowKey%20eq%20%27202603060461748%27 HTTP/1.1" 200 None
2026-03-27 14:32:16 [auto_selling_autotrader.co.uk] INFO: Scrape type for 202603060461748: 1
2026-03-27 14:32:16 [urllib3.connectionpool] DEBUG: https://alxsourcingstorageprod.table.core.windows.net:443 "POST /Tables HTTP/1.1" 409 None
2026-03-27 14:32:16 [urllib3.connectionpool] DEBUG: https://alxsourcingstorageprod.table.core.windows.net:443 "GET /ScrapedListings()?$filter=PartitionKey%20eq%20%27auto_selling_autotrader.co.uk%27%20and%20RowKey%20eq%20%27202602230143291%27 HTTP/1.1" 200 None
2026-03-27 14:32:16 [auto_selling_autotrader.co.uk] INFO: Scrape type for 202602230143291: 1
2026-03-27 14:32:16 [urllib3.connectionpool] DEBUG: https://alxsourcingstorageprod.table.core.windows.net:443 "POST /Tables HTTP/1.1" 409 None
2026-03-27 14:32:16 [urllib3.connectionpool] DEBUG: https://alxsourcingstorageprod.table.core.windows.net:443 "GET /ScrapedListings()?$filter=PartitionKey%20eq%20%27auto_selling_autotrader.co.uk%27%20and%20RowKey%20eq%20%27202602210112515%27 HTTP/1.1" 200 None
2026-03-27 14:32:16 [auto_selling_autotrader.co.uk] INFO: Scrape type for 202602210112515: 1
2026-03-27 14:32:16 [urllib3.connectionpool] DEBUG: https://alxsourcingstorageprod.table.core.windows.net:443 "POST /Tables HTTP/1.1" 409 None
2026-03-27 14:32:16 [urllib3.connectionpool] DEBUG: https://alxsourcingstorageprod.table.core.windows.net:443 "GET /ScrapedListings()?$filter=PartitionKey%20eq%20%27auto_selling_autotrader.co.uk%27%20and%20RowKey%20eq%20%27202602119856878%27 HTTP/1.1" 200 None
2026-03-27 14:32:16 [auto_selling_autotrader.co.uk] INFO: Scrape type for 202602119856878: 1
2026-03-27 14:32:16 [urllib3.connectionpool] DEBUG: https://alxsourcingstorageprod.table.core.windows.net:443 "POST /Tables HTTP/1.1" 409 None
2026-03-27 14:32:16 [urllib3.connectionpool] DEBUG: https://alxsourcingstorageprod.table.core.windows.net:443 "GET /ScrapedListings()?$filter=PartitionKey%20eq%20%27auto_selling_autotrader.co.uk%27%20and%20RowKey%20eq%20%27202602089769376%27 HTTP/1.1" 200 None
2026-03-27 14:32:16 [auto_selling_autotrader.co.uk] INFO: Scrape type for 202602089769376: 1
2026-03-27 14:32:16 [urllib3.connectionpool] DEBUG: https://alxsourcingstorageprod.table.core.windows.net:443 "POST /Tables HTTP/1.1" 409 None
2026-03-27 14:32:16 [urllib3.connectionpool] DEBUG: https://alxsourcingstorageprod.table.core.windows.net:443 "GET /ScrapedListings()?$filter=PartitionKey%20eq%20%27auto_selling_autotrader.co.uk%27%20and%20RowKey%20eq%20%27202601179226019%27 HTTP/1.1" 200 None
2026-03-27 14:32:16 [auto_selling_autotrader.co.uk] INFO: Scrape type for 202601179226019: 1
2026-03-27 14:32:16 [urllib3.connectionpool] DEBUG: https://alxsourcingstorageprod.table.core.windows.net:443 "POST /Tables HTTP/1.1" 409 None
2026-03-27 14:32:16 [urllib3.connectionpool] DEBUG: https://alxsourcingstorageprod.table.core.windows.net:443 "GET /ScrapedListings()?$filter=PartitionKey%20eq%20%27auto_selling_autotrader.co.uk%27%20and%20RowKey%20eq%20%27202512128552229%27 HTTP/1.1" 200 None
2026-03-27 14:32:16 [auto_selling_autotrader.co.uk] INFO: Scrape type for 202512128552229: 1
2026-03-27 14:32:16 [auto_selling_autotrader.co.uk] INFO: Quick mode: batching 10 detail requests (batch 1)
2026-03-27 14:32:16 [auto_selling_autotrader.co.uk] INFO: Quick mode: batching 2 detail requests (batch 2)
2026-03-27 14:32:16 [zyte_api._retry] DEBUG: Starting call to 'zyte_api._async.AsyncZyteAPI.get.<locals>.request', this is the 1st time calling it.
2026-03-27 14:32:16 [zyte_api._retry] DEBUG: Starting call to 'zyte_api._async.AsyncZyteAPI.get.<locals>.request', this is the 1st time calling it.
2026-03-27 14:32:18 [scrapy.core.engine] DEBUG: Crawled (200) <POST https://www.autotrader.co.uk/at-graphql> (referer: https://www.autotrader.co.uk/at-gateway?opname=SearchResultsListingsGridQuery&opname=SearchResultsFacetsWithGroupsQuery) ['zyte-api']
2026-03-27 14:32:18 [auto_selling_autotrader.co.uk] INFO: Extracted item from batch: None
2026-03-27 14:32:18 [crawlers.auto_selling.middlewares.id_gen_middleware] INFO: Generated ID for item: 974259cd-d0ba-5355-8eec-6bfdf9edef46 with identifier: 202603261038863
2026-03-27 14:32:18 [crawlers.middlewares.photo_download_middleware] INFO: Downloading 1 photos for item 974259cd-d0ba-5355-8eec-6bfdf9edef46
2026-03-27 14:32:18 [urllib3.connectionpool] DEBUG: https://alxsourcingstorageprod.table.core.windows.net:443 "POST /Tables HTTP/1.1" 409 None
2026-03-27 14:32:18 [urllib3.connectionpool] DEBUG: https://alxsourcingstorageprod.table.core.windows.net:443 "GET /PhotoDownloadCache()?$filter=PartitionKey%20eq%20%27auto_selling_autotrader.co.uk%27%20and%20RowKey%20eq%20%27https%3A%252F%252Fm.atcdn.co.uk%252Fa%252Fmedia%252Fe01d7bee96c4456a9e52c93c22d88313.jpg%27 HTTP/1.1" 200 None
2026-03-27 14:32:18 [crawlers.middlewares.photo_download_middleware] INFO: Image https://m.atcdn.co.uk/a/media/e01d7bee96c4456a9e52c93c22d88313.jpg not downloaded yet
2026-03-27 14:32:18 [auto_selling_autotrader.co.uk] INFO: Extracted item from batch: None
2026-03-27 14:32:18 [crawlers.auto_selling.middlewares.id_gen_middleware] INFO: Generated ID for item: af31ee35-34d3-52e9-82dc-f19adac01cd6 with identifier: 202603160742123
2026-03-27 14:32:18 [crawlers.middlewares.photo_download_middleware] INFO: Downloading 1 photos for item af31ee35-34d3-52e9-82dc-f19adac01cd6
2026-03-27 14:32:18 [urllib3.connectionpool] DEBUG: https://alxsourcingstorageprod.table.core.windows.net:443 "POST /Tables HTTP/1.1" 409 None
2026-03-27 14:32:18 [urllib3.connectionpool] DEBUG: https://alxsourcingstorageprod.table.core.windows.net:443 "GET /PhotoDownloadCache()?$filter=PartitionKey%20eq%20%27auto_selling_autotrader.co.uk%27%20and%20RowKey%20eq%20%27https%3A%252F%252Fm.atcdn.co.uk%252Fa%252Fmedia%252Ff2bd491f3a984914868382c9a7c0eabf.jpg%27 HTTP/1.1" 200 None
2026-03-27 14:32:18 [crawlers.middlewares.photo_download_middleware] INFO: Image https://m.atcdn.co.uk/a/media/f2bd491f3a984914868382c9a7c0eabf.jpg not downloaded yet
2026-03-27 14:32:18 [auto_selling_autotrader.co.uk] INFO: Extracted item from batch: None
2026-03-27 14:32:18 [crawlers.auto_selling.middlewares.id_gen_middleware] INFO: Generated ID for item: 6df70f7a-9ca5-5250-a94f-7cbed4efa748 with identifier: 202603140701425
2026-03-27 14:32:18 [crawlers.middlewares.photo_download_middleware] INFO: Downloading 1 photos for item 6df70f7a-9ca5-5250-a94f-7cbed4efa748
2026-03-27 14:32:18 [urllib3.connectionpool] DEBUG: https://alxsourcingstorageprod.table.core.windows.net:443 "POST /Tables HTTP/1.1" 409 None
2026-03-27 14:32:18 [urllib3.connectionpool] DEBUG: https://alxsourcingstorageprod.table.core.windows.net:443 "GET /PhotoDownloadCache()?$filter=PartitionKey%20eq%20%27auto_selling_autotrader.co.uk%27%20and%20RowKey%20eq%20%27https%3A%252F%252Fm.atcdn.co.uk%252Fa%252Fmedia%252F69c30d115922418786b999656535ee13.jpg%27 HTTP/1.1" 200 None
2026-03-27 14:32:18 [crawlers.middlewares.photo_download_middleware] INFO: Image https://m.atcdn.co.uk/a/media/69c30d115922418786b999656535ee13.jpg not downloaded yet
2026-03-27 14:32:18 [auto_selling_autotrader.co.uk] INFO: Extracted item from batch: None
2026-03-27 14:32:18 [crawlers.auto_selling.middlewares.id_gen_middleware] INFO: Generated ID for item: 871df821-de01-5244-9063-45ea1ccef53b with identifier: 202603060470389
2026-03-27 14:32:18 [crawlers.middlewares.photo_download_middleware] INFO: Downloading 1 photos for item 871df821-de01-5244-9063-45ea1ccef53b
2026-03-27 14:32:18 [urllib3.connectionpool] DEBUG: https://alxsourcingstorageprod.table.core.windows.net:443 "POST /Tables HTTP/1.1" 409 None
2026-03-27 14:32:18 [urllib3.connectionpool] DEBUG: https://alxsourcingstorageprod.table.core.windows.net:443 "GET /PhotoDownloadCache()?$filter=PartitionKey%20eq%20%27auto_selling_autotrader.co.uk%27%20and%20RowKey%20eq%20%27https%3A%252F%252Fm.atcdn.co.uk%252Fa%252Fmedia%252F6a61a25c3e1c43ffbab9668e47fbeefb.jpg%27 HTTP/1.1" 200 None
2026-03-27 14:32:18 [crawlers.middlewares.photo_download_middleware] INFO: Image https://m.atcdn.co.uk/a/media/6a61a25c3e1c43ffbab9668e47fbeefb.jpg not downloaded yet
2026-03-27 14:32:18 [auto_selling_autotrader.co.uk] INFO: Extracted item from batch: None
2026-03-27 14:32:18 [crawlers.auto_selling.middlewares.id_gen_middleware] INFO: Generated ID for item: d49cefb8-2285-5773-949e-deae4e7b6919 with identifier: 202603060462594
2026-03-27 14:32:18 [crawlers.middlewares.photo_download_middleware] INFO: Downloading 1 photos for item d49cefb8-2285-5773-949e-deae4e7b6919
2026-03-27 14:32:18 [urllib3.connectionpool] DEBUG: https://alxsourcingstorageprod.table.core.windows.net:443 "POST /Tables HTTP/1.1" 409 None
2026-03-27 14:32:18 [urllib3.connectionpool] DEBUG: https://alxsourcingstorageprod.table.core.windows.net:443 "GET /PhotoDownloadCache()?$filter=PartitionKey%20eq%20%27auto_selling_autotrader.co.uk%27%20and%20RowKey%20eq%20%27https%3A%252F%252Fm.atcdn.co.uk%252Fa%252Fmedia%252Fce971ae9d7a248cda16c4d42bd2bba09.jpg%27 HTTP/1.1" 200 None
2026-03-27 14:32:18 [crawlers.middlewares.photo_download_middleware] INFO: Image https://m.atcdn.co.uk/a/media/ce971ae9d7a248cda16c4d42bd2bba09.jpg not downloaded yet
2026-03-27 14:32:18 [auto_selling_autotrader.co.uk] INFO: Extracted item from batch: None
2026-03-27 14:32:18 [crawlers.auto_selling.middlewares.id_gen_middleware] INFO: Generated ID for item: 5821636a-81cb-5ff8-8018-d3d243d4b8a7 with identifier: 202603060461748
2026-03-27 14:32:18 [crawlers.middlewares.photo_download_middleware] INFO: Downloading 1 photos for item 5821636a-81cb-5ff8-8018-d3d243d4b8a7
2026-03-27 14:32:18 [urllib3.connectionpool] DEBUG: https://alxsourcingstorageprod.table.core.windows.net:443 "POST /Tables HTTP/1.1" 409 None
2026-03-27 14:32:19 [urllib3.connectionpool] DEBUG: https://alxsourcingstorageprod.table.core.windows.net:443 "GET /PhotoDownloadCache()?$filter=PartitionKey%20eq%20%27auto_selling_autotrader.co.uk%27%20and%20RowKey%20eq%20%27https%3A%252F%252Fm.atcdn.co.uk%252Fa%252Fmedia%252F300f30eb4fb247e59ae2e10a87eb8967.jpg%27 HTTP/1.1" 200 None
2026-03-27 14:32:19 [crawlers.middlewares.photo_download_middleware] INFO: Image https://m.atcdn.co.uk/a/media/300f30eb4fb247e59ae2e10a87eb8967.jpg not downloaded yet
2026-03-27 14:32:19 [auto_selling_autotrader.co.uk] INFO: Extracted item from batch: None
2026-03-27 14:32:19 [crawlers.auto_selling.middlewares.id_gen_middleware] INFO: Generated ID for item: d66d0515-5f45-5928-a736-7576880947a9 with identifier: 202602230143291
2026-03-27 14:32:19 [crawlers.middlewares.photo_download_middleware] INFO: Downloading 1 photos for item d66d0515-5f45-5928-a736-7576880947a9
2026-03-27 14:32:19 [urllib3.connectionpool] DEBUG: https://alxsourcingstorageprod.table.core.windows.net:443 "POST /Tables HTTP/1.1" 409 None
2026-03-27 14:32:19 [urllib3.connectionpool] DEBUG: https://alxsourcingstorageprod.table.core.windows.net:443 "GET /PhotoDownloadCache()?$filter=PartitionKey%20eq%20%27auto_selling_autotrader.co.uk%27%20and%20RowKey%20eq%20%27https%3A%252F%252Fm.atcdn.co.uk%252Fa%252Fmedia%252Fbbe25bfe791a44769c7c516c2e1782b6.jpg%27 HTTP/1.1" 200 None
2026-03-27 14:32:19 [crawlers.middlewares.photo_download_middleware] INFO: Image https://m.atcdn.co.uk/a/media/bbe25bfe791a44769c7c516c2e1782b6.jpg not downloaded yet
2026-03-27 14:32:19 [auto_selling_autotrader.co.uk] INFO: Extracted item from batch: None
2026-03-27 14:32:19 [crawlers.auto_selling.middlewares.id_gen_middleware] INFO: Generated ID for item: 489fb03a-9eca-5abc-97cf-3b22a8200458 with identifier: 202602210112515
2026-03-27 14:32:19 [crawlers.middlewares.photo_download_middleware] INFO: Downloading 1 photos for item 489fb03a-9eca-5abc-97cf-3b22a8200458
2026-03-27 14:32:19 [urllib3.connectionpool] DEBUG: https://alxsourcingstorageprod.table.core.windows.net:443 "POST /Tables HTTP/1.1" 409 None
2026-03-27 14:32:19 [urllib3.connectionpool] DEBUG: https://alxsourcingstorageprod.table.core.windows.net:443 "GET /PhotoDownloadCache()?$filter=PartitionKey%20eq%20%27auto_selling_autotrader.co.uk%27%20and%20RowKey%20eq%20%27https%3A%252F%252Fm.atcdn.co.uk%252Fa%252Fmedia%252Fa4b67838af784cc6850ab17f030ffb2e.jpg%27 HTTP/1.1" 200 None
2026-03-27 14:32:19 [crawlers.middlewares.photo_download_middleware] INFO: Image https://m.atcdn.co.uk/a/media/a4b67838af784cc6850ab17f030ffb2e.jpg not downloaded yet
2026-03-27 14:32:19 [auto_selling_autotrader.co.uk] INFO: Extracted item from batch: None
2026-03-27 14:32:19 [crawlers.auto_selling.middlewares.id_gen_middleware] INFO: Generated ID for item: 3468b15a-bdf0-51b1-84d6-d1539ff2f494 with identifier: 202602119856878
2026-03-27 14:32:19 [crawlers.middlewares.photo_download_middleware] INFO: Downloading 1 photos for item 3468b15a-bdf0-51b1-84d6-d1539ff2f494
2026-03-27 14:32:19 [urllib3.connectionpool] DEBUG: https://alxsourcingstorageprod.table.core.windows.net:443 "POST /Tables HTTP/1.1" 409 None
2026-03-27 14:32:19 [urllib3.connectionpool] DEBUG: https://alxsourcingstorageprod.table.core.windows.net:443 "GET /PhotoDownloadCache()?$filter=PartitionKey%20eq%20%27auto_selling_autotrader.co.uk%27%20and%20RowKey%20eq%20%27https%3A%252F%252Fm.atcdn.co.uk%252Fa%252Fmedia%252F84ed2d84b7b04fe4b35b9dda30cc46ec.jpg%27 HTTP/1.1" 200 None
2026-03-27 14:32:19 [crawlers.middlewares.photo_download_middleware] INFO: Image https://m.atcdn.co.uk/a/media/84ed2d84b7b04fe4b35b9dda30cc46ec.jpg not downloaded yet
2026-03-27 14:32:19 [auto_selling_autotrader.co.uk] INFO: Extracted item from batch: None
2026-03-27 14:32:19 [crawlers.auto_selling.middlewares.id_gen_middleware] INFO: Generated ID for item: 73d996ce-71d6-5e2b-8b11-8a08fd87c5f5 with identifier: 202602089769376
2026-03-27 14:32:19 [crawlers.middlewares.photo_download_middleware] INFO: Downloading 1 photos for item 73d996ce-71d6-5e2b-8b11-8a08fd87c5f5
2026-03-27 14:32:19 [urllib3.connectionpool] DEBUG: https://alxsourcingstorageprod.table.core.windows.net:443 "POST /Tables HTTP/1.1" 409 None
2026-03-27 14:32:19 [urllib3.connectionpool] DEBUG: https://alxsourcingstorageprod.table.core.windows.net:443 "GET /PhotoDownloadCache()?$filter=PartitionKey%20eq%20%27auto_selling_autotrader.co.uk%27%20and%20RowKey%20eq%20%27https%3A%252F%252Fm.atcdn.co.uk%252Fa%252Fmedia%252Fb76fa25739594f5292b1e661fa34ab7a.jpg%27 HTTP/1.1" 200 None
2026-03-27 14:32:19 [crawlers.middlewares.photo_download_middleware] INFO: Image https://m.atcdn.co.uk/a/media/b76fa25739594f5292b1e661fa34ab7a.jpg not downloaded yet
2026-03-27 14:32:19 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://m.atcdn.co.uk/a/media/e01d7bee96c4456a9e52c93c22d88313.jpg> (referer: https://www.autotrader.co.uk/)
2026-03-27 14:32:19 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://m.atcdn.co.uk/a/media/69c30d115922418786b999656535ee13.jpg> (referer: https://www.autotrader.co.uk/)
2026-03-27 14:32:19 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://m.atcdn.co.uk/a/media/84ed2d84b7b04fe4b35b9dda30cc46ec.jpg> (referer: https://www.autotrader.co.uk/)
2026-03-27 14:32:19 [common.azure_blobstore] INFO: Using managed identity for Azure Blob Storage
2026-03-27 14:32:19 [common.azure_blobstore] INFO: Using managed identity with client ID: 39192755-105d-4ce2-a21d-ab53b63f8b87
2026-03-27 14:32:19 [common.azure_blobstore] INFO: Creating container client for top-deal-listing
2026-03-27 14:32:19 [common.azure_blobstore] INFO: Container_client: <azure.storage.blob._container_client.ContainerClient object at 0x7ff6c99ca650>
2026-03-27 14:32:19 [common.azure_blobstore] INFO: Creating container if not exists
2026-03-27 14:32:19 [urllib3.connectionpool] DEBUG: Starting new HTTPS connection (1): login.microsoftonline.com:443
2026-03-27 14:32:19 [urllib3.connectionpool] DEBUG: https://login.microsoftonline.com:443 "POST /8ea908c1-4e85-4692-bc3f-3646b9b40891/oauth2/v2.0/token HTTP/1.1" 200 2113
2026-03-27 14:32:19 [urllib3.connectionpool] DEBUG: Starting new HTTPS connection (1): alxproduction.blob.core.windows.net:443
2026-03-27 14:32:19 [urllib3.connectionpool] DEBUG: https://alxproduction.blob.core.windows.net:443 "PUT /top-deal-listing?restype=container HTTP/1.1" 409 230
2026-03-27 14:32:19 [common.azure_blobstore] INFO: Container already exists, skipping creation
2026-03-27 14:32:19 [urllib3.connectionpool] DEBUG: https://alxproduction.blob.core.windows.net:443 "PUT /top-deal-listing/listing_974259cd-d0ba-5355-8eec-6bfdf9edef46/974259cd-d0ba-5355-8eec-6bfdf9edef46_image_0.jpg HTTP/1.1" 201 0
2026-03-27 14:32:19 [urllib3.connectionpool] DEBUG: https://alxsourcingstorageprod.table.core.windows.net:443 "POST /Tables HTTP/1.1" 409 None
2026-03-27 14:32:19 [urllib3.connectionpool] DEBUG: https://alxsourcingstorageprod.table.core.windows.net:443 "PATCH /PhotoDownloadCache(PartitionKey='auto_selling_autotrader.co.uk',RowKey='https%3A%252F%252Fm.atcdn.co.uk%252Fa%252Fmedia%252Fe01d7bee96c4456a9e52c93c22d88313.jpg') HTTP/1.1" 204 0
2026-03-27 14:32:19 [crawlers.middlewares.photo_download_middleware] INFO: Image https://m.atcdn.co.uk/a/media/e01d7bee96c4456a9e52c93c22d88313.jpg saved to cache
2026-03-27 14:32:19 [crawlers.middlewares.photo_download_middleware] INFO: Downloaded image 0 for item 974259cd-d0ba-5355-8eec-6bfdf9edef46
2026-03-27 14:32:19 [crawlers.middlewares.photo_download_middleware] INFO: All photos downloaded for item 974259cd-d0ba-5355-8eec-6bfdf9edef46 sending to next pipeline
2026-03-27 14:32:19 [crawlers.pipelines.post_to_azurebus_auto_selling] INFO: Azure Bus Pipeline processing item: 974259cd-d0ba-5355-8eec-6bfdf9edef46 with identifier 202603261038863
2026-03-27 14:32:19 [azure.servicebus._pyamqp._connection] INFO: Connection state changed: None -> <ConnectionState.START: 0>
2026-03-27 14:32:19 [azure.servicebus._pyamqp._connection] INFO: Connection state changed: <ConnectionState.START: 0> -> <ConnectionState.HDR_SENT: 2>
2026-03-27 14:32:19 [azure.servicebus._pyamqp._connection] INFO: Connection state changed: <ConnectionState.HDR_SENT: 2> -> <ConnectionState.HDR_SENT: 2>
2026-03-27 14:32:19 [azure.servicebus._pyamqp._connection] INFO: Connection state changed: <ConnectionState.HDR_SENT: 2> -> <ConnectionState.OPEN_PIPE: 4>
2026-03-27 14:32:19 [azure.servicebus._pyamqp.session] INFO: Session state changed: <SessionState.UNMAPPED: 0> -> <SessionState.BEGIN_SENT: 1>
2026-03-27 14:32:19 [azure.servicebus._pyamqp.link] INFO: Link state changed: <LinkState.DETACHED: 0> -> <LinkState.ATTACH_SENT: 1>
2026-03-27 14:32:19 [azure.servicebus._pyamqp.management_link] INFO: Management link receiver state changed: <LinkState.DETACHED: 0> -> <LinkState.ATTACH_SENT: 1>
2026-03-27 14:32:19 [azure.servicebus._pyamqp.link] INFO: Link state changed: <LinkState.DETACHED: 0> -> <LinkState.ATTACH_SENT: 1>
2026-03-27 14:32:19 [azure.servicebus._pyamqp.management_link] INFO: Management link sender state changed: <LinkState.DETACHED: 0> -> <LinkState.ATTACH_SENT: 1>
2026-03-27 14:32:19 [azure.servicebus._pyamqp._connection] INFO: Connection state changed: <ConnectionState.OPEN_PIPE: 4> -> <ConnectionState.OPEN_SENT: 7>
2026-03-27 14:32:19 [azure.servicebus._pyamqp._connection] INFO: Connection state changed: <ConnectionState.OPEN_SENT: 7> -> <ConnectionState.OPENED: 9>
2026-03-27 14:32:19 [azure.servicebus._pyamqp.session] INFO: Session state changed: <SessionState.BEGIN_SENT: 1> -> <SessionState.MAPPED: 3>
2026-03-27 14:32:19 [azure.servicebus._pyamqp.link] INFO: Link state changed: <LinkState.ATTACH_SENT: 1> -> <LinkState.ATTACHED: 3>
2026-03-27 14:32:19 [azure.servicebus._pyamqp.management_link] INFO: Management link receiver state changed: <LinkState.ATTACH_SENT: 1> -> <LinkState.ATTACHED: 3>
2026-03-27 14:32:19 [azure.servicebus._pyamqp.link] INFO: Link state changed: <LinkState.ATTACH_SENT: 1> -> <LinkState.ATTACHED: 3>
2026-03-27 14:32:19 [azure.servicebus._pyamqp.management_link] INFO: Management link sender state changed: <LinkState.ATTACH_SENT: 1> -> <LinkState.ATTACHED: 3>
2026-03-27 14:32:19 [azure.servicebus._pyamqp.cbs] DEBUG: CBS completed opening with status: <ManagementOpenResult.OK: 1>
2026-03-27 14:32:19 [urllib3.connectionpool] DEBUG: Starting new HTTPS connection (1): login.microsoftonline.com:443
2026-03-27 14:32:20 [urllib3.connectionpool] DEBUG: https://login.microsoftonline.com:443 "POST /8ea908c1-4e85-4692-bc3f-3646b9b40891/oauth2/v2.0/token HTTP/1.1" 200 2023
2026-03-27 14:32:20 [azure.identity._credentials.chained] INFO: DefaultAzureCredential acquired a token from WorkloadIdentityCredential
2026-03-27 14:32:20 [azure.servicebus._pyamqp.cbs] DEBUG: CBS update in progress. Token put time: 1774621940
2026-03-27 14:32:20 [azure.servicebus._pyamqp.cbs] DEBUG: CBS update in progress. Token put time: 1774621940
2026-03-27 14:32:20 [azure.servicebus._pyamqp.cbs] DEBUG: CBS Put token result (<ManagementExecuteOperationResult.OK: 0>), status code: 202, status_description: b'Accepted'.
2026-03-27 14:32:20 [azure.servicebus._pyamqp.cbs] DEBUG: CBS status check: state == <CbsAuthState.OK: 0>, expired == False, refresh required == False
2026-03-27 14:32:20 [azure.servicebus._pyamqp.link] INFO: Link state changed: <LinkState.DETACHED: 0> -> <LinkState.ATTACH_SENT: 1>
2026-03-27 14:32:20 [azure.servicebus._pyamqp.link] INFO: Link state changed: <LinkState.ATTACH_SENT: 1> -> <LinkState.ATTACHED: 3>
2026-03-27 14:32:20 [azure.servicebus._pyamqp.cbs] DEBUG: CBS status check: state == <CbsAuthState.OK: 0>, expired == False, refresh required == False
2026-03-27 14:32:20 [azure.servicebus._pyamqp.cbs] DEBUG: CBS status check: state == <CbsAuthState.OK: 0>, expired == False, refresh required == False
2026-03-27 14:32:20 [azure.servicebus._pyamqp.cbs] DEBUG: CBS status check: state == <CbsAuthState.OK: 0>, expired == False, refresh required == False
2026-03-27 14:32:20 [azure.servicebus._pyamqp.cbs] DEBUG: CBS status check: state == <CbsAuthState.OK: 0>, expired == False, refresh required == False
2026-03-27 14:32:20 [crawlers.pipelines.post_to_azurebus_auto_selling] DEBUG: [AzureBusPipeline] sent process item CloudEvent for item 974259cd-d0ba-5355-8eec-6bfdf9edef46
2026-03-27 14:32:20 [scrapy.core.scraper] DEBUG: Scraped from <200 https://m.atcdn.co.uk/a/media/e01d7bee96c4456a9e52c93c22d88313.jpg>
{'auction_title': '',
 'c_o_2_emission_value': 49,
 'color': 'White',
 'currency': 'GBP',
 'description': 'Introducing the stunning BMW i8, a true masterpiece of '
                'engineering and design. With only 34978 miles on the clock, '
                'this beauty is just waiting to be taken for a spin.\n'
                '\n'
                'Finished in a sleek White colour, this BMW i8 is sure to turn '
                "heads wherever you go. But it's not just about looks - this "
                'car is packed with extras to enhance your driving experience. '
                'From the 20in Alloy Wheels to the BMW i Spheric Full '
                'Perforated Leather interior in Carum Grey, every detail has '
                'been carefully considered.\n'
                '\n'
                'Enjoy crystal clear sound with the Speaker System by Harman '
                'Kardon and stay connected on the go with Internet '
                "capabilities. And let's not forget the eDrive Exterior Sound "
                'feature, adding an extra layer of excitement to your drive.\n'
                '\n'
                "Don't miss your chance to own this exceptional BMW i8. Visit "
                'us today and experience the thrill of driving this incredible '
                'machine for yourself.',
 'doors_number': 2,
 'emission_standard': 'Euro 6',
 'engine': '1.5L',
 'engine_horse_power': 357,
 'fuel_type': 'Petrol Plug-in Hybrid',
 'id': '974259cd-d0ba-5355-8eec-6bfdf9edef46',
 'is_damaged': False,
 'is_operable': True,
 'link_to_web_offer': 'https://www.autotrader.co.uk/car-details/202603261038863',
 'listing': 'directbuy',
 'location': 'Bexleyheath, Kent',
 'make': 'BMW',
 'model': 'i8',
 'odometer': 34978,
 'odometer_reading_unit': 'mi',
 'origin_country_code': 'GB',
 'original_photo_count': 53,
 'photos': [{'BlobStorageUrl': 'https://alxproduction.blob.core.windows.net/top-deal-listing/listing_974259cd-d0ba-5355-8eec-6bfdf9edef46/974259cd-d0ba-5355-8eec-6bfdf9edef46_image_0.jpg',
             'Extension': 'jpg',
             'FileSizeInBytes': 220787,
             'Height': 1440,
             'OriginUrl': 'https://m.atcdn.co.uk/a/media/e01d7bee96c4456a9e52c93c22d88313.jpg',
             'Width': 1920}],
 'price': 31950,
 'price_includes_vat': True,
 'registration_date': '2016-05-31',
 'seats_number': 4,
 'seller_name': None,
 'title': 'BMW 1.5 i8 4WD 3dr',
 'transmission': 'automatic',
 'year': 2016}
2026-03-27 14:32:20 [urllib3.connectionpool] DEBUG: https://alxsourcingstorageprod.table.core.windows.net:443 "POST /Tables HTTP/1.1" 409 None
2026-03-27 14:32:20 [urllib3.connectionpool] DEBUG: https://alxsourcingstorageprod.table.core.windows.net:443 "GET /ScrapedListings()?$filter=PartitionKey%20eq%20%27auto_selling_autotrader.co.uk%27%20and%20RowKey%20eq%20%27202603261038863%27 HTTP/1.1" 200 None
2026-03-27 14:32:20 [auto_selling_autotrader.co.uk] INFO: Saving data for 202603261038863: {'created_time': 1774621940.396811, 'last_price_update_time': 1774621940.396824}
2026-03-27 14:32:20 [urllib3.connectionpool] DEBUG: https://alxsourcingstorageprod.table.core.windows.net:443 "POST /Tables HTTP/1.1" 409 None
2026-03-27 14:32:20 [urllib3.connectionpool] DEBUG: https://alxsourcingstorageprod.table.core.windows.net:443 "PATCH /ScrapedListings(PartitionKey='auto_selling_autotrader.co.uk',RowKey='202603261038863') HTTP/1.1" 204 0
2026-03-27 14:32:20 [common.azure_blobstore] INFO: Creating container client for top-deal-listing
2026-03-27 14:32:20 [common.azure_blobstore] INFO: Container_client: <azure.storage.blob._container_client.ContainerClient object at 0x7ff6c99f30d0>
2026-03-27 14:32:20 [common.azure_blobstore] INFO: Creating container if not exists
2026-03-27 14:32:20 [urllib3.connectionpool] DEBUG: https://alxproduction.blob.core.windows.net:443 "PUT /top-deal-listing?restype=container HTTP/1.1" 409 230
2026-03-27 14:32:20 [common.azure_blobstore] INFO: Container already exists, skipping creation
2026-03-27 14:32:20 [urllib3.connectionpool] DEBUG: https://alxproduction.blob.core.windows.net:443 "PUT /top-deal-listing/listing_6df70f7a-9ca5-5250-a94f-7cbed4efa748/6df70f7a-9ca5-5250-a94f-7cbed4efa748_image_0.jpg HTTP/1.1" 201 0
2026-03-27 14:32:20 [urllib3.connectionpool] DEBUG: https://alxsourcingstorageprod.table.core.windows.net:443 "POST /Tables HTTP/1.1" 409 None
2026-03-27 14:32:20 [urllib3.connectionpool] DEBUG: https://alxsourcingstorageprod.table.core.windows.net:443 "PATCH /PhotoDownloadCache(PartitionKey='auto_selling_autotrader.co.uk',RowKey='https%3A%252F%252Fm.atcdn.co.uk%252Fa%252Fmedia%252F69c30d115922418786b999656535ee13.jpg') HTTP/1.1" 204 0
2026-03-27 14:32:20 [crawlers.middlewares.photo_download_middleware] INFO: Image https://m.atcdn.co.uk/a/media/69c30d115922418786b999656535ee13.jpg saved to cache
2026-03-27 14:32:20 [crawlers.middlewares.photo_download_middleware] INFO: Downloaded image 0 for item 6df70f7a-9ca5-5250-a94f-7cbed4efa748
2026-03-27 14:32:20 [crawlers.middlewares.photo_download_middleware] INFO: All photos downloaded for item 6df70f7a-9ca5-5250-a94f-7cbed4efa748 sending to next pipeline
2026-03-27 14:32:20 [common.azure_blobstore] INFO: Creating container client for top-deal-listing
2026-03-27 14:32:20 [common.azure_blobstore] INFO: Container_client: <azure.storage.blob._container_client.ContainerClient object at 0x7ff6cc903c10>
2026-03-27 14:32:20 [common.azure_blobstore] INFO: Creating container if not exists
2026-03-27 14:32:20 [urllib3.connectionpool] DEBUG: https://alxproduction.blob.core.windows.net:443 "PUT /top-deal-listing?restype=container HTTP/1.1" 409 230
2026-03-27 14:32:20 [common.azure_blobstore] INFO: Container already exists, skipping creation
2026-03-27 14:32:20 [urllib3.connectionpool] DEBUG: https://alxproduction.blob.core.windows.net:443 "PUT /top-deal-listing/listing_3468b15a-bdf0-51b1-84d6-d1539ff2f494/3468b15a-bdf0-51b1-84d6-d1539ff2f494_image_0.jpg HTTP/1.1" 201 0
2026-03-27 14:32:20 [urllib3.connectionpool] DEBUG: https://alxsourcingstorageprod.table.core.windows.net:443 "POST /Tables HTTP/1.1" 409 None
2026-03-27 14:32:20 [urllib3.connectionpool] DEBUG: https://alxsourcingstorageprod.table.core.windows.net:443 "PATCH /PhotoDownloadCache(PartitionKey='auto_selling_autotrader.co.uk',RowKey='https%3A%252F%252Fm.atcdn.co.uk%252Fa%252Fmedia%252F84ed2d84b7b04fe4b35b9dda30cc46ec.jpg') HTTP/1.1" 204 0
2026-03-27 14:32:20 [crawlers.middlewares.photo_download_middleware] INFO: Image https://m.atcdn.co.uk/a/media/84ed2d84b7b04fe4b35b9dda30cc46ec.jpg saved to cache
2026-03-27 14:32:20 [crawlers.middlewares.photo_download_middleware] INFO: Downloaded image 0 for item 3468b15a-bdf0-51b1-84d6-d1539ff2f494
2026-03-27 14:32:20 [crawlers.middlewares.photo_download_middleware] INFO: All photos downloaded for item 3468b15a-bdf0-51b1-84d6-d1539ff2f494 sending to next pipeline
2026-03-27 14:32:20 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://m.atcdn.co.uk/a/media/b76fa25739594f5292b1e661fa34ab7a.jpg> (referer: https://www.autotrader.co.uk/)
2026-03-27 14:32:20 [crawlers.pipelines.post_to_azurebus_auto_selling] INFO: Azure Bus Pipeline processing item: 6df70f7a-9ca5-5250-a94f-7cbed4efa748 with identifier 202603140701425
2026-03-27 14:32:20 [azure.servicebus._pyamqp.cbs] DEBUG: CBS status check: state == <CbsAuthState.OK: 0>, expired == False, refresh required == False
2026-03-27 14:32:20 [azure.servicebus._pyamqp.cbs] DEBUG: CBS status check: state == <CbsAuthState.OK: 0>, expired == False, refresh required == False
2026-03-27 14:32:20 [crawlers.pipelines.post_to_azurebus_auto_selling] DEBUG: [AzureBusPipeline] sent process item CloudEvent for item 6df70f7a-9ca5-5250-a94f-7cbed4efa748
2026-03-27 14:32:20 [scrapy.core.scraper] DEBUG: Scraped from <200 https://m.atcdn.co.uk/a/media/69c30d115922418786b999656535ee13.jpg>
{'auction_title': '',
 'c_o_2_emission_value': 164,
 'color': 'Black',
 'currency': 'GBP',
 'description': 'Introducing the stunning BMW 5 Series in sleek Black, '
                'boasting a range of luxurious extras to elevate your driving '
                'experience to new heights. With its 20-inch Alloy Wheels, '
                'Gesture Control, and Icon Adaptive LED Headlights, this car '
                'is a true standout on the road.\n'
                '\n'
                'Step inside the BMW 5 Series and experience ultimate comfort '
                'with Dakota Leather seats in Ivory White with Black Interior, '
                'Electric Glass Sunroof, and Fine-Wood Trim Fineline Ridge. '
                'Stay connected on the go with Enhanced Bluetooth Telephone '
                'Preparation and Wireless Charging, while enjoying '
                'crystal-clear sound with the Harman Kardon Loudspeaker '
                'System.\n'
                '\n'
                'Equipped with innovative features like Head-Up Display, '
                'High-Beam Assistant, and Reversing Assist Camera, the BMW 5 '
                'Series ensures a safe and seamless driving experience. The M '
                'Sport Plus Package, Variable Damper Control, and Tailgate '
                'Operation make every journey a pleasure.\n'
                '\n'
                "Don't miss out on the opportunity to own this exceptional BMW "
                '5 Series with its advanced technology, premium comfort, and '
                'striking design. Visit us today and experience luxury like '
                'never before.',
 'doors_number': 4,
 'emission_standard': 'Euro 6',
 'engine': '3L',
 'engine_horse_power': 335,
 'fuel_type': 'Petrol',
 'id': '6df70f7a-9ca5-5250-a94f-7cbed4efa748',
 'is_damaged': False,
 'is_operable': True,
 'link_to_web_offer': 'https://www.autotrader.co.uk/car-details/202603140701425',
 'listing': 'directbuy',
 'location': 'Bexleyheath, Kent',
 'make': 'BMW',
 'model': '5 Series',
 'odometer': 72000,
 'odometer_reading_unit': 'mi',
 'origin_country_code': 'GB',
 'original_photo_count': 88,
 'photos': [{'BlobStorageUrl': 'https://alxproduction.blob.core.windows.net/top-deal-listing/listing_6df70f7a-9ca5-5250-a94f-7cbed4efa748/6df70f7a-9ca5-5250-a94f-7cbed4efa748_image_0.jpg',
             'Extension': 'jpg',
             'FileSizeInBytes': 296706,
             'Height': 1440,
             'OriginUrl': 'https://m.atcdn.co.uk/a/media/69c30d115922418786b999656535ee13.jpg',
             'Width': 1920}],
 'price': 22750,
 'price_includes_vat': True,
 'registration_date': '2017-03-01',
 'seats_number': 5,
 'seller_name': None,
 'title': 'BMW 5 Series 3.0 540i xDrive M Sport Auto 4WD 4dr',
 'transmission': 'automatic',
 'trim': 'M Sport',
 'year': 2017}
2026-03-27 14:32:20 [urllib3.connectionpool] DEBUG: https://alxsourcingstorageprod.table.core.windows.net:443 "POST /Tables HTTP/1.1" 409 None
2026-03-27 14:32:20 [urllib3.connectionpool] DEBUG: https://alxsourcingstorageprod.table.core.windows.net:443 "GET /ScrapedListings()?$filter=PartitionKey%20eq%20%27auto_selling_autotrader.co.uk%27%20and%20RowKey%20eq%20%27202603140701425%27 HTTP/1.1" 200 None
2026-03-27 14:32:20 [auto_selling_autotrader.co.uk] INFO: Saving data for 202603140701425: {'created_time': 1774621940.916065, 'last_price_update_time': 1774621940.916078}
2026-03-27 14:32:20 [urllib3.connectionpool] DEBUG: https://alxsourcingstorageprod.table.core.windows.net:443 "POST /Tables HTTP/1.1" 409 None
2026-03-27 14:32:20 [urllib3.connectionpool] DEBUG: https://alxsourcingstorageprod.table.core.windows.net:443 "PATCH /ScrapedListings(PartitionKey='auto_selling_autotrader.co.uk',RowKey='202603140701425') HTTP/1.1" 204 0
2026-03-27 14:32:20 [crawlers.pipelines.post_to_azurebus_auto_selling] INFO: Azure Bus Pipeline processing item: 3468b15a-bdf0-51b1-84d6-d1539ff2f494 with identifier 202602119856878
2026-03-27 14:32:20 [azure.servicebus._pyamqp.cbs] DEBUG: CBS status check: state == <CbsAuthState.OK: 0>, expired == False, refresh required == False
2026-03-27 14:32:20 [azure.servicebus._pyamqp.cbs] DEBUG: CBS status check: state == <CbsAuthState.OK: 0>, expired == False, refresh required == False
2026-03-27 14:32:20 [crawlers.pipelines.post_to_azurebus_auto_selling] DEBUG: [AzureBusPipeline] sent process item CloudEvent for item 3468b15a-bdf0-51b1-84d6-d1539ff2f494
2026-03-27 14:32:20 [scrapy.core.scraper] DEBUG: Scraped from <200 https://m.atcdn.co.uk/a/media/84ed2d84b7b04fe4b35b9dda30cc46ec.jpg>
{'auction_title': '',
 'c_o_2_emission_value': 158,
 'color': 'Black',
 'currency': 'GBP',
 'description': 'Introducing the stunning Mercedes-Benz GLC 2.1 GLC 220 D '
                '4Matic AMG Line Premium+ Auto 4WD 5dr. This luxurious SUV is '
                'the epitome of style and performance, perfect for those who '
                'demand the best.\n'
                '\n'
                'With only 61,183 miles on the clock and 2 previous owners, '
                'this Mercedes-Benz GLC is in impeccable condition, ready to '
                'take on any adventure you throw its way.\n'
                '\n'
                'Finished in a sleek Black exterior, this vehicle exudes '
                'sophistication and class. The AMG Line Premium+ trim adds a '
                'touch of sportiness and luxury, making every drive a truly '
                'unforgettable experience.\n'
                '\n'
                "Don't miss your chance to own this exceptional Mercedes-Benz "
                'GLC. Visit us today and experience the ultimate in automotive '
                'excellence.',
 'doors_number': 5,
 'emission_standard': 'Euro 6',
 'engine': '2.1L',
 'engine_horse_power': 168,
 'fuel_type': 'Diesel',
 'id': '3468b15a-bdf0-51b1-84d6-d1539ff2f494',
 'is_damaged': False,
 'is_operable': True,
 'link_to_web_offer': 'https://www.autotrader.co.uk/car-details/202602119856878',
 'listing': 'directbuy',
 'location': 'Bexleyheath, Kent',
 'make': 'Mercedes-Benz',
 'model': 'GLC',
 'odometer': 61183,
 'odometer_reading_unit': 'mi',
 'origin_country_code': 'GB',
 'original_photo_count': 57,
 'photos': [{'BlobStorageUrl': 'https://alxproduction.blob.core.windows.net/top-deal-listing/listing_3468b15a-bdf0-51b1-84d6-d1539ff2f494/3468b15a-bdf0-51b1-84d6-d1539ff2f494_image_0.jpg',
             'Extension': 'jpg',
             'FileSizeInBytes': 286815,
             'Height': 1440,
             'OriginUrl': 'https://m.atcdn.co.uk/a/media/84ed2d84b7b04fe4b35b9dda30cc46ec.jpg',
             'Width': 1920}],
 'price': 18950,
 'price_includes_vat': True,
 'registration_date': '2019-01-31',
 'seats_number': 5,
 'seller_name': None,
 'title': 'Mercedes-Benz 2.1 GLC 220 D 4Matic AMG Line Premium+ Auto 4WD 5dr',
 'transmission': 'automatic',
 'trim': 'AMG Line',
 'year': 2019}
2026-03-27 14:32:20 [urllib3.connectionpool] DEBUG: https://alxsourcingstorageprod.table.core.windows.net:443 "POST /Tables HTTP/1.1" 409 None
2026-03-27 14:32:20 [urllib3.connectionpool] DEBUG: https://alxsourcingstorageprod.table.core.windows.net:443 "GET /ScrapedListings()?$filter=PartitionKey%20eq%20%27auto_selling_autotrader.co.uk%27%20and%20RowKey%20eq%20%27202602119856878%27 HTTP/1.1" 200 None
2026-03-27 14:32:20 [auto_selling_autotrader.co.uk] INFO: Saving data for 202602119856878: {'created_time': 1774621940.96529, 'last_price_update_time': 1774621940.9653}
2026-03-27 14:32:20 [urllib3.connectionpool] DEBUG: https://alxsourcingstorageprod.table.core.windows.net:443 "POST /Tables HTTP/1.1" 409 None
2026-03-27 14:32:21 [urllib3.connectionpool] DEBUG: https://alxsourcingstorageprod.table.core.windows.net:443 "PATCH /ScrapedListings(PartitionKey='auto_selling_autotrader.co.uk',RowKey='202602119856878') HTTP/1.1" 204 0
2026-03-27 14:32:21 [crawlers.pipelines.post_to_azurebus_auto_selling] INFO: Azure Bus Pipeline processing item: af31ee35-34d3-52e9-82dc-f19adac01cd6 with identifier 202603160742123
2026-03-27 14:32:21 [azure.servicebus._pyamqp.cbs] DEBUG: CBS status check: state == <CbsAuthState.OK: 0>, expired == False, refresh required == False
2026-03-27 14:32:21 [azure.servicebus._pyamqp.cbs] DEBUG: CBS status check: state == <CbsAuthState.OK: 0>, expired == False, refresh required == False
2026-03-27 14:32:21 [crawlers.pipelines.post_to_azurebus_auto_selling] DEBUG: [AzureBusPipeline] sent process item CloudEvent for item af31ee35-34d3-52e9-82dc-f19adac01cd6
2026-03-27 14:32:21 [scrapy.core.scraper] DEBUG: Scraped from Could not open CONNECT tunnel with proxy ddc.oxylabs.io:8000 [{'status': 429, 'reason': b'Too Many Requests'}]
{'auction_title': '',
 'c_o_2_emission_value': 152,
 'color': 'Grey',
 'currency': 'GBP',
 'description': 'Introducing the prestigious BMW 6 Series 2.0 630i M Sport '
                'Auto 5dr. This luxury vehicle has been meticulously cared for '
                'by its 2 previous owners, ensuring top-notch performance and '
                'style.\n'
                '\n'
                'Finished in a sleek Grey exterior, this BMW 6 Series is a '
                'true head-turner on the road. With extras like Remote '
                'Services and Sophisto Grey Xirallic, you can enjoy a seamless '
                'driving experience with added convenience and '
                'sophistication.\n'
                '\n'
                'Experience the ultimate blend of power, elegance, and '
                "cutting-edge technology with the BMW 6 Series. Don't miss out "
                'on the opportunity to own this exceptional vehicle. Contact '
                'us today to schedule a test drive and elevate your driving '
                'experience to the next level.',
 'doors_number': 5,
 'emission_standard': 'Euro 6',
 'engine': '2L',
 'engine_horse_power': 254,
 'fuel_type': 'Petrol',
 'id': 'af31ee35-34d3-52e9-82dc-f19adac01cd6',
 'is_damaged': False,
 'is_operable': True,
 'link_to_web_offer': 'https://www.autotrader.co.uk/car-details/202603160742123',
 'listing': 'directbuy',
 'location': 'Bexleyheath, Kent',
 'make': 'BMW',
 'model': '6 Series Gran Turismo',
 'odometer': 89573,
 'odometer_reading_unit': 'mi',
 'origin_country_code': 'GB',
 'original_photo_count': 56,
 'photo_urls': ['https://m.atcdn.co.uk/a/media/f2bd491f3a984914868382c9a7c0eabf.jpg'],
 'photos': [],
 'price': 13950,
 'price_includes_vat': True,
 'registration_date': '2018-04-30',
 'seats_number': 5,
 'seller_name': None,
 'title': 'BMW 6 SERIES 2.0 630I M Sport Auto 5dr',
 'transmission': 'automatic',
 'trim': 'M Sport',
 'year': 2018}
2026-03-27 14:32:21 [urllib3.connectionpool] DEBUG: https://alxsourcingstorageprod.table.core.windows.net:443 "POST /Tables HTTP/1.1" 409 None
2026-03-27 14:32:21 [urllib3.connectionpool] DEBUG: https://alxsourcingstorageprod.table.core.windows.net:443 "GET /ScrapedListings()?$filter=PartitionKey%20eq%20%27auto_selling_autotrader.co.uk%27%20and%20RowKey%20eq%20%27202603160742123%27 HTTP/1.1" 200 None
2026-03-27 14:32:21 [auto_selling_autotrader.co.uk] INFO: Saving data for 202603160742123: {'created_time': 1774621941.040537, 'last_price_update_time': 1774621941.040554}
2026-03-27 14:32:21 [urllib3.connectionpool] DEBUG: https://alxsourcingstorageprod.table.core.windows.net:443 "POST /Tables HTTP/1.1" 409 None
2026-03-27 14:32:21 [urllib3.connectionpool] DEBUG: https://alxsourcingstorageprod.table.core.windows.net:443 "PATCH /ScrapedListings(PartitionKey='auto_selling_autotrader.co.uk',RowKey='202603160742123') HTTP/1.1" 204 0
2026-03-27 14:32:21 [crawlers.pipelines.post_to_azurebus_auto_selling] INFO: Azure Bus Pipeline processing item: 871df821-de01-5244-9063-45ea1ccef53b with identifier 202603060470389
2026-03-27 14:32:21 [azure.servicebus._pyamqp.cbs] DEBUG: CBS status check: state == <CbsAuthState.OK: 0>, expired == False, refresh required == False
2026-03-27 14:32:21 [azure.servicebus._pyamqp.cbs] DEBUG: CBS status check: state == <CbsAuthState.OK: 0>, expired == False, refresh required == False
2026-03-27 14:32:21 [crawlers.pipelines.post_to_azurebus_auto_selling] DEBUG: [AzureBusPipeline] sent process item CloudEvent for item 871df821-de01-5244-9063-45ea1ccef53b
2026-03-27 14:32:21 [scrapy.core.scraper] DEBUG: Scraped from Could not open CONNECT tunnel with proxy ddc.oxylabs.io:8000 [{'status': 429, 'reason': b'Too Many Requests'}]
{'auction_title': '',
 'c_o_2_emission_value': 200,
 'color': 'Blue',
 'currency': 'GBP',
 'description': 'Introducing the stunning Mercedes-Benz C-Class 4.0 AMG C 63 S '
                'Premium Auto 2dr, a true masterpiece of automotive '
                'engineering. Finished in a sleek and sophisticated Blue, this '
                'luxury car is sure to turn heads wherever you go.\n'
                '\n'
                'Step inside the Mercedes-Benz C-Class and experience the '
                'pinnacle of comfort and style. The premium interior is '
                'crafted with the finest materials, ensuring a first-class '
                'driving experience every time you take the wheel.\n'
                '\n'
                "But don't be fooled by its elegance - under the hood lies a "
                'powerful 4.0-litre engine that delivers exhilarating '
                'performance. The AMG C 63 S model is designed for those who '
                'demand the very best in terms of speed and agility.\n'
                '\n'
                "Whether you're cruising down the motorway or navigating city "
                'streets, the Mercedes-Benz C-Class offers a smooth and '
                'responsive ride that will make every journey a pleasure.\n'
                '\n'
                "Don't miss your chance to own this exceptional vehicle. Visit "
                'our showroom today and experience the luxury of the '
                'Mercedes-Benz C-Class for yourself.',
 'doors_number': 2,
 'emission_standard': 'Euro 6',
 'engine': '4L',
 'engine_horse_power': 503,
 'fuel_type': 'Petrol',
 'id': '871df821-de01-5244-9063-45ea1ccef53b',
 'is_damaged': False,
 'is_operable': True,
 'link_to_web_offer': 'https://www.autotrader.co.uk/car-details/202603060470389',
 'listing': 'directbuy',
 'location': 'Bexleyheath, Kent',
 'make': 'Mercedes-Benz',
 'model': 'C Class',
 'odometer': 74583,
 'odometer_reading_unit': 'mi',
 'origin_country_code': 'GB',
 'original_photo_count': 64,
 'photo_urls': ['https://m.atcdn.co.uk/a/media/6a61a25c3e1c43ffbab9668e47fbeefb.jpg'],
 'photos': [],
 'price': 28950,
 'price_includes_vat': True,
 'registration_date': '2017-04-25',
 'seats_number': 4,
 'seller_name': None,
 'title': 'Mercedes-Benz C-Class 4.0 AMG C 63 S Premium Auto 2dr',
 'transmission': 'automatic',
 'trim': 'AMG S',
 'year': 2017}
2026-03-27 14:32:21 [urllib3.connectionpool] DEBUG: https://alxsourcingstorageprod.table.core.windows.net:443 "POST /Tables HTTP/1.1" 409 None
2026-03-27 14:32:21 [urllib3.connectionpool] DEBUG: https://alxsourcingstorageprod.table.core.windows.net:443 "GET /ScrapedListings()?$filter=PartitionKey%20eq%20%27auto_selling_autotrader.co.uk%27%20and%20RowKey%20eq%20%27202603060470389%27 HTTP/1.1" 200 None
2026-03-27 14:32:21 [auto_selling_autotrader.co.uk] INFO: Saving data for 202603060470389: {'created_time': 1774621941.09487, 'last_price_update_time': 1774621941.094883}
2026-03-27 14:32:21 [urllib3.connectionpool] DEBUG: https://alxsourcingstorageprod.table.core.windows.net:443 "POST /Tables HTTP/1.1" 409 None
2026-03-27 14:32:21 [urllib3.connectionpool] DEBUG: https://alxsourcingstorageprod.table.core.windows.net:443 "PATCH /ScrapedListings(PartitionKey='auto_selling_autotrader.co.uk',RowKey='202603060470389') HTTP/1.1" 204 0
2026-03-27 14:32:21 [crawlers.pipelines.post_to_azurebus_auto_selling] INFO: Azure Bus Pipeline processing item: d49cefb8-2285-5773-949e-deae4e7b6919 with identifier 202603060462594
2026-03-27 14:32:21 [azure.servicebus._pyamqp.cbs] DEBUG: CBS status check: state == <CbsAuthState.OK: 0>, expired == False, refresh required == False
2026-03-27 14:32:21 [azure.servicebus._pyamqp.cbs] DEBUG: CBS status check: state == <CbsAuthState.OK: 0>, expired == False, refresh required == False
2026-03-27 14:32:21 [crawlers.pipelines.post_to_azurebus_auto_selling] DEBUG: [AzureBusPipeline] sent process item CloudEvent for item d49cefb8-2285-5773-949e-deae4e7b6919
2026-03-27 14:32:21 [scrapy.core.scraper] DEBUG: Scraped from Could not open CONNECT tunnel with proxy ddc.oxylabs.io:8000 [{'status': 429, 'reason': b'Too Many Requests'}]
{'auction_title': '',
 'c_o_2_emission_value': 196,
 'color': 'Grey',
 'currency': 'GBP',
 'description': 'Introducing the Porsche Macan - a true masterpiece of '
                'engineering and design. This 3.0 Macan S Semi-Auto 4WD 5dr '
                'model is the epitome of luxury and performance.\n'
                '\n'
                'With a sleek Grey finish, this Porsche Macan is sure to turn '
                'heads wherever you go. It has been meticulously cared for by '
                'its 2 previous owners, ensuring that it is in pristine '
                'condition both inside and out.\n'
                '\n'
                'Step inside and experience the ultimate in comfort and '
                'sophistication. The Porsche Macan boasts a range of premium '
                'features, from leather upholstery to advanced technology '
                'systems, all designed to enhance your driving experience.\n'
                '\n'
                "But don't be fooled by its elegance - under the hood lies a "
                'powerful engine that delivers exhilarating performance on '
                "every journey. Whether you're cruising through the city "
                'streets or tackling winding country roads, the Porsche Macan '
                'will exceed your expectations.\n'
                '\n'
                "Don't miss your chance to own this stunning Porsche Macan. "
                'Visit us today and take it for a test drive to truly '
                'appreciate its beauty and capabilities. Experience the thrill '
                "of driving a Porsche - you won't be disappointed.",
 'doors_number': 5,
 'emission_standard': 'Euro 6',
 'engine': '3L',
 'engine_horse_power': 349,
 'fuel_type': 'Petrol',
 'id': 'd49cefb8-2285-5773-949e-deae4e7b6919',
 'is_damaged': False,
 'is_operable': True,
 'link_to_web_offer': 'https://www.autotrader.co.uk/car-details/202603060462594',
 'listing': 'directbuy',
 'location': 'Bexleyheath, Kent',
 'make': 'Porsche',
 'model': 'Macan',
 'odometer': 70521,
 'odometer_reading_unit': 'mi',
 'origin_country_code': 'GB',
 'original_photo_count': 70,
 'photo_urls': ['https://m.atcdn.co.uk/a/media/ce971ae9d7a248cda16c4d42bd2bba09.jpg'],
 'photos': [],
 'price': 28450,
 'price_includes_vat': True,
 'registration_date': '2019-03-18',
 'seats_number': 5,
 'seller_name': None,
 'title': 'Porsche 3.0 Macan S Semi-Auto 4WD 5dr',
 'transmission': 'automatic',
 'trim': 'S',
 'year': 2019}
2026-03-27 14:32:21 [urllib3.connectionpool] DEBUG: https://alxsourcingstorageprod.table.core.windows.net:443 "POST /Tables HTTP/1.1" 409 None
2026-03-27 14:32:21 [urllib3.connectionpool] DEBUG: https://alxsourcingstorageprod.table.core.windows.net:443 "GET /ScrapedListings()?$filter=PartitionKey%20eq%20%27auto_selling_autotrader.co.uk%27%20and%20RowKey%20eq%20%27202603060462594%27 HTTP/1.1" 200 None
2026-03-27 14:32:21 [auto_selling_autotrader.co.uk] INFO: Saving data for 202603060462594: {'created_time': 1774621941.146972, 'last_price_update_time': 1774621941.146986}
2026-03-27 14:32:21 [urllib3.connectionpool] DEBUG: https://alxsourcingstorageprod.table.core.windows.net:443 "POST /Tables HTTP/1.1" 409 None
2026-03-27 14:32:21 [urllib3.connectionpool] DEBUG: https://alxsourcingstorageprod.table.core.windows.net:443 "PATCH /ScrapedListings(PartitionKey='auto_selling_autotrader.co.uk',RowKey='202603060462594') HTTP/1.1" 204 0
2026-03-27 14:32:21 [crawlers.pipelines.post_to_azurebus_auto_selling] INFO: Azure Bus Pipeline processing item: 5821636a-81cb-5ff8-8018-d3d243d4b8a7 with identifier 202603060461748
2026-03-27 14:32:21 [azure.servicebus._pyamqp.cbs] DEBUG: CBS status check: state == <CbsAuthState.OK: 0>, expired == False, refresh required == False
2026-03-27 14:32:21 [azure.servicebus._pyamqp.cbs] DEBUG: CBS status check: state == <CbsAuthState.OK: 0>, expired == False, refresh required == False
2026-03-27 14:32:21 [crawlers.pipelines.post_to_azurebus_auto_selling] DEBUG: [AzureBusPipeline] sent process item CloudEvent for item 5821636a-81cb-5ff8-8018-d3d243d4b8a7
2026-03-27 14:32:21 [scrapy.core.scraper] DEBUG: Scraped from Could not open CONNECT tunnel with proxy ddc.oxylabs.io:8000 [{'status': 429, 'reason': b'Too Many Requests'}]
{'auction_title': '',
 'c_o_2_emission_value': 156,
 'color': 'Black',
 'currency': 'GBP',
 'description': 'Introducing the luxurious Mercedes-Benz GLE 2.1 GLE 250 D '
                '4Matic AMG Line Premium Auto 4WD 5dr. This stunning vehicle, '
                'finished in sleek Black, exudes sophistication and style.\n'
                '\n'
                'With only 2 previous owners, this Mercedes-Benz GLE has been '
                'meticulously cared for and is in pristine condition. The AMG '
                'Line Premium Auto 4WD ensures a smooth and powerful driving '
                'experience, perfect for navigating any road with ease.\n'
                '\n'
                'Step into the lap of luxury with the premium features of this '
                'Mercedes-Benz GLE, from the plush interior to the '
                "cutting-edge technology. Whether you're cruising through the "
                'city streets or embarking on a long journey, this vehicle '
                'offers comfort, performance, and elegance in one package.\n'
                '\n'
                "Don't miss your chance to own this exceptional Mercedes-Benz "
                'GLE. Contact us today to schedule a test drive and experience '
                'the ultimate in automotive luxury.',
 'doors_number': 5,
 'emission_standard': 'Euro 6',
 'engine': '2.1L',
 'engine_horse_power': 201,
 'fuel_type': 'Diesel',
 'id': '5821636a-81cb-5ff8-8018-d3d243d4b8a7',
 'is_damaged': False,
 'is_operable': True,
 'link_to_web_offer': 'https://www.autotrader.co.uk/car-details/202603060461748',
 'listing': 'directbuy',
 'location': 'Bexleyheath, Kent',
 'make': 'Mercedes-Benz',
 'model': 'GLE',
 'odometer': 79001,
 'odometer_reading_unit': 'mi',
 'origin_country_code': 'GB',
 'original_photo_count': 51,
 'photo_urls': ['https://m.atcdn.co.uk/a/media/300f30eb4fb247e59ae2e10a87eb8967.jpg'],
 'photos': [],
 'price': 16450,
 'price_includes_vat': True,
 'registration_date': '2016-09-13',
 'seats_number': 5,
 'seller_name': None,
 'title': 'Mercedes-Benz 2.1 GLE 250 D 4Matic AMG Line Premium Auto 4WD 5dr',
 'transmission': 'automatic',
 'trim': 'AMG Line',
 'year': 2016}
2026-03-27 14:32:21 [urllib3.connectionpool] DEBUG: https://alxsourcingstorageprod.table.core.windows.net:443 "POST /Tables HTTP/1.1" 409 None
2026-03-27 14:32:21 [urllib3.connectionpool] DEBUG: https://alxsourcingstorageprod.table.core.windows.net:443 "GET /ScrapedListings()?$filter=PartitionKey%20eq%20%27auto_selling_autotrader.co.uk%27%20and%20RowKey%20eq%20%27202603060461748%27 HTTP/1.1" 200 None
2026-03-27 14:32:21 [auto_selling_autotrader.co.uk] INFO: Saving data for 202603060461748: {'created_time': 1774621941.193145, 'last_price_update_time': 1774621941.193158}
2026-03-27 14:32:21 [urllib3.connectionpool] DEBUG: https://alxsourcingstorageprod.table.core.windows.net:443 "POST /Tables HTTP/1.1" 409 None
2026-03-27 14:32:21 [urllib3.connectionpool] DEBUG: https://alxsourcingstorageprod.table.core.windows.net:443 "PATCH /ScrapedListings(PartitionKey='auto_selling_autotrader.co.uk',RowKey='202603060461748') HTTP/1.1" 204 0
2026-03-27 14:32:21 [crawlers.pipelines.post_to_azurebus_auto_selling] INFO: Azure Bus Pipeline processing item: d66d0515-5f45-5928-a736-7576880947a9 with identifier 202602230143291
2026-03-27 14:32:21 [azure.servicebus._pyamqp.cbs] DEBUG: CBS status check: state == <CbsAuthState.OK: 0>, expired == False, refresh required == False
2026-03-27 14:32:21 [azure.servicebus._pyamqp.cbs] DEBUG: CBS status check: state == <CbsAuthState.OK: 0>, expired == False, refresh required == False
2026-03-27 14:32:21 [crawlers.pipelines.post_to_azurebus_auto_selling] DEBUG: [AzureBusPipeline] sent process item CloudEvent for item d66d0515-5f45-5928-a736-7576880947a9
2026-03-27 14:32:21 [scrapy.core.scraper] DEBUG: Scraped from Could not open CONNECT tunnel with proxy ddc.oxylabs.io:8000 [{'status': 429, 'reason': b'Too Many Requests'}]
{'auction_title': '',
 'c_o_2_emission_value': 194,
 'color': 'White',
 'currency': 'GBP',
 'description': 'Introducing the stunning BMW M4, a true masterpiece on '
                'wheels. With only 24,093 miles on the clock and 2 careful '
                'owners, fitted with 4 Michelin Pilot Sport tyres - this '
                'beauty is ready to hit the road in style.\n'
                '\n'
                'Finished in a sleek White exterior, this BMW M4 comes loaded '
                'with extras that will take your driving experience to the '
                'next level. From the Adaptive LED Headlights with High-Beam '
                'Assistant to the Extended Merino Leather interior in Black, '
                'every detail has been carefully crafted for luxury and '
                'performance.\n'
                '\n'
                "But that's not all - this M4 also features Extended Storage, "
                'Head-Up Display, High-Beam Assistant, Carbon Fibre Interior '
                'Trim Finisher, Harman Kardon Loudspeaker System, Reversing '
                'Assist Camera, Speed Limit Display, Sun Protection Glazing, '
                'and Surround-View technology.\n'
                '\n'
                "Don't miss your chance to own this exceptional BMW M4 with "
                'the Competition Edition Package. Visit us today and '
                'experience the thrill of driving a true icon.',
 'doors_number': 2,
 'emission_standard': 'Euro 6',
 'engine': '3L',
 'engine_horse_power': 444,
 'fuel_type': 'Petrol',
 'id': 'd66d0515-5f45-5928-a736-7576880947a9',
 'is_damaged': False,
 'is_operable': True,
 'link_to_web_offer': 'https://www.autotrader.co.uk/car-details/202602230143291',
 'listing': 'directbuy',
 'location': 'Bexleyheath, Kent',
 'make': 'BMW',
 'model': 'M4',
 'odometer': 24093,
 'odometer_reading_unit': 'mi',
 'origin_country_code': 'GB',
 'original_photo_count': 65,
 'photo_urls': ['https://m.atcdn.co.uk/a/media/bbe25bfe791a44769c7c516c2e1782b6.jpg'],
 'photos': [],
 'price': 31450,
 'price_includes_vat': True,
 'registration_date': '2018-06-29',
 'seats_number': 4,
 'seller_name': None,
 'title': 'BMW 3.0 M4 Competition Edition Package Semi-Auto 2dr',
 'transmission': 'automatic',
 'trim': 'Competition',
 'year': 2018}
2026-03-27 14:32:21 [urllib3.connectionpool] DEBUG: https://alxsourcingstorageprod.table.core.windows.net:443 "POST /Tables HTTP/1.1" 409 None
2026-03-27 14:32:21 [urllib3.connectionpool] DEBUG: https://alxsourcingstorageprod.table.core.windows.net:443 "GET /ScrapedListings()?$filter=PartitionKey%20eq%20%27auto_selling_autotrader.co.uk%27%20and%20RowKey%20eq%20%27202602230143291%27 HTTP/1.1" 200 None
2026-03-27 14:32:21 [auto_selling_autotrader.co.uk] INFO: Saving data for 202602230143291: {'created_time': 1774621941.24583, 'last_price_update_time': 1774621941.245844}
2026-03-27 14:32:21 [urllib3.connectionpool] DEBUG: https://alxsourcingstorageprod.table.core.windows.net:443 "POST /Tables HTTP/1.1" 409 None
2026-03-27 14:32:21 [urllib3.connectionpool] DEBUG: https://alxsourcingstorageprod.table.core.windows.net:443 "PATCH /ScrapedListings(PartitionKey='auto_selling_autotrader.co.uk',RowKey='202602230143291') HTTP/1.1" 204 0
2026-03-27 14:32:21 [crawlers.pipelines.post_to_azurebus_auto_selling] INFO: Azure Bus Pipeline processing item: 489fb03a-9eca-5abc-97cf-3b22a8200458 with identifier 202602210112515
2026-03-27 14:32:21 [azure.servicebus._pyamqp.cbs] DEBUG: CBS status check: state == <CbsAuthState.OK: 0>, expired == False, refresh required == False
2026-03-27 14:32:21 [azure.servicebus._pyamqp.cbs] DEBUG: CBS status check: state == <CbsAuthState.OK: 0>, expired == False, refresh required == False
2026-03-27 14:32:21 [crawlers.pipelines.post_to_azurebus_auto_selling] DEBUG: [AzureBusPipeline] sent process item CloudEvent for item 489fb03a-9eca-5abc-97cf-3b22a8200458
2026-03-27 14:32:21 [scrapy.core.scraper] DEBUG: Scraped from Could not open CONNECT tunnel with proxy ddc.oxylabs.io:8000 [{'status': 429, 'reason': b'Too Many Requests'}]
{'auction_title': '',
 'c_o_2_emission_value': 155,
 'color': 'Black',
 'currency': 'GBP',
 'description': 'Introducing the stunning Mini Hatch 2.0 John Cooper Works '
                '3dr, a true icon of British motoring excellence. With a sleek '
                'Black finish and a sporty design, this Mini Hatch is sure to '
                'turn heads wherever you go.\n'
                '\n'
                'This Mini Hatch comes with a range of fantastic extras, '
                'including 18in BOLA Alloy Wheels, Air Conditioning with '
                'Automatic Dual Zone control for maximum comfort, and the '
                'CHILI Pack for added style and convenience.\n'
                '\n'
                'Step inside and experience luxury with Dinamica-Leather '
                'Carbon Black seats, enhanced Bluetooth with USB Audio for '
                'seamless connectivity, and a Harman Kardon Hi-Fi System for '
                'an incredible audio experience.\n'
                '\n'
                'With features like Seat Heating for Driver and Front '
                'Passenger, Rain Sensor and Automatic Headlight Activation, '
                'and the MINI Excitement Pack, this Mini Hatch is designed to '
                'make every journey a pleasure.\n'
                '\n'
                "Don't miss out on this incredible opportunity to own a Mini "
                'Hatch with only 69859 miles on the clock. Contact us today to '
                'schedule a test drive and experience the thrill of driving a '
                'Mini Hatch for yourself.',
 'doors_number': 3,
 'emission_standard': 'Euro 6',
 'engine': '2L',
 'engine_horse_power': 228,
 'fuel_type': 'Petrol',
 'id': '489fb03a-9eca-5abc-97cf-3b22a8200458',
 'is_damaged': False,
 'is_operable': True,
 'link_to_web_offer': 'https://www.autotrader.co.uk/car-details/202602210112515',
 'listing': 'directbuy',
 'location': 'Bexleyheath, Kent',
 'make': 'MINI',
 'model': 'Hatch',
 'odometer': 69859,
 'odometer_reading_unit': 'mi',
 'origin_country_code': 'GB',
 'original_photo_count': 57,
 'photo_urls': ['https://m.atcdn.co.uk/a/media/a4b67838af784cc6850ab17f030ffb2e.jpg'],
 'photos': [],
 'price': 10450,
 'price_includes_vat': True,
 'registration_date': '2015-09-01',
 'seats_number': 4,
 'seller_name': None,
 'title': 'MINI Hatch 2.0 John Cooper Works 3dr',
 'transmission': 'manual',
 'trim': 'John Cooper Works',
 'year': 2015}
2026-03-27 14:32:21 [urllib3.connectionpool] DEBUG: https://alxsourcingstorageprod.table.core.windows.net:443 "POST /Tables HTTP/1.1" 409 None
2026-03-27 14:32:21 [urllib3.connectionpool] DEBUG: https://alxsourcingstorageprod.table.core.windows.net:443 "GET /ScrapedListings()?$filter=PartitionKey%20eq%20%27auto_selling_autotrader.co.uk%27%20and%20RowKey%20eq%20%27202602210112515%27 HTTP/1.1" 200 None
2026-03-27 14:32:21 [auto_selling_autotrader.co.uk] INFO: Saving data for 202602210112515: {'created_time': 1774621941.297823, 'last_price_update_time': 1774621941.297835}
2026-03-27 14:32:21 [urllib3.connectionpool] DEBUG: https://alxsourcingstorageprod.table.core.windows.net:443 "POST /Tables HTTP/1.1" 409 None
2026-03-27 14:32:21 [urllib3.connectionpool] DEBUG: https://alxsourcingstorageprod.table.core.windows.net:443 "PATCH /ScrapedListings(PartitionKey='auto_selling_autotrader.co.uk',RowKey='202602210112515') HTTP/1.1" 204 0
2026-03-27 14:32:21 [crawlers.middlewares.photo_download_middleware] DEBUG: Failed to download image 0 for item af31ee35-34d3-52e9-82dc-f19adac01cd6 due to Could not open CONNECT tunnel with proxy ddc.oxylabs.io:8000 [{'status': 429, 'reason': b'Too Many Requests'}]
2026-03-27 14:32:21 [crawlers.middlewares.photo_download_middleware] DEBUG: Failed to download image 0 for item 871df821-de01-5244-9063-45ea1ccef53b due to Could not open CONNECT tunnel with proxy ddc.oxylabs.io:8000 [{'status': 429, 'reason': b'Too Many Requests'}]
2026-03-27 14:32:21 [crawlers.middlewares.photo_download_middleware] DEBUG: Failed to download image 0 for item d49cefb8-2285-5773-949e-deae4e7b6919 due to Could not open CONNECT tunnel with proxy ddc.oxylabs.io:8000 [{'status': 429, 'reason': b'Too Many Requests'}]
2026-03-27 14:32:21 [crawlers.middlewares.photo_download_middleware] DEBUG: Failed to download image 0 for item 5821636a-81cb-5ff8-8018-d3d243d4b8a7 due to Could not open CONNECT tunnel with proxy ddc.oxylabs.io:8000 [{'status': 429, 'reason': b'Too Many Requests'}]
2026-03-27 14:32:21 [crawlers.middlewares.photo_download_middleware] DEBUG: Failed to download image 0 for item d66d0515-5f45-5928-a736-7576880947a9 due to Could not open CONNECT tunnel with proxy ddc.oxylabs.io:8000 [{'status': 429, 'reason': b'Too Many Requests'}]
2026-03-27 14:32:21 [crawlers.middlewares.photo_download_middleware] DEBUG: Failed to download image 0 for item 489fb03a-9eca-5abc-97cf-3b22a8200458 due to Could not open CONNECT tunnel with proxy ddc.oxylabs.io:8000 [{'status': 429, 'reason': b'Too Many Requests'}]
2026-03-27 14:32:21 [common.azure_blobstore] INFO: Creating container client for top-deal-listing
2026-03-27 14:32:21 [common.azure_blobstore] INFO: Container_client: <azure.storage.blob._container_client.ContainerClient object at 0x7ff6c985c050>
2026-03-27 14:32:21 [common.azure_blobstore] INFO: Creating container if not exists
2026-03-27 14:32:21 [urllib3.connectionpool] DEBUG: https://alxproduction.blob.core.windows.net:443 "PUT /top-deal-listing?restype=container HTTP/1.1" 409 230
2026-03-27 14:32:21 [common.azure_blobstore] INFO: Container already exists, skipping creation
2026-03-27 14:32:21 [urllib3.connectionpool] DEBUG: https://alxproduction.blob.core.windows.net:443 "PUT /top-deal-listing/listing_73d996ce-71d6-5e2b-8b11-8a08fd87c5f5/73d996ce-71d6-5e2b-8b11-8a08fd87c5f5_image_0.jpg HTTP/1.1" 201 0
2026-03-27 14:32:21 [urllib3.connectionpool] DEBUG: https://alxsourcingstorageprod.table.core.windows.net:443 "POST /Tables HTTP/1.1" 409 None
2026-03-27 14:32:21 [urllib3.connectionpool] DEBUG: https://alxsourcingstorageprod.table.core.windows.net:443 "PATCH /PhotoDownloadCache(PartitionKey='auto_selling_autotrader.co.uk',RowKey='https%3A%252F%252Fm.atcdn.co.uk%252Fa%252Fmedia%252Fb76fa25739594f5292b1e661fa34ab7a.jpg') HTTP/1.1" 204 0
2026-03-27 14:32:21 [crawlers.middlewares.photo_download_middleware] INFO: Image https://m.atcdn.co.uk/a/media/b76fa25739594f5292b1e661fa34ab7a.jpg saved to cache
2026-03-27 14:32:21 [crawlers.middlewares.photo_download_middleware] INFO: Downloaded image 0 for item 73d996ce-71d6-5e2b-8b11-8a08fd87c5f5
2026-03-27 14:32:21 [crawlers.middlewares.photo_download_middleware] INFO: All photos downloaded for item 73d996ce-71d6-5e2b-8b11-8a08fd87c5f5 sending to next pipeline
2026-03-27 14:32:21 [crawlers.pipelines.post_to_azurebus_auto_selling] INFO: Azure Bus Pipeline processing item: 73d996ce-71d6-5e2b-8b11-8a08fd87c5f5 with identifier 202602089769376
2026-03-27 14:32:21 [azure.servicebus._pyamqp.cbs] DEBUG: CBS status check: state == <CbsAuthState.OK: 0>, expired == False, refresh required == False
2026-03-27 14:32:21 [azure.servicebus._pyamqp.cbs] DEBUG: CBS status check: state == <CbsAuthState.OK: 0>, expired == False, refresh required == False
2026-03-27 14:32:21 [crawlers.pipelines.post_to_azurebus_auto_selling] DEBUG: [AzureBusPipeline] sent process item CloudEvent for item 73d996ce-71d6-5e2b-8b11-8a08fd87c5f5
2026-03-27 14:32:21 [scrapy.core.scraper] DEBUG: Scraped from <200 https://m.atcdn.co.uk/a/media/b76fa25739594f5292b1e661fa34ab7a.jpg>
{'auction_title': '',
 'c_o_2_emission_value': 103,
 'color': 'Silver',
 'currency': 'GBP',
 'description': 'Introducing the stunning Mercedes-Benz A-Class 2.1 A 200 D '
                'AMG Line Premium+ Auto 5dr. With only 37000 miles on the '
                'clock, this sleek and stylish vehicle is the epitome of '
                'luxury and performance.\n'
                '\n'
                'Finished in a sophisticated Silver colour, this Mercedes-Benz '
                'A-Class exudes elegance and class. The AMG Line Premium+ trim '
                'level offers a host of premium features, ensuring a '
                'comfortable and enjoyable driving experience every time you '
                'hit the road.\n'
                '\n'
                'From its powerful 2.1-litre diesel engine to its sleek and '
                'aerodynamic design, the Mercedes-Benz A-Class is a true '
                "standout on the road. Whether you're cruising through the "
                'city streets or embarking on a long-distance journey, this '
                'car delivers a smooth and responsive performance that will '
                'leave you feeling exhilarated.\n'
                '\n'
                "Don't miss your chance to own this exceptional Mercedes-Benz "
                "A-Class. With its low mileage and premium features, it's the "
                'perfect choice for those who demand the best in quality and '
                'style. Visit us today to experience the luxury and '
                'sophistication of the Mercedes-Benz A-Class for yourself.',
 'doors_number': 5,
 'emission_standard': 'Euro 6',
 'engine': '2.1L',
 'engine_horse_power': 134,
 'fuel_type': 'Diesel',
 'id': '73d996ce-71d6-5e2b-8b11-8a08fd87c5f5',
 'is_damaged': False,
 'is_operable': True,
 'link_to_web_offer': 'https://www.autotrader.co.uk/car-details/202602089769376',
 'listing': 'directbuy',
 'location': 'Bexleyheath, Kent',
 'make': 'Mercedes-Benz',
 'model': 'A Class',
 'odometer': 37000,
 'odometer_reading_unit': 'mi',
 'origin_country_code': 'GB',
 'original_photo_count': 44,
 'photos': [{'BlobStorageUrl': 'https://alxproduction.blob.core.windows.net/top-deal-listing/listing_73d996ce-71d6-5e2b-8b11-8a08fd87c5f5/73d996ce-71d6-5e2b-8b11-8a08fd87c5f5_image_0.jpg',
             'Extension': 'jpg',
             'FileSizeInBytes': 274552,
             'Height': 1440,
             'OriginUrl': 'https://m.atcdn.co.uk/a/media/b76fa25739594f5292b1e661fa34ab7a.jpg',
             'Width': 1920}],
 'price': 14450,
 'price_includes_vat': True,
 'registration_date': '2018-03-29',
 'seats_number': 5,
 'seller_name': None,
 'title': 'Mercedes-Benz A-Class 2.1 A 200 D AMG Line Premium+ Auto 5dr',
 'transmission': 'automatic',
 'trim': 'AMG Line',
 'year': 2018}
2026-03-27 14:32:21 [urllib3.connectionpool] DEBUG: https://alxsourcingstorageprod.table.core.windows.net:443 "POST /Tables HTTP/1.1" 409 None
2026-03-27 14:32:21 [urllib3.connectionpool] DEBUG: https://alxsourcingstorageprod.table.core.windows.net:443 "GET /ScrapedListings()?$filter=PartitionKey%20eq%20%27auto_selling_autotrader.co.uk%27%20and%20RowKey%20eq%20%27202602089769376%27 HTTP/1.1" 200 None
2026-03-27 14:32:21 [auto_selling_autotrader.co.uk] INFO: Saving data for 202602089769376: {'created_time': 1774621941.465586, 'last_price_update_time': 1774621941.465598}
2026-03-27 14:32:21 [urllib3.connectionpool] DEBUG: https://alxsourcingstorageprod.table.core.windows.net:443 "POST /Tables HTTP/1.1" 409 None
2026-03-27 14:32:21 [urllib3.connectionpool] DEBUG: https://alxsourcingstorageprod.table.core.windows.net:443 "PATCH /ScrapedListings(PartitionKey='auto_selling_autotrader.co.uk',RowKey='202602089769376') HTTP/1.1" 204 0
2026-03-27 14:32:25 [scrapy.core.engine] DEBUG: Crawled (200) <POST https://www.autotrader.co.uk/at-graphql> (referer: https://www.autotrader.co.uk/at-gateway?opname=SearchResultsListingsGridQuery&opname=SearchResultsFacetsWithGroupsQuery) ['zyte-api']
2026-03-27 14:32:25 [auto_selling_autotrader.co.uk] INFO: Extracted item from batch: None
2026-03-27 14:32:25 [crawlers.auto_selling.middlewares.id_gen_middleware] INFO: Generated ID for item: f0e16a1b-76d4-5647-b2f2-a33f92d0c3b6 with identifier: 202601179226019
2026-03-27 14:32:25 [crawlers.middlewares.photo_download_middleware] INFO: Downloading 1 photos for item f0e16a1b-76d4-5647-b2f2-a33f92d0c3b6
2026-03-27 14:32:26 [urllib3.connectionpool] DEBUG: https://alxsourcingstorageprod.table.core.windows.net:443 "POST /Tables HTTP/1.1" 409 None
2026-03-27 14:32:26 [urllib3.connectionpool] DEBUG: https://alxsourcingstorageprod.table.core.windows.net:443 "GET /PhotoDownloadCache()?$filter=PartitionKey%20eq%20%27auto_selling_autotrader.co.uk%27%20and%20RowKey%20eq%20%27https%3A%252F%252Fm.atcdn.co.uk%252Fa%252Fmedia%252F7fbc2650d51743f2a3d266e4399640b2.jpg%27 HTTP/1.1" 200 None
2026-03-27 14:32:26 [crawlers.middlewares.photo_download_middleware] INFO: Image https://m.atcdn.co.uk/a/media/7fbc2650d51743f2a3d266e4399640b2.jpg not downloaded yet
2026-03-27 14:32:26 [auto_selling_autotrader.co.uk] INFO: Extracted item from batch: None
2026-03-27 14:32:26 [crawlers.auto_selling.middlewares.id_gen_middleware] INFO: Generated ID for item: 50507e34-2243-5338-9417-cd7ba6d18965 with identifier: 202512128552229
2026-03-27 14:32:26 [crawlers.middlewares.photo_download_middleware] INFO: Downloading 1 photos for item 50507e34-2243-5338-9417-cd7ba6d18965
2026-03-27 14:32:26 [urllib3.connectionpool] DEBUG: https://alxsourcingstorageprod.table.core.windows.net:443 "POST /Tables HTTP/1.1" 409 None
2026-03-27 14:32:26 [urllib3.connectionpool] DEBUG: https://alxsourcingstorageprod.table.core.windows.net:443 "GET /PhotoDownloadCache()?$filter=PartitionKey%20eq%20%27auto_selling_autotrader.co.uk%27%20and%20RowKey%20eq%20%27https%3A%252F%252Fm.atcdn.co.uk%252Fa%252Fmedia%252F546150e296c54b40a742ad956a95ed15.jpg%27 HTTP/1.1" 200 None
2026-03-27 14:32:26 [crawlers.middlewares.photo_download_middleware] INFO: Image https://m.atcdn.co.uk/a/media/546150e296c54b40a742ad956a95ed15.jpg not downloaded yet
2026-03-27 14:32:26 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://m.atcdn.co.uk/a/media/546150e296c54b40a742ad956a95ed15.jpg> (referer: https://www.autotrader.co.uk/)
2026-03-27 14:32:26 [common.azure_blobstore] INFO: Creating container client for top-deal-listing
2026-03-27 14:32:26 [common.azure_blobstore] INFO: Container_client: <azure.storage.blob._container_client.ContainerClient object at 0x7ff6c97d7290>
2026-03-27 14:32:26 [common.azure_blobstore] INFO: Creating container if not exists
2026-03-27 14:32:26 [urllib3.connectionpool] DEBUG: https://alxproduction.blob.core.windows.net:443 "PUT /top-deal-listing?restype=container HTTP/1.1" 409 230
2026-03-27 14:32:26 [common.azure_blobstore] INFO: Container already exists, skipping creation
2026-03-27 14:32:26 [urllib3.connectionpool] DEBUG: https://alxproduction.blob.core.windows.net:443 "PUT /top-deal-listing/listing_50507e34-2243-5338-9417-cd7ba6d18965/50507e34-2243-5338-9417-cd7ba6d18965_image_0.jpg HTTP/1.1" 201 0
2026-03-27 14:32:26 [urllib3.connectionpool] DEBUG: https://alxsourcingstorageprod.table.core.windows.net:443 "POST /Tables HTTP/1.1" 409 None
2026-03-27 14:32:26 [urllib3.connectionpool] DEBUG: https://alxsourcingstorageprod.table.core.windows.net:443 "PATCH /PhotoDownloadCache(PartitionKey='auto_selling_autotrader.co.uk',RowKey='https%3A%252F%252Fm.atcdn.co.uk%252Fa%252Fmedia%252F546150e296c54b40a742ad956a95ed15.jpg') HTTP/1.1" 204 0
2026-03-27 14:32:26 [crawlers.middlewares.photo_download_middleware] INFO: Image https://m.atcdn.co.uk/a/media/546150e296c54b40a742ad956a95ed15.jpg saved to cache
2026-03-27 14:32:26 [crawlers.middlewares.photo_download_middleware] INFO: Downloaded image 0 for item 50507e34-2243-5338-9417-cd7ba6d18965
2026-03-27 14:32:26 [crawlers.middlewares.photo_download_middleware] INFO: All photos downloaded for item 50507e34-2243-5338-9417-cd7ba6d18965 sending to next pipeline
2026-03-27 14:32:26 [crawlers.pipelines.post_to_azurebus_auto_selling] INFO: Azure Bus Pipeline processing item: 50507e34-2243-5338-9417-cd7ba6d18965 with identifier 202512128552229
2026-03-27 14:32:26 [azure.servicebus._pyamqp.cbs] DEBUG: CBS status check: state == <CbsAuthState.OK: 0>, expired == False, refresh required == False
2026-03-27 14:32:26 [azure.servicebus._pyamqp.cbs] DEBUG: CBS status check: state == <CbsAuthState.OK: 0>, expired == False, refresh required == False
2026-03-27 14:32:26 [crawlers.pipelines.post_to_azurebus_auto_selling] DEBUG: [AzureBusPipeline] sent process item CloudEvent for item 50507e34-2243-5338-9417-cd7ba6d18965
2026-03-27 14:32:26 [scrapy.core.scraper] DEBUG: Scraped from <200 https://m.atcdn.co.uk/a/media/546150e296c54b40a742ad956a95ed15.jpg>
{'auction_title': '',
 'c_o_2_emission_value': 149,
 'color': 'Black',
 'currency': 'GBP',
 'description': 'Introducing the Volkswagen Tiguan 2.0 Tiguan R-Line TDI '
                'BlueMotion Technology 4Motion Semi-Auto 4WD 5dr, a sleek and '
                'powerful SUV that is sure to turn heads on the road. With '
                'only 67,134 miles on the clock and 1 previous owners, this '
                'Tiguan is in excellent condition and ready for its next '
                'adventure.\n'
                '\n'
                'Finished in a stylish Black exterior, this Volkswagen Tiguan '
                'not only looks the part but also delivers a smooth and '
                'efficient driving experience. The 4Motion Semi-Auto 4WD '
                'system ensures optimal traction and control in all road '
                'conditions, making it perfect for both city commutes and '
                'off-road escapades.\n'
                '\n'
                "Don't miss your chance to own this impressive Volkswagen "
                'Tiguan. Visit us today to experience the power and '
                'sophistication of this exceptional SUV for yourself.',
 'doors_number': 5,
 'emission_standard': 'Euro 6',
 'engine': '2L',
 'engine_horse_power': 148,
 'fuel_type': 'Diesel',
 'id': '50507e34-2243-5338-9417-cd7ba6d18965',
 'is_damaged': False,
 'is_operable': True,
 'link_to_web_offer': 'https://www.autotrader.co.uk/car-details/202512128552229',
 'listing': 'directbuy',
 'location': 'Bexleyheath, Kent',
 'make': 'Volkswagen',
 'model': 'Tiguan',
 'odometer': 67134,
 'odometer_reading_unit': 'mi',
 'origin_country_code': 'GB',
 'original_photo_count': 56,
 'photos': [{'BlobStorageUrl': 'https://alxproduction.blob.core.windows.net/top-deal-listing/listing_50507e34-2243-5338-9417-cd7ba6d18965/50507e34-2243-5338-9417-cd7ba6d18965_image_0.jpg',
             'Extension': 'jpg',
             'FileSizeInBytes': 292478,
             'Height': 1440,
             'OriginUrl': 'https://m.atcdn.co.uk/a/media/546150e296c54b40a742ad956a95ed15.jpg',
             'Width': 1920}],
 'price': 16450,
 'price_includes_vat': True,
 'registration_date': '2017-09-06',
 'seats_number': 5,
 'seller_name': None,
 'title': 'Volkswagen 2.0 Tiguan R-Line TDI BlueMotion Technology 4Motion '
          'Semi-Auto 4WD 5dr',
 'transmission': 'automatic',
 'trim': 'R-Line',
 'year': 2017}
2026-03-27 14:32:26 [urllib3.connectionpool] DEBUG: https://alxsourcingstorageprod.table.core.windows.net:443 "POST /Tables HTTP/1.1" 409 None
2026-03-27 14:32:26 [urllib3.connectionpool] DEBUG: https://alxsourcingstorageprod.table.core.windows.net:443 "GET /ScrapedListings()?$filter=PartitionKey%20eq%20%27auto_selling_autotrader.co.uk%27%20and%20RowKey%20eq%20%27202512128552229%27 HTTP/1.1" 200 None
2026-03-27 14:32:26 [auto_selling_autotrader.co.uk] INFO: Saving data for 202512128552229: {'created_time': 1774621946.366055, 'last_price_update_time': 1774621946.366063}
2026-03-27 14:32:26 [urllib3.connectionpool] DEBUG: https://alxsourcingstorageprod.table.core.windows.net:443 "POST /Tables HTTP/1.1" 409 None
2026-03-27 14:32:26 [urllib3.connectionpool] DEBUG: https://alxsourcingstorageprod.table.core.windows.net:443 "PATCH /ScrapedListings(PartitionKey='auto_selling_autotrader.co.uk',RowKey='202512128552229') HTTP/1.1" 204 0
2026-03-27 14:32:26 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://m.atcdn.co.uk/a/media/7fbc2650d51743f2a3d266e4399640b2.jpg> (referer: https://www.autotrader.co.uk/)
2026-03-27 14:32:26 [common.azure_blobstore] INFO: Creating container client for top-deal-listing
2026-03-27 14:32:26 [common.azure_blobstore] INFO: Container_client: <azure.storage.blob._container_client.ContainerClient object at 0x7ff6c97d2cd0>
2026-03-27 14:32:26 [common.azure_blobstore] INFO: Creating container if not exists
2026-03-27 14:32:26 [urllib3.connectionpool] DEBUG: https://alxproduction.blob.core.windows.net:443 "PUT /top-deal-listing?restype=container HTTP/1.1" 409 230
2026-03-27 14:32:26 [common.azure_blobstore] INFO: Container already exists, skipping creation
2026-03-27 14:32:26 [urllib3.connectionpool] DEBUG: https://alxproduction.blob.core.windows.net:443 "PUT /top-deal-listing/listing_f0e16a1b-76d4-5647-b2f2-a33f92d0c3b6/f0e16a1b-76d4-5647-b2f2-a33f92d0c3b6_image_0.jpg HTTP/1.1" 201 0
2026-03-27 14:32:26 [urllib3.connectionpool] DEBUG: https://alxsourcingstorageprod.table.core.windows.net:443 "POST /Tables HTTP/1.1" 409 None
2026-03-27 14:32:26 [urllib3.connectionpool] DEBUG: https://alxsourcingstorageprod.table.core.windows.net:443 "PATCH /PhotoDownloadCache(PartitionKey='auto_selling_autotrader.co.uk',RowKey='https%3A%252F%252Fm.atcdn.co.uk%252Fa%252Fmedia%252F7fbc2650d51743f2a3d266e4399640b2.jpg') HTTP/1.1" 204 0
2026-03-27 14:32:26 [crawlers.middlewares.photo_download_middleware] INFO: Image https://m.atcdn.co.uk/a/media/7fbc2650d51743f2a3d266e4399640b2.jpg saved to cache
2026-03-27 14:32:26 [crawlers.middlewares.photo_download_middleware] INFO: Downloaded image 0 for item f0e16a1b-76d4-5647-b2f2-a33f92d0c3b6
2026-03-27 14:32:26 [crawlers.middlewares.photo_download_middleware] INFO: All photos downloaded for item f0e16a1b-76d4-5647-b2f2-a33f92d0c3b6 sending to next pipeline
2026-03-27 14:32:26 [crawlers.pipelines.post_to_azurebus_auto_selling] INFO: Azure Bus Pipeline processing item: f0e16a1b-76d4-5647-b2f2-a33f92d0c3b6 with identifier 202601179226019
2026-03-27 14:32:26 [azure.servicebus._pyamqp.cbs] DEBUG: CBS status check: state == <CbsAuthState.OK: 0>, expired == False, refresh required == False
2026-03-27 14:32:26 [azure.servicebus._pyamqp.cbs] DEBUG: CBS status check: state == <CbsAuthState.OK: 0>, expired == False, refresh required == False
2026-03-27 14:32:26 [crawlers.pipelines.post_to_azurebus_auto_selling] DEBUG: [AzureBusPipeline] sent process item CloudEvent for item f0e16a1b-76d4-5647-b2f2-a33f92d0c3b6
2026-03-27 14:32:26 [scrapy.core.scraper] DEBUG: Scraped from <200 https://m.atcdn.co.uk/a/media/7fbc2650d51743f2a3d266e4399640b2.jpg>
{'auction_title': '',
 'c_o_2_emission_value': 110,
 'color': 'Grey',
 'currency': 'GBP',
 'description': 'Introducing the stunning Mercedes-Benz A-Class 2.0 A 200 D '
                'AMG Line Premium Auto 5dr. This exceptional vehicle, finished '
                'in a sleek Grey colour, exudes luxury and style at every '
                'turn.\n'
                '\n'
                'With only 69,739 miles on the clock and a total of 2 careful '
                'owners, this Mercedes-Benz A-Class is in impeccable '
                'condition, ready to elevate your driving experience to new '
                'heights.\n'
                '\n'
                'Experience the perfect blend of performance and '
                'sophistication with the AMG Line trim, offering a sporty edge '
                'to the already impressive A 200 D model. The Premium Auto '
                'features ensure a smooth and effortless drive, making every '
                'journey a pleasure.\n'
                '\n'
                "Don't miss your chance to own this premium Mercedes-Benz "
                'A-Class. Contact us today to arrange a test drive and '
                'discover the true meaning of driving excellence.',
 'doors_number': 5,
 'emission_standard': 'Euro 6',
 'engine': '2L',
 'engine_horse_power': 148,
 'fuel_type': 'Diesel',
 'id': 'f0e16a1b-76d4-5647-b2f2-a33f92d0c3b6',
 'is_damaged': False,
 'is_operable': True,
 'link_to_web_offer': 'https://www.autotrader.co.uk/car-details/202601179226019',
 'listing': 'directbuy',
 'location': 'Bexleyheath, Kent',
 'make': 'Mercedes-Benz',
 'model': 'A Class',
 'odometer': 69739,
 'odometer_reading_unit': 'mi',
 'origin_country_code': 'GB',
 'original_photo_count': 53,
 'photos': [{'BlobStorageUrl': 'https://alxproduction.blob.core.windows.net/top-deal-listing/listing_f0e16a1b-76d4-5647-b2f2-a33f92d0c3b6/f0e16a1b-76d4-5647-b2f2-a33f92d0c3b6_image_0.jpg',
             'Extension': 'jpg',
             'FileSizeInBytes': 277117,
             'Height': 1440,
             'OriginUrl': 'https://m.atcdn.co.uk/a/media/7fbc2650d51743f2a3d266e4399640b2.jpg',
             'Width': 1920}],
 'price': 14950,
 'price_includes_vat': True,
 'registration_date': '2019-04-30',
 'seats_number': 5,
 'seller_name': None,
 'title': 'Mercedes-Benz A-Class 2.0 A 200 D AMG Line Premium Auto 5dr',
 'transmission': 'automatic',
 'trim': 'AMG Line',
 'year': 2019}
2026-03-27 14:32:26 [urllib3.connectionpool] DEBUG: https://alxsourcingstorageprod.table.core.windows.net:443 "POST /Tables HTTP/1.1" 409 None
2026-03-27 14:32:26 [urllib3.connectionpool] DEBUG: https://alxsourcingstorageprod.table.core.windows.net:443 "GET /ScrapedListings()?$filter=PartitionKey%20eq%20%27auto_selling_autotrader.co.uk%27%20and%20RowKey%20eq%20%27202601179226019%27 HTTP/1.1" 200 None
2026-03-27 14:32:26 [auto_selling_autotrader.co.uk] INFO: Saving data for 202601179226019: {'created_time': 1774621946.655894, 'last_price_update_time': 1774621946.655902}
2026-03-27 14:32:26 [urllib3.connectionpool] DEBUG: https://alxsourcingstorageprod.table.core.windows.net:443 "POST /Tables HTTP/1.1" 409 None
2026-03-27 14:32:26 [urllib3.connectionpool] DEBUG: https://alxsourcingstorageprod.table.core.windows.net:443 "PATCH /ScrapedListings(PartitionKey='auto_selling_autotrader.co.uk',RowKey='202601179226019') HTTP/1.1" 204 0
2026-03-27 14:32:26 [scrapy.core.engine] INFO: Closing spider (finished)
2026-03-27 14:32:26 [crawlers.pipelines.save_to_cdlq_db_auto_selling] DEBUG: [SaveToCdlqDbAutoSelling] failed to close cursor: 'SaveToCdlqDbAutoSelling' object has no attribute 'cursor'
2026-03-27 14:32:26 [crawlers.pipelines.save_to_cdlq_db_auto_selling] DEBUG: [SaveToCdlqDbAutoSelling] failed to close connection: 'SaveToCdlqDbAutoSelling' object has no attribute 'conn'
2026-03-27 14:32:26 [azure.servicebus._pyamqp.cbs] DEBUG: CBS status check: state == <CbsAuthState.OK: 0>, expired == False, refresh required == False
2026-03-27 14:32:26 [azure.servicebus._pyamqp.cbs] DEBUG: CBS status check: state == <CbsAuthState.OK: 0>, expired == False, refresh required == False
2026-03-27 14:32:26 [crawlers.pipelines.post_to_azurebus_auto_selling] DEBUG: [AzureBusPipeline] sent process scrape complete CloudEvent for item bbc8d04a29e911f18f76c264925fbcfa
2026-03-27 14:32:26 [azure.servicebus._pyamqp.link] INFO: Link state changed: <LinkState.ATTACHED: 3> -> <LinkState.DETACH_SENT: 4>
2026-03-27 14:32:26 [azure.servicebus._pyamqp.link] INFO: Link state changed: <LinkState.ATTACHED: 3> -> <LinkState.DETACH_SENT: 4>
2026-03-27 14:32:26 [azure.servicebus._pyamqp.management_link] INFO: Management link receiver state changed: <LinkState.ATTACHED: 3> -> <LinkState.DETACH_SENT: 4>
2026-03-27 14:32:26 [azure.servicebus._pyamqp.link] INFO: Link state changed: <LinkState.ATTACHED: 3> -> <LinkState.DETACH_SENT: 4>
2026-03-27 14:32:26 [azure.servicebus._pyamqp.management_link] INFO: Management link sender state changed: <LinkState.ATTACHED: 3> -> <LinkState.DETACH_SENT: 4>
2026-03-27 14:32:26 [azure.servicebus._pyamqp.session] INFO: Session state changed: <SessionState.MAPPED: 3> -> <SessionState.END_SENT: 4>
2026-03-27 14:32:26 [azure.servicebus._pyamqp._connection] INFO: Connection state changed: <ConnectionState.OPENED: 9> -> <ConnectionState.CLOSE_SENT: 11>
2026-03-27 14:32:26 [azure.servicebus._pyamqp._connection] INFO: Connection state changed: <ConnectionState.CLOSE_SENT: 11> -> <ConnectionState.END: 13>
2026-03-27 14:32:26 [azure.servicebus._pyamqp.session] INFO: Session state changed: <SessionState.END_SENT: 4> -> <SessionState.DISCARDING: 6>
2026-03-27 14:32:26 [azure.servicebus._pyamqp.link] INFO: Link state changed: <LinkState.DETACH_SENT: 4> -> <LinkState.DETACHED: 0>
2026-03-27 14:32:26 [azure.servicebus._pyamqp.management_link] INFO: Management link sender state changed: <LinkState.DETACH_SENT: 4> -> <LinkState.DETACHED: 0>
2026-03-27 14:32:26 [azure.servicebus._pyamqp.link] INFO: Link state changed: <LinkState.DETACH_SENT: 4> -> <LinkState.DETACHED: 0>
2026-03-27 14:32:26 [azure.servicebus._pyamqp.management_link] INFO: Management link receiver state changed: <LinkState.DETACH_SENT: 4> -> <LinkState.DETACHED: 0>
2026-03-27 14:32:26 [azure.servicebus._pyamqp.link] INFO: Link state changed: <LinkState.DETACH_SENT: 4> -> <LinkState.DETACHED: 0>
2026-03-27 14:32:26 [crawlers.pipelines.post_to_azurebus_auto_selling] INFO: [AzureBusPipeline] connection closed
2026-03-27 14:32:26 [auto_selling_autotrader.co.uk] INFO: auto_selling_autotrader.co.uk Crawl ended with reason finished, scrape types: {<ScrapeType.NEW: 1>: 0, <ScrapeType.NEW_DUPLICATE_ID: 4>: 0, <ScrapeType.PRICE_UPDATE: 2>: 0, <ScrapeType.AUCTION_UPDATE: 3>: 0, <ScrapeType.SKIPPED: 0>: 0}
2026-03-27 14:32:26 [urllib3.connectionpool] DEBUG: https://alxsourcingstorageprod.table.core.windows.net:443 "POST /Tables HTTP/1.1" 409 None
2026-03-27 14:32:27 [urllib3.connectionpool] DEBUG: https://alxsourcingstorageprod.table.core.windows.net:443 "GET /ScrapedListings()?$filter=PartitionKey%20eq%20%27auto_selling_autotrader.co.uk%27%20and%20last_price_update_time%20lt%201773325946 HTTP/1.1" 200 None
2026-03-27 14:32:27 [scrapy.extensions.feedexport] INFO: Stored jsonlines feed (12 items) in: file:///var/lib/scrapyd/items/sourcing_v2/auto_selling_autotrader.co.uk/bbc8d04a29e911f18f76c264925fbcfa.jl
2026-03-27 14:32:27 [scrapy.statscollectors] INFO: Dumping Scrapy stats:
{'downloader/exception_count': 6,
 'downloader/exception_type_count/scrapy.core.downloader.handlers.http11.TunnelError': 6,
 'downloader/request_bytes': 98565,
 'downloader/request_count': 15,
 'downloader/request_method_count/GET': 12,
 'downloader/request_method_count/POST': 3,
 'downloader/response_bytes': 2390522,
 'downloader/response_count': 9,
 'downloader/response_status_count/200': 9,
 'elapsed_time_seconds': 17.102006,
 'feedexport/success_count/FileFeedStorage': 1,
 'finish_reason': 'finished',
 'finish_time': datetime.datetime(2026, 3, 27, 14, 32, 27, 81622, tzinfo=datetime.timezone.utc),
 'item_scraped_count': 12,
 'log_count/DEBUG': 212,
 'log_count/INFO': 190,
 'memusage/max': 148951040,
 'memusage/startup': 148951040,
 'photo_download_count': 6,
 'request_depth_max': 2,
 'response_received_count': 3,
 'scheduler/dequeued': 15,
 'scheduler/dequeued/memory': 15,
 'scheduler/enqueued': 15,
 'scheduler/enqueued/memory': 15,
 'scrape_type/new': 12,
 'scrapy-zyte-api/429': 0,
 'scrapy-zyte-api/attempts': 3,
 'scrapy-zyte-api/error_ratio': 0.0,
 'scrapy-zyte-api/errors': 0,
 'scrapy-zyte-api/fatal_errors': 0,
 'scrapy-zyte-api/mean_connection_seconds': 3.9461959761877856,
 'scrapy-zyte-api/mean_response_seconds': 4.103087444324046,
 'scrapy-zyte-api/processed': 3,
 'scrapy-zyte-api/request_args/customHttpRequestHeaders': 3,
 'scrapy-zyte-api/request_args/experimental.requestCookies': 2,
 'scrapy-zyte-api/request_args/experimental.responseCookies': 3,
 'scrapy-zyte-api/request_args/httpRequestBody': 3,
 'scrapy-zyte-api/request_args/httpRequestMethod': 3,
 'scrapy-zyte-api/request_args/httpResponseBody': 3,
 'scrapy-zyte-api/request_args/httpResponseHeaders': 3,
 'scrapy-zyte-api/request_args/url': 3,
 'scrapy-zyte-api/status_codes/200': 3,
 'scrapy-zyte-api/success': 3,
 'scrapy-zyte-api/success_ratio': 1.0,
 'scrapy-zyte-api/throttle_ratio': 0.0,
 'start_time': datetime.datetime(2026, 3, 27, 14, 32, 9, 979616, tzinfo=datetime.timezone.utc)}
2026-03-27 14:32:27 [scrapy.core.engine] INFO: Spider closed (finished)
2026-03-27 14:32:27 [asyncio] ERROR: Unclosed client session
client_session: <aiohttp.client.ClientSession object at 0x7ff6ccd4f3d0>
2026-03-27 14:32:27 [asyncio] ERROR: Unclosed client session
client_session: <aiohttp.client.ClientSession object at 0x7ff6ccd4fb50>