让我们考虑一下这个Python代码:
def process_payload(payload, url, headers): response = requests.post(url, headers=headers, json=payload) return responsedef parallel_group2(payloads, url, headers): with ThreadPoolExecutor() as executor: results = executor.map(process_payload,payloads, [url]*len(payloads), [headers]*len(payloads)) return list(results)def parallel_group(payloads, url, headers): with ThreadPoolExecutor() as executor: results = executor.map(requests.post, [url]*len(payloads), [headers]*len(payloads), payloads) return list(results)times = []# payloads grouped by 15payloads_grouped = [payloads[i:i+15] for i in range(0, len(payloads), 15)]print( "shape of payloads_grouped", len(payloads_grouped), " x ", len(payloads_grouped[0]))for i in range(3): start = time.time() with ThreadPoolExecutor() as executor: # results = executor.map(parallel_group2, payloads_grouped, [url]*len(payloads_grouped), [headers]*len(payloads_grouped)) results = executor.map(parallel_group, payloads_grouped, [url]*len(payloads_grouped), [headers]*len(payloads_grouped)) end = time.time() times.append(end-start) print( "Durations of iterations:", times)print( "Durations of iterations:", times)print( "Average time for 150 requests:", sum(times)/len(times))
当我使用parallel_group运行脚本时,我非常一致地得到了这些结果:
Durations of iterations: [5.246389389038086, 5.195073127746582, 5.278628587722778]Average time for 150 requests: 5.2400303681691485
当我使用parallel_group2运行时,结果看起来更像是这样:
Durations of iterations: [10.99542498588562, 9.43007493019104, 23.003321170806885]Average time for 150 requests: 10.142940362294516
有谁对Python多线程有很好的了解,能够解释为什么直接调用requests.post和调用一个仅执行requests.post的函数在多线程调用上会有如此大的差异吗?我完全不明白。
我多次运行了之前的代码,结果是一致的。
编辑:URL是OpenAI的聊天完成API =”api.openai.com/v1/chat/completions”
回答:
你的parallel_group函数并没有按你希望的方式工作。原因是你传递给requests.post的三个参数中,只有第一个是正确的(URL)。负载将被分配为data,而头信息将被分配给json。API很可能会返回错误,但你忽略了这种可能性