Description
Confirm this is an issue with the Python library and not an underlying OpenAI API
- This is an issue with the Python library
Describe the bug
When I enabled logprobs, I started to get consistent issue with one particular long response prompt. I already set max token 16384. Not sure if it is a library issue or api issue.
I started having issues when I enabled logprobs
logprobs=true
top_logprobs=1
Does it count towards max token (16384)?
Here are my experiments and request ids, getting this error “peer closed connection without sending complete message body (incomplete chunked read)”
Different finetuned models same prompt
OpenAI Request ID: req_cd79d67111c6231ece4da879c6120181
OpenAI Request ID: req_bdd68774e518adfa6983ba73425c26cb
Upgraded to latest packages: httpcore-1.0.9 openai-1.79.0 pydantic-2.11.4
OpenAI Request ID: req_94bbc34a05590b9a9654683b0ccc0593
tried openai_client = OpenAI(timeout=600), no effect
OpenAI Request ID: req_c1c6dbbc213df52df00abb3302ca405f
-use standard model gpt-4o-mini-2024-07-18
OpenAI Request ID: req_4b93428d151f2ce8091e69ab470128c5
When I disabled logprobs, no issue
OpenAI Request ID: req_320590488da2052e1318f0148cef0220 (Okay - 3764 chunks)
To Reproduce
Check the Request IDs
Code snippets
stream = openai_client.chat.completions.create(
model=current_model,
messages=messages,
temperature=0,
top_p=1,
max_tokens=16384,
stream=True,
logprobs=True,
top_logprobs=1
response_format={
"type": "json_schema",
"json_schema": {
"name": "section",
"strict": True,
"schema": {
"type": "object",
"properties": {
"sectionName": {
"type": "string"
},
"clauses": {
"type": "array",
"items": {
"type": "object",
"properties": {
"clauseNo": {
"type": "string"
},
"contractTypeApplicability": {
"type": "string"
},
"biasAssessment": {
"type": "string"
},
"justification": {
"type": "string"
},
},
"required": [
"clauseNo",
"contractTypeApplicability",
"biasAssessment",
"justification"
],
"additionalProperties": False
}
}
},
"required": [
"sectionName",
"clauses"
],
"additionalProperties": False
}
},
}
)OS
Windows
Python version
Python 3.12.1
Library version
openai-1.79.0