Api/generate endpoint

[gl2007]

For some reasons I can't explain Ollama's performance on my HP workstation sucks as compared to this for the same model. However, I want to use structured output which Ollama seems to support but not sure if this repo supports it?

If anyone knows and has been able to get it working, please give some details. Appreciate any help!

[onestardao]

You’re right — structured outputs are supported in Ollama, but whether this repo replicates that depends on how the inference server handles formatting/parsing under the hood.

For structured output (like JSON), one thing I’ve done in similar setups is manually enforce formatting in the prompt — something like:
Respond only in valid JSON. Use keys: message, status, tokens.
But if you're relying on automatic formatting (like with function_calling or json_mode), this repo might not mirror Ollama's behavior exactly, since it's closer to raw inference bindings.

As for Ollama vs this on the same model — hardware aside, the runtime stack (backends, quantization, server flags) can affect perf a lot more than it looks on paper. Feel free to drop more details if you want someone to cross-check!

— passing by on GitHub patrol