Adding VERL replacement for Tinker #264
Conversation
There was a problem hiding this comment.
Pull Request Overview
This PR adds a VERL-based training alternative for the 20 Questions agent and adjusts hyperparameters for the existing Tinker-based training. The key changes introduce a new algo_verl command-line option that enables local training when Tinker service is unavailable.
- Added
algo_verl()function with VERL configuration for local training - Updated training hyperparameters (batch_size, seed, concurrency, lora_rank)
- Added CLI parser for the new
verlsubcommand - Updated README to document the VERL alternative
Reviewed Changes
Copilot reviewed 2 out of 2 changed files in this pull request and generated 2 comments.
| File | Description |
|---|---|
| examples/tinker/q20_train.py | Added VERL training function, CLI support, and updated hyperparameters for Tinker training |
| examples/tinker/README.md | Updated documentation to mention VERL as an alternative training method |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
| train_dataset = cast(agl.Dataset[Q20Task], train_data.to_dict(orient="records")) # type: ignore | ||
| test_dataset = cast(agl.Dataset[Q20Task], test_data.to_dict(orient="records")) # type: ignore | ||
|
|
||
| verl.run(train_dataset=train_dataset, val_dataset=test_dataset) |
There was a problem hiding this comment.
The verl.run() call is missing the rollout agent. According to the VERL API pattern used in other examples (calc_x, spider), you should either pass the algorithm to trainer.fit(q20_agent, train_dataset, val_dataset) or ensure the rollout agent is registered with VERL. The current implementation provides datasets but no rollout function, so VERL won't know how to execute the tasks.
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
* Adding VERL replacement for Tinker * Apply suggestion from @Copilot Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> --------- Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
No description provided.