Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Save more detailed bet details in strategy simulation script #429

Merged
merged 3 commits into from
Sep 24, 2024
Merged

Save more detailed bet details in strategy simulation script #429

Changes from 1 commit
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
8a2a5f4
Save more detailed bet details in strategy simulation script
kongzii Sep 24, 2024
cef2795
add url
kongzii Sep 24, 2024
2e8fecf
remove extra strategy
kongzii Sep 24, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
35 changes: 32 additions & 3 deletions examples/monitor/match_bets_with_langfuse_traces.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -102,6 +102,7 @@ def get_outcome_for_trace(
KellyBettingStrategy(max_bet_amount=1),
KellyBettingStrategy(max_bet_amount=2),
KellyBettingStrategy(max_bet_amount=25),
KellyBettingStrategy(max_bet_amount=250),
MaxAccuracyWithKellyScaledBetsStrategy(max_bet_amount=1),
MaxAccuracyWithKellyScaledBetsStrategy(max_bet_amount=2),
MaxAccuracyWithKellyScaledBetsStrategy(max_bet_amount=25),
Expand All @@ -110,6 +111,8 @@ def get_outcome_for_trace(
MaxExpectedValueBettingStrategy(bet_amount=25),
]

overall_md = ""

print("# Agent Bet vs Simulated Bet Comparison")
for agent_name, private_key in agent_pkey_map.items():
print(f"\n## {agent_name}\n")
Expand Down Expand Up @@ -152,11 +155,12 @@ def get_outcome_for_trace(

print(f"Number of bets since {start_time}: {len(bets_with_traces)}\n")
if len(bets_with_traces) != len(bets):
raise ValueError(
f"{len(bets) - len(bets_with_traces)} bets do not have a corresponding trace"
print(
f"{len(bets) - len(bets_with_traces)} bets do not have a corresponding trace, ignoring them."
)

simulations: list[dict[str, Any]] = []
details = []
kongzii marked this conversation as resolved.
Show resolved Hide resolved

for strategy_idx, strategy in enumerate(strategies):
# "Born" agent with initial funding, simulate as if he was doing bets one by one.
Expand All @@ -175,6 +179,25 @@ def get_outcome_for_trace(
simulated_outcomes.append(simulated_outcome)
agent_balance += simulated_outcome.profit

details.append(
{
"market_p_yes": round(trace.market.current_p_yes, 4),
"agent_p_yes": round(trace.answer.p_yes, 4),
"agent_conf": round(trace.answer.confidence, 4),
"org_bet": round(bet.amount.amount, 4),
"sim_bet": round(simulated_outcome.size, 4),
"org_dir": bet.outcome,
"sim_dir": simulated_outcome.direction,
"org_profit": round(bet.profit.amount, 4),
"sim_profit": round(simulated_outcome.profit, 4),
}
)

details.sort(key=lambda x: x["sim_profit"], reverse=True)
pd.DataFrame.from_records(details).to_csv(
f"{agent_name} - {strategy} - all bets.csv", index=False
)
Comment on lines +197 to +199
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue

Sanitize file names to prevent invalid characters.

Using str(strategy) in file names may introduce characters invalid in file systems (e.g., <, >, /, :). This can cause errors when saving CSV files. Consider sanitizing the strategy representation to ensure file names are valid.

Apply this diff to sanitize the file names:

+                import re
+                strategy_name = re.sub(r'[<>:"/\\|?*]', '_', str(strategy))
+                filename = f"{agent_name}_{strategy_name}_all_bets.csv"
                 pd.DataFrame.from_records(details).to_csv(
-                    f"{agent_name} - {strategy} - all bets.csv", index=False
+                    filename, index=False
                 )
Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

Suggested change
pd.DataFrame.from_records(details).to_csv(
f"{agent_name} - {strategy} - all bets.csv", index=False
)
import re
strategy_name = re.sub(r'[<>:"/\\|?*]', '_', str(strategy))
filename = f"{agent_name}_{strategy_name}_all_bets.csv"
pd.DataFrame.from_records(details).to_csv(
filename, index=False
)


total_bet_amount = sum([bt.bet.amount.amount for bt in bets_with_traces])
total_bet_profit = sum([bt.bet.profit.amount for bt in bets_with_traces])
total_simulated_amount = sum([so.size for so in simulated_outcomes])
Expand Down Expand Up @@ -207,4 +230,10 @@ def get_outcome_for_trace(
}
)

print(pd.DataFrame.from_records(simulations).to_markdown(index=False))
overall_md += (
f"\n\n## {agent_name}\n\n{len(simulations)} bets\n\n"
+ pd.DataFrame.from_records(simulations).to_markdown(index=False)
)
Comment on lines +233 to +236
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue

Correct the number of bets displayed in the overall report.

Using len(simulations) represents the number of strategies simulated, not the number of bets placed. To display the correct number of bets, use len(bets_with_traces).

Apply this diff to correct the displayed number of bets:

             overall_md += (
-                f"\n\n## {agent_name}\n\n{len(simulations)} bets\n\n"
+                f"\n\n## {agent_name}\n\n{len(bets_with_traces)} bets\n\n"
                 + pd.DataFrame.from_records(simulations).to_markdown(index=False)
             )
Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

Suggested change
overall_md += (
f"\n\n## {agent_name}\n\n{len(simulations)} bets\n\n"
+ pd.DataFrame.from_records(simulations).to_markdown(index=False)
)
overall_md += (
f"\n\n## {agent_name}\n\n{len(bets_with_traces)} bets\n\n"
+ pd.DataFrame.from_records(simulations).to_markdown(index=False)
)


with open("match_bets_with_langfuse_traces_overall.md", "w") as overall_f:
overall_f.write(overall_md)
Loading