0843 I have reached the lab at https://colab.research.google.com/github/huggingface/deep-rl-class/blob/main... . I have fixed the email name. The unit has an additional special section added to include one of the many small new technologies that are emerging. I am holding the goal of completing the bare essentials of the lab.

0847 I have found the first task. I am experience confusion due to a habit of pressing shift-enter to insert a carriage return in web dialog boxes, which in colab leaves the editing and executes the box. I am excited to produce some horrific abuse of my very own.

0851 This is the solution I filled in: # TODO: Define a PPO MlpPolicy architecture # We use MultiLayerPerceptron (MLPPolicy) because the input is a vector, # if we had frames as input we would use CnnPolicy import stable_baselines3 model = stable_baselines3.PPO('MlpPolicy', env, verbose=1) This is the solution they provide: # SOLUTION # We added some parameters to fasten the training model = PPO( policy = 'MlpPolicy', env = env, n_steps = 1024, batch_size = 64, n_epochs = 4, gamma = 0.999, gae_lambda = 0.98, ent_coef = 0.01, verbose=1) I will copy their parameters over to my code, thinking briefly about each one. I recognise 3 of them. I recall that some of them were mentioned in the learning material, and I do not remember what they are.

0853 here is what I have. i did not look up the terms i was unsure of. i will instead move on with the lab. # TODO: Define a PPO MlpPolicy architecture # We use MultiLayerPerceptron (MLPPolicy) because the input is a vector, # if we had frames as input we would use CnnPolicy import stable_baselines3 model = stable_baselines3.PPO( 'MlpPolicy', # vector input, CnnPolicy is for images env, # environment objects to feed back with verbose=1, # output information n_steps = 1024, # number of steps policy takes in each parallel environment before updating batch_size = 64, # number of data items sent interdependently to the gpu when updating. faster, smoother & better results when this is higher. n_epochs = 4, # not sure, usually this means how many times to run over the data gamma = 0.999 # not sure, relates to PPO I think )

900 I found something scrolling up, and pasted it in. The box now looks like this: # TODO: Train it for 500,000 timesteps model.learn(total_timesteps=int(2e5)) # TODO: Specify file name for model and save the model to file model_name = "" I will first delete the model.learn line, which I pasted in, and retype it on my own, to produce a behavioral experience of doing so.

0910 I have a strong guess as to how the logging intervals work, and am waiting for a final test to run to run the 500k steps.

0912 This is the solution: # SOLUTION # Train it for 500,000 timesteps model.learn(total_timesteps=500000) # Save the model model_name = "ppo-LunarLander-v2" model.save(model_name) They did not add extra information like with model construction. It is indeed much less informative here, the available extra information. While waiting for the training to complete, I will first go to the next section and look at it. I am also considering looking up the parameters I did not know in the model construction. I am most familiar with learning by looking at the direct source code of the api function being called.

Here is my code: # TODO: Evaluate the agent # Create a new environment for evaluation import stable_baselines3.common.env_util eval_env = stable_baselines3.common.env_util.make_vec_env('LunarLander-v2', n_envs=4) # Evaluate the model with 10 evaluation episodes and deterministic=True import stable_baselines3.common.evaluation mean_reward, std_reward = stable_baselines3.common.evaluation.evaluate_policy(model, eval_env, n_eval_episodes=10, deterministic=True) # Print the results print(f'Rewards: mean={mean_reward} std={std_reward}') The model finished training, and I ran it. I think it does a total of 40 episodes because I passed a vectorised environment. It displayed a mean reward of around 251 and an std of around 20.5

0931 The leaderboard does not display anything for me, even with advertising javascript enabled, or trying in a different browser.

0938 my model did not save and package correctly due to the naming issue. i'm worried this may desynchronise the behaviors. i'm holding the intention of exploring the optuna hyperparameter notebook.

0947 I'm reading down through the optuna notebook. I'm actually reading it! I got as far as through setting the ranges of the hyperparameters. I feel able to do the cold water and eat breakfast, so I am holding that intention now.