Tips for ML Researchers

by John Schulman [joschu.net]

Main ideas:

• Be Goal driven

  Being idea driven can lead to your idea being scooped because everyone is reading the same stuff and probably working on same thing. Being goal driven gives you a different perspective.

• Aim high and climb incrementally

  x.com/fchollet also said "ambition in vision, but pragmatism in execution"

• Keep a notebook and review it

  • Keep note of your daily plans, ideas, stuff you did, the results
  • then every week or two, review the notes, summarize them

  This acts a record of idea, gives you perspective on what you are spending time on. And also acts as central place for results when you need to refer them later.

• Separate time for Personal development. Read books, theses, and papers. Implement the methods. Textbooks are dense and best for building up your foundational knowledge.

Also read:

• You and Your Research from Art of Doing Science and Engineering
• Principles of Effective Research by Michael Nielsen

by Eugene Vinitsky [eugenevinitsky.com]

• Use Hydra (or Pyrallis) for configs. Don't just write the hyperparameters/config parameters in code files.
• Use package manager to manage python version and packages. E.g. Conda, uv-pip, Poetry, pixi
• Use git, don't use jupyter notebooks (because they don't play nice with git)
• Don't tune hyperparameters by hand. Use Optuna, Ray Tune or custom tuner.
• Run experiments in cluster not in desktop (quick experiments = quicker progress)