Meta is dedicating 3 engineers to get the nogil patches into cpython. There are some other companies stepping up as well. This is huge this is the closest we have ever been to solving the issue of the GIL.

  • @mvirts@lemmy.world
    link
    fedilink
    710 months ago

    Basically there is no parallel execution of python code within a single process as long as the global interpreter lock exists. It prevents more than one python thread from running. However, many of the major libraries used in python call out to libraries that can and do release the gil to run native code in parallel.

    • Kale
      link
      fedilink
      English
      5
      edit-2
      10 months ago

      The GIL only executes one thread at a time. A python program can be multithreaded, but only only thread runs in CPython at a time. If one thread does a system call (like copying a file), then when the python thread is sleeping, the system call can still run in the OS, so there are situations where multithreading can speed up Python programs, even running one thread at a time.

      You can run multiple instances of CPython, which is called multiprocessing, and each instance will run one python thread at a time. With different memory space, so all process communication has to be handled manually (afaik, by definition, threads share the same memory space, processes do not).

      Any library calls not written in Python don’t run in the interpreter, so most common critical things aren’t limited too badly. For example, I install a NumPy and SciPy library which are compiled against Intel’s MKL library. Any NumPy operations execute in MKL, not the Python interpreter, so are almost as fast as writing the program in C and compiling against MKL myself. And I can write Python and NumPy code about 10x faster than C/MKL. And if I’m on a computer that doesn’t have MKL, I can install a different NumPy library and it will execute just fine without changing the code.

      There’s a book called “high performance Python” that helped me figure out a lot of this.

      Edit: thought I was posting on the grandparents post instead of the parent post. Sorry.